Shengma teaching sequence calligraphy rubbing fusion repairing method based on multi-condition diffusion model

Abstract

The invention aims to provide a multi-condition diffusion model-based Shengtuo calligraphy rubbing fusion restoration method, which comprises the following steps of: firstly, acquiring a plurality of Shengtuo calligraphy rubbing images from different sources, preprocessing the images, and constructing a standardized training data set; designing a multi-image feature extraction and alignment module to obtain a fused structural feature tensor; constructing a core network structure of the multi-conditional diffusion model, and supporting guidance of a plurality of conditional vectors; jointly inputting the obtained fusion structure feature tensor and a plurality of condition guide items into a diffusion model to generate a fused and repaired potential image; and finally, restoring the potential image obtained by reconstruction into a final output image through a reverse mapping module. According to the method, the problems of blurring, pen lacking, breakage and non-uniform styles during fusion and restoration of the rubbing image in the prior art are solved.

Description

Technical Field

[0001] This invention belongs to the field of calligraphy image restoration and deep learning technology, specifically involving a method for the fusion and restoration of the Preface to the Sacred Teachings calligraphy rubbings based on a multi-condition diffusion model. Background Technology

[0002] With the deepening of ancient book digitization, cultural relic preservation, and calligraphy art research, an increasing number of precious calligraphy rubbings are being collected, archived, and studied. However, due to their age and poor storage conditions, calligraphy rubbings often suffer from structural defects, blurred strokes, and mottled ink, which seriously affects their digital reconstruction and subsequent cultural exploration value. In order to improve the readability and restoration integrity of calligraphy images, researchers have recently attempted to use computer vision technology to restore and enhance damaged rubbing images.

[0003] Because the same character shape often exists in different versions of calligraphy rubbings from multiple sources, such as the famous collection of Wang Xizhi's Preface to the Sacred Teachings, the three best rubbings are the Shaanxi History Museum version, the Liu Tieyun version, and the Mohuang version. However, these three rubbings each have their own advantages and disadvantages, and possess complementary information. Therefore, fusing and restoring these multi-source calligraphy rubbing images is expected to achieve high-quality character shape reconstruction with complete structure, natural lines, and unified style while preserving the calligraphic artistic style. However, most existing methods focus on single-image restoration, image denoising, or style transfer, lacking effective modeling and feature fusion mechanisms for the structural differences between multi-source images. This limits the restoration effect and makes it difficult to fully utilize the potential of multi-source information.

[0004] Especially in unsupervised scenarios, the lack of high-quality labels or standard answers poses greater challenges to model training and evaluation, and there is an urgent need for an innovative method that can effectively integrate multi-source structural information in the latent space and improve the quality of repair. Summary of the Invention

[0005] The purpose of this invention is to provide a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, which solves the problems of blurring, missing strokes, breaks, and inconsistent styles in the fusion and restoration of rubbing images in the prior art.

[0006] The technical solution adopted in this invention is a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, which is implemented according to the following steps: Step 1: Collect images of rubbings of the Preface to the Sacred Teachings from multiple sources, preprocess the images, and construct a standardized training dataset; Step 2: Design a multi-image feature extraction and alignment module to obtain the fused structural feature tensor; Step 3: Construct the core network structure of the multi-conditional diffusion model and support multiple conditional vector guidance; Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model to generate the fused and repaired latent image. Step 5: Use the reverse mapping module to restore the reconstructed latent image to the final output image.

[0007] The invention is further characterized in that, Step 1 is implemented in the following steps: Step 1.1: Collect image data of the calligraphy rubbings of the "Preface to the Sacred Teachings" from multiple sources. The collected image data includes original rubbings from different historical versions, different scanning resolutions, and different storage formats. This ensures that the data covers various differences such as font style, ink density, paper material and texture, clarity of characters, and rubbing process characteristics, in order to enhance the robustness of the subsequent model to style differences. Step 1.2: Perform basic image preprocessing on the acquired image data, including noise removal, image enhancement, and color normalization. Noise removal is used to eliminate high-frequency interference introduced by scanning; image enhancement improves the clarity of image edges and strokes; color normalization is used to unify the brightness, contrast, and tone style of different images, and improve the overall image structure consistency. Step 1.3: Perform data augmentation operations to expand the sample size and improve the model's generalization ability. Augmentation operations include: slight rotation, scaling, mirror flipping, Gaussian noise superposition, and slight blurring. Step 1.4: Number and classify the processed image samples according to a unified naming rule to construct a standardized dataset, and divide it into a training set and a validation set in a 9:1 ratio. The test set uses all the single-character images of the reserved whole rubbing. Each subset contains character block image samples from different style sources and different structural variations to support the comprehensive training needs of subsequent feature modeling, style alignment and image fusion.

[0008] In step 2, the feature extraction module uses a ResNet50 network pre-trained on ImageNet and removes the last fully connected classification layer to extract features of each rubbing character. The overall hierarchical structure of ResNet50 is as follows: the line passes through a 7×7 convolutional layer with a stride of 2, then through a 3×3 max pooling layer with a stride of 2, and then through 4 residual block groups. Each group contains multiple Bottleneck residual blocks, and each Bottleneck residual block contains 3 convolutional layers: a 1×1 convolution for dimensionality reduction, reducing the number of channels to 1 / 4 of the output channel number, followed by a 3×3 convolution to extract features while keeping the number of channels unchanged, and finally a 1×1 convolution to increase the dimensionality to the original output channels. The output is a four-dimensional tensor [B,C,H,W], where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width. The feature alignment module uses variable convolution, which adds a learnable offset to each sampling point based on the previous feature extraction step. The offset is generated by convolving the input feature map with another 3×3 convolutional layer.

[0009] Step 2 is implemented in the following steps: The feature extraction and alignment module is used to extract latent feature tensors from multiple rubbings of the *Preface to the Sacred Teachings* calligraphy from different sources. Specifically, for each preprocessed rubbing image of [1,3,224,224], after being input into the feature extraction module, it first undergoes a huge 7x7 convolution, resulting in a tensor of [1,64,112,112]. Then, a 3x3 max-pooling layer is applied, followed by four residual block groups to obtain the final four-dimensional tensor with a structure of [1,2048, 7, 7]. This multi-residual block approach effectively extracts and fuses features layer by layer, from low-level to high-level semantic features, ultimately obtaining the latent feature tensors of the image, denoted as […]. Subsequently, these tensors are input into the feature alignment module. Then, for each feature map that needs alignment, they are all input into the same offset generation network with shared weights to generate their respective offsets. Finally, a variability convolution is applied to perform the alignment operation to obtain a fused feature tensor. , as the initial potential input for the diffusion model.

[0010] In step 3, the core network is designed based on the U-Net architecture and supports multiple conditional vector guidance. The network structure specifically includes: Input layer: Receives three inputs – a noisy latent image generated by the diffusion process at time step t. Conditional guiding vector and time step encoding ; Conditional Vector Extraction Layer: Constructs two independent discriminative networks to extract two high-level abstract attributes, sharpness and smoothness, from the features obtained from the previous feature extraction module; The conditional processing unit is constructed by first concatenating two vectors in the last dimension to obtain a vector of shape [B, 2×D]. Then, the concatenated vector is passed through three fully connected layers, with an activation function such as LeakyReLU added after each layer. The first layer expands the dimension, expanding the original [B, 2×D] to [B, 4×D]. The second layer compresses the dimension, compressing the vector to [B, 2×D]. The last layer projects to the target dimension and outputs the final conditional vector c with shape [B,D_target]. The sharpness guide vectors generated by the external discriminant network for the input image are respectively... and smoothness guiding vector The input is fed into the conditional processing unit, where it undergoes fully connected mapping and nonlinear activation to obtain a vector representation of uniform length. The two representations are then concatenated and merged into a comprehensive conditional vector using the following formula. :

[0011] Where ReLU is the activation function, Wc is the weight matrix, and bc is the bias vector; The encoder module consists of multiple levels of convolutional units, each level containing a convolutional layer, a normalization layer, and an activation function. It progressively extracts multi-scale spatial features through downsampling, and fuses conditional vectors using the FiLM method at each encoding stage. In the modulation feature extraction process, the multi-scale feature map output by the encoder is retained in the decoding stage through skip connections; Conditional fusion module: In feature maps at various scales, the conditional modulation layer... Generate scaling factor With offset coefficient And perform feature modulation according to the following formula:

[0012] Decoder module: The structure is symmetrical to that of the encoder, including upsampling operation, skip connection feature fusion, convolutional layer, normalization layer and activation function. FiLM modulation is also introduced in the decoding stage so that conditional guidance continues to play a role in the image restoration process. Output layer: A 1×1 convolution is used to map the number of channels to the same dimension as the input noise tensor, and the output noise residual is predicted. Used for diffusion reverse process update .

[0013] In step 3, the sharpness discrimination network transforms the feature map [B, C, H, W] into a [B, C] vector containing global channel statistics through global average pooling. Then, a 1x1 convolution is used to reduce the number of channels to a smaller value, resulting in the feature map F_detail [B, C_low, H, W]. The standard deviation of F_detail in the spatial dimension [H, W] is calculated to obtain a vector [B, C_low]. Finally, the output vectors of the two channels are concatenated along the channel dimension to obtain the hybrid feature V_fusion [B, C + C_low]. The concatenated hybrid feature V_fusion is then passed through 2-3 fully connected layers, including activation functions such as LeakyReLU and Dropout for non-linear transformation and compression. The final output is a low-dimensional, abstract guiding vector representing sharpness. The shape is [B, D]; The fluency discrimination network first reshapes the feature map [B, C, H, W] into [B, C, N], where N = H * W. That is, [B, C, N] is treated as a sequence containing N feature points. Then, by calculating query, key, and value vectors, each point in the sequence interacts with all other points. Next, through attention weights, it learns the correspondence and association between distant strokes. Finally, the output of the self-attention module is globally averaged and pooled to obtain a [B, C] vector, which encapsulates global structural information. The pooled features are then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression, resulting in a low-dimensional, abstract guiding vector representing fluency. The shape is also [B, D].

[0014] Step 4 is implemented in the following steps: Step 4.1: Convert the fusion structural feature tensor obtained in Step 3 into a fusion tensor. Compared with the sharpness guide vector pre-built in step 2 and smoothness guiding vector In a common-input multi-condition diffusion model, the sharpness guiding vector serves as both the initial state and the controlling guiding condition for diffusion. and smoothness guiding vector Conditional guiding terms are formed by combining them in a splicing manner. The splicing formula is:

[0015] Step 4.2: In the forward reasoning process of the diffusion model, firstly, the initial latent tensor... Gaussian noise is injected to construct the noise evolution trajectory during the diffusion process; subsequently, a denoising network with U-Net as the backbone and integrating a multi-condition injection mechanism is used to receive the current noise state at each diffusion time step t. Conditional vector Including time-encoded information, predicted noise residuals, and the conditional denoising network representation: ; in, This is a potential representation of the current noisy image. For conditional vectors, For diffusion time steps; Step 4.3: Repeat the denoising process of the conditional diffusion model to gradually recover the latent representation of the target fused image from random noise, and obtain a high-quality fused repair latent image representation within the preset number of diffusion steps.

[0016] Step 5 is implemented in the following steps: Step 5.1: Reconstruct the fused latent image obtained from the multi-condition diffusion model in Step 4. As input to the inverse mapping module, the latent image is a low-dimensional, denoised latent representation that carries rich structural and stylistic information; Step 5.2: The reverse mapping module uses a pre-trained decoder network to map the latent representation back to the original image space, restoring the details and texture features of the image. Step 5.3: The decoder consists of multiple layers of transposed convolution, upsampling operation ReLU activation function and batch normalization layer. By expanding the spatial size of the latent image layer by layer, it restores the stroke structure, edge sharpness and ink texture in the calligraphy image, and realizes the conversion from latent space to pixel space. Step 5.4: During training, a combined loss function is used for optimization, in the following form:

[0017] in, This is the primary loss term, used to optimize the model's prediction accuracy for noise. Guiding conditions for a combination of clarity and smoothness. and These represent the losses in perceived consistency and style consistency, respectively. , These are weight hyperparameters; Step 5.5: The decoder output is processed through post-processing operations such as image enhancement, edge sharpening, and artifact suppression to generate a final high-quality, natural, and smooth fused rubbing image.

[0018] The beneficial effects of this invention are as follows: The method for fusing and restoring rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model first collects images of rubbings of the Preface to the Sacred Teachings from multiple sources. A shared coding network is used to extract the latent features of each rubbing image, and a structure alignment module is used to align and fuse the features of multiple images to obtain a unified structural feature vector. Subsequently, a diffusion model containing multiple conditional guiding terms is constructed, including a sharpness guiding term and a fluency guiding term, which are used to control the detail sharpness and stroke fluency of the generated image, respectively. By setting different weights for the two guiding terms, controllable optimization of the restoration target is achieved, guiding the diffusion process to gradually generate a high-quality calligraphy image that combines sharpness and fluency. This method can effectively fuse information from multiple rubbings, enhancing the image and unifying the style while preserving the original appearance of the calligraphy, and has high value for cultural document restoration and digital application. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the method for the restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, as described in this invention. Figure 2 This is a schematic diagram of the overall network architecture of the method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, as proposed in this invention. Figure 3 This is a schematic diagram of the encoder module used in this invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0021] This invention relates to a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, combined with... Figure 1 , Figure 2 The specific steps are as follows: Step 1: Collect images of calligraphy rubbings of the Preface to the Sacred Teachings from multiple sources, perform preprocessing operations such as size normalization, noise removal, and character block cropping on the images, and construct a standardized training dataset; Step 1 is implemented in the following steps: Step 1.1: Collect image data of calligraphy rubbings of the "Preface to the Sacred Teachings" from multiple sources. The collected image data includes, but is not limited to, original rubbing images from different historical versions, different scanning resolutions, and different storage formats. Ensure that it covers a variety of differences such as font style, ink density, paper material and texture, clarity of characters and rubbing process characteristics, so as to enhance the robustness of the subsequent model to style differences. Step 1.2: Perform basic image preprocessing on the acquired image data, including noise removal, image enhancement, and color normalization. Noise removal is used to eliminate high-frequency interference introduced by scanning; image enhancement improves the clarity of image edges and strokes; color normalization is used to unify the brightness, contrast, and tone style of different images, and improve the overall image structure consistency. Step 1.3: Perform data augmentation operations to expand the sample size and improve the model's generalization ability. Augmentation operations include, but are not limited to: slight rotation (±5° to ±15°), scaling (±5% to ±10%), mirror flipping, Gaussian noise addition, and slight blurring. During augmentation, the glyph structure and style features are preserved to ensure the effectiveness and representativeness of the augmented image in the style fusion training task.

[0022] Step 1.4: Number and classify the processed image samples according to a unified naming rule to construct a standardized dataset, and divide it into a training set and a validation set in a 9:1 ratio. The test set uses all the single-character images of the reserved whole rubbing. Each subset contains character block image samples from different style sources and different structural variations to support the comprehensive training needs of subsequent feature modeling, style alignment and image fusion.

[0023] Step 2: Design a multi-image feature extraction and alignment module to obtain the fused structural feature tensor, which serves as the initial potential input for the diffusion model; Step 2 is implemented in the following steps: The feature extraction module uses a ResNet50 network pre-trained on ImageNet and removes the last fully connected classification layer to extract features from each rubbing character. The overall hierarchical structure of ResNet50 is as follows: a line passes through a 7×7 convolutional layer with a stride of 2, then through a 3×3 max pooling layer with a stride of 2, followed by four residual block groups. Each group contains multiple Bottleneck residual blocks, and each Bottleneck residual block contains three convolutional layers: a 1×1 convolution for dimensionality reduction, reducing the number of channels to 1 / 4 of the output channels; a 3×3 convolution for feature extraction while keeping the number of channels unchanged; and a final 1×1 convolution for dimensionality up to the original output channels. The output is a four-dimensional tensor [B,C,H,W], where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width. It contains rich spatial and semantic information.

[0024] The feature alignment module uses variable convolution, which adds a learnable offset to each sampling point based on the previous feature extraction step. The offset is generated by convolving the input feature map with another 3×3 convolutional layer. The feature extraction and alignment module is used to extract latent feature tensors from multiple rubbings of the *Preface to the Sacred Teachings* calligraphy from different sources. Specifically, for each preprocessed rubbing image [1,3,224,224], after input to the feature extraction module, it first undergoes a large 7x7 convolution, resulting in a tensor of [1,64,112,112]. The purpose is to quickly and coarsely extract low-level features and significantly reduce image resolution. Then, a 3x3 max-pooling layer is applied to further compress the data and enhance the translation invariance of the features. Finally, four residual blocks are used to obtain the final four-dimensional tensor, with a structure of [1,2048, 7, 7]. This multi-residual block approach effectively extracts and fuses features layer by layer from low-level to high-level semantic features, ultimately obtaining the latent feature tensors of the image, denoted as […]. Subsequently, these tensors are input into the feature alignment module. Then, for each feature map that needs alignment, they are all input into the same offset generation network with shared weights to generate their respective offsets. Finally, a variability convolution is applied to perform the alignment operation to obtain a fused feature tensor. , as the initial potential input for the diffusion model.

[0025] Step 3: Construct the core network structure of the multi-condition diffusion model. The network adopts a diffusion generation model based on the U-Net architecture and supports multiple conditional vector guidance. Step 3 is implemented in the following steps: The core network is designed based on the U-Net architecture and supports multiple conditional vector guidance. The network structure specifically includes: Input layer: Receives three inputs – a noisy latent image generated by the diffusion process at time step t. Conditional guiding vector and time step encoding ; Conditional Vector Extraction Layer: Constructs two independent discriminative networks to extract two high-level abstract attributes, sharpness and smoothness, from the features obtained from the previous feature extraction module; The sharpness discrimination network transforms the feature map [B, C, H, W] into a [B, C] vector through global average pooling, containing global channel statistics that reflect overall ink density, contrast, and average sharpness. Then, a 1x1 convolution reduces the number of channels to a smaller value, resulting in the feature map F_detail [B, C_low, H, W]. The standard deviation of F_detail in the spatial dimension [H, W] is calculated to obtain a [B, C_low] vector. The standard deviation effectively reflects the richness of texture and local contrast; a higher value generally indicates sharper and clearer details. Finally, the output vectors of the two pathways are concatenated along the channel dimension to obtain the hybrid feature V_fusion [B, C + C_low]. This concatenated hybrid feature V_fusion is then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression. The final output is a low-dimensional, abstract guiding vector representing sharpness. The shape is [B, D]; The fluency discrimination network first reshapes the feature map [B, C, H, W] into [B, C, N], where N = H * W. This treats [B, C, N] as a sequence of N feature points. Then, by calculating query, key, and value vectors, each point in the sequence interacts with all other points. Next, through attention weights, it learns the correspondence and relationships between distant strokes, such as judging whether the starting and ending strokes are continuous and whether the structure is balanced. Finally, the output of the self-attention module is globally averaged and pooled to obtain a [B, C] vector. This vector encapsulates global structural information. The pooled features are then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression, resulting in a low-dimensional, abstract guiding vector representing fluency. The shape is also [B, D]; The conditional processing unit is constructed by first concatenating two vectors in the last dimension to obtain a vector of shape [B, 2×D]. This ensures that both sharpness and fluency information are preserved and fed into subsequent networks. The concatenated vector is then passed through three fully connected layers, each followed by an activation function such as LeakyReLU. The first layer expands the dimension, increasing the original [B, 2×D] to [B, 4×D]. The second layer compresses the dimension, reducing the vector to [B, 2×D]. The final layer projects the vector onto the target dimension, outputting the final conditional vector c with shape [B, D_target]. The sharpness guide vectors generated by the external discriminant network for the input image are respectively... and smoothness guiding vector The input is fed into the conditional processing unit, where it undergoes fully connected mapping and nonlinear activation to obtain a vector representation of uniform length. The two representations are then concatenated and merged into a comprehensive conditional vector using the following formula. :

[0026] Where ReLU is the activation function, Wc is the weight matrix, and bc is the bias vector; Combination Figure 3 The encoder module consists of multiple levels of convolutional units, each level containing a convolutional layer, a normalization layer, and an activation function. It progressively extracts multi-scale spatial features through downsampling. In each encoding stage, it fuses conditional vectors using the Feature-wise Linear Modulation (FiLM) method. In the modulation feature extraction process, the multi-scale feature map output by the encoder is retained in the decoding stage through skip connections; Conditional fusion module: In feature maps at various scales, the conditional modulation layer... Generate scaling factor With offset coefficient (Obtained by mapping from fully connected layers), and feature modulation is performed according to the following formula:

[0027] Decoder module: The structure is symmetrical to that of the encoder, including upsampling operation, skip connection feature fusion, convolutional layer, normalization layer and activation function. FiLM modulation is also introduced in the decoding stage so that conditional guidance continues to play a role in the image restoration process. Output layer: A 1×1 convolution is used to map the number of channels to the same dimension as the input noise tensor, and the output noise residual is predicted. Used for diffusion reverse process update .

[0028] Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model to generate the fused and repaired latent image. Step 4 is implemented in the following steps: Step 4.1: Convert the fusion structural feature tensor obtained in Step 3 into a fusion tensor. Compared with the sharpness guide vector pre-built in step 2 and smoothness guiding vector In a common-input multi-condition diffusion model, the sharpness guiding vector serves as both the initial state and the controlling guiding condition for diffusion. and smoothness guiding vector Conditional guiding terms are formed by combining them in a splicing manner. The splicing formula is:

[0029] Step 4.2: In the forward reasoning process of the diffusion model, firstly, the initial latent tensor... Gaussian noise is injected to construct the noise evolution trajectory during the diffusion process; subsequently, a denoising network with U-Net as the backbone and integrating a multi-condition injection mechanism is used to receive the current noise state at each diffusion time step t. Conditional vector Including time-encoded information, predicted noise residuals, and the conditional denoising network representation: ; in, This is a potential representation of the current noisy image. For conditional vectors, For diffusion time steps; Step 4.3: Repeat the denoising process of the conditional diffusion model to gradually recover the latent representation of the target fused image from random noise, and obtain a high-quality fused repair latent image representation within the preset number of diffusion steps.

[0030] Step 5: Use the reverse mapping module to restore the reconstructed latent image to the final output image.

[0031] Step 5 is implemented in the following steps: Step 5.1: Reconstruct the fused latent image obtained from the multi-condition diffusion model in Step 4. As input to the inverse mapping module, the latent image is a low-dimensional, denoised latent representation that carries rich structural and stylistic information; Step 5.2: The reverse mapping module uses a pre-trained decoder network to map the latent representation back to the original image space, restoring the details and texture features of the image. Step 5.3: The decoder consists of multiple layers of transposed convolution, upsampling operation ReLU activation function and batch normalization layer. By expanding the spatial size of the latent image layer by layer, it restores the stroke structure, edge sharpness and ink texture in the calligraphy image, and realizes the conversion from latent space to pixel space. Step 5.4: During training, a combined loss function is used for optimization, in the following form:

[0032] in, This is the primary loss term, used to optimize the model's prediction accuracy for noise. Guiding conditions for a combination of clarity and smoothness. and These represent the losses in perceived consistency and style consistency, respectively. , These are weight hyperparameters; Step 5.5: The decoder output is processed through post-processing operations such as image enhancement, edge sharpening, and artifact suppression to generate a final high-quality, natural, and smooth fused rubbing image.

[0033] Example 1 This invention relates to a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, combined with... Figure 1 , Figure 2 The specific steps are as follows: Step 1: Collect images of calligraphy rubbings of the Preface to the Sacred Teachings from multiple sources, perform preprocessing operations such as size normalization, noise removal, and character block cropping on the images, and construct a standardized training dataset; Step 2: Design a multi-image feature extraction and alignment module to extract the latent structural feature tensors of the multiple rubbing images, and perform spatial alignment and fusion of the latent features through feature matching or attention mechanism to obtain the fused structural feature tensor as the initial latent input of the diffusion model; Step 3: Construct the core network structure of the multi-condition diffusion model. The network adopts a diffusion generation model based on the U-Net architecture and supports multiple conditional vectors for guidance. During the diffusion generation process, the fused latent structural features are received as the main input and guided by at least two independent conditional vectors to gradually complete the image reconstruction and style optimization. Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model, perform multiple rounds of diffusion denoising process, and gradually generate the fused and repaired potential image. Step 5: The reconstructed latent image is restored to the final output image through the reverse mapping module, resulting in a rubbing image of the Sacred Preface with clear structure, natural brushstrokes, and unified style.

[0034] Example 2 This invention relates to a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, combined with... Figure 1 , Figure 2 The specific steps are as follows: Step 1: Collect images of calligraphy rubbings of the Preface to the Sacred Teachings from multiple sources, perform preprocessing operations such as size normalization, noise removal, and character block cropping on the images, and construct a standardized training dataset; Step 1 is implemented in the following steps: Step 1.1: Collect image data of calligraphy rubbings of the "Preface to the Sacred Teachings" from multiple sources. The collected image data includes, but is not limited to, original rubbing images from different historical versions, different scanning resolutions, and different storage formats. Ensure that it covers a variety of differences such as font style, ink density, paper material and texture, clarity of characters and rubbing process characteristics, so as to enhance the robustness of the subsequent model to style differences. Step 1.2: Perform basic image preprocessing on the acquired image data, including noise removal, image enhancement, and color normalization. Noise removal is used to eliminate high-frequency interference introduced by scanning; image enhancement improves the clarity of image edges and strokes; color normalization is used to unify the brightness, contrast, and tone style of different images, and improve the overall image structure consistency. Step 1.3: Perform data augmentation operations to expand the sample size and improve the model's generalization ability. Augmentation operations include, but are not limited to: slight rotation (±5° to ±15°), scaling (±5% to ±10%), mirror flipping, Gaussian noise addition, and slight blurring. During augmentation, the glyph structure and style features are preserved to ensure the effectiveness and representativeness of the augmented image in the style fusion training task.

[0035] Step 1.4: Number and classify the processed image samples according to a unified naming rule to construct a standardized dataset, and divide it into a training set and a validation set in a 9:1 ratio. The test set uses all the single-character images of the reserved whole rubbing. Each subset contains character block image samples from different style sources and different structural variations to support the comprehensive training needs of subsequent feature modeling, style alignment and image fusion.

[0036] Step 2: Design a multi-image feature extraction and alignment module to obtain the fused structural feature tensor, which serves as the initial potential input for the diffusion model; Step 3: Construct the core network structure of the multi-condition diffusion model. The network adopts a diffusion generation model based on the U-Net architecture and supports multiple conditional vector guidance. Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model to generate the fused and repaired latent image. Step 5: Use the reverse mapping module to restore the reconstructed latent image to the final output image.

[0037] Example 3 This invention relates to a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, combined with... Figure 1 , Figure 2 The specific steps are as follows: Step 1: Collect images of calligraphy rubbings of the Preface to the Sacred Teachings from multiple sources, perform preprocessing operations such as size normalization, noise removal, and character block cropping on the images, and construct a standardized training dataset; Step 1 is implemented in the following steps: Step 1.1: Collect image data of calligraphy rubbings of the "Preface to the Sacred Teachings" from multiple sources. The collected image data includes, but is not limited to, original rubbing images from different historical versions, different scanning resolutions, and different storage formats. Ensure that it covers a variety of differences such as font style, ink density, paper material and texture, clarity of characters and rubbing process characteristics, so as to enhance the robustness of the subsequent model to style differences. Step 1.2: Perform basic image preprocessing on the acquired image data, including noise removal, image enhancement, and color normalization. Noise removal is used to eliminate high-frequency interference introduced by scanning; image enhancement improves the clarity of image edges and strokes; color normalization is used to unify the brightness, contrast, and tone style of different images, and improve the overall image structure consistency. Step 1.3: Perform data augmentation operations to expand the sample size and improve the model's generalization ability. Augmentation operations include, but are not limited to: slight rotation (±5° to ±15°), scaling (±5% to ±10%), mirror flipping, Gaussian noise addition, and slight blurring. During augmentation, the glyph structure and style features are preserved to ensure the effectiveness and representativeness of the augmented image in the style fusion training task.

[0038] Step 1.4: Number and classify the processed image samples according to a unified naming rule to construct a standardized dataset, and divide it into a training set and a validation set in a 9:1 ratio. The test set uses all the single-character images of the reserved whole rubbing. Each subset contains character block image samples from different style sources and different structural variations to support the comprehensive training needs of subsequent feature modeling, style alignment and image fusion.

[0039] Step 2: Design a multi-image feature extraction and alignment module to obtain the fused structural feature tensor, which serves as the initial potential input for the diffusion model; Step 2 is implemented in the following steps: The feature extraction module uses a ResNet50 network pre-trained on ImageNet and removes the last fully connected classification layer to extract features from each rubbing character. The overall hierarchical structure of ResNet50 is as follows: a line passes through a 7×7 convolutional layer with a stride of 2, then through a 3×3 max pooling layer with a stride of 2, followed by four residual block groups. Each group contains multiple Bottleneck residual blocks, and each Bottleneck residual block contains three convolutional layers: a 1×1 convolution for dimensionality reduction, reducing the number of channels to 1 / 4 of the output channels; a 3×3 convolution for feature extraction while keeping the number of channels unchanged; and a final 1×1 convolution for dimensionality up to the original output channels. The output is a four-dimensional tensor [B,C,H,W], where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width. It contains rich spatial and semantic information.

[0040] The feature alignment module uses variable convolution, which adds a learnable offset to each sampling point based on the previous feature extraction step. The offset is generated by convolving the input feature map with another 3×3 convolutional layer. The feature extraction and alignment module is used to extract latent feature tensors from multiple rubbings of the *Preface to the Sacred Teachings* calligraphy from different sources. Specifically, for each preprocessed rubbing image [1,3,224,224], after input to the feature extraction module, it first undergoes a large 7x7 convolution, resulting in a tensor of [1,64,112,112]. The purpose is to quickly and coarsely extract low-level features and significantly reduce image resolution. Then, a 3x3 max-pooling layer is applied to further compress the data and enhance the translation invariance of the features. Finally, four residual blocks are used to obtain the final four-dimensional tensor, with a structure of [1,2048, 7, 7]. This multi-residual block approach effectively extracts and fuses features layer by layer from low-level to high-level semantic features, ultimately obtaining the latent feature tensors of the image, denoted as […]. Subsequently, these tensors are input into the feature alignment module. Then, for each feature map that needs alignment, they are all input into the same offset generation network with shared weights to generate their respective offsets. Finally, a variability convolution is applied to perform the alignment operation to obtain a fused feature tensor. , as the initial potential input for the diffusion model.

[0041] Step 3: Construct the core network structure of the multi-condition diffusion model. The network adopts a diffusion generation model based on the U-Net architecture and supports multiple conditional vector guidance. Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model to generate the fused and repaired latent image. Step 5: Use the reverse mapping module to restore the reconstructed latent image to the final output image.

[0042] Example 4 This invention relates to a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, combined with... Figure 1 , Figure 2 The specific steps are as follows: Step 1: Collect images of calligraphy rubbings of the Preface to the Sacred Teachings from multiple sources, perform preprocessing operations such as size normalization, noise removal, and character block cropping on the images, and construct a standardized training dataset; Step 1 is implemented in the following steps: Step 1.1: Collect image data of calligraphy rubbings of the "Preface to the Sacred Teachings" from multiple sources. The collected image data includes, but is not limited to, original rubbing images from different historical versions, different scanning resolutions, and different storage formats. Ensure that it covers a variety of differences such as font style, ink density, paper material and texture, clarity of characters and rubbing process characteristics, so as to enhance the robustness of the subsequent model to style differences. Step 1.2: Perform basic image preprocessing on the acquired image data, including noise removal, image enhancement, and color normalization. Noise removal is used to eliminate high-frequency interference introduced by scanning; image enhancement improves the clarity of image edges and strokes; color normalization is used to unify the brightness, contrast, and tone style of different images, and improve the overall image structure consistency. Step 1.3: Perform data augmentation operations to expand the sample size and improve the model's generalization ability. Augmentation operations include, but are not limited to: slight rotation (±5° to ±15°), scaling (±5% to ±10%), mirror flipping, Gaussian noise addition, and slight blurring. During augmentation, the glyph structure and style features are preserved to ensure the effectiveness and representativeness of the augmented image in the style fusion training task.

[0043] Step 1.4: Number and classify the processed image samples according to a unified naming rule to construct a standardized dataset, and divide it into a training set and a validation set in a 9:1 ratio. The test set uses all the single-character images of the reserved whole rubbing. Each subset contains character block image samples from different style sources and different structural variations to support the comprehensive training needs of subsequent feature modeling, style alignment and image fusion.

[0044] Step 2: Design a multi-image feature extraction and alignment module to obtain the fused structural feature tensor, which serves as the initial potential input for the diffusion model; Step 2 is implemented in the following steps: The feature extraction module uses a ResNet50 network pre-trained on ImageNet and removes the last fully connected classification layer to extract features from each rubbing character. The overall hierarchical structure of ResNet50 is as follows: a line passes through a 7×7 convolutional layer with a stride of 2, then through a 3×3 max pooling layer with a stride of 2, followed by four residual block groups. Each group contains multiple Bottleneck residual blocks, and each Bottleneck residual block contains three convolutional layers: a 1×1 convolution for dimensionality reduction, reducing the number of channels to 1 / 4 of the output channels; a 3×3 convolution for feature extraction while keeping the number of channels unchanged; and a final 1×1 convolution for dimensionality up to the original output channels. The output is a four-dimensional tensor [B,C,H,W], where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width. It contains rich spatial and semantic information.

[0045] The feature alignment module uses variable convolution, which adds a learnable offset to each sampling point based on the previous feature extraction step. The offset is generated by convolving the input feature map with another 3×3 convolutional layer. The feature extraction and alignment module is used to extract latent feature tensors from multiple rubbings of the *Preface to the Sacred Teachings* calligraphy from different sources. Specifically, for each preprocessed rubbing image [1,3,224,224], after input to the feature extraction module, it first undergoes a large 7x7 convolution, resulting in a tensor of [1,64,112,112]. The purpose is to quickly and coarsely extract low-level features and significantly reduce image resolution. Then, a 3x3 max-pooling layer is applied to further compress the data and enhance the translation invariance of the features. Finally, four residual blocks are used to obtain the final four-dimensional tensor, with a structure of [1,2048, 7, 7]. This multi-residual block approach effectively extracts and fuses features layer by layer from low-level to high-level semantic features, ultimately obtaining the latent feature tensors of the image, denoted as […]. Subsequently, these tensors are input into the feature alignment module. Then, for each feature map that needs alignment, they are all input into the same offset generation network with shared weights to generate their respective offsets. Finally, a variability convolution is applied to perform the alignment operation to obtain a fused feature tensor. , as the initial potential input for the diffusion model.

[0046] Step 3: Construct the core network structure of the multi-condition diffusion model. The network adopts a diffusion generation model based on the U-Net architecture and supports multiple conditional vector guidance. Step 3 is implemented in the following steps: The core network is designed based on the U-Net architecture and supports multiple conditional vector guidance. The network structure specifically includes: Input layer: Receives three inputs – a noisy latent image generated by the diffusion process at time step t. Conditional guiding vector and time step encoding ; Conditional Vector Extraction Layer: Constructs two independent discriminative networks to extract two high-level abstract attributes, sharpness and smoothness, from the features obtained from the previous feature extraction module; The sharpness discrimination network transforms the feature map [B, C, H, W] into a [B, C] vector through global average pooling, containing global channel statistics that reflect overall ink density, contrast, and average sharpness. Then, a 1x1 convolution reduces the number of channels to a smaller value, resulting in the feature map F_detail [B, C_low, H, W]. The standard deviation of F_detail in the spatial dimension [H, W] is calculated to obtain a [B, C_low] vector. The standard deviation effectively reflects the richness of texture and local contrast; a higher value generally indicates sharper and clearer details. Finally, the output vectors of the two pathways are concatenated along the channel dimension to obtain the hybrid feature V_fusion [B, C + C_low]. This concatenated hybrid feature V_fusion is then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression. The final output is a low-dimensional, abstract guiding vector representing sharpness. The shape is [B, D]; The fluency discrimination network first reshapes the feature map [B, C, H, W] into [B, C, N], where N = H * W. This treats [B, C, N] as a sequence of N feature points. Then, by calculating query, key, and value vectors, each point in the sequence interacts with all other points. Next, through attention weights, it learns the correspondence and relationships between distant strokes, such as judging whether the starting and ending strokes are continuous and whether the structure is balanced. Finally, the output of the self-attention module is globally averaged and pooled to obtain a [B, C] vector. This vector encapsulates global structural information. The pooled features are then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression, resulting in a low-dimensional, abstract guiding vector representing fluency. The shape is also [B, D]; The conditional processing unit is constructed by first concatenating two vectors in the last dimension to obtain a vector of shape [B, 2×D]. This ensures that both sharpness and fluency information are preserved and fed into subsequent networks. The concatenated vector is then passed through three fully connected layers, each followed by an activation function such as LeakyReLU. The first layer expands the dimension, increasing the original [B, 2×D] to [B, 4×D]. The second layer compresses the dimension, reducing the vector to [B, 2×D]. The final layer projects the vector onto the target dimension, outputting the final conditional vector c with shape [B, D_target]. The sharpness guide vectors generated by the external discriminant network for the input image are respectively... and smoothness guiding vector The input is fed into the conditional processing unit, where it undergoes fully connected mapping and nonlinear activation to obtain a vector representation of uniform length. The two representations are then concatenated and merged into a comprehensive conditional vector using the following formula. :

[0047] Where ReLU is the activation function, Wc is the weight matrix, and bc is the bias vector; The encoder module consists of multiple levels of convolutional units, each level containing a convolutional layer, a normalization layer, and an activation function. It progressively extracts multi-scale spatial features through downsampling. In each encoding stage, it fuses conditional vectors using Feature-wise Linear Modulation (FiLM). In the modulation feature extraction process, the multi-scale feature map output by the encoder is retained in the decoding stage through skip connections; Conditional fusion module: In feature maps at various scales, the conditional modulation layer... Generate scaling factor With offset coefficient (Obtained by mapping from fully connected layers), and feature modulation is performed according to the following formula:

[0048] Decoder module: The structure is symmetrical to that of the encoder, including upsampling operation, skip connection feature fusion, convolutional layer, normalization layer and activation function. FiLM modulation is also introduced in the decoding stage so that conditional guidance continues to play a role in the image restoration process. Output layer: A 1×1 convolution is used to map the number of channels to the same dimension as the input noise tensor, and the output noise residual is predicted. Used for diffusion reverse process update .

[0049] Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model to generate the fused and repaired latent image. Step 5: Use the reverse mapping module to restore the reconstructed latent image to the final output image.

[0050] Example 5 This invention relates to a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, combined with... Figure 1 , Figure 2 The specific steps are as follows: Step 1: Collect images of calligraphy rubbings of the Preface to the Sacred Teachings from multiple sources, perform preprocessing operations such as size normalization, noise removal, and character block cropping on the images, and construct a standardized training dataset; Step 1 is implemented in the following steps: Step 1.1: Collect image data of calligraphy rubbings of the "Preface to the Sacred Teachings" from multiple sources. The collected image data includes, but is not limited to, original rubbing images from different historical versions, different scanning resolutions, and different storage formats. Ensure that it covers a variety of differences such as font style, ink density, paper material and texture, clarity of characters and rubbing process characteristics, so as to enhance the robustness of the subsequent model to style differences. Step 1.2: Perform basic image preprocessing on the acquired image data, including noise removal, image enhancement, and color normalization. Noise removal is used to eliminate high-frequency interference introduced by scanning; image enhancement improves the clarity of image edges and strokes; color normalization is used to unify the brightness, contrast, and tone style of different images, and improve the overall image structure consistency. Step 1.3: Perform data augmentation operations to expand the sample size and improve the model's generalization ability. Augmentation operations include, but are not limited to: slight rotation (±5° to ±15°), scaling (±5% to ±10%), mirror flipping, Gaussian noise addition, and slight blurring. During augmentation, the glyph structure and style features are preserved to ensure the effectiveness and representativeness of the augmented image in the style fusion training task.

[0051] Step 1.4: Number and classify the processed image samples according to a unified naming rule to construct a standardized dataset, and divide it into a training set and a validation set in a 9:1 ratio. The test set uses all the single-character images of the reserved whole rubbing. Each subset contains character block image samples from different style sources and different structural variations to support the comprehensive training needs of subsequent feature modeling, style alignment and image fusion.

[0052] Step 2: Design a multi-image feature extraction and alignment module to obtain the fused structural feature tensor, which serves as the initial potential input for the diffusion model; Step 2 is implemented in the following steps: The feature extraction module uses a ResNet50 network pre-trained on ImageNet and removes the last fully connected classification layer to extract features from each rubbing character. The overall hierarchical structure of ResNet50 is as follows: a line passes through a 7×7 convolutional layer with a stride of 2, then through a 3×3 max pooling layer with a stride of 2, followed by four residual block groups. Each group contains multiple Bottleneck residual blocks, and each Bottleneck residual block contains three convolutional layers: a 1×1 convolution for dimensionality reduction, reducing the number of channels to 1 / 4 of the output channels; a 3×3 convolution for feature extraction while keeping the number of channels unchanged; and a final 1×1 convolution for dimensionality up to the original output channels. The output is a four-dimensional tensor [B,C,H,W], where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width. It contains rich spatial and semantic information.

[0053] The feature alignment module uses variable convolution, which adds a learnable offset to each sampling point based on the previous feature extraction step. The offset is generated by convolving the input feature map with another 3×3 convolutional layer. The feature extraction and alignment module is used to extract latent feature tensors from multiple rubbings of the *Preface to the Sacred Teachings* calligraphy from different sources. Specifically, for each preprocessed rubbing image [1,3,224,224], after input to the feature extraction module, it first undergoes a large 7x7 convolution, resulting in a tensor of [1,64,112,112]. The purpose is to quickly and coarsely extract low-level features and significantly reduce image resolution. Then, a 3x3 max-pooling layer is applied to further compress the data and enhance the translation invariance of the features. Finally, four residual blocks are used to obtain the final four-dimensional tensor, with a structure of [1,2048, 7, 7]. This multi-residual block approach effectively extracts and fuses features layer by layer from low-level to high-level semantic features, ultimately obtaining the latent feature tensors of the image, denoted as […]. Subsequently, these tensors are input into the feature alignment module. Then, for each feature map that needs alignment, they are all input into the same offset generation network with shared weights to generate their respective offsets. Finally, a variability convolution is applied to perform the alignment operation to obtain a fused feature tensor. , as the initial potential input for the diffusion model.

[0054] Step 3: Construct the core network structure of the multi-condition diffusion model. The network adopts a diffusion generation model based on the U-Net architecture and supports multiple conditional vector guidance. Step 3 is implemented in the following steps: The core network is designed based on the U-Net architecture and supports multiple conditional vector guidance. The network structure specifically includes: Input layer: Receives three inputs – a noisy latent image generated by the diffusion process at time step t. Conditional guiding vector and time step encoding ; Conditional Vector Extraction Layer: Constructs two independent discriminative networks to extract two high-level abstract attributes, sharpness and smoothness, from the features obtained from the previous feature extraction module; The sharpness discrimination network transforms the feature map [B, C, H, W] into a [B, C] vector through global average pooling, containing global channel statistics that reflect overall ink density, contrast, and average sharpness. Then, a 1x1 convolution reduces the number of channels to a smaller value, resulting in the feature map F_detail [B, C_low, H, W]. The standard deviation of F_detail in the spatial dimension [H, W] is calculated to obtain a [B, C_low] vector. The standard deviation effectively reflects the richness of texture and local contrast; a higher value generally indicates sharper and clearer details. Finally, the output vectors of the two pathways are concatenated along the channel dimension to obtain the hybrid feature V_fusion [B, C + C_low]. This concatenated hybrid feature V_fusion is then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression. The final output is a low-dimensional, abstract guiding vector representing sharpness. The shape is [B, D]; The fluency discrimination network first reshapes the feature map [B, C, H, W] into [B, C, N], where N = H * W. This treats [B, C, N] as a sequence of N feature points. Then, by calculating query, key, and value vectors, each point in the sequence interacts with all other points. Next, through attention weights, it learns the correspondence and relationships between distant strokes, such as judging whether the starting and ending strokes are continuous and whether the structure is balanced. Finally, the output of the self-attention module is globally averaged and pooled to obtain a [B, C] vector. This vector encapsulates global structural information. The pooled features are then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression, resulting in a low-dimensional, abstract guiding vector representing fluency. The shape is also [B, D]; The conditional processing unit is constructed by first concatenating two vectors in the last dimension to obtain a vector of shape [B, 2×D]. This ensures that both sharpness and fluency information are preserved and fed into subsequent networks. The concatenated vector is then passed through three fully connected layers, each followed by an activation function such as LeakyReLU. The first layer expands the dimension, increasing the original [B, 2×D] to [B, 4×D]. The second layer compresses the dimension, reducing the vector to [B, 2×D]. The final layer projects the vector onto the target dimension, outputting the final conditional vector c with shape [B, D_target]. The sharpness guide vectors generated by the external discriminant network for the input image are respectively... and smoothness guiding vector The input is fed into the conditional processing unit, where it undergoes fully connected mapping and nonlinear activation to obtain a vector representation of uniform length. The two representations are then concatenated and merged into a comprehensive conditional vector using the following formula. :

[0055] Where ReLU is the activation function, Wc is the weight matrix, and bc is the bias vector; The encoder module consists of multiple levels of convolutional units, each level containing a convolutional layer, a normalization layer, and an activation function. It progressively extracts multi-scale spatial features through downsampling. In each encoding stage, it fuses conditional vectors using Feature-wise Linear Modulation (FiLM). In the modulation feature extraction process, the multi-scale feature map output by the encoder is retained in the decoding stage through skip connections; Conditional fusion module: In feature maps at various scales, the conditional modulation layer... Generate scaling factor With offset coefficient (Obtained by mapping from fully connected layers), and feature modulation is performed according to the following formula:

[0056] Decoder module: The structure is symmetrical to that of the encoder, including upsampling operation, skip connection feature fusion, convolutional layer, normalization layer and activation function. FiLM modulation is also introduced in the decoding stage so that conditional guidance continues to play a role in the image restoration process. Output layer: A 1×1 convolution is used to map the number of channels to the same dimension as the input noise tensor, and the output noise residual is predicted. Used for diffusion reverse process update .

[0057] Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model to generate the fused and repaired latent image. Step 4 is implemented in the following steps: Step 4.1: Convert the fusion structural feature tensor obtained in Step 3 into a fusion tensor. Compared with the sharpness guide vector pre-built in step 2 and smoothness guiding vector In a common-input multi-condition diffusion model, the sharpness guiding vector serves as both the initial state and the controlling guiding condition for diffusion. and smoothness guiding vector Conditional guiding terms are formed by combining them in a splicing manner. The splicing formula is:

[0058] Step 4.2: In the forward reasoning process of the diffusion model, firstly, the initial latent tensor... Gaussian noise is injected to construct the noise evolution trajectory during the diffusion process; subsequently, a denoising network with U-Net as the backbone and integrating a multi-condition injection mechanism is used to receive the current noise state at each diffusion time step t. Conditional vector Including time-encoded information, predicted noise residuals, and the conditional denoising network representation: ; in, This is a potential representation of the current noisy image. For conditional vectors, For diffusion time steps; Step 4.3: Repeat the denoising process of the conditional diffusion model to gradually recover the latent representation of the target fused image from random noise, and obtain a high-quality fused repair latent image representation within the preset number of diffusion steps.

[0059] Step 5: Use the reverse mapping module to restore the reconstructed latent image to the final output image.

[0060] Example 6 This invention relates to a method for the fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, combined with... Figure 1 , Figure 2 The specific steps are as follows: Step 1: Collect images of calligraphy rubbings of the Preface to the Sacred Teachings from multiple sources, perform preprocessing operations such as size normalization, noise removal, and character block cropping on the images, and construct a standardized training dataset; Step 1 is implemented in the following steps: Step 1.1: Collect image data of calligraphy rubbings of the "Preface to the Sacred Teachings" from multiple sources. The collected image data includes, but is not limited to, original rubbing images from different historical versions, different scanning resolutions, and different storage formats. Ensure that it covers a variety of differences such as font style, ink density, paper material and texture, clarity of characters and rubbing process characteristics, so as to enhance the robustness of the subsequent model to style differences. Step 1.2: Perform basic image preprocessing on the acquired image data, including noise removal, image enhancement, and color normalization. Noise removal is used to eliminate high-frequency interference introduced by scanning; image enhancement improves the clarity of image edges and strokes; color normalization is used to unify the brightness, contrast, and tone style of different images, and improve the overall image structure consistency. Step 1.3: Perform data augmentation operations to expand the sample size and improve the model's generalization ability. Augmentation operations include, but are not limited to: slight rotation (±5° to ±15°), scaling (±5% to ±10%), mirror flipping, Gaussian noise addition, and slight blurring. During augmentation, the glyph structure and style features are preserved to ensure the effectiveness and representativeness of the augmented image in the style fusion training task.

[0061] Step 1.4: Number and classify the processed image samples according to a unified naming rule to construct a standardized dataset, and divide it into a training set and a validation set in a 9:1 ratio. The test set uses all the single-character images of the reserved whole rubbing. Each subset contains character block image samples from different style sources and different structural variations to support the comprehensive training needs of subsequent feature modeling, style alignment and image fusion.

[0062] Step 2: Design a multi-image feature extraction and alignment module to obtain the fused structural feature tensor, which serves as the initial potential input for the diffusion model; Step 2 is implemented in the following steps: The feature extraction module uses a ResNet50 network pre-trained on ImageNet and removes the last fully connected classification layer to extract features from each rubbing character. The overall hierarchical structure of ResNet50 is as follows: a line passes through a 7×7 convolutional layer with a stride of 2, then through a 3×3 max pooling layer with a stride of 2, followed by four residual block groups. Each group contains multiple Bottleneck residual blocks, and each Bottleneck residual block contains three convolutional layers: a 1×1 convolution for dimensionality reduction, reducing the number of channels to 1 / 4 of the output channels; a 3×3 convolution for feature extraction while keeping the number of channels unchanged; and a final 1×1 convolution for dimensionality up to the original output channels. The output is a four-dimensional tensor [B,C,H,W], where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width. It contains rich spatial and semantic information.

[0063] The feature alignment module uses variable convolution, which adds a learnable offset to each sampling point based on the previous feature extraction step. The offset is generated by convolving the input feature map with another 3×3 convolutional layer. The feature extraction and alignment module is used to extract latent feature tensors from multiple rubbings of the *Preface to the Sacred Teachings* calligraphy from different sources. Specifically, for each preprocessed rubbing image [1,3,224,224], after input to the feature extraction module, it first undergoes a large 7x7 convolution, resulting in a tensor of [1,64,112,112]. The purpose is to quickly and coarsely extract low-level features and significantly reduce image resolution. Then, a 3x3 max-pooling layer is applied to further compress the data and enhance the translation invariance of the features. Finally, four residual blocks are used to obtain the final four-dimensional tensor, with a structure of [1,2048, 7, 7]. This multi-residual block approach effectively extracts and fuses features layer by layer from low-level to high-level semantic features, ultimately obtaining the latent feature tensors of the image, denoted as […]. Subsequently, these tensors are input into the feature alignment module. Then, for each feature map that needs alignment, they are all input into the same offset generation network with shared weights to generate their respective offsets. Finally, a variability convolution is applied to perform the alignment operation to obtain a fused feature tensor. , as the initial potential input for the diffusion model.

[0064] Step 3: Construct the core network structure of the multi-condition diffusion model. The network adopts a diffusion generation model based on the U-Net architecture and supports multiple conditional vector guidance. Step 3 is implemented in the following steps: The core network is designed based on the U-Net architecture and supports multiple conditional vector guidance. The network structure specifically includes: Input layer: Receives three inputs – a noisy latent image generated by the diffusion process at time step t. Conditional guiding vector and time step encoding ; Conditional Vector Extraction Layer: Constructs two independent discriminative networks to extract two high-level abstract attributes, sharpness and smoothness, from the features obtained from the previous feature extraction module; The sharpness discrimination network transforms the feature map [B, C, H, W] into a [B, C] vector through global average pooling, containing global channel statistics that reflect overall ink density, contrast, and average sharpness. Then, a 1x1 convolution reduces the number of channels to a smaller value, resulting in the feature map F_detail [B, C_low, H, W]. The standard deviation of F_detail in the spatial dimension [H, W] is calculated to obtain a [B, C_low] vector. The standard deviation effectively reflects the richness of texture and local contrast; a higher value generally indicates sharper and clearer details. Finally, the output vectors of the two pathways are concatenated along the channel dimension to obtain the hybrid feature V_fusion [B, C + C_low]. This concatenated hybrid feature V_fusion is then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression. The final output is a low-dimensional, abstract guiding vector representing sharpness. The shape is [B, D]; The fluency discrimination network first reshapes the feature map [B, C, H, W] into [B, C, N]. N = H * W, treating [B, C, N] as a sequence containing N feature points. Then, by calculating query, key, and value vectors, each point in the sequence interacts with all other points. Next, through attention weights, the system learns the correspondence and connections between distant strokes, such as determining whether the starting and ending strokes are continuous and whether the structure is balanced. Finally, the output of the self-attention module is globally averaged and pooled to obtain a [B, C] vector. This vector encapsulates global structural information. The pooled features are then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression, resulting in a low-dimensional, abstract guiding vector representing fluency. The shape is also [B, D]; The conditional processing unit is constructed by first concatenating two vectors in the last dimension to obtain a vector of shape [B, 2×D]. This ensures that both sharpness and fluency information are preserved and fed into subsequent networks. The concatenated vector is then passed through three fully connected layers, each followed by an activation function such as LeakyReLU. The first layer expands the dimension, increasing the original [B, 2×D] to [B, 4×D]. The second layer compresses the dimension, reducing the vector to [B, 2×D]. The final layer projects the vector onto the target dimension, outputting the final conditional vector. c The shape is [B, D_target]; The sharpness guide vectors generated by the external discriminant network for the input image are respectively... and smoothness guiding vector The input is fed into the conditional processing unit, where it undergoes fully connected mapping and nonlinear activation to obtain a vector representation of uniform length. The two representations are then concatenated and merged into a comprehensive conditional vector using the following formula. c :

[0065] Where ReLU is the activation function, Wc is the weight matrix, and bc is the bias vector; The encoder module consists of multiple levels of convolutional units, each level containing a convolutional layer, a normalization layer, and an activation function. It progressively extracts multi-scale spatial features through downsampling. In each encoding stage, it fuses conditional vectors using Feature-wise Linear Modulation (FiLM). c In the modulation feature extraction process, the multi-scale feature map output by the encoder is retained in the decoding stage through skip connections; Conditional fusion module: In feature maps at various scales, the conditional modulation layer... c Generate scaling factor With offset coefficient (Obtained by mapping from fully connected layers), and feature modulation is performed according to the following formula:

[0066] Decoder module: The structure is symmetrical to that of the encoder, including upsampling operation, skip connection feature fusion, convolutional layer, normalization layer and activation function. FiLM modulation is also introduced in the decoding stage so that conditional guidance continues to play a role in the image restoration process. Output layer: A 1×1 convolution is used to map the number of channels to the same dimension as the input noise tensor, and the output noise residual is predicted. Used for diffusion reverse process update .

[0067] Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model to generate the fused and repaired latent image. Step 5: Use the reverse mapping module to restore the reconstructed latent image to the final output image.

[0068] Step 5 is implemented in the following steps: Step 5.1: Reconstruct the fused latent image obtained from the multi-condition diffusion model in Step 4. As input to the inverse mapping module, the latent image is a low-dimensional, denoised latent representation that carries rich structural and stylistic information; Step 5.2: The reverse mapping module uses a pre-trained decoder network to map the latent representation back to the original image space, restoring the details and texture features of the image. Step 5.3: The decoder consists of multiple layers of transposed convolution, upsampling operation ReLU activation function and batch normalization layer. By expanding the spatial size of the latent image layer by layer, it restores the stroke structure, edge sharpness and ink texture in the calligraphy image, and realizes the conversion from latent space to pixel space. Step 5.4: During training, a combined loss function is used for optimization, in the following form:

[0069] in, This is the primary loss term, used to optimize the model's prediction accuracy for noise. Guiding conditions for a combination of clarity and smoothness. and These represent the losses in perceived consistency and style consistency, respectively. , These are weight hyperparameters; Step 5.5: The decoder output is processed through post-processing operations such as image enhancement, edge sharpening, and artifact suppression to generate a final high-quality, natural, and smooth fused rubbing image.

Claims

1. A method for the restoration and fusion of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model, characterized in that, The specific steps are as follows: Step 1: Collect images of rubbings of the Preface to the Sacred Teachings from multiple sources, preprocess the images, and construct a standardized training dataset; Step 2: Design a multi-image feature extraction and alignment module to obtain the fused structural feature tensor; Step 3: Construct the core network structure of the multi-conditional diffusion model and support multiple conditional vector guidance; Step 4: Input the fusion structural feature tensor obtained in Step 2 and the multiple conditional guidance terms in Step 3 into the diffusion model to generate the fused and repaired latent image. Step 5: Use the reverse mapping module to restore the reconstructed latent image to the final output image.

2. The method for fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model according to claim 1, characterized in that, Step 1 is implemented in the following steps: Step 1.1: Collect image data of the calligraphy rubbings of the "Preface to the Sacred Teachings" from multiple sources. The collected image data includes original rubbings from different historical versions, different scanning resolutions, and different storage formats. This ensures that the data covers various differences such as font style, ink density, paper material and texture, clarity of characters, and rubbing process characteristics, in order to enhance the robustness of the subsequent model to style differences. Step 1.2: Perform basic image preprocessing on the acquired image data, including noise removal, image enhancement, and color normalization. Noise removal is used to eliminate high-frequency interference introduced by scanning; image enhancement improves the clarity of image edges and strokes; color normalization is used to unify the brightness, contrast, and tone style of different images, and improve the overall image structure consistency. Step 1.3: Perform data augmentation operations to expand the sample size and improve the model's generalization ability. Augmentation operations include: slight rotation, scaling, mirror flipping, Gaussian noise superposition, and slight blurring. Step 1.4: Number and classify the processed image samples according to a unified naming rule to construct a standardized dataset, and divide it into a training set and a validation set in a 9:1 ratio. The test set uses all the single-character images of the reserved whole rubbing. Each subset contains character block image samples from different style sources and different structural variations to support the comprehensive training needs of subsequent feature modeling, style alignment and image fusion.

3. The method for fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model according to claim 2, characterized in that, In step 2, the feature extraction module uses a ResNet50 network pre-trained on ImageNet and removes the last fully connected classification layer to extract features of each rubbing character. The overall hierarchical structure of ResNet50 is as follows: the line passes through a 7×7 convolutional layer with a stride of 2, then through a 3×3 max pooling layer with a stride of 2, and then through 4 residual block groups. Each group contains multiple Bottleneck residual blocks, and each Bottleneck residual block contains 3 convolutional layers: a 1×1 convolution for dimensionality reduction, reducing the number of channels to 1 / 4 of the output channel number; a 3×3 convolution for feature extraction while keeping the number of channels unchanged; and a final 1×1 convolution for dimensionality up to the original output channels. The output is a four-dimensional tensor [B,C,H,W], where B represents the batch size, C represents the number of channels, H represents the height, and W represents the width. The feature alignment module uses variable convolution, which adds a learnable offset to each sampling point based on the previous feature extraction step. The offset is generated by convolving the input feature map with another 3×3 convolutional layer.

4. The method for fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model according to claim 3, characterized in that, Step 2 is implemented in the following steps: The feature extraction and alignment module is used to extract latent feature tensors from multiple rubbings of the *Preface to the Sacred Teachings* calligraphy from different sources. Specifically, for each preprocessed rubbing image of [1,3,224,224], after being input into the feature extraction module, it first undergoes a huge 7x7 convolution, resulting in a tensor of [1,64,112,112]. Then, a 3x3 max-pooling layer is applied, followed by four residual block groups to obtain the final four-dimensional tensor with a structure of [1,2048, 7, 7]. This multi-residual block approach effectively extracts and fuses features layer by layer, from low-level to high-level semantic features, ultimately obtaining the latent feature tensors of the image, denoted as […]. Subsequently, these tensors are input into the feature alignment module. Then, for each feature map that needs alignment, they are all input into the same offset generation network with shared weights to generate their respective offsets. Finally, a variability convolution is applied to perform the alignment operation to obtain a fused feature tensor. , as the initial potential input for the diffusion model.

5. The method for fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model according to claim 4, characterized in that, In step 3, the core network is designed based on the U-Net architecture and supports multiple conditional vector guidance. The network structure specifically includes: Input layer: Receives three inputs – a noisy latent image generated by the diffusion process at time step t. Conditional guiding vector and time step encoding ; Conditional Vector Extraction Layer: Constructs two independent discriminative networks to extract two high-level abstract attributes, sharpness and smoothness, from the features obtained from the previous feature extraction module; The conditional processing unit is constructed by first concatenating two vectors in the last dimension to obtain a vector of shape [B, 2×D]. Then, the concatenated vector is passed through three fully connected layers, with an activation function such as LeakyReLU added after each layer. The first layer expands the dimension, expanding the original [B, 2×D] to [B, 4×D]. The second layer compresses the dimension, compressing the vector to [B, 2×D]. The last layer projects to the target dimension and outputs the final conditional vector c with shape [B,D_target]. The sharpness guide vectors generated by the external discriminant network for the input image are respectively... and smoothness guiding vector The input is fed into the conditional processing unit, where it undergoes fully connected mapping and nonlinear activation to obtain a vector representation of uniform length. The two representations are then concatenated and merged into a comprehensive conditional vector using the following formula. : Where ReLU is the activation function, Wc is the weight matrix, and bc is the bias vector; The encoder module consists of multiple levels of convolutional units, each level containing a convolutional layer, a normalization layer, and an activation function. It progressively extracts multi-scale spatial features through downsampling, and fuses conditional vectors using the FiLM method at each encoding stage. In the modulation feature extraction process, the multi-scale feature map output by the encoder is retained in the decoding stage through skip connections; Conditional fusion module: In feature maps at various scales, the conditional modulation layer... Generate scaling factor With offset coefficient And perform feature modulation according to the following formula: Decoder module: The structure is symmetrical to that of the encoder, including upsampling operation, skip connection feature fusion, convolutional layer, normalization layer and activation function. FiLM modulation is also introduced in the decoding stage so that conditional guidance continues to play a role in the image restoration process. Output layer: A 1×1 convolution is used to map the number of channels to the same dimension as the input noise tensor, and the output noise residual is predicted. Used for diffusion reverse process update .

6. The method for fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model according to claim 5, characterized in that, In step 3, the sharpness discrimination network transforms the feature map [B, C, H, W] into a [B, C] vector containing global channel statistics through global average pooling. Then, a 1x1 convolution is used to reduce the number of channels to a smaller value, resulting in the feature map F_detail [B, C_low, H, W]. The standard deviation of F_detail in the spatial dimension [H, W] is calculated to obtain a vector [B, C_low]. Finally, the output vectors of the two channels are concatenated along the channel dimension to obtain the hybrid feature V_fusion [B, C + C_low]. The concatenated hybrid feature V_fusion is then passed through 2-3 fully connected layers, including activation functions such as LeakyReLU and Dropout for non-linear transformation and compression. The final output is a low-dimensional, abstract guiding vector representing sharpness. The shape is [B, D]; The fluency discrimination network first reshapes the feature map [B, C, H, W] into [B, C, N], where N = H * W. That is, [B, C, N] is treated as a sequence containing N feature points. Then, by calculating query, key, and value vectors, each point in the sequence interacts with all other points. Next, through attention weights, it learns the correspondence and association between distant strokes. Finally, the output of the self-attention module is globally averaged and pooled to obtain a [B, C] vector, which encapsulates global structural information. The pooled features are then passed through 2-3 fully connected layers, incorporating activation functions such as LeakyReLU and Dropout for non-linear transformation and compression, resulting in a low-dimensional, abstract guiding vector representing fluency. The shape is also [B, D].

7. The method for fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model according to claim 6, characterized in that, Step 4 is implemented in the following steps: Step 4.1: Convert the fusion structural feature tensor obtained in Step 3 into a fusion tensor. Compared with the sharpness guide vector pre-built in step 2 and smoothness guiding vector In a common-input multi-condition diffusion model, the sharpness guiding vector serves as both the initial state and the controlling guiding condition for diffusion. and smoothness guiding vector Conditional guiding terms are formed by combining them in a splicing manner. The splicing formula is: Step 4.2: In the forward reasoning process of the diffusion model, firstly, the initial latent tensor... Inject Gaussian noise to construct the noise evolution trajectory during the diffusion process; Subsequently, a denoising network with U-Net as its backbone and integrating a multi-condition injection mechanism is adopted to receive the current noise state at each diffusion time step t. Conditional vector Including time-encoded information, predicted noise residuals, and the conditional denoising network representation: ; in, This is a potential representation of the current noisy image. For conditional vectors, For diffusion time steps; Step 4.3: Repeat the denoising process of the conditional diffusion model to gradually recover the latent representation of the target fused image from random noise, and obtain a high-quality fused repair latent image representation within the preset number of diffusion steps.

8. The method for fusion and restoration of rubbings of the Preface to the Sacred Teachings based on a multi-condition diffusion model according to claim 7, characterized in that, Step 5 is implemented in the following steps: Step 5.1: Reconstruct the fused latent image obtained from the multi-condition diffusion model in Step 4. As input to the inverse mapping module, the latent image is a low-dimensional, denoised latent representation that carries rich structural and stylistic information; Step 5.2: The reverse mapping module uses a pre-trained decoder network to map the latent representation back to the original image space, restoring the details and texture features of the image. Step 5.3: The decoder consists of multiple layers of transposed convolution, upsampling operation ReLU activation function and batch normalization layer. By expanding the spatial size of the latent image layer by layer, it restores the stroke structure, edge sharpness and ink texture in the calligraphy image, and realizes the conversion from latent space to pixel space. Step 5.4: During training, a combined loss function is used for optimization, in the following form: in, This is the primary loss term, used to optimize the model's prediction accuracy for noise. Guiding conditions for a combination of clarity and smoothness. and These represent the losses in perceived consistency and style consistency, respectively. , These are weight hyperparameters; Step 5.5: The decoder output is processed through post-processing operations such as image enhancement, edge sharpening, and artifact suppression to generate a final high-quality, natural, and smooth fused rubbing image.

Images