A method and system for generating digital rubbings based on diffusion model style transfer

Abstract

This invention discloses a method and system for generating digital rubbings based on diffusion model style transfer. The method involves collecting point cloud data from the surface of a stone inscription and performing a first preprocessing step to obtain a source content image. Rubbing images with different styles are then selected and subjected to a second preprocessing step to obtain a target style image, which is used as a pre-training dataset. The structural and semantic information of the obtained source content image is decoupled, and the structural latent code and semantic latent code are passed through a structure-preserving network. This network effectively preserves the overall contour, geometric features, and shape information of the content image during style transfer, avoiding the detail loss problem that easily occurs in existing methods. The diffusion model combines the semantic information of the content image and the semantic features of the target style image to generate high-quality digital rubbings that retain both the original image content and the target style, solving the technical problem of low-quality generation due to detail loss in existing digital rubbing generation methods.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and relates to a method for generating digital rubbings, specifically a method and system for generating digital rubbings based on diffusion model style transfer. Background Technology

[0002] Rubbings are an important tool for replicating and disseminating stone inscriptions. However, the traditional rubbing process carries the risk of damaging the artifacts; therefore, the State Administration of Cultural Heritage prohibits the making of rubbings for artifacts from the Yuan Dynasty and earlier. With the development of digital technology, generating digital rubbings using digital images of artifacts and deep learning methods has become a research hotspot.

[0003] Digital rubbing production mainly includes two methods: traditional image processing and deep learning. In the first method, traditional image processing, Zhu Xiaoli et al. used Photoshop in 2008 to conduct preliminary experiments on digital rubbings of relief images. Their paper described operations such as extraction, grayscale conversion, high-contrast preservation, thresholding, background filling, and image inversion on the original artifact images. In 2009, Zhu Xiaoli et al. used MATLAB to read stone carving images, perform background filling, grayscale conversion, and image enhancement. While traditional image processing methods have achieved initial success in generating digital rubbings, the process is relatively cumbersome and the results are inconsistent due to the large number of manual processing steps. Furthermore, manually rendering text effects requires significant manpower and time. The second method, deep learning-based digital rubbing generation, focuses primarily on images. Deep learning technology was initially applied to artistic paintings, landscapes, and portrait photographs. Current research is gradually extending to areas such as font style conversion and artistic font generation. Existing digital rubbing generation technologies for inscriptions mainly fall into two categories: digital rubbing generation based on 3D model depth and digital rubbing generation based on style transfer. In 2018, Zhang Yuanlin et al. used digital rubbing generation based on 3D model depth. This method requires complete depth information in the rubbing data and combines it with edge detection algorithms to obtain the digital rubbing. It has high requirements for the rubbing data, and the generated digital rubbing is only a black and white image reflecting the structural features of the font. The digital rubbing generation method based on style transfer can preserve the font structure while retaining the texture details of traditional rubbings. Therefore, this invention focuses on the digital rubbing generation method based on style transfer. In style transfer, the content of the image is taken as the main body, and the style is taken as a transformable item. Under the premise of maintaining the original image content features, different types of style features are switched. The style can be the color, texture, style, and other characteristics of the image. Digital rubbing production includes two feature images, where the content is defined as the overall structure of the inscription font, and the style is defined as the texture details of the target rubbing. The style transfer applied to digital rubbing generation is the product of fusing the font structure of the inscription and the style and texture of the rubbing.

[0004] With the rapid development and application of artificial intelligence, machine learning and deep learning have also risen in the field of computer science, driving another leap forward in style transfer. In 2016, Gatys, Ecker, and Bethge pioneered the use of neural networks for style transfer, sparking a surge of interest in deep learning-based image style transfer. In early 2017, Isola et al. attempted to train generative adversarial networks, limited to only two styles. In 2018, Choi et al. proposed StarGAN, which uses a single hot vector to process multiple predefined styles, but training requires a large dataset to handle new styles.

[0005] Image style transfer uses images as the target, content as the subject, and style as the transformable element. It switches between different style features while maintaining the original image's content characteristics. Style can be features such as color, texture, and pattern. Image style transfer includes methods that iterate through image pixel values ​​by calculating gradient values ​​and methods that iteratively generate image models. The former generates higher-quality images but has limited application and low generation speed. The latter addresses the low efficiency of model generation through forward propagation, but requires training a specific feedforward network for each target style, resulting in low overall model operability. In general, style transfer is a technique used to convert content images into images of other styles, primarily using two methods: image-based iteration and model-based iteration. Image-based iteration methods are more effective but inefficient, typically only applicable to single images; while model-based iteration methods address speed and computational cost issues, requiring a separate feedforward network to be trained for each target style image, thus reducing flexibility.

[0006] Diffusion models (DMs) have demonstrated superior performance in image generation tasks in recent years. Unlike traditional GANs and VAEs, diffusion models generate high-quality images by progressively adding noise to image data and learning the inverse process. The advantages of diffusion models include: High-fidelity image generation: Diffusion models can generate high-resolution and high-fidelity images, outperforming traditional generative models in visual quality. Stable training process: The training process of diffusion models is relatively stable and less prone to mode collapse, giving them an advantage in generating diverse images. Good structure preservation: Due to the progressive image generation characteristic of diffusion models, they excel in preserving image structure, better retaining the details and structural information of the input image.

[0007] Diffusion models are an important class of methods in generative modeling, and have made significant progress in image generation and image processing tasks in recent years. The core idea of ​​diffusion models is to disrupt the data distribution by progressively adding noise and then recovering the original data by learning the inverse process. The Denoising Diffusion Probabilistic Model (DDPM) proposed by Ho et al. in 2020 is an important representative of diffusion models. DDPM uses a denoising network to progressively remove noise and is optimized through variational inference. DDPM performs well in generating high-resolution and high-fidelity images. However, its generation process is slow, requiring multiple steps of progressive denoising; DDPM has been able to generate high-quality images without adversarial training. However, a major limitation of DDPM is that it requires simulating a multi-step Markov chain to generate each sample, which is relatively time-consuming. To accelerate this sampling process, Song et al. proposed the Denoising Diffusion Implicit Model (DDIM). DDIM is a more efficient implicit probabilistic iterative model that maintains the same training process as DDPM while accelerating the sampling process by reusing the network trained by DDPM. The basic idea behind accelerated sampling is that the original generation process involves stepwise sampling from the sequence [1, ..., T]. To accelerate the process, we consider sampling from the subsequence {τ1, ..., τ...}. S},τ1>τ2>...>

[0008] Sampling reduces the number of sampling steps by skipping steps. For example, the original training of the DDPM network contains 1000 steps, but during sampling, only 50 steps can be evenly selected from those 1000 steps to sample the image. DDIM's main contribution is that, while reusing the network trained by DDPM, it proposes a generation method that can adjust the variance, resulting in better generation performance when the number of steps is small.

[0009] The technique of stylizing input images using only a single reference image originated from Neural Style Transfer (NST), first introduced by Gatys et al. However, this method requires optimization each time a stylized image is generated, a problem solved by Johnson et al.'s introduction of image transformation networks for fast stylization. However, traditional NST methods have limited ability to capture semantic information from both the input and stylized images. For GAN-based models, one-off image stylization is achieved using one-off adaptation methods, which aim to transfer the generator to a new domain using a single reference image. One-off adaptation methods typically involve fine-tuning the generator using only a single reference image. Once the generator is fine-tuned, these methods can unconditionally generate stylized images and also stylize the input image using GAN inversion techniques. While StyleGAN-NADA is suitable for one-off image stylization tasks, its capabilities are limited, primarily because it was originally developed for text-driven style transfer purposes. The first successful GAN-based one-off adaptation method was MTG, which fine-tuned the generator and mitigated overfitting by using CLIP orientation loss. Other work has focused on improving generation quality, content preservation, and entity transfer.

[0010] Existing image stylization techniques primarily rely on Generative Adversarial Networks (GANs) and diffusion models. These methods often struggle to maintain the structural integrity of the input image during the stylization process. This is especially true when dealing with complex structural images rarely encountered during training, where existing methods frequently suffer from structural distortion. While GANs have achieved some success in style transfer tasks, they still face the challenge of separating the structure and style of the reference image during inference. Recently, diffusion models have demonstrated superior performance in tasks such as image generation, high-fidelity image generation, and text-based image manipulation. However, these methods mainly focus on building diffusion model frameworks suitable for stylization tasks, rather than prioritizing the preservation of the input image's structural integrity. Summary of the Invention

[0011] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for generating digital rubbings based on diffusion model style transfer, thereby solving the technical problem of low generation quality caused by the loss of details in existing digital rubbing generation methods.

[0012] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0013] A method for generating stylized digital rubbings based on a diffusion model includes the following steps:

[0014] Step 1: Collect point cloud data of the surface of the inscription and perform the first preprocessing to obtain the source content image;

[0015] Step 2: Select rubbing images with different styles, perform a second preprocessing on them to obtain target style images, and use them as a pre-training dataset;

[0016] Step 3: Decouple the structural and semantic information of the source content image obtained in Step 1 to obtain the structural latent code. With semantic latent coding

[0017] Step four: Encode a target style image from the pre-training dataset obtained in step two to generate a style latent code.

[0018] Step 5: Calculate the structural latent encoding of the source content image obtained in Step 3. and semantic latent coding The input is fed into a structure preservation network, and the output is a structure latent code. and semantic latent coding

[0019] The structure-preserving network includes multiple sequentially connected input layers, multi-layer convolutional networks, and output layers;

[0020] The input layer is used to receive the semantic latent code of the source content image extracted by the semantic encoder. The multi-layer convolutional network is used to extract and enhance the semantic latent coding of the source content image. The output layer is used to generate the final structural latent code.

[0021] Step six, convert the structural latent code obtained in step five. and semantic latent coding and the style latent encoding of the target style image obtained in step four. The input is fed into the diffusion model, and a directional CLIP loss is introduced to generate a stylized image;

[0022] The diffusion model is used to preserve the structural latent encoding of the source content image. and semantic latent coding Based on this, stylized images are generated;

[0023] The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the style latent encoding of the generated stylized image matches that of the target style image. Maintaining feature consistency enables the diffusion model to generate stylized images that are consistent with both the target style image and the source content image.

[0024] This invention also includes the following technical features:

[0025] In step five, the input layer is a convolutional layer with a 1x1 kernel; the multi-layer convolutional network includes two sequentially connected SPN blocks, each SPN block including a group normalization and Swish activation function, a 1x1 convolutional layer, a group normalization, a time-step embedding layer, a Swish activation function, and a 1x1 convolutional layer, all sequentially connected; the output layer is a convolutional layer with a 1x1 kernel.

[0026] The structural latent coding This includes overall outline, geometric features, and shape information; the semantic latent encoding This includes semantic information, object category, and location; the style latent encoding. This includes color, texture, and artistic style.

[0027] The directional CLIP loss includes cross-domain loss, intra-domain loss, and reconstruction loss;

[0028] The cross-domain loss is used to ensure that the generated stylized image retains the structural latent coding of the source content image during cross-domain style transfer. Feature and semantic latent encoding Features can also introduce the style latent coding of the target style image. feature;

[0029] The in-domain loss is used to ensure that the generated stylized image has structural features that are latently encoded with the source content image structure. The characteristics remain consistent;

[0030] The reconstruction loss is used to ensure that the generated stylized image retains the structural latent coding of the source content image after style transfer. Semantic latent coding Style latent coding of target style images

[0031] The first preprocessing includes the following steps:

[0032] S1, Remove irrelevant background data and noise from the point cloud data;

[0033] S2, performs hole-filling operation for holes caused by incomplete acquisition or noise during the point cloud data acquisition process;

[0034] S3 performs re-meshization, smoothing, and noise filtering on the point cloud data processed by S1 and S2.

[0035] S4 projects the point cloud data onto a two-dimensional plane to generate a depth image, i.e., the source content image.

[0036] The second preprocessing includes the following steps:

[0037] Step 2.1: Select rubbings with different calligraphic styles, colors, and designs;

[0038] Step 2.2: Use median filtering image denoising method to remove noise and texture interference from the rubbing image to obtain the target style image.

[0039] A system for generating digital rubbings based on diffusion model style transfer, comprising a data processing unit, a feature extraction unit, a structure preservation unit, and a stylized image generation unit, based on the method for generating digital rubbings based on diffusion model style transfer.

[0040] The data processing unit is used to collect point cloud data of the surface of the stele and perform a first preprocessing to obtain the source content image; and to filter out rubbing images with different styles, perform a second preprocessing to obtain the target style image, and use it as a pre-training dataset.

[0041] The feature extraction unit is used to decouple the structural and semantic information of the source content image to obtain the structural latent code. With semantic latent coding And encode a target style image from the pre-trained dataset obtained in step two to generate a style latent code.

[0042] The structure-preserving unit is used to employ a structure-preserving network to perform structural latent encoding on the source content image. and semantic latent coding Process the code to output the latent structural encoding. and semantic latent coding

[0043] The structure-preserving network includes multiple sequentially connected input layers, multi-layer convolutional networks, and output layers;

[0044] The input layer is used to receive the semantic latent code of the source content image extracted by the semantic encoder. The multi-layer convolutional network is used to extract and enhance the semantic latent coding of the source content image. The output layer is used to generate the final structural latent code.

[0045] The stylized image generation unit is used to employ a diffusion model to latently encode the structure. and semantic latent coding and the style latent coding of the target style image The image is processed and a directional CLIP loss is introduced to generate a stylized image.

[0046] The diffusion model is used to preserve the structural latent encoding of the source content image. and semantic latent coding Based on this, stylized images are generated;

[0047] The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the style latent encoding of the generated stylized image matches that of the target style image. Maintaining feature consistency enables the diffusion model to generate stylized images that are consistent with both the target style image and the source content image.

[0048] Compared with the prior art, the beneficial technical effects of this invention are:

[0049] (I) This invention uses a structure-preserving network to preserve the overall outline, geometric features and shape information of the content image during style transfer, avoiding the detail loss problem that is easy to occur in existing methods. The diffusion model combines the semantic information of the content image and the semantic features of the target style image to generate a high-quality digital rubbing that retains both the original image content and the target style, thus solving the technical problem of low generation quality caused by the loss of details in existing digital rubbing generation methods.

[0050] (II) This invention enhances the ability to process complex image structures by combining a diffusion model with a structure-preserving network, and can generate high-quality digital rubbings even with limited training data. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the process of the present invention;

[0052] Figure 2 This is a schematic diagram illustrating the principle of the present invention;

[0053] Figure 3 This is a schematic diagram of the SPN block structure in this invention;

[0054] Figure 4 This is a schematic diagram of the source content image in this invention;

[0055] Figure 5 This is a schematic diagram illustrating the generation of stylized images in this invention.

[0056] The specific content of the present invention will be further explained in detail below with reference to the embodiments. Detailed Implementation

[0057] It should be noted that, unless otherwise specified, all components in this invention are those known in the art.

[0058] The following are specific embodiments of the present invention. It should be noted that the present invention is not limited to the following specific embodiments. All equivalent modifications made based on the technical solutions of this application fall within the protection scope of the present invention.

[0059] This invention presents a method for generating stylized digital rubbings based on a diffusion model, specifically including the following steps:

[0060] Step 1: Collect point cloud data of the surface of the inscription and perform the first preprocessing to obtain the source content image;

[0061] Step 2: Select rubbing images with different styles, perform a second preprocessing on them to obtain target style images, and use them as a pre-training dataset;

[0062] Step 3: Decouple the structural and semantic information of the source content image obtained in Step 1 to obtain the structural latent code. With semantic latent coding

[0063] Step four: Encode a target style image from the pre-training dataset obtained in step two to generate a style latent code.

[0064] Step 5: Calculate the structural latent encoding of the source content image obtained in Step 3. and semantic latent coding The input is fed into a structure preservation network, and the output is a structure latent code. and semantic latent coding

[0065] Structure-preserving networks consist of multiple sequentially connected input layers, multi-layer convolutional networks, and output layers;

[0066] The input layer is used to receive the semantic latent code of the source content image extracted by the semantic encoder. Multilayer convolutional networks are used to extract and enhance the semantic latent coding of source content images. The output layer is used to generate the final structural latent code.

[0067] Step six, convert the structural latent code obtained in step five. and semantic latent coding and the style latent encoding of the target style image obtained in step four. The input is fed into the diffusion model, and a directional CLIP loss is introduced to generate a stylized image;

[0068] The diffusion model is used to preserve the structural latent encoding of the source content image. and semantic latent coding Based on this, stylized images are generated;

[0069] Directional CLIP loss is used to optimize the diffusion model, i.e., to ensure that the generated stylized image shares the style latent encoding with the target stylized image. Maintaining feature consistency enables the diffusion model to generate stylized images that are consistent with both the target style image and the source content image.

[0070] In step three, the structural information of the source content image is encoded into structural latent coding. Ensure that the basic structure of the image is preserved during style transfer. Simultaneously, encode the semantic information of the content image into semantic latent code. This encoding preserves high-level features in the image.

[0071] Step 4, Style Latent Coding It captures the low-level visual features of the style images.

[0072] In step five, to merge the time information for each time step, the SPN block employs sinusoidal position embedding, with each time step denoted as ψ. A time step embedding layer (TE) is also merged, consisting of two linear layers. Furthermore, to adjust the SPN block at the current time step, this invention uses the time step embedding as the scale and shift parameter for group normalization.

[0073] By using a structure-preserving network, the overall outline, geometric features, and shape information of the content image can be well preserved during style transfer, avoiding the detail loss problem that is prone to occur in existing methods. The diffusion model combines the semantic information of the content image and the semantic features of the target style image to generate high-quality digital rubbings that retain both the original image content and the target style. This solves the technical problem of low generation quality caused by detail loss that is prone to occur in existing digital rubbing generation methods.

[0074] Specifically, in step five, the input layer is a convolutional layer with a 1x1 kernel; the multi-layer convolutional network includes two sequentially connected SPN blocks, each SPN block including sequentially connected group normalization and Swish activation functions, 1x1 convolutional layers, group normalization, time-step embedding layers, Swish activation functions, and 1x1 convolutional layers; the output layer is a convolutional layer with a 1x1 kernel.

[0075] In the above technical solution, the latent encoding of the source content image structure is obtained after further extraction via SPN blocks.

[0076] Specifically, structural latent coding Including overall outline, geometric features, and shape information; semantic latent encoding This includes semantic information, object category and location; style latent encoding. This includes color, texture, and artistic style.

[0077] Specifically, directional CLIP loss includes cross-domain loss, intra-domain loss, and reconstruction loss;

[0078] Cross-domain loss is used to ensure that the generated stylized image retains the structural latent coding of the source content image during cross-domain style transfer. Feature and semantic latent encoding Features can also introduce the style latent coding of the target style image. feature;

[0079] In-domain loss is used to ensure that the generated stylized image is structurally similar to the source content image in terms of latent structural coding. The characteristics remain consistent;

[0080] The reconstruction loss is used to ensure that the generated stylized image retains the structural latent coding of the source content image after style transfer. Semantic latent coding Style latent coding of target style images

[0081] In the above technical solution, the diffusion model is optimized by CLIP directional loss, so that it can better bridge the differences between the source content image and the target style image.

[0082] The specific calculation method for the loss function is as follows:

[0083] (1) Cross-Domain Loss: Utilizing the CLIP image encoder E I Extract latent vectors from the input source image. Latent vectors of target style images Subtracting these semantically aligned latent vectors yields the semantically desired direction. Equations 1 and 2 are used to calculate the variation and cosine similarity between the target style image and the source content image, respectively. Equation 3 is used to evaluate the similarity between the two directions.

[0084]

[0085] L cross =1-sin(v) style ,v in (3)

[0086] in:

[0087] The goal of cross-domain loss is to align directional changes from domain A to domain B, ensuring that the source content image... arrive Changes and changes from target style images arrive The changes remain consistent.

[0088] (2) In-Domain Loss: Ensures the generation of the final stylized image I style In terms of structural features, it is similar to the source content image I cont To maintain consistency, this invention uses pixel-level L1 loss and feature-level perceptual loss. Intra-domain variations are calculated using Equations 4 and 5. Then, Equation 6 is used to determine the similarity between the two directions.

[0089]

[0090] L In =1-sin(v) A ,v B (6)

[0091] in:

[0092] The purpose of intra-domain loss is to reduce unexpected directional changes during stylization, which can lead to over-reflection in the stylized image. This is achieved by measuring the similarity of changes within domain A and domain B. Similar to cross-domain loss, intra-domain changes are calculated using Equations 4 and 5. Subsequently, the similarity between the two directions is determined using Equation 6.

[0093] (3) Reconstruction Loss: This is achieved by analyzing the source content image... and target style images The characteristic loss L extracted re-image Perceptual similarity loss L re-lpips and CLIP embedding loss L re-clip The total reconstruction loss is calculated by comparing the sums of the two equations, and can be calculated using Equation 7-9. The total reconstruction loss can be calculated using Equation 10.

[0094]

[0095] L recon =λ re-image L re-image +λ re-lpips L re-lpips +λ re-clip L re-clip (10)

[0096] in:

[0097] The reconstruction loss measures the difference between the generated stylized image and the source content image, ensuring that the stylized image preserves as much content and detail as possible from the source image. The reconstruction loss is a weighted sum of the feature loss, perceptual similarity loss, and CLIP embedding loss described above.

[0098] (4) Total loss function: The total loss function can be expressed by Equation 11.

[0099] L total =λ cross L cross +λ in L in +L recon (11)

[0100] in:

[0101] The total loss is the weighted sum of the cross-domain loss, intra-domain loss, and reconstruction loss mentioned above.

[0102] Specifically, the first preprocessing includes the following steps:

[0103] S1, Remove irrelevant background data and noise from the point cloud data;

[0104] S2, performs hole-filling operation for holes caused by incomplete acquisition or noise during the point cloud data acquisition process;

[0105] S3 performs re-meshization, smoothing, and noise filtering on the point cloud data processed by S1 and S2.

[0106] S4 projects the point cloud data onto a two-dimensional plane to generate a depth image, i.e., the source content image.

[0107] In the above technical solution, the hole-patch operation ensures the smoothness and continuity of the model's surface. A threshold is set to 1% of the total point cloud area based on visual effect and data integrity; when the area of ​​a hole exceeds this threshold, patching is performed. The solution also includes adjusting, rotating, and cropping the model's parameters to ensure that source content images with better performance are used as pre-training samples.

[0108] Specifically, the second preprocessing includes the following steps:

[0109] Step 2.1: Select rubbings with different calligraphic styles, colors, and designs;

[0110] Step 2.2: Use median filtering image denoising method to remove noise and texture interference from the rubbing image to obtain the target style image.

[0111] The above technical solution ensures that the image is clean and clear; at the same time, it appropriately increases the contrast of the rubbing image, making the boundary between the subject and the background more distinct and enhancing the stylistic features.

[0112] The images of the rubbings are sourced from the resource database of the National Digital Library of China - Ancient Chinese Books Resource Database - Inscriptions and Rubbings Collection.

[0113] This invention also provides a system for generating digital rubbings based on diffusion model style transfer, and a method for generating digital rubbings based on diffusion model style transfer, including a data processing unit, a feature extraction unit, a structure preservation unit, and a stylized image generation unit.

[0114] The data processing unit is used to collect point cloud data of the surface of the stele and perform a first preprocessing to obtain the source content image; and to filter out rubbing images with different styles, perform a second preprocessing to obtain the target style image, and use it as a pre-training dataset.

[0115] The feature extraction unit is used to decouple the structural and semantic information of the source content image to obtain the structural latent code. With semantic latent coding And encode a target style image from the pre-trained dataset obtained in step two to generate a style latent code.

[0116] Structure Preservation Units are used to encode the structure of source content images using structure preservation networks. and semantic latent coding Process the code to output the latent structural encoding. and semantic latent coding

[0117] Structure-preserving networks consist of multiple sequentially connected input layers, multi-layer convolutional networks, and output layers;

[0118] The input layer is used to receive the semantic latent code of the source content image extracted by the semantic encoder. Multilayer convolutional networks are used to extract and enhance the semantic latent coding of source content images. The output layer is used to generate the final structural latent code.

[0119] Stylized image generation units are used to latently encode structural information using a diffusion model. and semantic latent coding and the style latent coding of the target style image The image is processed and a directional CLIP loss is introduced to generate a stylized image.

[0120] The diffusion model is used to preserve the structural latent encoding of the source content image. and semantic latent coding Based on this, stylized images are generated;

[0121] Directional CLIP loss is used to optimize the diffusion model, i.e., to ensure that the generated stylized image shares the style latent encoding with the target stylized image. Maintaining feature consistency enables the diffusion model to generate stylized images that are consistent with both the target style image and the source content image.

Claims

1. A method for generating stylized digital rubbings based on a diffusion model, characterized in that, Specifically, the following steps are included: Step 1: Collect point cloud data of the surface of the inscription and perform the first preprocessing to obtain the source content image; Step 2: Select rubbing images with different styles, perform a second preprocessing on them to obtain target style images, and use them as a pre-training dataset; Step 3: Decouple the structural and semantic information of the source content image obtained in Step 1 to obtain the structural latent code. With semantic latent coding ; Step four: Encode a target style image from the pre-training dataset obtained in step two to generate a style latent code. ; Step 5: Calculate the structural latent encoding of the source content image obtained in Step 3. and semantic latent coding The input is fed into a structure preservation network, and the output is a structure latent code. and semantic latent coding ; The structural latent coding This includes overall outline, geometric features, and shape information; the semantic latent encoding This includes semantic information, object category, and location; the style latent encoding. Including color, texture, and art style; The structure-preserving network includes multiple sequentially connected input layers, multi-layer convolutional networks, and output layers; The input layer is used to receive the semantic latent code of the source content image extracted by the semantic encoder. The multi-layer convolutional network is used to extract and enhance the semantic latent coding of the source content image. The output layer is used to generate the final structural latent code. ; Step six, convert the structural latent code obtained in step five. and semantic latent coding and the style latent encoding of the target style image obtained in step four. The input is fed into the diffusion model, and a directional CLIP loss is introduced to generate a stylized image; The diffusion model is used to preserve the structural latent encoding of the source content image. and semantic latent coding Based on this, stylized images are generated; The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the style latent encoding of the generated stylized image matches that of the target style image. Maintaining feature consistency enables the diffusion model to generate stylized images that are consistent with the style of the target style image and the content of the source content image; The directional CLIP loss includes cross-domain loss, intra-domain loss, and reconstruction loss; The cross-domain loss is used to ensure that the generated stylized image retains the structural latent coding of the source content image during cross-domain style transfer. Feature and semantic latent encoding Features can also introduce the style latent coding of the target style image. feature; The in-domain loss is used to ensure that the generated stylized image has structural features that are latently encoded with the source content image structure. The characteristics remain consistent; The reconstruction loss is used to ensure that the generated stylized image retains the structural latent coding of the source content image after style transfer. Semantic latent coding Style latent coding of target style images .

2. The method for generating stylized digital rubbings based on a diffusion model as described in claim 1, characterized in that, In step five, the input layer is a convolutional layer with a 1x1 kernel; the multi-layer convolutional network includes two sequentially connected SPN blocks, each SPN block including a group normalization and Swish activation function, a 1x1 convolutional layer, a group normalization, a time-step embedding layer, a Swish activation function, and a 1x1 convolutional layer, all sequentially connected; the output layer is a convolutional layer with a 1x1 kernel.

3. The method for generating stylized digital rubbings based on a diffusion model as described in claim 1, characterized in that, The first preprocessing includes the following steps: S1, Remove irrelevant background data and noise from the point cloud data; S2, performs hole-filling operation for holes caused by incomplete acquisition or noise during the point cloud data acquisition process; S3 performs re-meshization, smoothing, and noise filtering on the point cloud data processed by S1 and S2. S4 projects the point cloud data onto a two-dimensional plane to generate a depth image, i.e., the source content image.

4. The method for generating stylized digital rubbings based on a diffusion model as described in claim 1, characterized in that, The second preprocessing includes the following steps: Step 2.1: Select rubbings with different calligraphic styles, colors, and designs; Step 2.2: Use median filtering image denoising method to remove noise and texture interference from the rubbing image to obtain the target style image.

5. A system for generating digital rubbings based on diffusion model style transfer, characterized in that, The method for generating digital rubbings based on diffusion model style transfer according to any one of claims 1 to 4 includes a data processing unit, a feature extraction unit, a structure preservation unit, and a stylized image generation unit; The data processing unit is used to collect point cloud data of the surface of the inscription and perform a first preprocessing to obtain the source content image; And select rubbing images with different styles, perform a second preprocessing on them to obtain target style images, and use them as a pre-training dataset; The feature extraction unit is used to decouple the structural and semantic information of the source content image to obtain the structural latent code. With semantic latent coding ; and encode a target style image from the pre-trained dataset obtained in step two to generate a style latent code. ; The structure-preserving unit is used to employ a structure-preserving network to perform structural latent encoding on the source content image. and semantic latent coding Process the code to output the latent structural encoding. and semantic latent coding .

Images