Method and system for generating digital rubbing based on diffusion model style migration

Abstract

The invention discloses a method and system for generating a digital rubbing based on diffusion model style migration, and the method comprises the steps: collecting point cloud data of an inscription surface, and carrying out the first preprocessing of the point cloud data, and obtaining a source content picture; screening out rubbing images with different styles, performing second preprocessing on the rubbing images to obtain a target style picture, and taking the target style picture as a pre-training data set; structural and semantic information decoupling is carried out on the obtained source content picture, a structural latent code # imgabs0 # and a semantic latent code # imgabs1 # are obtained, the overall contour, geometric features and shape information of the content image can be well reserved in the style migration process through a structure maintaining network, the problem that details are prone to being lost in an existing method is avoided, and the method is suitable for being applied to the style migration process of the content image. The diffusion model combines the semantic information of the content image and the semantic features of the target style image to generate a high-quality digital rubbing which retains the original image content and has the target style, and the technical problem that the generation quality is not high due to the fact that an existing digital rubbing generation method is prone to detail loss is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and relates to a method for generating digital rubbings, and specifically to a method and system for generating digital rubbings based on diffusion model style transfer. Background Art

[0002] Rubbings are an important tool for copying and disseminating stone inscriptions. However, the traditional rubbing process carries the risk of damaging cultural relics, so the State Administration of Cultural Heritage prohibits the production of rubbings of cultural relics from the Yuan Dynasty and before. With the development of digital technology, the use of digital images of cultural relics and deep learning methods to generate digital rubbings has become a research hotspot.

[0003] Digital rubbings production mainly includes two methods based on traditional image processing and deep learning. In the first traditional image processing method for digital rubbings generation, in 2008, Zhu Xiaoli et al. used Photoshop software to realize preliminary experiments on digital rubbings of relief images. In the paper, the original cultural relic images were extracted, grayed, high contrast retained, thresholded, background filled, inverted, etc.; in 2009, Zhu Xiaoli et al. used MATLAB software to realize the reading, background filling, graying, and image enhancement of stone carving images. Although the traditional image processing method has preliminarily completed the generation of digital rubbings, the process is relatively cumbersome and the results are uneven, because a large number of manual processing steps are required, and manual rendering of text effects requires a lot of manpower and time costs. The second method of digital rubbings generation based on deep learning, the research based on deep learning mainly focuses on images. Deep learning technology was originally applied to images such as artistic paintings, landscapes, and portraits. At present, the research direction has gradually extended to the fields of font style conversion and artistic word generation. The existing digital rubbing generation technology of inscriptions is mainly divided into two types: digital rubbing generation technology based on three-dimensional model depth and digital rubbing generation technology based on style transfer. In 2018, Zhang Yuanlin et al. used the digital rubbing generation technology based on three-dimensional model depth. This method requires the rubbing data to have complete depth information, and combines the edge detection algorithm to obtain the digital rubbing. The rubbing data has high requirements, and the generated digital rubbing only reflects the black and white image of the font structure characteristics. The digital rubbing generation method based on style transfer can retain the texture details of traditional rubbings while retaining the font structure, so the focus of the present invention is on the digital rubbing generation method based on style transfer. In style transfer, the content in the image is taken as the main body, and the style is taken as the transformable item. On the premise that the content maintains the content characteristics of the original image, different types of style features are switched. The style can be the color, texture, style and other characteristics of the image. The production of digital rubbings includes two feature images, in which the content is defined as the overall structure of the inscription font, and the style is defined as the texture details of the target rubbing. The style transfer applied to the generation of digital rubbings is the product of the fusion of the font structure of the inscription and the style texture of the rubbing.

[0004] With the rapid development and application of artificial intelligence, machine learning and deep learning have also emerged in the field of computer science, leading to another leap forward in style transfer. In 2016, Gatys, Ecker, and Bethge pioneered the use of neural networks for style transfer, which triggered a wave of image style transfer based on deep learning. In early 2017, Isola et al. tried to train a generative adversarial network, which was limited to two styles. In 2018, Choi et al. proposed StarGAN to use a hot vector to process multiple predefined styles, but in order to process new styles, training requires a large amount of data set support.

[0005] Image style transfer takes the image as the transfer target, the content in the image as the subject, and the style as the transformable item. On the premise that the content maintains the original image content characteristics, different types of style features are switched. The style can be the color, texture, style and other characteristics of the image. Image style transfer includes the method of calculating the gradient value to iterate the image pixel value and the method of iteratively generating the image model. The former generates high-quality images, but the application is single and the image generation rate is low. The latter solves the problem of low model generation efficiency by forward propagation, but for a specific target style, a specific feedforward network needs to be trained, which makes the overall operability of the model low. In general, style transfer is a technology used to convert content images into images of other styles. There are two main methods: image-based iteration and model-based iteration. The image-based iteration method has better results but low efficiency and is usually only applicable to a single image; while the model-based iteration method solves the speed and computational cost problems, but requires a feedforward network to be trained for each target style image, so it has low flexibility.

[0006] Diffusion Models (DMs) have demonstrated superior performance in image generation tasks in recent years. Unlike traditional GANs and VAEs, diffusion models generate high-quality images by gradually adding noise to image data and learning the inverse process. The advantages of diffusion models include: High-fidelity image generation: Diffusion models are able to generate high-resolution and high-fidelity images, which outperform traditional generative models in visual quality. Stable training process: The training process of the diffusion model is relatively stable and is not prone to mode collapse, which makes it advantageous in generating diverse images. Good structure preservation ability: Due to the characteristic of the diffusion model to gradually generate images, it performs well in preserving image structure and can better preserve the details and structural information of the input image.

[0007] Diffusion models are an important class of methods in the field of generative modeling, and have made significant progress in image generation and image processing tasks in recent years. The core idea of ​​the diffusion model is to destroy the data distribution by gradually adding noise, and restore the noise to the original data by learning the inverse process. The denoising diffusion probability model (DDPM) model proposed by Ho et al. in 2020 is an important representative of the diffusion model. DDPM uses a denoising network to gradually remove noise and optimizes it through variational inference. DDPM performs well in image generation quality and can generate high-resolution and high-fidelity images. However, its generation process is slow and requires multiple steps of gradual denoising; DDPM has been able to generate high-quality pictures without adversarial training. However, a major limitation of DDPM is that they need to generate each sample by simulating a multi-step Markov chain, which is relatively time-consuming. In order to speed up this sampling process, Song et al. proposed the denoising diffusion implicit model (DDIM). DDIM is a more efficient implicit probability iterative model that maintains the same training process as DDPM and accelerates the sampling process while reusing the network trained by DDPM. The basic idea of ​​accelerated sampling is that the original generation process is to sample gradually from the sequence [1, ..., T], and the acceleration process considers sampling from the subsequence {τ1, ..., τ S},τ1>τ2>...>

[0008] Sampling, reducing the number of sampling steps by skipping steps. For example, the original training of the DDPM network contains 1000 steps, but when sampling, only 50 steps can be evenly selected from the 1000 steps, and the image is sampled with these 50 steps. The contribution of DDIM is mainly to propose a generation form that can adjust the variance under the premise of reusing the network trained by DDPM. When the number of steps is small, the generation effect is better with a small variance.

[0009] The technique of stylizing an input image with only one reference image originated from Neural Style Transfer (NST) first introduced by Gatys et al. However, this method requires optimization every time a stylized image is generated, a problem that was solved by Johnson et al. after they introduced image transformation networks for fast stylization. However, traditional NST methods are limited in their ability to capture the semantic information of the input and style images. For GAN-based models, one-shot image stylization is achieved by using one-shot adaptation methods, which aim to transfer the generator to a new domain using a single reference image. One-shot adaptation methods usually involve fine-tuning the generator using only a single reference image. Once the generator is fine-tuned, these methods can unconditionally generate stylized images and also stylize the input image by using GAN inversion techniques. StyleGAN-NADA, while suitable for one-shot image stylization tasks, has limited capabilities, mainly due to the fact that it was originally developed for text-driven style transfer purposes. The first successful GAN-based one-shot adaptation method is MTG, which fine-tunes the generator and alleviates the overfitting problem by using the CLIP directional loss. Other works focus on improving generation quality, content preservation, and entity transfer.

[0010] Existing image stylization techniques mainly rely on generative adversarial networks (GANs) and diffusion models. These methods often have difficulty maintaining the structural integrity of the input image in the process of image stylization. In particular, when dealing with complex structure images that are rarely seen during training, existing methods often suffer from structural distortion problems. Although GANs have achieved some success in style transfer tasks, there is still the challenge of difficulty in separating the structure and style of the reference image during inference. Recently, diffusion models have shown excellent performance in tasks such as image generation, high-fidelity image generation, and text-based image manipulation. However, these methods mainly focus on building a diffusion model framework suitable for stylization tasks, rather than prioritizing maintaining the structural integrity of the input image. Summary of the invention

[0011] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a method and system for generating digital rubbings based on diffusion model style transfer, so as to solve the technical problem that the digital rubbing generation method in the prior art is prone to detail loss, resulting in low generation quality.

[0012] In order to solve the above technical problems, the present invention adopts the following technical solutions to achieve the above problems:

[0013] A method for generating stylized digital rubbings based on a diffusion model specifically comprises the following steps:

[0014] Step 1: collect point cloud data of the surface of the stele, and perform a first preprocessing on it to obtain a source content image;

[0015] Step 2: Screen out rubbing images with different styles, perform a second preprocessing on them to obtain target style images, and use them as a pre-training data set;

[0016] Step 3: Decouple the structure and semantic information of the source content image obtained in step 1 to obtain the structural latent code Semantic latent coding

[0017] Step 4: Encode a target style image in the pre-training dataset obtained in step 2 to generate a style latent code

[0018] Step 5: Encode the structure of the source content image obtained in step 3 and semantic latent coding Input into the structure-preserving network and output the structure latent code and semantic latent coding

[0019] The structure-preserving network includes a plurality of sequentially connected input layers, a multi-layer convolutional network and an output layer;

[0020] The input layer is used to receive the semantic latent code of the source content picture extracted by the semantic encoder The multi-layer convolutional network is used to extract and enhance the semantic latent coding of the source content image. The output layer is used to generate the final structural latent code

[0021] Step 6: Encode the structure latent code obtained in step 5 and semantic latent coding And the style latent code of the target style image obtained in step 4 Input into the diffusion model, introduce directional CLIP loss, and generate stylized images;

[0022] The diffusion model is used to maintain the structure of the source content picture while latent encoding and semantic latent coding Generate stylized images based on

[0023] The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the generated stylized image is consistent with the style latent encoding of the target style image. The features remain consistent, enabling the diffusion model to generate stylized images that are consistent with the style of the target style image and consistent with the content of the source content image.

[0024] The present invention also includes the following technical features:

[0025] In step 5, the input layer is a convolution layer with a 1x1 convolution kernel; the multi-layer convolution network includes two SPN blocks connected in sequence, each of the SPN blocks includes group normalization and Swish activation function, 1x1 convolution layer, group normalization, time step embedding layer, Swish activation function and 1x1 convolution layer connected in sequence; the output layer is a convolution layer with a 1x1 convolution kernel.

[0026] The structural latent code Including overall outline, geometric features and shape information; the semantic latent coding Including semantic information, object category and location; the style latent coding This includes colors, textures, and art styles.

[0027] The directional CLIP loss includes cross-domain loss, intra-domain loss and reconstruction loss;

[0028] The cross-domain loss is used to ensure that the generated stylized image can maintain the structural latent code of the source content image during the cross-domain style transfer process. Features and Semantic Latents Features, and can also introduce style latent coding of the target style image feature;

[0029] The domain loss is used to ensure that the generated stylized image has the same structural features as the source content picture structure latent code The characteristics remain consistent;

[0030] The reconstruction loss is used to ensure that the generated stylized image can retain the structural latent code of the source content image after style transfer. Semantic latent coding and the style latent code of the target style image

[0031] The first pre-processing comprises the following steps:

[0032] S1, remove irrelevant background data and noise on the point cloud data;

[0033] S2, filling holes caused by incomplete collection or noise during the collection of point cloud data;

[0034] S3, regridding, smoothing and noise filtering the point cloud data processed by S1 and S2;

[0035] S4, projects the point cloud data onto a two-dimensional plane to generate a depth image, i.e., the source content image.

[0036] The second pretreatment comprises the following steps:

[0037] Step 2.1, select rubbing images with different scripts, colors and styles;

[0038] Step 2.2, use the median filter image denoising method to remove the noise and texture interference in the rubbing image to obtain the target style image.

[0039] A system for generating digital rubbings based on diffusion model style transfer, based on the method for generating digital rubbings based on diffusion model style transfer, comprising a data processing unit, a feature extraction unit, a structure preservation unit and a stylized image generation unit;

[0040] The data processing unit is used to collect point cloud data of the surface of the stele, and perform a first preprocessing on it to obtain a source content image; and to screen out rubbing images with different styles, perform a second preprocessing on them to obtain a target style image, and use it as a pre-training data set;

[0041] The feature extraction unit is used to decouple the structure and semantic information of the source content image to obtain the structural latent code Semantic latent coding And encode a target style picture in the pre-training dataset obtained in step 2 to generate style latent code

[0042] The structure preserving unit is used to use a structure preserving network to encode the structure of the source content picture and semantic latent coding Process and output structure latent code and semantic latent coding

[0043] The structure-preserving network includes a plurality of sequentially connected input layers, a multi-layer convolutional network and an output layer;

[0044] The input layer is used to receive the semantic latent code of the source content picture extracted by the semantic encoder The multi-layer convolutional network is used to extract and enhance the semantic latent coding of the source content image. The output layer is used to generate the final structural latent code

[0045] The stylized image generation unit is used to use a diffusion model to encode the structure latent and semantic latent coding And the style latent code of the target style image Processing and introducing directional CLIP loss to generate stylized images;

[0046] The diffusion model is used to maintain the structure of the source content picture while latent encoding and semantic latent coding Generate stylized images based on

[0047] The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the generated stylized image is consistent with the style latent encoding of the target style image. The features remain consistent, enabling the diffusion model to generate stylized images that are consistent with the style of the target style image and consistent with the content of the source content image.

[0048] Compared with the prior art, the present invention has the following beneficial technical effects:

[0049] (I) The present invention uses a structure-preserving network to well preserve the overall outline, geometric features and shape information of the content image during the style transfer process, avoiding the problem of detail loss that is prone to occur in existing methods. The diffusion model combines the semantic information of the content image and the semantic features of the target style image to generate a high-quality digital rubbing that retains the original image content and has the target style, thereby solving the technical problem of low generation quality caused by the problem of detail loss that is prone to occur in the digital rubbing generation method in the prior art.

[0050] (II) The present invention enhances the processing capability of complex image structures by combining the diffusion model with the structure-preserving network, and can generate high-quality digital rubbings even when the training data is limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the process of the present invention;

[0052] Figure 2 It is a schematic diagram of the principle of the present invention;

[0053] Figure 3 It is a schematic diagram of the structure of the SPN block in the present invention;

[0054] Figure 4 is a schematic diagram of a source content picture in the present invention;

[0055] Figure 5 A schematic diagram of the generation of a stylized image in the present invention.

[0056] The specific contents of the present invention are further explained in detail below in conjunction with embodiments. DETAILED DESCRIPTION

[0057] It should be noted that, unless otherwise specified, all components in the present invention are components known in the art.

[0058] Specific embodiments of the present invention are given below. It should be noted that the present invention is not limited to the following specific embodiments, and all equivalent changes made on the basis of the technical solution of this application fall within the protection scope of the present invention.

[0059] The present invention provides a method for generating stylized digital rubbings based on a diffusion model, which specifically comprises the following steps:

[0060] Step 1: collect point cloud data of the surface of the stele, and perform a first preprocessing on it to obtain a source content image;

[0061] Step 2: Screen out rubbing images with different styles, perform a second preprocessing on them to obtain target style images, and use them as a pre-training data set;

[0062] Step 3: Decouple the structure and semantic information of the source content image obtained in step 1 to obtain the structural latent code Semantic latent coding

[0063] Step 4: Encode a target style image in the pre-training dataset obtained in step 2 to generate a style latent code

[0064] Step 5: Encode the structure of the source content image obtained in step 3 and semantic latent coding Input into the structure-preserving network and output the structure latent code and semantic latent coding

[0065] The structure-preserving network includes multiple sequentially connected input layers, multi-layer convolutional networks, and output layers;

[0066] The input layer is used to receive the semantic latent code of the source content image extracted by the semantic encoder Multi-layer convolutional networks are used to extract and enhance the semantic latent codes of source content images. The output layer is used to generate the final structural latent code

[0067] Step 6: Encode the structure latent code obtained in step 5 and semantic latent coding And the style latent code of the target style image obtained in step 4 Input into the diffusion model, introduce directional CLIP loss, and generate stylized images;

[0068] The diffusion model is used to maintain the structure of the source content image latent code and semantic latent coding Generate stylized images based on

[0069] The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the generated stylized image is consistent with the style latent encoding of the target style image. The features remain consistent, enabling the diffusion model to generate stylized images that are consistent with the style of the target style image and consistent with the content of the source content image.

[0070] In step 3, the structural information of the source content image is encoded into a structural latent code Ensure that the basic structure of the image is preserved during the style transfer process. At the same time, the semantic information of the content image is encoded as a semantic latent This encoding preserves the high-level features in the image.

[0071] Step 4: Style Latent Coding Captures the low-level visual features of the style image.

[0072] In step 5, in order to incorporate the temporal information of each time step, the SPN block uses sinusoidal position embedding, and each time step is represented as ψ. A time step embedding layer (TE) is also incorporated, which consists of two linear layers. In addition, in order to adjust the SPN block on the current time step, the present invention uses the time step embedding as the scale and shift parameter of the group normalization.

[0073] Through the structure-preserving network, the overall outline, geometric features and shape information of the content image can be well preserved during the style transfer process, avoiding the problem of detail loss that is prone to occur in existing methods. The diffusion model combines the semantic information of the content image and the semantic features of the target style image to generate high-quality digital rubbings that not only retain the original image content but also have the target style, solving the technical problem of low generation quality caused by the detail loss problem that is prone to occur in the digital rubbing generation method in the existing technology.

[0074] Specifically, in step 5, the input layer is a convolution layer with a 1x1 convolution kernel; the multi-layer convolution network includes two sequentially connected SPN blocks, each SPN block includes sequentially connected group normalization and Swish activation function, 1x1 convolution layer, group normalization, time step embedding layer, Swish activation function and 1x1 convolution layer; the output layer is a convolution layer with a 1x1 convolution kernel.

[0075] In the above technical solution, the source content picture structure latent code is obtained after the SPN block is extracted again

[0076] Specifically, structural latent coding Including overall outline, geometric features and shape information; semantic latent coding Including semantic information, object category and location; style latent coding This includes colors, textures, and art styles.

[0077] Specifically, the directional CLIP loss includes cross-domain loss, intra-domain loss, and reconstruction loss;

[0078] The cross-domain loss is used to ensure that the generated stylized image can maintain the structural latent code of the source content image during the cross-domain style transfer process. Features and Semantic Latents Features, and can also introduce style latent coding of the target style image feature;

[0079] The in-domain loss is used to ensure that the generated stylized image has the same structural features as the source content image structure latent coding The characteristics remain consistent;

[0080] The reconstruction loss is used to ensure that the generated stylized image can preserve the structural latent code of the source content image after style transfer. Semantic latent coding and the style latent code of the target style image

[0081] In the above technical solution, the diffusion model is optimized through CLIP directional loss, so that it can better bridge the difference between the source content picture and the target style picture.

[0082] The specific calculation method of the loss function is as follows:

[0083] (1) Cross-Domain Loss: Using CLIP image encoder E I , extract the input source content image potential vector and the latent vector of the target style image Subtracting these semantically aligned latent vectors will yield the semantically desired direction. The change and cosine similarity of the target style image and the source content image are calculated using Equation 1 and Equation 2, respectively. Equation 3 is used to evaluate the similarity of the two directions.

[0084]

[0085] L cross =1-sin(v style ,v in ) (3)

[0086] in:

[0087] The goal of the cross-domain loss is to align the directional changes from domain A to domain B, ensuring that the source content image arrive The change from the target style image arrive The changes remain consistent.

[0088] (2) In-Domain Loss: Ensures the generation of the final stylized image I style In terms of structural features, it is similar to the source content image I cont Keeping the same, the present invention uses the pixel-level L1 loss and the feature-level perceptual loss. The intra-domain variation is calculated using Equation 4 and Equation 5. Then Equation 6 is used to determine the similarity between the two directions.

[0089]

[0090] L In =1-sin(v A ,v B ) (6)

[0091] in:

[0092] The purpose of the intra-domain loss is to reduce unexpected directional changes during the stylization process, which may cause the style image to be overly reflective. This is achieved by measuring the similarity of the changes within domain A and domain B. Similar to the cross-domain loss, the intra-domain change is calculated using Equation 4 and Equation 5. Subsequently, the similarity between the two directions is determined by Equation 6.

[0093] (3) Reconstruction Loss: By reconstructing the original image from the source content and target style images The characteristic loss L extracted from re-image , perceptual similarity loss L re-lpips and the CLIP embedding loss L re-clip The total reconstruction loss can be calculated by comparing the sum of , which can be calculated using formula 7-9, and the total reconstruction loss can be calculated using formula 10.

[0094]

[0095] L recon =λ re-image L re-image +λ re-lpips L re-lpips +λ re-clip L re-clip (10)

[0096] in:

[0097] The reconstruction loss measures the difference between the generated stylized image and the source content image to ensure that the stylized image can retain the content and details of the source content image as much as possible. The reconstruction loss is the weighted sum of the above-mentioned characteristic loss, perceptual similarity loss, and CLIP embedding loss.

[0098] (4) Total loss function: The total loss function can be expressed by Formula 11.

[0099] L total =λ cross L cross +λ in L in +L recon (11)

[0100] in:

[0101] The total loss is the weighted sum of the above cross-domain loss, intra-domain loss and reconstruction loss.

[0102] Specifically, the first preprocessing includes the following steps:

[0103] S1, remove irrelevant background data and noise on the point cloud data;

[0104] S2, filling holes caused by incomplete collection or noise during the collection of point cloud data;

[0105] S3, regridding, smoothing and noise filtering the point cloud data processed by S1 and S2;

[0106] S4, projects the point cloud data onto a two-dimensional plane to generate a depth image, i.e., the source content image.

[0107] In the above technical solution, the hole filling operation is performed to ensure the smoothness and continuity of the model surface. The threshold is set to %1 of the total area of ​​the point cloud according to the visual effect and data integrity. When the area of ​​the hole exceeds this threshold, it is repaired; it also includes parameter adjustment, rotation and cropping of the model to ensure that the source content picture with better effect is obtained as the pre-training sample.

[0108] Specifically, the second preprocessing includes the following steps:

[0109] Step 2.1, select rubbing images with different scripts, colors and styles;

[0110] Step 2.2, use the median filter image denoising method to remove the noise and texture interference in the rubbing image to obtain the target style image.

[0111] In the above technical solution, a clean and clear image is ensured; at the same time, the contrast of the rubbing image is appropriately increased to make the boundary between the subject and the background clearer, thereby enhancing the style characteristics.

[0112] The source of the rubbing images is the resource library of China National Digital Library - Chinese Ancient Books Resource Library - Essence of Stele and Calligraphy.

[0113] The present invention also provides a system for generating digital rubbings based on diffusion model style transfer, and a method for generating digital rubbings based on diffusion model style transfer, comprising a data processing unit, a feature extraction unit, a structure preservation unit and a stylized image generation unit;

[0114] The data processing unit is used to collect point cloud data of the surface of the stele, and perform a first preprocessing on it to obtain a source content image; and to screen out rubbing images with different styles, perform a second preprocessing on them to obtain a target style image, and use it as a pre-training data set;

[0115] The feature extraction unit is used to decouple the structure and semantic information of the source content image to obtain the structural latent code Semantic latent coding And encode a target style picture in the pre-training dataset obtained in step 2 to generate style latent code

[0116] The structure preserving unit is used to encode the structure of the source content image using the structure preserving network. and semantic latent coding Process and output structure latent code and semantic latent coding

[0117] The structure-preserving network includes multiple sequentially connected input layers, multi-layer convolutional networks, and output layers;

[0118] The input layer is used to receive the semantic latent code of the source content image extracted by the semantic encoder Multi-layer convolutional networks are used to extract and enhance the semantic latent codes of source content images. The output layer is used to generate the final structural latent code

[0119] The stylized image generation unit is used to encode the structure latently using a diffusion model and semantic latent coding And the style latent code of the target style image Processing and introducing directional CLIP loss to generate stylized images;

[0120] The diffusion model is used to maintain the structure of the source content image latent code and semantic latent coding Generate stylized images based on

[0121] The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the generated stylized image is consistent with the style latent encoding of the target style image. The features remain consistent, enabling the diffusion model to generate stylized images that are consistent with the style of the target style image and consistent with the content of the source content image.

Claims

1. A method for generating stylized digital rubbings based on a diffusion model, characterized in that: The specific steps include: Step 1: collect point cloud data of the surface of the stele, and perform a first preprocessing on it to obtain a source content image; Step 2: Screen out rubbing images with different styles, perform a second preprocessing on them to obtain target style images, and use them as a pre-training data set; Step 3: Decouple the structure and semantic information of the source content image obtained in step 1 to obtain the structural latent code Semantic latent coding Step 4: Encode a target style image in the pre-training dataset obtained in step 2 to generate a style latent code Step 5: Encode the structure of the source content image obtained in step 3 and semantic latent coding Input into the structure-preserving network and output the structure latent code and semantic latent coding The structure-preserving network includes a plurality of sequentially connected input layers, a multi-layer convolutional network and an output layer; The input layer is used to receive the semantic latent code of the source content picture extracted by the semantic encoder The multi-layer convolutional network is used to extract and enhance the semantic latent coding of the source content image. The output layer is used to generate the final structural latent code Step 6: Encode the structure latent code obtained in step 5 and semantic latent coding And the style latent code of the target style image obtained in step 4 Input into the diffusion model, introduce directional CLIP loss, and generate stylized images; The diffusion model is used to maintain the structure of the source content picture while latent encoding and semantic latent coding Generate stylized images based on The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the generated stylized image is consistent with the style latent encoding of the target style image. The features remain consistent, enabling the diffusion model to generate stylized images that are consistent with the style of the target style image and consistent with the content of the source content image.

2. The method for generating stylized digital rubbings based on a diffusion model as claimed in claim 1, characterized in that: In step 5, the input layer is a convolution layer with a 1x1 convolution kernel; the multi-layer convolution network includes two SPN blocks connected in sequence, each of the SPN blocks includes group normalization and Swish activation function, 1x1 convolution layer, group normalization, time step embedding layer, Swish activation function and 1x1 convolution layer connected in sequence; the output layer is a convolution layer with a 1x1 convolution kernel.

3. The method for generating stylized digital rubbings based on a diffusion model as claimed in claim 1, characterized in that: The structural latent code Including overall outline, geometric features and shape information; the semantic latent coding Including semantic information, object category and location; the style latent coding This includes colors, textures, and art styles.

4. The method for generating stylized digital rubbings based on a diffusion model as claimed in claim 1, characterized in that: The directional CLIP loss includes cross-domain loss, intra-domain loss and reconstruction loss; The cross-domain loss is used to ensure that the generated stylized image can maintain the structural latent code of the source content image during the cross-domain style transfer process. Features and Semantic Latents Features, and can also introduce style latent coding of the target style image feature; The domain loss is used to ensure that the generated stylized image has the same structural features as the source content picture structure latent code The characteristics remain consistent; The reconstruction loss is used to ensure that the generated stylized image can retain the structural latent code of the source content image after style transfer. Semantic latent coding and the style latent code of the target style image 5. The method for generating stylized digital rubbings based on a diffusion model as claimed in claim 1, characterized in that: The first pre-processing comprises the following steps: S1, remove irrelevant background data and noise on the point cloud data; S2, filling holes caused by incomplete collection or noise during the collection of point cloud data; S3, regridding, smoothing and noise filtering the point cloud data processed by S1 and S2; S4, projects the point cloud data onto a two-dimensional plane to generate a depth image, i.e., the source content image.

6. The method for generating stylized digital rubbings based on a diffusion model as claimed in claim 1, characterized in that: The second pretreatment comprises the following steps: Step 2.1, select rubbing images with different scripts, colors and styles; Step 2.2, use the median filter image denoising method to remove the noise and texture interference in the rubbing image to obtain the target style image.

7. A system for generating digital rubbings based on diffusion model style transfer, characterized in that: A method for generating digital rubbings based on diffusion model style transfer according to any one of claims 1 to 6, comprising a data processing unit, a feature extraction unit, a structure preservation unit and a stylized image generation unit; The data processing unit is used to collect point cloud data of the surface of the stele, and perform a first preprocessing on the point cloud data to obtain a source content image; and screening out rubbing images with different styles, performing a second preprocessing on them to obtain target style images, and using them as pre-training data sets; The feature extraction unit is used to decouple the structure and semantic information of the source content image to obtain the structural latent code Semantic latent coding And encode a target style picture in the pre-training dataset obtained in step 2 to generate style latent code The structure preserving unit is used to use a structure preserving network to encode the structure of the source content picture and semantic latent coding Process and output structure latent code and semantic latent coding The structure-preserving network includes a plurality of sequentially connected input layers, a multi-layer convolutional network and an output layer; The input layer is used to receive the semantic latent code of the source content picture extracted by the semantic encoder The multi-layer convolutional network is used to extract and enhance the semantic latent coding of the source content image. The output layer is used to generate the final structural latent code The stylized image generation unit is used to use a diffusion model to encode the structure latent and semantic latent coding And the style latent code of the target style image Processing and introducing directional CLIP loss to generate stylized images; The diffusion model is used to maintain the structure of the source content picture while latent encoding and semantic latent coding Generate stylized images based on The directional CLIP loss is used to optimize the diffusion model, that is, to ensure that the generated stylized image is consistent with the style latent encoding of the target style image. The features remain consistent, enabling the diffusion model to generate stylized images that are consistent with the style of the target style image and consistent with the content of the source content image.

Images