Denoising network model based on diffusion model and oracle rubbing image restoration method

Abstract

The invention discloses a denoising network model based on a diffusion model and an oracle rubbing image restoration method, which can improve the universality of the restoration method, improve the diversity of restoration results, reduce the calculation overhead and reduce the manual dependence. The denoising network model is adjusted based on an existing U-Net network and comprises a convolutional trunk network, an encoder, a middle layer, a decoder, a deep convolutional layer and a feedforward neural network which are connected in sequence; segmenting the input image by the convolutional trunk network; the encoder comprises a plurality of groups of cascaded paired visual state space blocks and image merging blocks; the decoder comprises a plurality of groups of cascaded paired visual state space blocks and image expansion blocks; performing down-sampling and up-sampling operations on the image merging block and the image expansion block respectively; the visual state space block adopts the architecture of an existing Mama module, and a two-dimensional selective scanning module is selected to replace an S6 module of the Mama module.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing algorithm design and development, and in particular to a denoising network model based on a diffusion model and an oracle bone rubbing image restoration method. Background Art

[0002] In the field of oracle bone rubbing image restoration, existing technologies mainly rely on digital image processing technology and manual intervention, including methods based on geometry and morphology. This method usually removes noise manually and retains the character stroke area, which has the following main disadvantages: (1) Lack of versatility - Most existing methods are based on low-level features of images, lack neural network support, are mainly suitable for single-word restoration, lack the ability to generalize to the entire oracle bone rubbing image, and are not adaptable to unknown data sets, making it difficult to become a general method.

[0003] (2) Limited diversity: Most existing methods adopt staged repair, and each stage can only generate a certain generation result. This single generation feature limits the possibility of repair.

[0004] (3) High computational overhead - In existing denoising networks, most denoising networks adopt the U-Net structure based on multi-head self-attention modules and residual connection modules. The computational complexity of the self-attention mechanism is proportional to the square of the input size, which is extremely expensive for high-resolution images, and there is an urgent need to improve computational efficiency.

[0005] (4) Heavy reliance on human participation - existing methods require human intervention, and the restoration effect depends largely on the experience of the intervener, and is highly dependent on experienced oracle bone restoration workers. Summary of the invention

[0006] The purpose of the present invention is to provide a denoising network model based on a diffusion model and an oracle bone rubbing image restoration method to solve the problems existing in the prior art, improve the versatility of the restoration method, enhance the diversity of the restoration results, reduce computational overhead, and reduce manual dependence.

[0007] In order to achieve the above object, one of the solutions of the present invention is: A denoising network model based on a diffusion model, which is adjusted based on the existing U-Net network, including a sequentially connected convolutional backbone network, encoder, intermediate layer, decoder and deep convolutional layer and feedforward neural network; The input image is segmented by the convolutional backbone network; The encoder includes several groups of cascaded paired visual state space blocks and image merging blocks; the decoder includes several groups of cascaded paired visual state space blocks and image expansion blocks; the image merging block and the image expansion block perform downsampling and upsampling operations respectively; The visual state space block adopts the architecture of the existing Mamba module and selects a two-dimensional selective scanning module to replace the S6 module of the Mamba module. Its workflow is as follows: an image matrix processed by a convolutional backbone network is taken as input. After the initial linear embedding layer, the output is divided into two information flows, one of which is transmitted through a The deep convolution layer then enters the two-dimensional selective scanning module through the SiLU activation function. The output of the two-dimensional selective scanning module passes through a normalization layer and is then added to the output of another information stream. The result of the addition is then activated by the SiLU activation function.

[0008] In the adjusted U-Net network, for a given height , width The input size is The convolutional backbone network divides the input image into multiple image blocks and then converts it into a one-dimensional sequence. The encoder is loaded with a pre-trained visual state space model and first resizes the feature dimensions of the image patch to , and the image is processed by several groups of cascaded 2 visual state space blocks and image merging blocks, so that the feature dimension of the image is doubled in each cascade stage of the encoder, while the image resolution is halved; among them, the image expansion block creates hierarchical features by reducing the image resolution and deepening the number of channels, and the visual state space block focuses on learning these feature representations; the output resolution of each stage of the encoder is , , and ; Several groups of cascaded 2 visual state space blocks and image expansion blocks in the decoder process the image so that the feature dimension of the image is halved in each cascade stage of the decoder, while the resolution of the image is doubled; Skip connections are established between the visual state space blocks of each layer of the encoder and decoder, and the skip connections are used to merge shallow and deep features to enhance the spatial details lost during the downsampling process; the middle layer includes two visual state space blocks, and a linear weight prediction branch is added after the first visual state space block.

[0009] Preferably, in each layer of the encoder, two consecutive visual state space blocks are used to process the linear mapping layer. The input image is divided into 1 / 4 size of the feature map of the previous layer in each downsampling process, and then they are connected and normalized by layer normalization to halve the length and width of the feature map and double the feature dimension. The decoder uses two consecutive visual state space blocks for feature reconstruction and uses an image expansion layer to upsample the deep features, halving the feature dimension and doubling the resolution.

[0010] The second solution of the present invention is: A method for restoring an oracle bone rubbing image, using the denoising network model based on a diffusion model, comprises two parts: a mask reconstruction step and a resampling step; the mask reconstruction step uses the known area of ​​the oracle bone rubbing image to be restored as a condition, and predicts and reconstructs the unknown area covered by the mask through a pre-trained denoising network model; the resampling step corrects the erroneous generation preference and artifacts caused by the difference between the training data of the denoising network model and the real data distribution, and by introducing the concept of jump length, gradually integrates semantic information in the denoising process to generate an image that is more coordinated with the known area.

[0011] The oracle bone rubbing image restoration method specifically comprises: Step 1: Pre-training First, the denoising network model needs to be pre-trained on a large amount of oracle bone data to learn the structure and texture features of the image; Step 2: Forward noise addition The oracle bone rubbing image and the corresponding mask, as well as the randomly sampled Gaussian noise, are used as the input of the denoising network model; Step 3: Reverse denoising The denoising network model recursively samples from the noise data by predicting the parameters of the Gaussian distribution, and gradually removes the noise added in step 2 according to the sampling results to generate a denoised high-quality output image; Step 4: Image Output Steps 2 and 3 are repeated, and an expected output image is obtained after multiple iterations. The expected output image is the restored oracle bone rubbing image.

[0012] After adopting the above technical solution, the present invention has the following technical effects: (I) Universal extension - This invention is the first to introduce the diffusion model into the downstream task of oracle bone rubbing image restoration. By introducing the diffusion model in the deep learning method, it is possible to achieve universal restoration of all oracle bone rubbing images by relying on its powerful modeling ability of the entire data set, so that the restoration method is no longer limited to a single character or a certain type of character, thereby improving universality.

[0013] (II) Improved diversity - The present invention adopts an end-to-end diffusion model to perform diversified restoration of oracle bone rubbing images. Specifically, it gradually removes noise through the reverse diffusion process, which can introduce randomness in each iteration, thereby generating a variety of possible restoration results, effectively overcoming the problem of single output that may be caused by the existing staged method; in addition, the diffusion model supports conditional generation and allows the input of prompt word information during the restoration process, further improving the diversity of the restoration results.

[0014] (III) Reducing computational overhead: The present invention integrates the visual state space model VMamba into the U-net architecture to improve the traditional U-net architecture. Relying on the advantage of low resource consumption of state space model reasoning, the efficiency and quality of restoration are improved without sacrificing restoration performance.

[0015] (IV) Reducing manual dependence - Since the present invention adopts machine learning methods for training, the dependence on manual processing is greatly reduced, and the prediction effect is stable, which reduces the dependence on the restorer's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is the overall flow chart of the present invention; Figure 2 A schematic diagram of the Mamba-UNet architecture of the denoising network model of the present invention; Figure 3 A comparison diagram of the efficient 2D scanning of the denoising network model of the present invention and the traditional selective scanning; Figure 4 is a repair flow chart of the image repair method of the present invention; Figure 5 A comparison chart of the repair quality between a specific embodiment of the present invention and an existing method; Figure 6 The figure is a comparison chart of computing resources and time overhead of a specific embodiment of the present invention and an existing method. DETAILED DESCRIPTION

[0017] First, let’s clarify some concepts: (1) Visual State Space Model (VMamba) VMamba is a visual model that combines the linear complexity of CNN and the global receptive field of Transformer. It uses the S6 model (described below) to process sequence data and implements four-way scanning through the CSM module to obtain global information while maintaining linear complexity. VMamba performs well in tasks such as image classification, see Figure 2 In part (b), the present invention mainly designs a set of oracle bone repair models based on this model.

[0018] (2) Visual State Space Block (VSS) VSS is a core component in the VMamba model. It aims to improve the model's ability to process visual data while maintaining a linear growth in computational complexity by combining the concepts of the selective scanning mechanism and the state-space model.

[0019] (3) Diffusion model The diffusion model is a generative model that transforms data into random noise by gradually adding noise and then learns how to reconstruct the data from the noise.

[0020] (4) Upsampling The input is the original image, and the output is the image with improved resolution based on the original image. The function is to improve the image resolution.

[0021] (5) Downsampling / Pooling The input is the original image, and the output is an image with a lower resolution than the original image, that is, the image matrix is ​​reduced while retaining the image features as much as possible to form an image thumbnail, which is then connected to obtain the common features of multiple images.

[0022] (6) Convolution The convolution kernel (a small matrix) corresponds to a part of the input matrix to be processed, multiplies them one by one, and then adds the results to get a number; this step is used to sharpen image features for subsequent feature extraction.

[0023] (7) Mask It is usually a two-dimensional array of the same size as the original image, and its value indicates whether the pixel at the corresponding position is retained (usually represented by 1) or blocked (usually represented by 0). In the present invention, the binary mask is used to mark the position that needs to be repaired and block other parts for protection.

[0024] (8) Gaussian noise Gaussian noise is a kind of statistical noise, and is used in the present invention to simulate the degradation process of an image, specifically, by gradually adding Gaussian noise to the image to simulate the degradation process of the image from being clear to being gradually blurred.

[0025] (9) Mamba module The Mamba module is a new deep learning architecture. Its core idea is to use the structured state space model (SSM) to efficiently process sequence data. Originally used for natural language processing, this invention uses its improved model VMamba for visual processing. The main process of Mamba is: ① Input data is fed into the model, and Mamba processes the data through its selective state space mechanism to capture the data features; ② The model uses hardware-aware algorithms for parallel computing, which greatly optimizes the computing efficiency on the GPU; ③Through a simplified state-space model architecture, the Mamba module can capture long-distance dependencies in sequence data while maintaining computational efficiency.

[0026] (10) 2D-Selective-Scan for Vision Data (SS2D) SS2D is a visual data processing module in the VMamba model. It expands 2D image features into 1D sequences through four-way scanning, processes these sequences with the S6 module, and then merges the results back into 2D feature maps. This design allows the model to process images efficiently and effectively capture global information.

[0027] (11) Structured State Space for Sequences (S6) A deep learning model for processing sequence data; it enhances the model's ability to process input sequences by introducing a selection mechanism, allowing the model to dynamically adjust its focus based on the relevance of the input, thereby improving the efficiency and accuracy of processing long sequence data.

[0028] (12) Sigmoid Linear Unit (SiLU activation function) A function that combines the characteristics of the Sigmoid function and the identity function, with an output value between 0 and 1, can adaptively adjust the signal flow, and helps to alleviate the gradient disappearance problem; in the present invention, it determines whether to activate a certain part of the neural network based on the strength of the input signal.

[0029] (13) Transformer architecture The Transformer architecture is a deep learning model based on the self-attention mechanism, originally designed for natural language processing tasks; its core consists of an encoder and a decoder, both of which use self-attention layers to process sequence data, allowing the model to capture dependencies at any position in the sequence. It is currently the most commonly used image processing architecture.

[0030] (14) U-net network U-Net is a convolutional network specially designed for image segmentation, which consists of an encoder and a decoder to form a U-shaped structure; the encoder gradually reduces the spatial dimension of the image and extracts features; the decoder gradually restores the spatial dimension of the image, and combines the feature map of the encoder with the feature map of the decoder through jump connections to improve the accuracy of segmentation. The present invention integrates VSS blocks on the basis of U-net to improve Mamba-Unet.

[0031] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0032] refer to Figure 1-3 As shown, the present invention discloses a denoising network model based on a diffusion model. The denoising process is an important part of the oracle bone restoration process and is used to process image noise and increase image clarity. Figure 2 In part (b), the denoising network model is adjusted based on the existing U-Net network, including a sequentially connected convolutional trunk (STEM) network (a network formed by connecting convolutional layers and downsampling layers), an encoder, an intermediate layer, a decoder, a depthwise convolutional neural network (DWCNN) and a feedforward neural network (FFN). The architecture of the feedforward neural network is shown in Figure 2 part (c) of The input image is segmented by the STEM network; The encoder includes several groups of cascaded paired visual state space blocks (VSS blocks) and image merging blocks (PM blocks); the decoder includes several groups of cascaded paired visual state space blocks and image extension blocks (PE blocks); the image merging blocks and image extension blocks perform downsampling (which can be considered as shrinking the image and then merging it) and upsampling (i.e. increasing the resolution of the input image) operations respectively; See also Figure 2 In part (a), the architecture of the VSS block is very similar to that of the Mamba module. The main difference between the two is that the VSS block uses a two-dimensional selective scanning module (SS2D) to replace the S6 module of the Mamba module, making the selective scanning more efficient and able to adapt to two-dimensional visual data.

[0033] The above VSS block is the key module of this denoising network model, which is used to perform privilege extraction on the image and keep the image size unchanged. The purpose of stacking VSS blocks multiple times is to extract the hidden features in the image. Specifically, the workflow of the VSS block is as follows: an image matrix processed by the STEM network is used as input. After the initial linear embedding layer, the output is divided into two information flows. One information flow passes through a The deep convolution layer is then passed through the SiLU activation function to enter the core SS2D module. The output of the SS2D module passes through a normalization layer and is then added to the output of another information stream. The result of the addition is then activated by the SiLU activation function. In view of the fact that the selective scan has converted the two-dimensional data into one-dimensional data, the present invention does not use the position embedding technology in the visual transformer commonly used for image processing. Here, the initial linear embedding layer is to preliminarily extract features and reduce the matrix dimension in preparation for subsequent feature extraction; the SiLU activation function is used to extract nonlinear features; and the normalization layer is used to limit the scan value to between 0 and 1 for easy processing.

[0034] It is worth noting that the architectural design of the VSS block improves on the layer normalization → multi-head self-attention → layer normalization → multi-layer perceptron (MLP) structure of the Transformer protocol. The VSS block completely omits the multi-perceptron layer and only uses the Sigmoid linear unit (SiLU activation function) immediately after the residual connection of the previous layer to capture the nonlinear features in the image. This structure significantly reduces the large number of parameters introduced by the stacked hidden layers of the multi-layer perceptron, so the VSS block usually requires fewer parameters than the attention-based block, and can stack more blocks within a similar overall model depth budget, or use fewer blocks to achieve the desired effect. The introduction of the VSS block enables the present invention to have higher architectural efficiency, better performance indicators and excellent scalability, making it a key component in the present invention.

[0035] A specific embodiment of the above-mentioned denoising network model is shown below.

[0036] In the adjusted U-Net network, for a given height , width The input size is The STEM network divides the input image into multiple image blocks and then converts it into a one-dimensional sequence. The encoder is loaded with a pre-trained visual state space model (VMamba), which first resizes the feature dimensions of the image patch to (Not specifically limited, can be adjusted according to actual needs), and the image is processed by several groups of cascaded 2 VSS blocks and image merging blocks, so that the feature dimension of the image is doubled in each cascade stage of the encoder, while the image resolution is halved; among them, the image expansion block creates hierarchical features by reducing the image resolution and deepening the number of channels, and the VSS block focuses on learning these feature representations; the output resolution of each stage of the encoder is , , and ; The following explains several key steps and innovations of the denoising network model: (1) Decoder: Several groups of cascaded 2 VSS blocks and image expansion blocks in the decoder process the image so that the feature dimension of the image is halved at each cascade stage of the decoder, while the resolution of the image is doubled, in order to achieve the same feature size output in the style of the encoder; (2) Skip connection and weight prediction branch: Skip connections are established between the VSS blocks of each layer of the encoder and decoder. The skip connections are used to merge shallow and deep features to enhance the spatial details lost in the downsampling process. The middle layer includes two VSS blocks, and a linear weight prediction branch is added after the first VSS block based on the design idea of ​​U-net. The weight prediction branch and skip connection ensure the retention of spatial details and restoration diversity in the restoration process of this model.

[0037] (3) Image feature learning: Furthermore, in each layer of the encoder, two consecutive VSS blocks are used to process the linear mapping layer. The first step is to learn features from a low-resolution image of a certain dimension and downsample it by using a small-sized block merging operation. This is done three times in total. Each downsampling process divides the input image into 1 / 4 the size of the feature map of the previous layer, and then connects them and normalizes them through layer normalization, thereby halving the length and width of the feature map and doubling the feature dimension. The decoder uses two consecutive VSS blocks for feature reconstruction and uses an image expansion layer to upsample the deep features, thereby halving the feature dimension and doubling the resolution.

[0038] (4) Efficient 2D scanning: The most critical part of the present invention is the efficient 2D scanning mechanism embedded in the VSS block, which can be considered as an improved version of the selective scanning mechanism. The cross scanning method used in VMamba requires traversing the image block of the entire row or column axis, which will consume a lot of computing resources. Therefore, combined with previous research, the VSS block of the present invention adopts an efficient 2D scanning method SS2D to traverse the spatial domain and convert any non-causal visual image into an ordered sequence of blocks; it also solves the direction sensitivity problem encountered in the scanning process. This method replaces the traditional cross scanning module (CSM) and effectively reduces the time complexity from Reduce to ,in Indicates the step size. For details on efficient 2D scanning methods, see Figure 3 , Figure 3 The upper part of the figure shows the cross-scanning mode (CSM) in the VSS block, which requires a full traversal of each image block. In contrast, the lower part shows the efficient scanning mode with a step size of (In the figure, =2 as an example), significantly reduces the time complexity of two-dimensional scanning. This process decomposes the complete scanning method into local and global sparse forms, and reduces the computational complexity by skipping the sampling of the local receptive field and selectively scanning smaller blocks of the feature map. , we use Interval sampling block to blocks, while in SS2D they are , which reduces the number of tags processed in each scan and merge operation, from Reduce to , thereby improving the efficiency of feature extraction. At the same time, although the significant reduction in the number of sampling times may cause some of the image's hidden deeper feature details to be ignored, considering that the object of restoration targeted by the present invention is oracle bone inscriptions, whose images are in black and white and mainly focus on basic strokes, and do not care about features such as image details and colors, the efficient scanning method proposed by the present invention is suitable for oracle bone inscription restoration, and can greatly reduce the scanning time without basically affecting the restoration effect.

[0039] See also Figure 4 The present invention also discloses a method for restoring an oracle bone rubbing image, using the above-mentioned denoising network model based on the diffusion model, including a mask reconstruction step ( Figure 4 The upper part) and the resampling step ( Figure 4The lower half of the image); the mask reconstruction step uses the known area of ​​the oracle bone rubbing image to be repaired as a condition, and predicts and reconstructs the unknown area covered by the mask through the pre-trained denoising network model, with the purpose of gradually removing noise and restoring the original structure of the image while keeping the known area unchanged; the resampling step corrects the erroneous generation preference and artifacts caused by the difference between the training data of the denoising network model and the real data distribution, and gradually integrates semantic information in the denoising process by introducing the concept of jump length, generating an image that is more coordinated with the known area, while reducing possible artificial traces, such as checkerboard and radial images. These two parts together ensure that the oracle bone rubbing image repaired by the denoising network model and method of the present invention is not only consistent with the original image in style, but also more reasonable in semantics, effectively improving the correctness of the repair.

[0040] The following is a specific embodiment of the above repair method, which specifically includes: Step 1: Pre-training In order to enable the denoising network model to handle the task of repairing oracle bone rubbings, it is first necessary to pre-train the denoising network model on a large amount of oracle bone data to learn the structure and texture features of the image.

[0041] Step 2: Forward noise addition The oracle bone rubbing image, the corresponding mask, and the randomly sampled Gaussian noise are used as the input of the denoising network model. The forward denoising process is to add noise to the oracle bone rubbing image and multiply the obtained image with the corresponding mask to obtain the result of the protected part + noise.

[0042] Step 3: Reverse denoising The denoising network model recursively samples from the noise data by predicting the parameters of the Gaussian distribution, and gradually removes the noise added in step 2 according to the sampling results to generate a denoised high-quality output image. The resampling step mentioned above is applied in the denoising process. After each denoising, a resampling operation is performed. This operation converts the output of the model into Backward diffusion to to achieve (see Figure 1 , 2 ), which is done by sampling from a specific distribution. Although this operation may reduce the clarity of the output and introduce noise, it still preserves the generated region Part of the information in, thus generating a new ,Should Not only with known areas It is more coordinated and contains conditional information from AI-generated areas and known areas to ensure that the repaired image area is visually consistent and coherent with other parts of the original image.

[0043] Step 4: Image Output Repeat steps 2 and 3. Figure 1 After multiple iterations of the OB block, an output image that meets expectations (the image quality meets the standards for cultural relics restoration) is obtained. The output image that meets expectations is the restored oracle bone rubbing image.

[0044] The technical effect of the present invention will be illustrated by the following comparative experiments: Here, the model of the present invention is named Orpaint. In order to highlight the role of integrating the VSS module into the denoising network and reduce the difference caused by the selection of different principle models, the present invention will use various models with similar principles for comparison. Specifically, the most representative models are selected from three mainstream zero-sample-based image restoration methods for comparison. For example: for the projection-based method, the Repaint model is selected; for the decomposition-based method, the denoising diffusion restoration model (DDRM) is selected; for the posterior estimation-based method, the diffuse posterior sampling (DPS) model is selected for comparison. The overall effect is as follows: Comparison table of repair effects of various models

[0045] See table above and Figure 5-6 By comparing the relevant data in , we can see that: (a) In terms of restoration quality, under various large-area mask types, Repaint and the present invention both achieve the best performance: specifically, when the mask area is small, Repaint integrated with the multi-head self-attention block performs more stably; and as the mask area increases, the performance of the present invention integrated with the VSS module gradually improves, even surpassing Repaint. It can be seen that the present invention has a good restoration effect on large-area damaged oracle bones and basically guarantees the original restoration effect.

[0046] (b) In terms of computing resources and time consumption, since the matrix singular value decomposition technology used by DDRM can be accelerated by parallelization methods, its time consumption is the lowest compared with other methods. Repaint takes the longest time due to a large number of self-attention calculations and residual connections. Although the present invention is not as time-consuming as DDRM, it is better than Repaint with a similar architecture in terms of model complexity, number of parameters, and time consumption. It can be seen that the improved Mamba-Unet of the present invention does accelerate the speed of repair.

[0047] In general, the present invention finds a good balance between the resources consumed by repair and the repair effect, while maximizing the repair effect, greatly reducing the time and computing resources consumed by repair.

[0048] At the same time, it should be noted that compared with other existing technologies, the key step of the present invention - efficient 2D scanning is proposed in the context of oracle bone inscription restoration, and is not applicable in other image restoration scenarios. This is because of the characteristics of oracle bone inscription restoration, namely: the image is in black and white; only the oracle bone inscription strokes are focused on; no attention is paid to image color, pixels and other features; and the image resolution is not required to be high. Based on these characteristics, efficient 2D scanning can greatly improve scanning efficiency while ignoring a small number of features, which is also the uniqueness of the present invention compared with other similar inventions.

[0049] Through the above scheme, the present invention dynamically adjusts the variance during the sampling process to adapt to the complexity of different image features, and introduces a non-Markov diffusion process, which allows the randomness to be controlled when generating samples, achieving a trade-off between sample quality and computational efficiency. Compared with traditional restoration methods, the present invention has the following technical effects: (I) Universal extension - This invention is the first to introduce the diffusion model into the downstream task of oracle bone rubbing image restoration. By introducing the diffusion model in the deep learning method, it is possible to achieve universal restoration of all oracle bone rubbing images by relying on its powerful modeling ability of the entire data set, so that the restoration method is no longer limited to a single character or a certain type of character, thereby improving universality.

[0050] (II) Improved diversity - The present invention adopts an end-to-end diffusion model to perform diversified restoration of oracle bone rubbing images. Specifically, it gradually removes noise through the reverse diffusion process, which can introduce randomness in each iteration, thereby generating a variety of possible restoration results, effectively overcoming the problem of single output that may be caused by the existing staged method; in addition, the diffusion model supports conditional generation and allows the input of prompt word information during the restoration process, further improving the diversity of the restoration results.

[0051] (III) Reducing computational overhead: The present invention integrates the visual state space model VMamba into the U-net architecture to improve the traditional U-net architecture. Relying on the advantage of low resource consumption of state space model reasoning, the efficiency and quality of restoration are improved without sacrificing restoration performance.

[0052] (IV) Reducing manual dependence - Since the present invention adopts machine learning methods for training, the dependence on manual processing is greatly reduced, and the prediction effect is stable, which reduces the dependence on the restorer's experience.

[0053] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A denoising network model based on a diffusion model, characterized in that: Adjustment based on the existing U-Net network, including sequentially connected convolutional backbone network, encoder, intermediate layer, decoder and deep convolutional layer and feedforward neural network; The input image is segmented by the convolutional backbone network; The encoder includes several groups of cascaded paired visual state space blocks and image merging blocks; the decoder includes several groups of cascaded paired visual state space blocks and image expansion blocks; the image merging block and the image expansion block perform downsampling and upsampling operations respectively; The visual state space block adopts the architecture of the existing Mamba module and selects a two-dimensional selective scanning module to replace the S6 module of the Mamba module. Its workflow is as follows: an image matrix processed by a convolutional backbone network is taken as input. After the initial linear embedding layer, the output is divided into two information flows, one of which is transmitted through a The deep convolution layer then enters the two-dimensional selective scanning module through the SiLU activation function. The output of the two-dimensional selective scanning module passes through a normalization layer and is then added to the output of another information stream. The result of the addition is then activated by the SiLU activation function.

2. The denoising network model based on the diffusion model as claimed in claim 1, characterized in that: In the adjusted U-Net network, for a given height , width The input size is The convolutional backbone network divides the input image into multiple image blocks and then converts it into a one-dimensional sequence. The encoder is loaded with a pre-trained visual state space model and first resizes the feature dimensions of the image patch to , and the image is processed by several groups of cascaded 2 visual state space blocks and image merging blocks, so that the feature dimension of the image is doubled in each cascade stage of the encoder, while the image resolution is halved; among them, the image expansion block creates hierarchical features by reducing the image resolution and deepening the number of channels, and the visual state space block focuses on learning these feature representations; the output resolution of each stage of the encoder is , , and ; Several groups of cascaded 2 visual state space blocks and image expansion blocks in the decoder process the image so that the feature dimension of the image is halved in each cascade stage of the decoder, while the resolution of the image is doubled; Skip connections are established between the visual state space blocks of each layer of the encoder and decoder, and the skip connections are used to merge shallow and deep features to enhance the spatial details lost during the downsampling process; the middle layer includes two visual state space blocks, and a linear weight prediction branch is added after the first visual state space block.

3. The denoising network model based on the diffusion model as claimed in claim 2, characterized in that: In each layer of the encoder, two consecutive visual state space blocks are used to process the linear mapping layer. The input image is divided into 1 / 4 size of the feature map of the previous layer in each downsampling process, and then they are connected and normalized by layer normalization to halve the length and width of the feature map and double the feature dimension. The decoder uses two consecutive visual state space blocks for feature reconstruction and uses an image expansion layer to upsample the deep features, halving the feature dimension and doubling the resolution.

4. A method for restoring an image of an oracle bone rubbing, using a denoising network model based on a diffusion model as claimed in any one of claims 1 to 3, characterized in that: It includes two parts: mask reconstruction step and resampling step; The mask reconstruction step uses the known area of ​​the oracle bone rubbing image to be restored as a condition, and predicts and reconstructs the unknown area covered by the mask through the pre-trained denoising network model; The resampling step corrects the incorrect generation preferences and artifacts caused by the difference between the training data of the denoising network model and the real data distribution. By introducing the concept of jump length, semantic information is gradually integrated in the denoising process to generate images that are more coordinated with known regions.

5. The oracle bone rubbing image restoration method as claimed in claim 4, characterized in that include: Step 1: Pre-training First, the denoising network model needs to be pre-trained on a large amount of oracle bone data to learn the structure and texture features of the image; Step 2: Forward noise addition The oracle bone rubbing image and the corresponding mask, as well as the randomly sampled Gaussian noise, are used as the input of the denoising network model; Step 3: Reverse denoising The denoising network model recursively samples from the noise data by predicting the parameters of the Gaussian distribution, and gradually removes the noise added in step 2 according to the sampling results to generate a denoised high-quality output image; Step 4: Image Output Steps 2 and 3 are repeated, and an expected output image is obtained after multiple iterations. The expected output image is the restored oracle bone rubbing image.

Images