A method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration

Abstract

The present invention discloses a method for removing highlight from ceramic artifact images based on a dual attention mechanism and detail restoration, comprising the following steps: 1. Cropping the collected original highlight image to 200×200 based on its center; 2. Extracting hypercolumn features of the image using a pre-trained VGG-19 network, and restoring it to 200×200 by upsampling; 3. Concatenating the extracted feature map with the original highlight image as an enhanced input of the network, extracting image features and expanding the dimension of the feature map through a Conv-ReLU sequence; 4. First, extracting and learning feature information of the feature map through a plurality of residual blocks embedded in CBAM modules, and then obtaining features of a wider range of perception field of view using a detail restoration module; 5. Using a layer of ordinary convolution, the spatial information extracted by the local multi-scale feature extraction module is interacted with the information extracted by the smoothed dilated convolution, and then the feature map is restored to 200×200 to obtain a highlight-free image, thereby effectively restoring the color and texture detail information of the image.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics processing, specifically relating to a method for desaturating ceramic artifact images based on dual attention mechanism kernel detail restoration. Background Technology

[0002] China boasts a long history of ceramics, with ceramic artists throughout the ages creating a diverse array of artworks under their rich imaginations. These ceramics are a precious historical and cultural heritage, reflecting the social development and cultural conditions of their time. Due to the unique nature of the raw materials, they are prone to damage, warping, cracking, and staining during long-term preservation. Therefore, to maximize the lifespan of ancient pottery and restore its original appearance, allowing viewers to reflect on history, it is essential to protect and pass on ancient ceramic artifacts. With the development and widespread use of computers, digital technology offers a good way to "bring artifacts to life." However, ceramics are highly reflective and have a high gloss level. When photographing smooth porcelain surfaces, specular reflection can easily occur, creating highlights that significantly degrade image quality and lose some of the desired information. This has a serious impact on subsequent artifact preservation and restoration work. According to statistics, the Palace Museum's digital cultural relics database, which contains the most cultural relics, has collected 377,173 pieces / sets of ceramics. Most of the images of cultural relics displayed in the database still have specular reflections, which have a certain impact on the overall texture of the porcelain cultural relics and the public's perception. Therefore, removing the highlights from the surface of porcelain cultural relics is of great significance for promoting the dissemination and protection of cultural relics.

[0003] Since Shafer proposed the two-color reflection model for image specular removal in 1985, many other specular removal methods have emerged. Early methods involved acquiring multiple images using a moving light source or camera, then applying polarization filtering or color information for specular removal. Boult and Wolff used polarization filters to separate the reflection component from the grayscale image based on the polarization properties of light waves. While this method could remove specular highlights, it was limited by hardware constraints. Nayar et al., building on Boult and Wolff's work, introduced color factors and proposed a color segmentation method that used the light intensity changes caused by polarization filters to remove specular highlights. This method was feasible for images without texture details, but its specular removal effect was not ideal for complex natural scenes, and it could not acquire multiple images simultaneously, resulting in poor adaptability. Therefore, a single-image specular removal method based on the two-color reflection model, color space, and adjacent pixels was proposed. However, Shen's method blurs a large area of ​​detail in the result, and even adds black information in near-white areas; Akashi's method cannot completely remove highlights and adds more noise to the entire image, resulting in a loss of detail.

[0004] With the development of deep learning, models for specular removal based on deep learning methods have emerged. Among them, Fu et al. proposed a unified framework for joint specular detection and removal, and designed multiple extended spatial context feature aggregation (DSCFA) modules to obtain context features at different scales. This accurately detects the location of specular highlights and then removes them. Although it can remove specular regions, there are still differences between the detailed information and the ground truth map. Therefore, it cannot achieve the ideal effect when processing colored lighting scenes.

[0005] In short, existing image highlight removal methods suffer from the following problems: First, the hue-saturation ambiguity problem has not been effectively solved. Pixels with low saturation may be misidentified as highlight pixels, causing visual defects in that area. For example, traditional highlight removal methods cannot perform semantic disambiguation between pure white or near-white areas and highlights, resulting in black blocks. Second, the problem of image restoration for images with complex texture details and rich colors has not been fully solved. Recent deep learning-based highlight removal methods can remove highlights, but their network architecture lacks flexibility and performs poorly on highly targeted highlight images, with errors still existing in the restoration of true color and texture information. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method for de-highlighting ceramic artifact images based on dual attention mechanism kernel detail restoration. This method can not only effectively restore the color and texture details of the image, but also has high timeliness.

[0007] To achieve the above objectives, the present invention employs the following technical solution:

[0008] A method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration includes the following steps:

[0009] Step 1: Using the center of the collected original highlight image as a reference, uniformly scale and crop it to 200 pixels × 200 pixels to obtain the preprocessed highlight image A;

[0010] Step 2: Input the preprocessed specular image A into the pre-trained VGG-19 network to extract hypercolumn features, and then upsample it to restore it to a 200-pixel × 200-pixel feature map.

[0011] Step 3: Concatenate the feature map extracted in Step 2 with the original highlight image as the enhanced input to the network, thereby helping the network learn semantic cues from the input image. Extract image features and expand the feature map dimension through Conv-ReLU sequence.

[0012] Step 4: Extract and learn the feature information of the feature map output in Step 3 by using multiple residual blocks embedded with CBAM modules, and then use the detail recovery module to obtain features of a wider range of receptive vision.

[0013] The detail restoration module consists of a local multi-scale feature extraction module and three smooth dilated convolution modules;

[0014] Step 5: Use a single ordinary convolution layer to interact with the spatial information extracted by the local multi-scale feature extraction module and the information extracted by the smooth dilation convolution layer, thereby enriching the detail information of the feature map. Then restore the feature map to 200 pixels × 200 pixels to obtain the image D without highlights.

[0015] Furthermore, the Conv-ReLU sequence in step 3 is a sequence composed of 3*3 convolutions and ReLU activation functions, and the BN layer following the 3*3 convolutions has been removed.

[0016] Furthermore, the CBAM module in step 4 includes a channel attention module and a spatial attention module. The attention weight coefficients obtained by the channel attention module and the spatial attention module are multiplied with the input feature map F to adaptively refine the features.

[0017] Furthermore, the specific process of the CBAM module in step 4 extracting and learning the feature information of the feature map is as follows:

[0018] Step 4.1.1: First, the feature map F is compressed in spatial dimension by max pooling and average pooling to form two weight vectors [C,1,1]. Then, the two weight vectors are input into the same multilayer perceptron and mapped to the weights of each channel. Then, the sigmoid function is used to activate the attention weights [C,1,1] of the output channel, and the weights are multiplied element-wise with the original features [C,H,W] to obtain the feature map F. 1 ;

[0019] Step 4.1.2: First, extract the feature map F. 1 As input to the spatial attention module, the input is compressed along the channel dimension using max pooling and average pooling to obtain two feature maps. These two feature maps are then stacked and subjected to a convolutional operation to obtain two weight vectors of [1, H, W]. The two weight vectors of [1, H, W] are then multiplied element-wise with the original input feature [C, H, W] to obtain the feature map F. 2 .

[0020] Furthermore, the specific process of obtaining features of a wider receptive field of view using the detail restoration module in step 4 is as follows:

[0021] Step 4.2.1: The local multi-scale feature extraction module first performs convolution operations in parallel using four types of convolution kernels: 3*3, 5*5, 7*7, and 9*9. Then, it performs dimensionality reduction operations on features of different scales through 1*1 convolution. Next, it uses deconvolution to restore the feature map to 200 pixels × 200 pixels, then aggregates it with the original features, and then performs a convolution operation.

[0022] Step 4.2.2: Connect three smooth dilated convolutional modules with dilation rates of 1, 3 and 5 in parallel in the local multi-scale feature extraction module, and fuse their output features.

[0023] Furthermore, the total loss function used in steps 2 to 5 includes a pixel loss function, a feature loss function, and an adversarial loss function, wherein:

[0024] The pixel loss function is expressed as:

[0025]

[0026] In the formula, where and represent the gradient operators in the x and y directions, respectively, and D represents the true specular-free image. The image representing the predicted image without highlights has α = 0.2 and β = 0.3.

[0027] The feature loss function is expressed as:

[0028]

[0029] In the formula, φ l In this context, 'l' represents the 'l' layer in VGG-19, and 'λ' represents the 'l' layer. l Represents the balancing weight;

[0030] The adversarial loss function is expressed as:

[0031]

[0032] In the formula, σ represents the sigmoid activation function, and C is the non-transform discriminant function. Representative discriminator network;

[0033] Therefore, the total loss function is expressed as:

[0034] l = w1l pixel +w2l feat +w3l adv .

[0035] In the formula, w1 = 1, w2 = 0.1, and w3 = 0.01.

[0036] Compared with the prior art, the present invention has the following beneficial technical effects:

[0037] First, a pre-trained VGG-19 network is used to extract features from the highlight image as the network's augmented input. Then, Conv-ReLU sequences are used to extract feature information. Next, multiple residual blocks embedded with CBAM modules are used to fuse the extracted feature information. Multiple connected residual blocks embedded with CBAM modules are used to transform the feature information. To obtain features from a larger receptive field, a detail restoration module is designed. This module consists of a traditional local multi-scale feature extraction block and three smooth dilated convolutional blocks. Since the feature correlation weakens after dilated convolution, a regular convolution layer is added after the detail restoration module to restore the feature map to its original size. The predicted de-highlighted image is then output. It can be seen that this invention utilizes the idea of ​​deep learning, which can quickly obtain a highlight-free image from a single highlight image. It requires fewer parameters to process the network, shortens the training time, has high real-time performance, and can better restore the texture, color, and other details of the image, resulting in a good highlight removal effect. Attached Figure Description

[0038] Figure 1 This is the original highlight image selected for this invention;

[0039] Figure 2 This is a schematic diagram of the CBAM module of the present invention;

[0040] Figure 3 This is a schematic diagram of the residual block embedded in CBAM according to the present invention;

[0041] Figure 4 This is a schematic diagram of the detailed recovery module of the present invention;

[0042] Figure 5 This is a schematic diagram of the image desaturation network structure based on dual attention mechanism and detail restoration of the present invention;

[0043] Figure 6 This is the specular-free image D predicted by the present invention;

[0044] Figure 7 These are the original highlight image a and the predicted non-highlight image b selected in this invention;

[0045] Figure 8 These are the original highlight image e and the predicted non-highlight image f selected in this invention. Detailed Implementation

[0046] The specific content of the present invention will be further explained in detail below with reference to the embodiments.

[0047] like Figure 5As shown, the method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration includes the following steps:

[0048] Step 1: Using the center of the original highlight image as a reference, uniformly scale and crop it to 200 pixels × 200 pixels, resulting in the following: Figure 1 The preprocessed highlight image A is shown below;

[0049] Step 2: Input the preprocessed highlight image into the pre-trained VGG-19 network to extract hypercolumn features, and then upsample it to restore it to a 200-pixel × 200-pixel feature map.

[0050] Step 3: Concatenate the feature map extracted in Step 2 with the original specular image as the enhanced input to the network, thereby helping the network learn semantic cues from the input image. Extract image features and expand the feature map dimension through Conv-ReLU sequence.

[0051] The Conv-ReLU sequence is a sequence of 3*3 convolutions and ReLU activation functions with BN layers removed. Removing BN layers is crucial for performance optimization, as the error increases and network stability deteriorates when the batch size becomes too small.

[0052] Step 4: First, extract and learn the feature information of the feature map output in Step 3 using multiple residual blocks embedded with CBAM modules to prevent gradient vanishing caused by the deep layers of the convolutional neural network. Then, use the detail recovery module to obtain features of a wider receptive field. The structure of the residual blocks embedded with CBAM modules is as follows: Figure 3 As shown, the structure of the CBAM module is as follows: Figure 2 As shown, the CBAM module includes a channel attention module and a spatial attention module. The attention weight coefficients obtained from the channel attention module and the spatial attention module are multiplied by the input feature map to adaptively refine the features. The specific process of the CBAM module extracting and learning the feature information of the feature map is as follows:

[0053] Step 4.1.1: First, the feature map F is compressed in spatial dimension by max pooling and average pooling to form two weight vectors [C,1,1]. Then, the two weight vectors are input into the same multilayer perceptron and mapped to the weights of each channel. Then, the sigmoid function is used to activate the attention weights [C,1,1] of the output channel, and the weights are multiplied element-wise with the original features [C,H,W] to obtain the feature map F. 1 ;

[0054] Step 4.1.2: First, extract the feature map F. 1As input to the spatial attention module, the input is compressed along the channel dimension using max pooling and average pooling to obtain two feature maps. These two feature maps are then stacked and subjected to a convolutional operation to obtain two weight vectors of [1, H, W]. The two weight vectors of [1, H, W] are then multiplied element-wise with the original input feature [C, H, W] to obtain the feature map F. 2 ;

[0055] The structure of the detail recovery module is as follows: Figure 4 As shown, it consists of a traditional local multi-scale feature extraction module and three smooth dilated convolution modules. The specific process of obtaining features of a wider receptive field of view using the detail restoration module is as follows:

[0056] Step 4.2.1: The local multi-scale feature extraction module first performs convolution operations in parallel using four types of convolution kernels: 3*3, 5*5, 7*7, and 9*9. Then, it performs dimensionality reduction operations on features of different scales through 1*1 convolution. Next, it uses deconvolution to restore the feature map to 200 pixels × 200 pixels, then aggregates it with the original features, and then performs a convolution operation.

[0057] Step 4.2.2: Connect three smooth dilated convolutional modules with dilation rates of 1, 3 and 5 in parallel in the local multi-scale feature extraction module, and fuse their output features;

[0058] Step 5: Since the feature correlation weakens after dilated convolution, a regular convolution layer is added after the detail restoration module to restore the feature map to its original 200 pixels × 200 pixels. The spatial information extracted by the local multi-scale feature extraction module interacts with the information extracted by the smooth dilation convolution, thereby enriching the detail information of the feature map, resulting in... Figure 6 The image D shown is without highlights;

[0059] contrast Figure 1 and Figure 6 It can be seen that during the process of removing highlights, the ceramic artifacts retained a clear background and details and color information of the decorative patterns, without any visual distortion.

[0060] In this embodiment, steps 2 through 5 are as follows: Figure 5 The total loss function used in the training process of the ceramic artifact image desaturation network based on dual attention mechanism and detail restoration, as shown, includes pixel loss function, feature loss function, and adversarial loss function, where:

[0061] To suppress distortion of texture and color information in the image, a pixel loss function is used to calculate the difference between the specular-free image D and the true value. The pixel-by-pixel difference, expressed by the pixel loss function, is as follows:

[0062]

[0063] In the formula, where and represent the gradient operators in the x and y directions, respectively, and D represents the true specular-free image. The image representing the predicted image without highlights has α = 0.2 and β = 0.3.

[0064] To achieve a more realistic visual effect, the feature loss function is defined using the activations from the pre-trained VGG-19 network in ImageNet, expressed as:

[0065]

[0066] In the formula, φ l In this context, 'l' represents the 'l' layer in VGG-19, and 'λ' represents the 'l' layer. l Represents the balanced weights, and the feature loss function is used in the conv3_2, conv4_2 and conv5_2 layers of VGG-19.

[0067] To improve the realism and similarity of the obtained specular-removed image, a discriminator network is employed. To evaluate image quality, the adversarial loss function is expressed as:

[0068]

[0069]

[0070] In the formula, σ represents the sigmoid activation function, and C is the non-transform discriminant function;

[0071] Therefore, the total loss function is expressed as:

[0072] l = w1l pixel +w2l feat +w3l adv

[0073] In the formula, w1 = 1, w2 = 0.1, and w3 = 0.01.

[0074] See Figure 7 After the original highlight image a is processed by the highlight removal method for ceramic artifact images based on dual attention mechanism and detail restoration proposed in this embodiment, the output is a highlight-free image b. It can be seen that a large area of ​​highlights on the original highlight image a has been removed, and no black information blocks or noise information have been generated, so the overall visual effect is not affected.

[0075] See Figure 8After the original highlight image e is processed by the highlight removal method for ceramic artifact images based on dual attention mechanism and detail restoration proposed in this embodiment, the output is a highlight-free image f. The highlight areas of the original highlight image e have been completely removed, and the information of the non-highlight areas has been completely preserved.

[0076] contrast Figure 7 a and b and Figure 8 As can be seen from e and f, the method for removing highlights from ceramic artifact images based on dual attention mechanism and detail restoration proposed in this embodiment can better restore the texture, color and other details of the image during the highlight removal process, and the image does not have any distortion, thus having a good highlight removal effect.

Claims

1. A method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration, characterized in that, Includes the following steps: Step 1: Using the center of the collected original highlight image as a reference, uniformly scale and crop it to 200 pixels × 200 pixels to obtain the preprocessed highlight image A; Step 2: Input the preprocessed highlight image into the pre-trained VGG-19 network to extract hypercolumn features, and then upsample it to restore it to a 200-pixel × 200-pixel feature map. Step 3: Concatenate the feature map extracted in Step 2 with the original highlight image as the enhanced input to the network, thereby helping the network learn semantic cues from the input image. Extract image features and expand the feature map dimension through Conv-ReLU sequence. Step 4: Extract and learn the feature information of the feature map output in Step 3 by using multiple residual blocks embedded with CBAM modules, and then use the detail recovery module to obtain features of a wider range of receptive vision. The detail restoration module consists of a local multi-scale feature extraction module and three smooth dilated convolution modules; Step 5: Use a single ordinary convolution layer to interact with the spatial information extracted by the local multi-scale feature extraction module and the information extracted by the smooth dilation convolution layer, thereby enriching the detail information of the feature map. Then restore the feature map to 200 pixels × 200 pixels to obtain the image D without highlights.

2. The method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration according to claim 1, characterized in that, The Conv-ReLU sequence in step 3 is a sequence consisting of a 3*3 convolution and a ReLU activation function, with the BN layer after the 3*3 convolution removed.

3. The method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration according to claim 1, characterized in that, The CBAM module in step 4 includes a channel attention module and a spatial attention module. The attention weight coefficients obtained by the channel attention module and the spatial attention module are multiplied by the input feature map F to achieve adaptive feature refinement.

4. The method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration according to claim 1, characterized in that, The specific process of the CBAM module extracting and learning the feature information of the feature map in step 4 is as follows: Step 4.1.1: First, the feature map F is compressed in spatial dimension by max pooling and average pooling to form two weight vectors [C,1,1]. Then, the two weight vectors are input into the same multilayer perceptron and mapped to the weights of each channel. Then, the sigmoid function is used to activate the attention weights [C,1,1] of the output channel, and the weights are multiplied element-wise with the original features [C,H,W] to obtain the feature map F. 1 ; Step 4.1.2: First, extract the feature map F. 1 As input to the spatial attention module, the input is compressed along the channel dimension using max pooling and average pooling to obtain two feature maps. These two feature maps are then stacked and subjected to a convolutional operation to obtain two weight vectors of [1, H, W]. The two weight vectors of [1, H, W] are then multiplied element-wise with the original input feature [C, H, W] to obtain the feature map F. 2 .

5. The method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration according to claim 1, characterized in that, The specific process of obtaining features of a wider range of receptive vision using the detail restoration module in step 4 is as follows: Step 4.2.1: The local multi-scale feature extraction module first performs convolution operations in parallel using four types of convolution kernels: 3*3, 5*5, 7*7, and 9*9. Then, it performs dimensionality reduction operations on features of different scales through 1*1 convolution. Next, it uses deconvolution to restore the feature map to 200 pixels × 200 pixels, then aggregates it with the original features, and then performs a convolution operation. Step 4.2.2: Connect three smooth dilated convolutional modules with dilation rates of 1, 3 and 5 in parallel in the local multi-scale feature extraction module, and fuse their output features.

6. The method for desaturating ceramic artifact images based on dual attention mechanism and detail restoration according to claim 1, characterized in that, The total loss function used in steps 2 to 5 includes a pixel loss function, a feature loss function, and an adversarial loss function, wherein: The pixel loss function is expressed as: In the formula, where and represent the gradient operators in the x and y directions, respectively, and D represents the true specular-free image. The image representing the predicted image without highlights has α = 0.2 and β = 0.

3. The feature loss function is expressed as: In the formula, φ l In this context, 'l' represents the 'l' layer in VGG-19, and 'λ' represents the 'l' layer. l Represents the balancing weight; The adversarial loss function is expressed as: In the formula, σ represents the sigmoid activation function, and C is the non-transform discriminant function. This represents the discriminator network; therefore, the total loss function is expressed as: l=w1l pixel +w2l feat +w3l adv In the formula, w1 = 1, w2 = 0.1, and w3 = 0.01.

Images