A grayscale image dequantization method based on deep learning

Abstract

The present invention addresses the technical issues in existing image restoration models, such as the need for a large amount of redundant training, excessive consumption of computing resources, the incomplete feature weights contained in the trained model, and the three-channel RGB image, which interferes with the feature capture of some three-dimensional information in the graphics. A grayscale image dequantization method based on deep learning is proposed, which is a fast dequantization method that combines deep learning models with image-based three-dimensional reconstruction problems. The present invention combines a simplified super-resolution restoration model obtained by deep learning with three-dimensional reconstruction methods to efficiently restore the missing dimensional information of objects in the image through feature channels using deep learning methods.

Description

Technical Field

[0001] The present invention relates to the field of grayscale image dequantization methods, and in particular to a grayscale image dequantization method based on deep learning. Background Art

[0002] With the rapid commercial development of the Internet of Things and virtual reality technologies, there is a growing need to represent the physical world in three dimensions using computers. 3D reconstruction is an interdisciplinary field, involving advanced mathematics, computer vision, computer graphics, and other fields. Specifically, it involves using sensors such as cameras to capture real-life objects and scenes and processing them using computer vision techniques to produce a complete 3D model of the object. With the development of advanced scanning equipment, it is now possible to obtain high-precision data from the surfaces of 3D objects in a short period of time. Mainstream technologies include 3D point clouds, multi-view stereo geometry, mesh reconstruction and optimization, texture mapping, Markov random fields, and image inverse quantization. 3D reconstruction is currently a core technology in augmented reality (AR), mixed reality (MR), robotic navigation, and autonomous driving.

[0003] Image super-resolution reconstruction technology uses low-quality, low-resolution images to generate high-quality, high-resolution images, automatically completing missing details. Existing image restoration models, whether based on CNNs or transformers, often require extensive redundant training, consuming excessive computing resources. Furthermore, the network layers are relatively shallow, so the feature weights contained in the trained models are often incomplete. Furthermore, they are often three-channel RGB images, taking into account factors such as color space, which may interfere with the capture of some three-dimensional information in the image. Summary of the Invention

[0004] The present invention performs dequantization on grayscale images representing height maps, i.e., converting discrete integer values ​​into continuous floating-point values. Grayscale images are represented in a lossy compressed format (e.g., JPG). Therefore, the goal of the present invention is to use a deep neural network to restore the structure and details of grayscale images after quantization and lossy compression. To achieve this goal, it is necessary to construct a corresponding dataset, a deep neural network model, and a loss function based on geometric quantities.

[0005] A drawback of traditional deep learning training models, such as CNN, is that the convolution kernel size of CNN-based methods remains fixed, making them inflexible for establishing relationships between pixels. To expand the receptive field, the depth and complexity of the network topology are continuously increased to recover details, directly leading to increased computational effort. Another example is the traditional transformer model, where Super Resolution (SR) input images are large, and the computational effort of the Visual Transformer (ViT) is squared with the image size, making it inconvenient to use the ViT model. In methods such as SwinIR, the receptive field of features extracted using 1x1 convolution is very small, which in turn affects the subsequent calculation of Self Attention (SA). Furthermore, calculating SA within a small window limits the ability to model long-range features. SwinIR has many redundant and fragmented components, such as relative position encoding, masking mechanisms, LayerNorm, and several sub-branches.

[0006] Transformers have not only gradually come to dominate the field of computer vision but have also found widespread application in image restoration. However, the self-attention computations in Transformers are computationally expensive, and some operations may be redundant for super-resolution images. This limits the scope of self-attention (SA) calculations, thus impacting some aspects of super-resolution image restoration performance. For example, traditional methods propose calculating SA in channel space. However, this often sacrifices useful spatial information, such as texture and structure, which can be crucial for specific downstream tasks.

[0007] In order to solve the above technical problems, the technical solution of the present invention is to provide a grayscale image dequantization method based on deep learning, comprising the following steps:

[0008] Step 1: Obtain a height grayscale image that can reflect the height of the object to be repaired;

[0009] Step 2: Data preprocessing: removing the background outside the object to be repaired from the highly grayscale image, retaining the grayscale image of the object to be repaired, and obtaining a grayscale image;

[0010] Step 3: Construct an image restoration network model based on grouped multi-scale self-attention and accelerated self-attention calculation, and use a 3×3 convolution operation to extract shallow features in the grayscale image;

[0011] Step 4: extracting deep features from the grayscale image using a set of cascaded grouped self-attention calculation modules;

[0012] Step 5: Using the shallow features and the deep features, a height map of the object to be repaired is obtained through reconstruction.

[0013] Preferably, the step three includes the following steps:

[0014] Input the grayscale image obtained in step 2 into the image restoration network model, where the grayscale image is the LR image X l ∈R 3×H×W , where H and W are the height and width of the LR image respectively;

[0015] Use the single layer of 3×3 convolution operation to extract shallow features X s ∈R C×H×W :

[0016] X s =H SF (X l )

[0017] Where C is the number of channels of the intermediate features, H SF (·) represents shallow feature extraction.

[0018] Preferably, in step 4, the grouped self-attention calculation module is an efficient long-distance attention module, and step 4 includes the following steps:

[0019] The shallow feature X obtained in step 3 s Input to the deep feature extraction module H DF (·), the deep feature extraction module H DF (·) Includes several cascaded efficient long-range attention modules;

[0020] X d =H DF (X s )

[0021] where X d ∈R C×H×W is the output result.

[0022] Preferably, the step 4 includes the following steps:

[0023] Step 41: Local feature extraction: Input the shallow feature X extracted in step 3 s , first enters a translation convolution layer and then goes to a ReLU layer for activation, and finally passes through a translation convolution layer;

[0024] The translation convolution layer consists of a set of translation operations and a 1×1 convolution. First, the input features of the translation convolution layer are evenly divided into five groups. Four of the groups of features are moved along the spatial dimensions of up, down, left, and right, while the other group remains unchanged to obtain the extracted feature map.

[0025] Step 42: Group multi-scale self-attention calculation:

[0026] First, the extracted feature map is cyclically shifted along the diagonal direction to obtain the cyclically shifted feature map X N×C×H×W ;

[0027] The cyclically shifted feature map X N×C×H×W As input, the SA value is calculated:

[0028] The feature map X N×C×H×W Perform two 1×1 convolution batch normalizations, reshaping the result of one of them twice into and Multiplying them again gives (1) The other result is reshaped once to obtain (2) Then multiply (1) and (2) to get the result Finally Reshape again to get the final SA value output result Y N×C×H×W ; That is, the dependency relationship between images output by the model is finally obtained;

[0029] The calculation of the SA value is a cascade process. The result obtained by performing two convolution operations and one reshaping operation in one of the previous SA calculation processes is directly multiplied with the result of one convolution and reshaping of the current SA calculation result to obtain a new SA value. This allows the attention map calculated by the previous module to be directly reused by the subsequent modules to directly obtain the calculation result SA.

[0030] The feature map X obtained by the calculation of the cyclic shift N×C×H×W Based on the inter-image dependency, a reverse shift operation is performed along the diagonal to obtain the final result of the original feature map order.

[0031] Preferably, the step five comprises the following steps:

[0032] X d and X s As input, the height map of the object to be repaired is obtained after reconstruction:

[0033] X h =H RC (X s +X d )

[0034] Among them H RC It is a reconstruction module, which includes the following two operations: first, a 3×3 convolution operation, and then a pixel shuffling operation;

[0035] For simplicity, the loss function is calculated using a large number of N-channel real HR (high-resolution) images. The network parameters are then optimized by minimizing the pixel-level L1 loss and the normal vector error L2 of the model surface height value.

[0036]

[0037]

[0038] Among them, X in the L1 loss function h，i is the i-th channel feature of the high-definition grayscale image output by the reconstruction module, X t，i is the i-th channel feature of the original real label image; n(h o (u, v)) is the original normal vector, and n(Φ(u, v)) is the result predicted by the model;

[0039] The Adam optimizer is used for optimization, which can improve its performance in low-level vision tasks.

[0040] The present invention has the following characteristics and beneficial effects:

[0041] The three-dimensional reconstruction network adopted by the present invention can repair the main structure of most height images with clear and sharp edges in actual use. It is proved that in the image restoration and reconstruction task, the restoration model of the original high-quality image can be further extended to the restoration of three-dimensional details of the image. Compared with the traditional modeling method, the present invention adopts a three-dimensional reconstruction scheme for height image restoration based on deep learning, which is low in cost and highly applicable, and can be classified and reconstructed for different image types, which greatly improves the accuracy of restored details. In addition, we have optimized the complex structure of the image restoration network and reduced a lot of redundant calculations. Using the above technical solution, only the grayscale image of the target can be input, and the depth of the grayscale value on each pixel represents the height information of the point. The lighter the color, the larger the height value. Then, the simplified and modified SwinIR model is used to remove noise interference in the image and retain high-frequency image details to capture features, and then the image is completed and repaired into a continuous three-dimensional graphic.

[0042] This invention provides a method that combines a simplified super-resolution restoration model based on deep learning with 3D reconstruction. This method efficiently restores missing dimensional information of objects in images through feature channels using deep learning methods. For example, given one or more images of an object, a deep learning super-resolution model is used to extract scene and object height information. Based on this information, the object's virtual 3D structure is reconstructed and restored, and details are completed to present the object's complete shape and details. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the implementation examples of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the examples or the description of the prior art.

[0044] Figure 1 This is a comparison chart of the effects of converting a grayscale image directly into a three-dimensional model and converting it into a three-dimensional model after dequantization in an implementation example of the present invention.

[0045] Figure 2 This is a schematic diagram of the overall process of grayscale image dequantization proposed by the present invention.

[0046] Figure 3 Schematic diagram of the process of the image restoration module in the present invention.

[0047] Figure 4 This is a specific schematic diagram of the translation convolution operation of the image restoration module in the present invention.

[0048] Figure 5 This is a schematic diagram of the accelerated self-attention calculation process used in the present invention. Different features are divided into multiple scales for self-attention calculation, and finally aggregated, which greatly reduces the amount of calculation. DETAILED DESCRIPTION

[0049] This paper proposes an efficient long-range attention network model for 3D reconstruction of deep images. The SwinIR model used for image restoration is simplified and modified to simply establish a LR to HR image mapping by sequentially stacking local feature lifting operations and SA. Through shift-convolution and a self-attention computation module with shared attention, an efficient long-range attention network can be easily obtained for SR. The network consists of several accelerated self-attention computation modules, two 3×3 convolutional layers, and a pixel reassembly operation layer, with the modified self-attention computation module being the core of the model architecture.

[0050] The self-attention calculation module has two key parts, including a local feature extraction module and a grouped multi-scale self-attention calculation module. Both modules use the residual learning strategy.

[0051] 1. Local feature extraction:

[0052] Local feature extraction uses a combination of two shift-convolutions and ReLUs. Specifically, shift-convolution consists of four shift operations and a 1×1 convolution. The shift operation shifts the input features up, down, left, and right. The resulting five sets of features are then input and then subjected to a 1×1 convolution for dimensionality reduction and information aggregation. Without introducing additional learnable parameters or computational overhead, shift-convolution achieves the same receptive field as a 3×3 convolution and requires only the same computational effort as a 1×1 convolution.

[0053] 2. Grouped multi-dimensional attention calculation:

[0054] The calculation of attention in this invention is different from the conventional transformer model. Because for a C×H×W feature map, the computational complexity of the self-attention algorithm using an M×M non-overlapping window is 2M 2 HWC. The window size M determines the range of self-attention calculations. A larger M indeed captures more self-similar information. However, as M increases, the computational resource consumption increases exponentially. To more efficiently calculate long-range self-attention, the input features are first divided into K groups. Self-attention is then performed on different groups at different window sizes. Finally, a 1×1 convolution is used to aggregate information across different features.

[0055] 3.Accelerated self-attention(ASA)

[0056] When performing SA calculations again, an improved method, ASA, was adopted. That is, BN (batch normalization) was used instead of the widely used LN (layer normalization). One of the reasons is that the BN calculation can be incorporated into the convolution operation without incurring additional computational overhead. LN decomposes the SA calculation into many element-level operations, which is less efficient. BN not only stabilizes training but can also be incorporated into convolution for acceleration during the inference phase. In addition, SwinIR's SA is calculated in an embedded Gaussian space with three independent 1×1 convolutions. The ASA module merges the calculations in a symmetric Gaussian space, which can save the computational complexity of a 1×1 convolution, further optimizing the computational complexity and memory pressure of SA without sacrificing its super-resolution performance.

[0057] 4. Shared Attention

[0058] In addition, to further accelerate SA computation across the entire model, a shared attention mechanism is implemented between adjacent SA modules. The attention score calculated in the i-th SA module can be directly reused by the subsequent n SA modules of the same scale. This avoids 2N reconstructions and N 1×1 convolutions across the n subsequent SA modules. Testing has shown that using the shared attention mechanism only degrades SR performance when n is small (e.g., 1 or 2), but it saves significant computational resources during inference.

[0059] 5. Move the window

[0060] Because the SA calculation mechanism on grouped multi-scale windows still lacks information connection between local windows of the same scale, the shift window mechanism in the SwinIR model is further optimized on the overall network, and a simple and effective shift scheme is proposed for SR tasks.

[0061] First, the features are circularly shifted along the diagonal direction, and self-attention is calculated on the shifted features. Then, the calculated feature results are reversely circularly shifted. In the previous self-attention calculation module, the circular shift of half the window size will lead to a new feature map division and introduce connections between adjacent non-overlapping windows. Although some boundary damaged pixels will be moved to farther areas due to the circular shift, since these pixels only occupy a small part of the entire feature map in the super-resolution task, the impact on the final super-resolution performance is negligible. And thanks to the circular shift mechanism, we can remove the mask mechanism and relative position encoding applicable to SwinIR, making the network more concise and efficient.

[0062] The overall process of the present invention is shown in Figure 1 , including the following steps:

[0063] Step 1: Dataset acquisition: Get a high-resolution grayscale image of the object to be repaired.

[0064] Step 2: Data preprocessing. Because this solution primarily targets grayscale input images and captures height information based on color depth, the image must be pre-converted into a unified format. The background is completely black, retaining only the grayscale image of the object to be restored.

[0065] Step 3: Build an efficient image restoration network model based on grouped multi-scale self-attention and accelerated self-attention calculation

[0066] First, the pre-processed grayscale image is input to the image restoration network model. The pre-processed grayscale image is a degraded LR (low resolution) image X l ∈R 3×H×W , H and W are the height and width of the LR image respectively.

[0067] First, shallow feature extraction is performed, namely H SF (·), which consists of only a single layer of 3×3 convolution to extract X s ∈R C ×H×W :

[0068] X s =H SF (X l )

[0069] Where C is the number of channels of the intermediate features

[0070] Step 4: Then use the X obtained in step 3 s Input to the deep feature extraction module H DF (·), H DF (·) Consists of M cascaded efficient long-range attention modules.

[0071] X d =H DF (X s )

[0072] where X d ∈R C×H×W is the output result.

[0073] Step 5: Then convert the previously output result X d and X s As input, a high-resolution grayscale image is reconstructed:

[0074] X h =H RC (X s +X d )

[0075] Among them H RC It is a reconstruction module, which includes the following two operations: first, a 3×3 convolution operation, and then a pixel shuffling operation.

[0076] For simplicity, the loss function is calculated using a large number of N-channel real HR (high-resolution) images. The network parameters are then optimized by minimizing the pixel-level L1 loss and the normal vector error L2 of the model surface height value.

[0077]

[0078]

[0079] Among them, X in the L1 loss function h，i is the i-th channel feature of the high-definition grayscale image output by the reconstruction module, X t，iis the i-th channel feature of the original true label image. In the L2 loss function, n(h o (u, v)) is the original normal vector, and n(Φ(u, v)) is the result predicted by the model.

[0080] The Adam optimizer is used for optimization, which can improve its performance in low-level vision tasks.

[0081] Among them, the core deep feature extraction module in step 4 is composed of multiple cascaded self-attention calculation modules. Figure 2 , mainly includes the following two steps using a staggered learning strategy:

[0082] Step 41: Local feature extraction, input the feature map extracted in step 3, first enter a translation convolution layer and then to a simple ReLU layer activation, and finally pass through a translation convolution layer. Figure 3 This is the flow chart of the translational convolution in this invention. A translational convolution consists of a set of translation operations and a 1×1 convolution. First, the input features are evenly divided into five groups. Four of these groups are shifted along the up, down, left, and right spatial dimensions, leaving the last group unchanged. This allows the subsequent 1×1 convolution to utilize information from neighboring pixels. This results in an extracted feature map.

[0083] Step 42: Group multi-scale self-attention calculation.

[0084] Sub-step 421: In order to increase the information connection between different windows of the same scale, the present invention performs a cyclic shift operation on the extracted feature map along the diagonal direction before calculating the SA to obtain a shifted feature map.

[0085] Sub-step 422: SA calculation needs to be performed in the self-attention calculation module. In order to speed up the calculation, this invention adopts an optimized method for improvement. The feature map after the cyclic shift is input for SA calculation. A simple description is: the layer normalization widely used in the traditional transformer model is removed, and batch normalization is used to stabilize the training process. And the three independent layers of θ, Φ, g in the embedded Gaussian space are set to θ = Φ, saving one convolution calculation in each SA calculation. The specific method is: input X N×C×H×W Perform two 1×1 convolution batch normalizations respectively, and reshape the result of one of them twice into and Multiplying them again gives (1) The other result is reshaped once to obtain (2) Then multiply the results of (1) and (2) to get the result Finally, the result is reshaped again to obtain the final ASA output result YN×C×H×W . Finally, the dependency data of the image is obtained.

[0086] Sub-step 423: A shared attention mechanism is used between adjacent SA modules to further accelerate and reduce consumption. The specific implementation is: in the cascaded SA calculation module, the result obtained by performing two convolution and reshaping operations on one of the previous SA modules is directly multiplied with the result of one convolution and reshaping operation in this module, so that the attention map calculated by the previous module can be directly reused by the subsequent modules to directly obtain the calculation result SA. See the specific process for details. Figure 5 .

[0087] Sub-step 424: Finally, the feature map obtained by the circular shift calculation in sub-step 421 is shifted in the opposite direction along the diagonal to obtain the final result of the original feature map sequence.

[0088] The above is merely a basic example of the present invention, and is intended only to help understand the technical solution and core concept of the present invention. It should be noted that those skilled in the art may make various improvements and modifications to the present invention without departing from the principles of the present invention, and such improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A grayscale image dequantization method based on deep learning, characterized in that: The following steps are involved: Step 1: Obtain a high-resolution grayscale image that can reflect the height of the object to be repaired; Step 2: Data preprocessing; Removing the background outside the object to be repaired from the highly grayscale image, retaining the grayscale image of the object to be repaired, to obtain a grayscale image; Step 3: Construct an image restoration network model based on grouped multi-scale self-attention and accelerated self-attention calculation, and use a 3×3 convolution operation to extract shallow features in the grayscale image; Step 4: extracting deep features from the grayscale image using a set of cascaded grouped self-attention calculation modules; Step 5: Using the shallow features and the deep features, a height map corresponding to the object to be repaired is obtained through reconstruction; The step 4 includes the following steps: Step 41: Local feature extraction: Input the shallow features extracted in step 3 , first enters a translation convolution layer and then goes to a ReLU layer for activation, and finally passes through a translation convolution layer; The translation convolution layer consists of a set of translation operations and a 1×1 convolution. First, the input features of the translation convolution layer are evenly divided into five groups. Four of the groups of features are moved along the spatial dimensions of up, down, left, and right, while the other group remains unchanged to obtain the extracted feature map. Step 42: Group multi-scale self-attention calculation: First, the extracted feature map is cyclically shifted along the diagonal direction to obtain the cyclically shifted feature map. ; The cyclically shifted feature map As input, the SA value is calculated: The feature map Perform two 1×1 convolution batch normalizations, reshaping the result of one of them twice into and , and then multiply to get (1) , and the other result is reshaped to obtain (2) , then multiply (1) and (2) to get the result , and finally Reshape again to get the final SA value output ; That is, the dependency relationship between images output by the model is finally obtained; The calculation of the SA value is a cascade process. The result obtained by performing two convolution operations and one reshaping operation in one of the previous SA calculation processes is directly multiplied with the result of one convolution and reshaping of the current SA calculation result to obtain a new SA value. This allows the attention map calculated by the previous module to be directly reused by the subsequent modules to directly obtain the calculation result SA. The feature map obtained by calculating the cyclic shift Based on the inter-image dependency, a reverse shift operation is performed along the diagonal to obtain the final result of the original feature map order.

2. The grayscale image dequantization method based on deep learning according to claim 1, wherein: The step three comprises the following steps: Input the grayscale image obtained in step 2 into the image restoration network model, where the grayscale image is an LR image. , where H and W are the height and width of the LR image respectively; Use the 3×3 convolution operation of a single layer to extract shallow features : ； Where C is the number of channels of the intermediate features, Stands for shallow feature extraction.

3. The grayscale image dequantization method based on deep learning according to claim 2, wherein: In step 4, the grouped self-attention calculation module is an efficient long-distance attention module, and step 4 includes the following steps: The shallow features obtained in step 3 Input to the deep feature extraction module , the deep feature extraction module It includes several cascaded efficient long-distance attention modules; ； in is the output result.

4. A grayscale image dequantization method based on deep learning according to any one of claim 3, characterized in that: The step five comprises the following steps: Will and As input, the height map of the object to be repaired is obtained after reconstruction: ； in It is a reconstruction module, which includes the following two operations: first, a 3×3 convolution operation, and then a pixel shuffling operation; For simplicity, the loss function is calculated using a large number of N-channel real HR (high-resolution) images. The network parameters are then optimized by minimizing the pixel-level L1 loss and the normal vector error L2 of the model surface height value. ； ； Among them, the L1 loss function is the i-th channel feature of the high-definition grayscale image output by the reconstruction module, is the i-th channel feature of the original true label image; in the L2 loss function is the original normal vector, The result predicted by the model; The Adam optimizer is used for optimization, which can improve its performance in low-level vision tasks.

Images