Reversible halftone method and system based on dynamic noise

Abstract

The invention relates to the technical field of image hard copy copying, in particular to a reversible halftone method and system based on dynamic noise, and the method comprises the steps: constructing a reversible halftone model which comprises a halftone generation network and a recovery network; enabling the original color image to pass through a halftone generation network of a reversible halftone model, and outputting a reversible halftone image; and the reversible halftone image passes through a recovery network, and a recovered color image is output. According to the method, the quality of the halftone image is improved, the detail accuracy and the color rendition degree in the reversible halftone image are improved, the recovered color image is ensured to reflect all details and color ranges of the original color image as much as possible, and the image processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image hard copy reproduction, and in particular to a reversible half-tone method and system based on dynamic noise. Background Art

[0002] Halftoning technology is the process of converting a continuous tone image into an image with only a limited number of colors (usually black and white) while maintaining the visual grayscale effect. This is especially important in monochrome printing and display devices. Halftoning technology simulates different grayscale levels by adjusting the size and distribution of black dots, so that rich grayscale and details can be displayed in black and white printing. It is widely used in newspaper, magazine, book printing and various artistic creations.

[0003] Existing halftoning technologies are mainly divided into traditional methods and deep learning-based methods.

[0004] Traditional halftone algorithms mainly include ordered dithering algorithms, error diffusion algorithms, and search-based algorithms. The ordered dithering-based method periodically divides the continuous tone image into many small blocks, compares each small block with a pre-set dithering array, and generates the final halftone image. This type of algorithm has high parallelism and the fastest processing speed, but the generated halftone image quality is poor. The error diffusion algorithm maintains the principle of local tone unchanged. In each step of processing, only one pixel is quantized, and the quantization error generated is diffused to adjacent unprocessed pixels through different weights. Although the error diffusion algorithm will produce unpleasant visual artifacts, it maintains a good balance between quality and efficiency, so it is widely used. The search-based method regards halftone as an optimization problem. By designing a halftone evaluation index and then using greedy or simulated annealing algorithms for optimization, a good halftone effect can be achieved, but its search time is usually unacceptable on real-time devices.

[0005] Deep learning algorithms based on neural networks are a new development direction for halftone technology. Choi et al. used two deep generative models to simulate the Direct Binary Search algorithm (DBS). The generated halftone images have less noise than those generated by the DBS algorithm and do not contain artifacts related to the error diffusion algorithm, but they are very time-consuming and cannot be processed in real time. Jiang et al. regarded halftone as a reinforcement learning problem, cleverly solved the problem that the gradient cannot be back-propagated after the threshold operation, and introduced a new suppression loss function to achieve lightweight CNN to generate blue noise halftones that meet visual quality. However, the above methods did not take into account the reversibility of halftones when designing. This problem needs to be solved by inverse halftone technology.

[0006] Inverse halftoning is a technique that converts a halftone image back to its original or nearly original continuous tone image. This process is often used in image processing, editing, and analysis, especially when it is necessary to recover higher quality image data from printed materials or processed images. However, it is challenging to recover a continuous tone image with accurate details from a halftone image that has lost information.

[0007] The most direct inverse halftone method is to use a low-pass filtering method to remove noise in the high-frequency area of ​​the image. In order to improve the blurring of image edges and details caused by low-pass filtering, researchers have proposed many enhanced fast filtering algorithms. For example, Xiong et al. used wavelet transform to separate background and edges and perform noise reduction in the high-frequency area of ​​inverse halftone. Subsequently, Ste regarded halftone restoration as an estimation of each pixel and used the maximum a posteriori probability (MAP) to accurately reconstruct the smooth area of ​​the image. Mese and Guo introduced an inverse halftone method using a lookup table (LUT) with prior information. In order to reduce complexity and improve efficiency, the researchers optimized the LUT in subsequent studies. Son et al. proposed a method based on local learning dictionary pairs to provide better detail and contour representation through feature clustering. In addition, deep convolutional neural networks have become the main method for solving various image reconstruction problems and have achieved state-of-the-art performance on different datasets. Hou et al. trained CNN as a nonlinear transformation function and developed the first deep learning-based inverse halftone and compression solution. Kim et al. proposed an automatic de-screening method for halftone images combining contextual hierarchical information. Xia et al. combined global enhancement and detail enhancement through a progressive residual network to enhance the restoration effect. Li et al. further optimized the recovery effect and performance through multi-scale input. However, halftone images have lost a lot of information during their creation process. Even the most advanced inverse halftone methods can only restore an approximate grayscale version. Restoring color from gray halftone images is still a difficult problem.

[0008] Reversible halftone technology is a new concept proposed by researchers in recent years. This technology embeds implicit information to generate a reversible halftone image with the ability to restore to the original version, so that the original continuous tone image can be accurately restored from the reversible halftone image, ensuring the quality of the image. This is particularly important for applications that require high-quality image reconstruction, such as archive protection, high-end printing, and professional photography. Xia et al. proposed a reversible deep halftone method. This method introduces dynamic Gaussian noise to enable the CNN network to generate non-periodic reversible halftone patterns. Although the reversible halftone image generated by their model has good restorability, it does not achieve strong blue noise characteristics, and the proposed two-stage training scheme is complex and unstable. Based on Xia's method, Lau et al. introduced an inverse halftone network as a brightness predictor, so that the model pays more attention to the color restoration ability of the reversible halftone, and generated a reversible halftone that can be fully restored. However, the differentiation of the restoration task leads to the loss of the blue noise characteristics of the reversible halftone image. At the same time, a large number of parameters are piled up, which leads to high computational complexity and is difficult to apply to industrial equipment.

[0009] In summary, the quality of the halftone image generated by the existing reversible halftoning technology and its restoration accuracy are low, and the high computational complexity leads to reduced image processing efficiency. Summary of the invention

[0010] Therefore, the technical problem to be solved by the present invention is to overcome the problems in the prior art that the quality of the halftone image generated and its restoration accuracy are low, and the high computational complexity leads to reduced image processing efficiency.

[0011] In order to solve the above technical problems, the present invention provides a reversible halftoning method based on dynamic noise, comprising:

[0012] Construct a reversible halftone model, including a halftone generation network and a restoration network;

[0013] The halftone generation network includes an input module, a local feature extraction module, two downsampling convolution modules, a high-level feature extraction module, two upsampling convolution modules and an output module connected in sequence;

[0014] The local feature extraction module and the advanced feature extraction module both include a number of feature extraction residual blocks; each feature extraction residual block includes a dynamic noise module, a ReLU activation function and a convolution layer connected in sequence, and the input features of the feature extraction residual block and the output features of the ReLU activation function are added through a jump connection as the output features of the feature extraction residual block; the dynamic noise module generates random noise and adaptively adjusts the intensity of the random noise to obtain adaptive random noise; the adaptive random noise is spliced ​​with the input features of the dynamic noise module and then subjected to a single-layer convolution to obtain the output features of the dynamic noise module;

[0015] The restoration network includes several lightweight residual blocks and flat convolutional layers;

[0016] The original color image is passed through the halftone generation network of the reversible halftone model to output a reversible halftone image; the reversible halftone image is then passed through the restoration network to output a restored color image.

[0017] Preferably, the local feature extraction module includes 2 feature extraction residual blocks; and the high-level feature extraction module includes 4 feature extraction residual blocks.

[0018] Preferably, the formula of the dynamic noise module is expressed as f(I O +N), where I O represents the input features of the dynamic noise module, N represents adaptive random noise, and f represents a single layer of convolution;

[0019] The dynamic noise module adaptively adjusts the intensity of random noise. The formula is:

[0020]

[0021] Among them, α(t) is the noise intensity at time step t, α0 is the initial noise intensity, is the hue loss value at time step t, is the initial hue loss value and γ is a hyperparameter.

[0022] Preferably, the output module comprises a standard residual block and a Gumbel-Softmax layer.

[0023] Preferably, the restoration network includes 7 lightweight residual blocks, each lightweight residual block includes two convolutional layers and a ReLU activation function, and the input features of the lightweight residual block and the output features of the ReLU activation function are added through a jump connection as the output features of the lightweight residual block.

[0024] Preferably, the original color image is passed through a halftone generation network of a reversible halftone model to output a reversible halftone image; the reversible halftone image is then passed through a restoration network to output a restored color image, including:

[0025] The original color image is input into the input module of the halftone generation network to obtain input features; the input features are passed through the local feature extraction module to obtain initial features; the initial features are passed through two downsampling convolution modules to obtain downsampling features; the downsampling features are passed through the high-level feature extraction module to further extract image semantic information and compensate for noise to obtain high-level features; the high-level features are passed through two upsampling convolution modules to obtain upsampling features; the upsampling features are passed through the output module to obtain a reversible halftone image;

[0026] The reversible halftone image is input into the restoration network and passed through several lightweight residual blocks to obtain the restoration features; the restoration features are passed through a flat convolution layer to output a restored color image.

[0027] Preferably, when training the reversible halftone model, the total loss function includes a reversible halftone loss and a restoration loss;

[0028] The reversible halftone loss includes content loss, tone consistency loss and spectrum loss;

[0029] The reversible halftone image is subjected to a pre-trained inverse halftone model to obtain a restored grayscale image; the restored grayscale image and the grayscale image of the original color image are input into the convolutional network VGG-19, and the content loss is constructed based on the feature maps extracted from each layer of the convolutional network VGG-19 based on the restored grayscale image and the grayscale image of the original color image;

[0030] The tone consistency loss is constructed based on the mean square error calculated after the reversible halftone image and the original color image are passed through a Gaussian filter;

[0031] Construct spectrum loss based on spectrum of reversible halftone image and ideal spectrum;

[0032] The restoration loss includes pixel L1 loss and spatial frequency L1 loss;

[0033] The pixel L1 loss and spatial frequency L1 loss are constructed based on the pixel and spectrum differences between the restored color image and the original color image, respectively.

[0034] Preferably, in the reversible halftone loss, the formula for the content loss is expressed as:

[0035]

[0036] Among them, i, j represents the spatial position of the feature map, k represents the channel index of the feature map, and F content Represents the grayscale image of the original color image, F generated Represents the restored grayscale image obtained by the pre-trained inverse halftone model of the reversible halftone image. and Respectively represent F content and F generated Feature maps extracted from the Lth layer of the convolutional network VGG-19;

[0037] The formula for the hue consistency loss is expressed as:

[0038] Among them, O h Represents a reversible halftone image, I crepresents the original color image, G{·} represents the Gaussian filter with a convolution kernel size of 11×11, represents the average operator of all input images in the training set;

[0039] The spectrum loss is expressed as:

[0040]

[0041] Among them, FFT generated Represents the spectrum of a reversible halftone image, FFT idea represents the ideal spectrum, and f represents a halftone image with constant grayscale.

[0042] Preferably, in the recovery loss, the formula of the pixel L1 loss is expressed as:

[0043]

[0044] Among them, O c Represents the restored color image, I c represents the original color image, n represents the total number of image pixels, and u represents the index of the pixel;

[0045] The formula for the spatial frequency L1 loss is expressed as:

[0046]

[0047] Here, FFT(·) represents fast Fourier transform.

[0048] The present invention also provides a reversible halftone system based on dynamic noise, comprising:

[0049] A model building module, used to build a reversible halftone model, including a halftone generation network and a restoration network;

[0050] The halftone generation network includes an input module, a local feature extraction module, two downsampling convolution modules, a high-level feature extraction module, two upsampling convolution modules and an output module connected in sequence;

[0051] The local feature extraction module and the advanced feature extraction module both include a number of feature extraction residual blocks; each feature extraction residual block includes a dynamic noise module, a ReLU activation function and a convolution layer connected in sequence, and the input features of the feature extraction residual block and the output features of the ReLU activation function are added through a jump connection as the output features of the feature extraction residual block; the dynamic noise module generates random noise and adaptively adjusts the intensity of the random noise to obtain adaptive random noise; the adaptive random noise is spliced ​​with the input features of the dynamic noise module and then subjected to a single-layer convolution to obtain the output features of the dynamic noise module;

[0052] The restoration network includes several lightweight residual blocks and flat convolutional layers;

[0053] The reversible halftone module is used to pass the original color image through the halftone generation network of the reversible halftone model to output a reversible halftone image; the reversible halftone image is then passed through the restoration network to output a restored color image.

[0054] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0055] The reversible halftone method based on dynamic noise described in the present invention adopts a U-shaped network structure in the halftone generation network, the downsampling convolution module gradually reduces the image spatial resolution while extracting global information, and the upsampling module restores the image spatial resolution while reconstructing the detail information of the original color image, helping the model to optimize feature expression from both global and local levels. The feature extraction residual block used in the local feature extraction module and the advanced feature extraction module is based on a lightweight design, which solves the problem of insufficient network depth or excessive computational overhead in traditional methods, and reduces the number of parameters and computational complexity of the model while retaining sufficient expression capabilities. In addition, the feature extraction residual block uses a dynamic noise module to introduce noise of different intensities, and by dynamically adjusting the noise intensity, the reversible halftone image is continuously perturbed during the generation process, avoiding the risk of the network falling into a local flat solution, ensuring that the reversible halftone image output by the halftone generation network has better smoothness, and improving the quality and restoration accuracy of the reversible halftone image.

[0056] Furthermore, the present invention introduces spectrum loss in the inverse halftone loss, introduces spatial frequency L1 loss in the restoration loss, maps the image from the spatial domain to the frequency domain through Fourier transform, pays more attention to the distribution of high-frequency components, provides global feature constraints, and makes up for the local defects of pixel-level loss, so that the generated reversible halftone image and restored color image can better retain the details in the original color image. In addition, the frequency domain loss and the pixel-level loss complement each other. The present invention optimizes the generation results in both the spatial domain and the frequency domain, further improving the restoration performance of the reversible halftone image.

[0057] Therefore, the reversible halftone image generated by the present invention is visually closer to the input image, the quality of the halftone image is improved, the detail accuracy and color reproduction in the reversible halftone image are improved, and the restored color image is guaranteed to reflect all the details and color range of the original color image as much as possible. In addition, the present invention reduces the number of parameters in the reversible halftone model to speed up the image processing speed, reduces the computing resources consumed in the image processing process, improves the image processing efficiency, and makes the reversible halftone technology suitable for real-time processing and large-scale image data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0059] Figure 1 is a structural diagram of the reversible halftone model of the present invention;

[0060] Figure 2 This is the structural diagram of the dynamic noise module;

[0061] Figure 3 This is a visual comparison of the flatness of the CNN network halftone, where Figure 3 (a) is the input image. Figure 3 (b) and (c) are the intermediate feature map and output halftone map of the traditional CNN, respectively. Figure 3 (d) and (e) are the intermediate feature map and output halftone map of CNN with NIB added, respectively. Figure 3 (f) and (g) are the intermediate feature map and output halftone map of CNN with dynamic noise module added, respectively;

[0062] Figure 4 is the visual quality comparison result of the halftone image, where Figure 4 (a) is the input continuous grayscale image. Figure 4 (b) in the figure is the halftone image output by the Ostromouhkov method. Figure 4 (c) in the figure is the halftone image output by the TRH method. Figure 4 (d) in the figure is the halftone image output by the RVH method. Figure 4 (e) is a halftone image output by the method of the present invention;

[0063] Figure 5 is the visual comparison map of the restored color image, where Figure 5 (a) is the original color image and its details. Figure 5 (b) in the figure is the reversible halftone image and its local magnification, the restored color image and its local magnification output by the RVH method. Figure 5 (c) is a reversible halftone image and a partial enlarged image thereof, a restored color image and a partial enlarged image thereof output by the method of the present invention, Figure 5 (d) in the figure is the restored color image output by the MPRL+IaCol method;

[0064] Figure 6 This is a comparison of the reversible halftone image using NIB and using the dynamic noise module. Figure 6 (a) is the input grayscale image. Figure 6 (b) is a reversible halftone image obtained using NIB. Figure 6(c) is a reversible halftone image obtained using a dynamic noise module; Figure 6 (d) is a partial enlarged view of (a); Figure 6 (e) is a partial enlarged view of (b); Figure 6 (f) is a partial enlarged view of (c). DETAILED DESCRIPTION

[0065] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0066] Embodiment 1

[0067] Reference Figure 1 As shown, the present invention provides a reversible halftone method based on dynamic noise, which generates a reversible halftone image and restores a color image by constructing a reversible halftone model.

[0068] The reversible halftone model includes a halftone generation network and a restoration network.

[0069] The halftone generation network adopts a U-shaped network structure, including an input module, a local feature extraction module, two downsampling convolution modules, a high-level feature extraction module, two upsampling convolution modules and an output module connected in sequence.

[0070] The input module includes a convolution layer and a ReLU activation function, which is used to receive the input original color image and extract the initial features. The convolution kernel size of the convolution layer is 3×3, the step size is 1, and the padding is 1.

[0071] The local feature extraction module includes two feature extraction residual blocks. Each feature extraction residual block includes a dynamic noise module, a ReLU activation function and a convolution layer connected in sequence, and the input features of the feature extraction residual block and the output features of the ReLU activation function are added through a jump connection as the output features of the feature extraction residual block.

[0072] A typical convolutional neural network (CNN) consists of convolutional layers (with bias terms), pooling layers, activation functions, and fully connected layers. When the input is flat, it cannot introduce spatial variations in the output. This is due to the paradigm of spatially sharing convolution kernels in the convolution operation, which Xia et al. call "flatness degradation". To address this problem, they proposed a noise excitation block (NIB) that can be used in CNNs as a network-independent plug-in. Although NIB can circumvent the problem of flatness degradation, the more pressing flaw is the lack of noise perturbation, which usually leads to overly regular textures guided by error diffusion. When noise is added only at the input stage, this perturbation will gradually weaken in the network's convolution operation.

[0073] Existing technologies such as NIB use fixed-intensity noise, which uses the same noise distribution and intensity for all inputs throughout the training and generation process, and cannot adaptively adjust the noise level according to the characteristics of the input data or feature layer. When some layers require stronger noise to break the flatness problem, fixed noise may not be enough. When some layer features are fragile, fixed-intensity noise may destroy important features.

[0074] Therefore, the present invention uses a dynamic noise module to introduce noise of different intensities in each feature extraction residual block, and dynamically adjusts the noise intensity so that the reversible halftone image is continuously perturbed during the generation process. The key idea of ​​the dynamic noise module is to introduce spatially varying feature representations without polluting the integrity of the original input information. In addition, the randomness of the noise is beneficial for jitter processing because it focuses on the overall pattern distribution rather than individual pixel values, thereby providing sufficient jitter dependency for the network.

[0075] The structure of the dynamic noise module refers to Figure 2 The dynamic noise module generates random noise and adaptively adjusts the intensity of the random noise to obtain adaptive random noise; the adaptive random noise is concatenated with the input features of the dynamic noise module and then subjected to a single-layer convolution to obtain the output features of the dynamic noise module, which is expressed as f(I O +N), where I O represents the input features of the dynamic noise module, N represents adaptive random noise, and f represents a single layer of convolution.

[0076] Specifically, the dynamic noise module adaptively adjusts the intensity of random noise, and the formula is:

[0077]

[0078] Among them, α(t) is the noise intensity at time step t, α0 is the initial noise intensity, is the hue loss value at time step t, is the initial hue loss value and γ is a hyperparameter that controls the noise intensity response.

[0079] The adaptive design of noise diversity in the dynamic noise module avoids the risk of the network falling into a local flat solution, solves the limitations of the fixed noise design, ensures that the reversible halftone image output by the halftone generation network has better smoothness, and improves the quality and restoration accuracy of the reversible halftone image. Figure 3 This is a visual comparison of the flatness of the CNN network halftone, where Figure 3 (a) is the input image. Figure 3 (b) and (c) in the figure are the intermediate feature map and output halftone map of the traditional CNN, respectively. Figure 3(d) and (e) are the intermediate feature map and output halftone map of CNN with NIB added, respectively. Figure 3 (f) and (g) are the intermediate feature map and output halftone map of CNN with dynamic noise module added, respectively.

[0080] The downsampling convolution module includes two convolutional layers connected in sequence and a ReLU activation function, which is used to gradually reduce the resolution of the feature map while increasing the number of channels to extract more abstract features, usually containing global information. The downsampled feature map contains high-level semantic information of the image, but its local details may be lost due to the reduction in spatial resolution.

[0081] Preferably, the convolution kernel size of each convolution layer in the downsampling convolution module is 3×3, the stride is 2, and the padding is 1.

[0082] The high-level feature extraction module includes four feature extraction residual blocks, which are used to further extract high-level features of the image with semantic information and compensate for noise.

[0083] The upsampling convolution module includes an upsampling layer, two convolution layers and a ReLU activation function connected in sequence, which is used to restore the feature map to the input size by gradually restoring the spatial resolution and help reconstruct the image details. In this process, the upsampling convolution module will fuse the information from the corresponding layer of the downsampling convolution module to obtain local detail information.

[0084] Preferably, the upsampling layer uses bilinear interpolation or deconvolution to upsample the feature map. The convolution kernel size of each convolution layer in the upsampling convolution module is 3×3, the step size is 1, and the padding is 1.

[0085] The output module includes a standard residual block and a Gumbel-Softmax layer. The Gumbel-Softmax layer is used to quantize the output of the halftone generation network to 0 or 1 to obtain a reversible halftone image.

[0086] The standard residual block includes two structures consisting of a convolutional layer and a ReLU activation function and a skip connection.

[0087] The present invention utilizes the Gumbel-softmax re-parameterization strategy to avoid interference with model gradient propagation at the output of the halftone generation network, realizes direct quantization of the network output to 0 or 1, and enhances the correspondence between the halftone generation network and the grayscale value of the real image.

[0088] The halftone generation network solves the problem of insufficient multi-scale information extraction of the input original color image through the downsampling convolution module and the upsampling convolution module, and helps the model optimize feature expression from both global and local levels. At the same time, when extracting features, the feature extraction residual block used by the halftone generation network is based on a lightweight design, which solves the problem of insufficient network depth or excessive computational overhead in traditional methods, while retaining sufficient expression capabilities. The number of parameters and computational complexity of the model are reduced. In addition, the halftone generation network constructed by the present invention introduces noise of different intensities in each feature extraction residual block, which solves the problem of network flatness degradation, so that the network is continuously disturbed during the learning process, thereby enhancing the recovery ability of the generated reversible halftone image.

[0089] The restoration network consists of 7 lightweight residual blocks and flat convolutional layers.

[0090] Each lightweight residual block is used to extract reversible halftone features and reconstruct grayscale information. It includes two convolutional layers and a ReLU activation function, and the input features of the lightweight residual block and the output features of the ReLU activation function are added through a jump connection as the output features of the lightweight residual block.

[0091] The flat convolution layer is used to generate the restored color image, which is a single convolution layer with a convolution kernel size of 1×1, a stride of 1, and padding of 0.

[0092] The original color image is passed through the halftone generation network of the reversible halftone model to output a reversible halftone image; the reversible halftone image is then passed through the restoration network to output a restored color image, including:

[0093] The original color image is input into the input module of the halftone generation network to obtain input features; the input features are passed through the local feature extraction module to obtain initial features; the initial features are passed through two downsampling convolution modules to obtain downsampling features; the downsampling features are passed through the high-level feature extraction module to further extract image semantic information and compensate for noise to obtain high-level features; the high-level features are passed through two upsampling convolution modules to obtain upsampling features; the upsampling features are passed through the output module to obtain a reversible halftone image;

[0094] The reversible halftone image is input into the restoration network and passed through several lightweight residual blocks to obtain the restoration features; the restoration features are passed through a flat convolution layer to output a restored color image.

[0095] The total loss function during the training of the reversible halftone model includes the reversible halftone loss and the restoration loss, and the formula is:

[0096] L=L h +L r

[0097] Among them, L represents the total loss function, L h Represents the reversible halftone loss, L r Indicates recovery of losses.

[0098] The reversible halftone loss includes content loss, tone consistency loss and spectrum loss, and the formula is:

[0099] L h =ω1L content +ω2L tone +ω3L freq

[0100] Among them, L content Indicates content loss, L tone represents the hue consistency loss, L freq represents spectrum loss; ω1, ω2 and v3 are the loss weights of content loss, tone consistency loss and spectrum loss respectively. In this embodiment, the values ​​are ω1=1, ω2=0.4, and v3=0.1.

[0101] The content loss is used to measure the similarity between the generated halftone image and the content image. Specifically, the present invention evaluates the content loss by calculating the feature difference between the generated reversible halftone image and the content image at some intermediate layers (usually convolutional layers), that is, using a pre-trained inverse halftone model to restore the grayscale image from the reversible halftone image, and calculating the content error between the grayscale image and the original color image.

[0102] In this embodiment, the reversible halftone image is subjected to a pre-trained inverse halftone model to obtain a restored grayscale image; the restored grayscale image and the grayscale image of the original color image are input into the convolutional network VGG-19, and the content loss is constructed based on the feature maps extracted from each layer of the convolutional network VGG-19 based on the restored grayscale image and the grayscale image of the original color image. The formula is expressed as follows:

[0103]

[0104] Among them, i, j represents the spatial position of the feature map, k represents the channel index of the feature map, and F content Represents the grayscale image of the original color image, F generated Represents the restored grayscale image obtained by the pre-trained inverse halftone model of the reversible halftone image. and Respectively represent F content and F generated Feature maps extracted from layer L of the convolutional network VGG-19.

[0105] Preferably, the pre-trained inverse halftoning model adopts a model in Deep inverse halftoning via progressively residual learning.

[0106] The tone consistency loss is used to ensure that the generated reversible halftone image remains visually similar to the input original color image. The original color image I c and reversible halftone image O h The mean square error after Gaussian filtering is as follows:

[0107]

[0108] Among them, O h Represents a reversible halftone image, I c represents the original color image, G(·) represents the Gaussian filter with a convolution kernel size of 11×11, represents the average operator over all input images in the training set.

[0109] Most existing methods rely on pixel-level losses, such as L1 or L2 losses, which are difficult to effectively capture the global frequency domain features of the image, and the difference in visual quality between halftone images and grayscale images is often reflected in the mismatch of the distribution of high-frequency and low-frequency components. The reversible halftone images generated by existing methods may have abrupt high-frequency noise in the transition area, affecting the visual effect. Since the variance is used to measure the degree of data discreteness, the distribution characteristics and fluctuation degree of halftone points can be explicitly characterized. Based on these analyses, the present invention designs a spectral loss to constrain the consistency of the generated halftone image with the target image in the frequency domain, especially focusing on the high-frequency part (the distribution of edges and details). The spectral loss uses the mean square error (MSE) between the spectrum corresponding to the reversible halftone image generated by the model and the ideal spectrum as a loss function to accurately evaluate the model's ability to maintain the blue noise characteristics of the image, and to improve the smoothness and visual perception quality of the halftone image by reducing unnecessary high-frequency oscillations in the frequency domain.

[0110] Specifically, the spectrum loss is constructed based on the spectrum of the reversible halftone image and the ideal spectrum, and the formula is:

[0111]

[0112] Among them, FFT generated Represents the spectrum of a reversible halftone image, FFT idea represents the ideal spectrum, and f represents a halftone image with constant grayscale.

[0113] Since spectral analysis is only applicable to halftone images with constant grayscale, the present invention optimizes the loss function through additional small batches of constant grayscale images when training the model.

[0114] Spectral loss maps the image from the spatial domain to the frequency domain through Fourier transform, which can provide global feature constraints and make up for the local defects of pixel-level loss. It has better constraints on the overall distribution of low-frequency and smooth areas to improve the overall consistency of the generated reversible halftone image. Spectral loss focuses on the distribution of high-frequency components, so that the generated reversible halftone image and restored color image can better preserve or reproduce details.

[0115] The spectral constraint in the reversible halftone loss effectively suppresses the pseudo high-frequency noise in the halftone image and improves the smoothness of the generated image. Secondly, human vision is more sensitive to frequency domain features, especially the mid-frequency and high-frequency parts. The introduction of spectral loss can better optimize the perceptual quality of the generated image. In addition, spectral loss can complement pixel-level losses such as L1 / L2, optimize the generated results in both the spatial domain and the frequency domain, and improve the restoration performance of reversible halftone images and grayscale images.

[0116] The restoration loss includes pixel L1 loss and spatial frequency L1 loss, and the formula is:

[0117] L r =L pixel +L FFT

[0118] Among them, L pixel represents pixel L1 loss, L FFT Indicates spatial frequency L1 loss.

[0119] The pixel L1 loss is constructed based on the difference between the restored color image and the original color image in pixels. The formula is:

[0120]

[0121] Among them, O c Represents the restored color image, I c Represents the original color image, n represents the total number of image pixels, and u represents the index of the pixel.

[0122] The spatial frequency L1 loss is constructed based on the difference in spectrum between the restored color image and the original color image. The formula is:

[0123]

[0124] Wherein, FFT(·) represents fast Fourier transform, which is used to convert image signals from the spatial domain to the frequency domain.

[0125] Since grayscale image restoration requires reconstruction of image details (high-frequency information) and global smoothness (low-frequency information), and the traditional restoration loss function's constraints on local pixels may not fully describe the differences in frequency components, the restoration loss used in this invention constrains the restored grayscale image to be close to the original image in the frequency domain, ensuring that both high-frequency details and low-frequency smooth parts can be accurately restored. This enhances the ability to restore textures and details and avoids over-smoothing.

[0126] In summary, the reversible halftone method based on dynamic noise described in the present invention adopts a U-shaped network structure in the halftone generation network, the downsampling convolution module gradually reduces the image spatial resolution while extracting global information, and the upsampling module restores the image spatial resolution while reconstructing the detail information of the original color image, helping the model to optimize feature expression from both global and local levels. The feature extraction residual block used in the local feature extraction module and the advanced feature extraction module is based on a lightweight design, which solves the problem of insufficient network depth or excessive computational overhead in traditional methods, and reduces the number of parameters and computational complexity of the model while retaining sufficient expression capabilities. In addition, the feature extraction residual block uses a dynamic noise module to introduce noise of different intensities, and by dynamically adjusting the noise intensity, the reversible halftone image is continuously perturbed during the generation process, avoiding the risk of the network falling into a local flat solution, ensuring that the reversible halftone image output by the halftone generation network has better smoothness, and improving the quality and restoration accuracy of the reversible halftone image.

[0127] Furthermore, the present invention introduces spectrum loss in the inverse halftone loss, introduces spatial frequency L1 loss in the restoration loss, maps the image from the spatial domain to the frequency domain through Fourier transform, pays more attention to the distribution of high-frequency components, provides global feature constraints, and makes up for the local defects of pixel-level loss, so that the generated reversible halftone image and restored color image can better retain the details in the original color image. In addition, the frequency domain loss and the pixel-level loss complement each other. The present invention optimizes the generation results in both the spatial domain and the frequency domain, further improving the restoration performance of the reversible halftone image.

[0128] Therefore, the reversible halftone image generated by the present invention is visually closer to the input image, the quality of the halftone image is improved, the detail accuracy and color reproduction in the reversible halftone image are improved, and the restored color image is guaranteed to reflect all the details and color range of the original color image as much as possible. In addition, the present invention reduces the number of parameters in the reversible halftone model to speed up the image processing speed, reduces the computing resources consumed in the image processing process, improves the image processing efficiency, and makes the reversible halftone technology suitable for real-time processing and large-scale image data processing.

[0129] Embodiment 2

[0130] This embodiment verifies the effectiveness of the reversible halftoning method based on dynamic noise described in the first embodiment through experiments.

[0131] In terms of halftones, the visual quality comparison results of halftone images refer to Figure 4 As shown, Figure 4 (a) is the input continuous grayscale image. Figure 4 (b) in the figure is the halftone image output by the Ostromouhkov method. Figure 4 (c) in the figure is the halftone image output by the TRH method. Figure 4 (d) in the figure is the halftone image output by the RVH method. Figure 4 (e) in FIG. 1 is a halftone image output by the method of the present invention. Figure 4 It can be seen that the snail part shows the advantages of the method of the present invention in the halftone details. The reversible halftone proposed by the present invention retains the recognizable structure of the original image. However, the classic traditional methods completely destroy the vein network because these methods do not take into account the structural details of the image. Even DBS only continuously optimizes the halftone image pixel by pixel through the HVS model. Compared with RVH, the visual effect of the method of the present invention is almost the same, and the method of the present invention has a more efficient running time.

[0132] The results of the quantitative evaluation of the halftone algorithm are shown in Table 1. It can be seen that the method of the present invention achieves the best performance in SSIM and is also competitive in PSNR.

[0133] Table 1. Quantitative evaluation results of halftone algorithm

[0134]

[0135] In terms of the inverse halftone restoration effect, in order to evaluate the ability of the method of the present invention in reconstructing continuous tones and fine details, the currently optimal inverse halftone method MPRL was selected as the restoration of the grayscale image and combined with the currently most advanced automatic colorization method IA-COL to compare the restored color image output by the present invention. In addition, this embodiment also compares the method of the present invention with the RVH method to confirm that the method of the present invention is superior to the RVH in restoration ability.

[0136] The results of the quantitative evaluation of the inverse halftone algorithm are shown in Table 2. This comparison may not be appropriate because the input of the inverse halftone model is a halftone image obtained by error diffusion, while the network input of the present invention is a reversible halftone image with embedded information. However, by comparing with these advanced methods, the effect of the present invention on image restoration can be reflected to a certain extent. As can be seen from Table 2, the present invention method exceeds RVH in terms of PSNR and SSIM.

[0137] Table 2. Quantitative evaluation results of inverse halftoning algorithms

[0138]

[0139]

[0140] Figure 5 To restore the visual contrast map of the color image, Figure 5 (a) is the original color image and its details. Figure 5 (b) in the figure is the reversible halftone image and its local magnification, the restored color image and its local magnification output by the RVH method. Figure 5 (c) is a reversible halftone image and a partial enlarged image thereof, a restored color image and a partial enlarged image thereof output by the method of the present invention, Figure 5 (d) in the figure is the restored color image output by the MPRL+IaCol method. It can be seen that the method of the present invention restores finer details and more realistic colors than the RVH method and the MPRL+IaCol method. In fact, inverse halftoning relies on inferring the original information from the halftone image, while automatic colorization extends seemingly reasonable colors to grayscale images. Compared with the RVH method, the method of the present invention introduces noise in each layer of the network and embeds more hidden information in the generation process of reversible halftones.

[0141] In terms of the restoration effect of reversible halftone, NIB is compared with the dynamic noise module proposed in the present invention. Figure 6 Comparison of reversible halftone images using NIB and using the dynamic noise module. Figure 6 (a) is the input grayscale image. Figure 6 (b) is a reversible halftone image obtained using NIB. Figure 6 (c) is a reversible halftone image obtained using a dynamic noise module; Figure 6 (d) is a partial enlarged view of (a); Figure 6 (e) is a partial enlarged view of (b); Figure 6 (f) in FIG. 1 is a partial enlarged view of (c). Compared with the existing reversible halftone method, the method of the present invention can produce better texture, such as Figure 6 As shown in (c) in the figure, other reversible halftone methods have obvious artificial textures, such as Figure 6 As shown in (b) in the figure. In the criteria for evaluating halftone images, the uniformity and discreteness of dot distribution are particularly important, which represents better blue noise characteristics. Figure 6In , samples generated by the model components dynamic noise module and noise induced enhancement (NIB) are shown, which can easily select the optimal halftone image. Therefore, the dynamic noise module proposed in this invention guides accurate decisions at each layer of the model, which is impossible to achieve with previous reversible halftoning methods using NIB and NCB.

[0142] In terms of running time, this embodiment evaluates the running time of each algorithm on a 512*512 “Lenna” test case, and the results are shown in Table 3.

[0143] Table 3. Comparison of running time and model parameters

[0144]

[0145] As can be seen from Table 3, in the reversible halftone part, compared with RVH and TRH, the parameters of the model of the present invention are reduced by 96.77%, the processing time is reduced by 57.28%, and the running time is comparable to the running time of the traditional error diffusion. In the image restoration part, compared with RVH and TRH, the model of the present invention not only improves the processing speed, but also achieves a certain improvement in image quality. The method of the present invention proves that lightweight CNN has the ability to produce better and faster reversible halftone images.

[0146] Embodiment 3

[0147] Based on the reversible halftoning method based on dynamic noise described in the first embodiment, this embodiment proposes a reversible halftoning system based on dynamic noise, including:

[0148] A model building module, used to build a reversible halftone model, including a halftone generation network and a restoration network;

[0149] The halftone generation network includes an input module, a local feature extraction module, two downsampling convolution modules, a high-level feature extraction module, two upsampling convolution modules and an output module connected in sequence;

[0150] The local feature extraction module and the advanced feature extraction module both include a number of feature extraction residual blocks; each feature extraction residual block includes a dynamic noise module, a ReLU activation function and a convolution layer connected in sequence, and the input features of the feature extraction residual block and the output features of the ReLU activation function are added through a jump connection as the output features of the feature extraction residual block; the dynamic noise module generates random noise and adaptively adjusts the intensity of the random noise to obtain adaptive random noise; the adaptive random noise is spliced ​​with the input features of the dynamic noise module and then subjected to a single-layer convolution to obtain the output features of the dynamic noise module;

[0151] The restoration network includes several lightweight residual blocks and flat convolutional layers;

[0152] The reversible halftone module is used to pass the original color image through the halftone generation network of the reversible halftone model to output a reversible halftone image; the reversible halftone image is then passed through the restoration network to output a restored color image.

[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0157] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A reversible halftoning method based on dynamic noise, characterized in that: include: Construct a reversible halftone model, including a halftone generation network and a restoration network; The halftone generation network includes an input module, a local feature extraction module, two downsampling convolution modules, a high-level feature extraction module, two upsampling convolution modules and an output module connected in sequence; The local feature extraction module and the high-level feature extraction module both include a number of feature extraction residual blocks; Each feature extraction residual block includes a dynamic noise module, a ReLU activation function and a convolution layer connected in sequence, and the input features of the feature extraction residual block and the output features of the ReLU activation function are added through a jump connection as the output features of the feature extraction residual block; the dynamic noise module generates random noise and adaptively adjusts the intensity of the random noise to obtain adaptive random noise; the adaptive random noise is spliced ​​with the input features of the dynamic noise module and then subjected to a single-layer convolution to obtain the output features of the dynamic noise module; The restoration network includes several lightweight residual blocks and flat convolutional layers; The original color image is passed through the halftone generation network of the reversible halftone model to output a reversible halftone image; the reversible halftone image is then passed through the restoration network to output a restored color image.

2. A reversible halftoning method based on dynamic noise according to claim 1, characterized in that: The local feature extraction module includes 2 feature extraction residual blocks; the high-level feature extraction module includes 4 feature extraction residual blocks.

3. A reversible halftoning method based on dynamic noise according to claim 1, characterized in that: The formula of the dynamic noise module is expressed as f(I O +N), where I O represents the input features of the dynamic noise module, N represents adaptive random noise, and f represents a single layer of convolution; The dynamic noise module adaptively adjusts the intensity of random noise. The formula is: Among them, α(t) is the noise intensity at time step t, α0 is the initial noise intensity, is the hue loss value at time step t, is the initial hue loss value and γ is a hyperparameter.

4. A reversible halftoning method based on dynamic noise according to claim 1, characterized in that: The output module includes a standard residual block and a Gumbel-Softmax layer.

5. A reversible halftoning method based on dynamic noise according to claim 1, characterized in that: The restoration network includes 7 lightweight residual blocks, each of which includes two convolutional layers and a ReLU activation function, and the input features of the lightweight residual block and the output features of the ReLU activation function are added through jump connections as the output features of the lightweight residual block.

6. A reversible halftoning method based on dynamic noise according to claim 1, characterized in that: The original color image is passed through the halftone generation network of the reversible halftone model to output a reversible halftone image; The reversible halftone image is then passed through the restoration network to output a restored color image, including: The original color image is input into the input module of the halftone generation network to obtain input features; the input features are passed through the local feature extraction module to obtain initial features; the initial features are passed through two downsampling convolution modules to obtain downsampling features; the downsampling features are passed through the high-level feature extraction module to further extract image semantic information and compensate for noise to obtain high-level features; the high-level features are passed through two upsampling convolution modules to obtain upsampling features; the upsampling features are passed through the output module, A reversible halftone image is obtained; The reversible halftone image is input into the restoration network and passed through several lightweight residual blocks to obtain the restored features; The restored features are passed through a flat convolution layer to output a restored color image.

7. A reversible halftoning method based on dynamic noise according to claim 1, characterized in that: When training the reversible halftone model, the total loss function includes the reversible halftone loss and the restoration loss; The reversible halftone loss includes content loss, tone consistency loss and spectrum loss; The reversible halftone image is subjected to a pre-trained inverse halftone model to obtain a restored grayscale image; the restored grayscale image and the grayscale image of the original color image are input into the convolutional network VGG-19, and the content loss is constructed based on the feature maps extracted from each layer of the convolutional network VGG-19 based on the restored grayscale image and the grayscale image of the original color image; The tone consistency loss is constructed based on the mean square error calculated after the reversible halftone image and the original color image are passed through a Gaussian filter; Construct spectrum loss based on spectrum of reversible halftone image and ideal spectrum; The restoration loss includes pixel L1 loss and spatial frequency L1 loss; The pixel L1 loss and spatial frequency L1 loss are constructed based on the pixel and spectrum differences between the restored color image and the original color image, respectively.

8. A reversible halftoning method based on dynamic noise according to claim 7, characterized in that: In the reversible halftone loss, the formula for the content loss is expressed as: Among them, i, j represents the spatial position of the feature map, k represents the channel index of the feature map, and F content Represents the grayscale image of the original color image, F generated Represents the restored grayscale image obtained by the pre-trained inverse halftone model of the reversible halftone image. and Respectively represent F comtent and F generated Feature maps extracted from the Lth layer of the convolutional network VGG-19; The formula for the hue consistency loss is expressed as: Among them, O h Represents a reversible halftone image, I c represents the original color image, G(·) represents the Gaussian filter with a convolution kernel size of 11×11, represents the average operator of all input images in the training set; The spectrum loss is expressed as: Among them, FFT generated Represents the spectrum of a reversible halftone image, FFT idea represents the ideal spectrum, and f represents a halftone image with constant grayscale.

9. A reversible halftoning method based on dynamic noise according to claim 7, characterized in that: In the restoration loss, the formula of the pixel L1 loss is expressed as: Among them, O c Represents the restored color image, I c represents the original color image, n represents the total number of image pixels, and u represents the index of the pixel; The formula for the spatial frequency L1 loss is expressed as: Here, FFT(·) represents fast Fourier transform.

10. A reversible halftone system based on dynamic noise, characterized in that: include: A model building module, used to build a reversible halftone model, including a halftone generation network and a restoration network; The halftone generation network includes an input module, a local feature extraction module, two downsampling convolution modules, a high-level feature extraction module, two upsampling convolution modules and an output module connected in sequence; The local feature extraction module and the high-level feature extraction module both include a number of feature extraction residual blocks; Each feature extraction residual block includes a dynamic noise module, a ReLU activation function and a convolution layer connected in sequence, and the input features of the feature extraction residual block and the output features of the ReLU activation function are added through a jump connection as the output features of the feature extraction residual block; the dynamic noise module generates random noise and adaptively adjusts the intensity of the random noise to obtain adaptive random noise; the adaptive random noise is spliced ​​with the input features of the dynamic noise module and then subjected to a single-layer convolution to obtain the output features of the dynamic noise module; The restoration network includes several lightweight residual blocks and flat convolutional layers; The reversible halftone module is used to pass the original color image through the halftone generation network of the reversible halftone model to output a reversible halftone image; the reversible halftone image is then passed through the restoration network to output a restored color image.

Images