Image processing method, image processing device, and computer-readable storage medium
By acquiring residual feature images through trained neural networks and performing residual adjustment, the problem of insufficient sharpness in image deblurring is solved, and the accuracy of image recognition is improved, especially in license plate recognition, where the recognition accuracy is significantly improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BOE TECHNOLOGY GROUP CO LTD
- Filing Date
- 2021-04-20
- Publication Date
- 2026-05-29
Smart Images
Figure CN115222608B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of image processing, in particular to an image processing method, an image processing device and a non-transitory computer readable storage medium. BACKGROUND
[0002] Image deblurring has been a very concerned technical hotspot in the field of image processing, which can be applied to photo sharpening, video sharpening and a large number of computer vision scenes. SUMMARY
[0003] The present disclosure provides an image processing method, an image processing device and a non-transitory computer readable storage medium.
[0004] The present disclosure provides an image processing method, comprising:
[0005] obtaining an input image to be processed and a weight coefficient;
[0006] processing the input image by using a trained first neural network to obtain a target output image, the definition of the target output image being greater than the definition of the input image;
[0007] The processing the input image by using a trained first neural network to obtain a target output image comprises:
[0008] obtaining a residual feature image of the input image, the channel number of the residual feature image being the same as the channel number of the input image;
[0009] generating a residual adjustment image according to the residual feature image and the weight coefficient, the channel number of the residual adjustment image being the same as the channel number of the input image;
[0010] fusing the residual adjustment image and the input image to obtain the target output image.
[0011] In some embodiments, the residual feature image of the input image is obtained, specifically comprising:
[0012] performing convolution processing on the input image to obtain a first intermediate image, the channel number of the first intermediate image being greater than the channel number of the input image;
[0013] performing m times of first feature transformation based on the first intermediate image; wherein m is a positive integer, each time the first feature transformation comprises: performing residual feature extraction on a first image to be processed to obtain a first residual image; and performing down-sampling on the first residual image to obtain a down-sampled image; wherein,
[0014] When m=1, the first image to be processed by the first feature transformation is the first intermediate image; when m>1, the first image to be processed by the first feature transformation is the first intermediate image; and the first image to be processed by each subsequent first feature transformation is the downsampled image generated by the previous first feature transformation.
[0015] Residual features are extracted from the downsampled image generated by the last first feature transformation to generate a second residual image;
[0016] Based on the second residual image, m second feature transformations are performed; wherein, the i-th second feature transformation includes: upsampling the second image to be processed to obtain an upsampled image; then obtaining a superimposed image of the upsampled image and the first residual image generated by the (m-i+1)-th first feature transformation, and extracting residual features from the superimposed image to obtain a third residual image; i is a positive integer less than or equal to m, where,
[0017] When m = 1, the second image to be processed in the second feature transformation is the second residual image; when m > 1, the second image to be processed in the first second feature transformation is the second residual image; the second image to be processed in each subsequent second feature transformation is the third residual image generated by the previous second feature transformation.
[0018] The third residual image generated by the last second feature transformation is convolved to obtain the residual feature image;
[0019] The number of channels in the first residual image, the second residual image, and the third residual image are all greater than the number of channels in the input image.
[0020] In some embodiments, obtaining the residual feature image of the input image specifically includes:
[0021] The input image is convolved to obtain a first intermediate image, wherein the number of channels in the first intermediate image is greater than the number of channels in the input image;
[0022] Two first feature transformations are performed based on the first intermediate image; wherein...
[0023] The first feature transformation includes: extracting residual features from the first intermediate image to obtain a first residual image; and downsampling the first residual image to obtain a downsampled image of the first feature transformation.
[0024] The second first feature transformation includes: extracting residual features from the downsampled image of the first first feature transformation to obtain a first residual image; and downsampling the first residual image to obtain the downsampled image of the second first feature transformation.
[0025] Residual features are extracted from the downsampled image generated by the second first feature transformation to generate a second residual image;
[0026] Two second feature transformations are performed based on the second residual image; wherein...
[0027] The first second feature transformation includes: upsampling the second residual image and obtaining a superimposed image of the upsampled image and the first residual image in the second first feature transformation; extracting residual features from the superimposed image to obtain a third residual image of the first second feature transformation;
[0028] The second feature transformation includes: upsampling the third residual image generated by the first feature transformation, obtaining the superimposed image of the upsampled image and the first residual image generated by the first feature transformation, and extracting residual features from the superimposed image to obtain the third residual image of the second feature transformation.
[0029] The third residual image generated by the second feature transformation is convolved to obtain the residual feature image;
[0030] The number of channels in the first residual image, the second residual image, and the third residual image are all greater than the number of channels in the input image.
[0031] In some embodiments, residual feature extraction is performed on the first image to be processed, specifically including:
[0032] The first residual block of the trained first neural network is used to extract residual features from the first image to be processed. The first residual block includes at least two first convolutional layers.
[0033] In some embodiments, residual feature extraction is performed on the downsampled image generated by the last first feature transformation, specifically including:
[0034] The second residual block of the trained first neural network is used to extract residual features from the downsampled image generated by the last first feature transformation to generate an intermediate residual image.
[0035] The third residual block of the trained first neural network is used to extract residual features from the intermediate residual image to generate the second residual image;
[0036] The second residual block includes at least two second convolutional layers, and the third residual block includes at least two third convolutional layers.
[0037] In some embodiments, residual feature extraction is performed on the superimposed image of the upsampled image and the first residual image generated by the (m-i+1)th first feature transformation, specifically including:
[0038] The third residual image is generated by extracting residual features from the superimposed image using the fourth residual block of the trained first neural network.
[0039] The fourth residual block includes at least two fourth convolutional layers.
[0040] In some embodiments, the weighting coefficient is between 0 and 4;
[0041] The residual adjusted image is fused with the input image to obtain the target output image, specifically including:
[0042] The residual adjusted image is superimposed on the input image to obtain the initial output image;
[0043] Determine whether each pixel value in the initial output image is within a preset range. If all pixel values in the initial output image are within the preset range, then the initial output image is used as the target output image. If at least one pixel value is not within the preset range, adjust the corresponding pixel value to the preset range according to a preset rule, and the adjusted initial output image is used as the target output image.
[0044] In some embodiments, the preset rules include:
[0045] For any pixel value in the initial output image, if the pixel value is less than the minimum value of the preset range, the pixel value is adjusted to the minimum value of the preset range; if the pixel value is greater than the maximum value of the preset range, the pixel value is adjusted to the maximum value of the preset range.
[0046] In some embodiments, the method further includes: alternatingly training the first neural network to be trained and the discriminant network to be trained to obtain the trained first neural network and the trained discriminant network;
[0047] The training of the discriminant network to be trained includes:
[0048] The first sample image and sample weight coefficients are provided to the current first neural network so that the current first neural network outputs a first image with improved clarity.
[0049] The first enhanced image and the original sample image corresponding to the first sample image are provided to the current discrimination network, and the parameters of the current discrimination network are adjusted according to the loss function of the discrimination network so that the output of the parameter-tuned discrimination network can characterize whether the input of the discrimination network is the input of the first neural network or the discrimination result of the original sample image.
[0050] The first neural network to be trained is trained, including:
[0051] The second sample image and the sample weight coefficients are provided to the current first neural network to enable the first neural network to generate a second sharpness-enhanced image.
[0052] The second sharpness-enhanced image is input into the parameter-tuned discriminant network so that the parameter-tuned discriminant network generates a second discrimination result based on the second sharpness-enhanced image;
[0053] Based on the second loss function, the parameters of the current first neural network are adjusted to obtain the updated first neural network;
[0054] In the process of alternating training of the first neural network to be trained and the discriminant network to be trained, the sample weight coefficient is a fixed value.
[0055] In some embodiments, the sample weight coefficient is 1.
[0056] In some embodiments, the first loss function includes a cross-entropy loss function.
[0057] In some embodiments, the second loss function includes the superposition of a first loss term, a second loss term, and a third loss term;
[0058] The first loss term is obtained based on the adversarial loss of the second sample image.
[0059] The second loss term is obtained based on the perceptual loss between the second sample image and its corresponding original sample image;
[0060] The third loss term is obtained based on the L1 loss between the second sharpness-enhanced image and its corresponding original sample image.
[0061] In some embodiments, the second loss term is:
[0062] Where β is a preset weight, and y is the original sample image corresponding to the second sharpness-enhanced image. The image is enhanced to the second level of clarity;
[0063]
[0064] Here, j is the number of layers in the preset optimization network, C is the number of channels in the output image of the preset network layer, H is the height of the output image of the preset network layer, and W is the width of the output image of the preset network layer. When the original sample image corresponding to the second sharpness-enhanced image is input into the preset optimization network, the output result of the preset network layer; The output result of the preset network layer when the second enhanced image is input to the preset optimization network; both the first sample image and the second sample image are license plate images, and the preset optimization network adopts the LPR license plate recognition network.
[0065] This disclosure also provides an image processing apparatus, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the image processing method described above.
[0066] This disclosure also provides a non-transient computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described image processing method. Attached Figure Description
[0067] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:
[0068] Figure 1 This is a flowchart of an image processing method in some embodiments of this disclosure.
[0069] Figure 2 This is a flowchart illustrating the processing of an input image using a trained first neural network in some embodiments of this disclosure.
[0070] Figure 3 This is a schematic diagram of the specific architecture of a trained first neural network in some embodiments of this disclosure.
[0071] Figure 4 This is a flowchart illustrating an optional implementation of step S21 provided in some embodiments of this disclosure.
[0072] Figure 5 This is a schematic diagram of a first residual block provided in some embodiments of this disclosure.
[0073] Figure 6 This is a flowchart illustrating an optional implementation of step S23 provided in some embodiments of this disclosure.
[0074] Figure 7 This is a schematic diagram of a discrimination network provided in some embodiments of this disclosure.
[0075] Figure 8 This is a schematic diagram illustrating the process of training a generative adversarial network as provided in some embodiments of this disclosure.
[0076] Figure 9A This is a schematic diagram of the license plate image to be processed.
[0077] Figure 9B This is a schematic diagram showing the effect of processing a license plate image using the image processing method of this embodiment. Detailed Implementation
[0078] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0079] The main component of a deep learning system is a convolutional neural network (CNN), which can be used in fields including image recognition, face recognition, text recognition, animal recognition, and image processing. For example, a CNN is used to process images; its input and output are images. Optionally, a CNN includes multiple convolutional layers, multiple upsampling layers, and multiple downsampling layers. For example, each of the multiple convolutional layers can process an image. Upsampling and downsampling layers can resize the input image to a size corresponding to a specific convolutional layer.
[0080] In embodiments of this disclosure, the term "convolution kernel" refers to a two-dimensional matrix used in the convolution process. Optionally, each item in a plurality of items in the two-dimensional matrix has a specific value.
[0081] In embodiments of this disclosure, the term "convolution" refers to the process of processing an image. A convolution kernel is used for convolution. Each pixel of the input image has a value, and the convolution kernel starts at one pixel in the input image and moves sequentially over each pixel in the input image. At each location of the convolution kernel, the kernel overlaps with several pixels in the image based on its scale. At the location of the convolution kernel, the value of one pixel among the several overlapping pixels is multiplied by a corresponding value of the convolution kernel to obtain a multiplied value for one pixel among the several overlapping pixels. Subsequently, all multiplied values of the overlapping pixels are summed to obtain a sum corresponding to the location of the convolution kernel in the input image. By moving the convolution kernel over each pixel of the input image, all sums corresponding to all locations of the convolution kernel are collected and output to form the output image. In one example, convolution can use different convolution kernels to extract different features from the input image. In another example, the convolution process can use different convolution kernels to add more features to the input image.
[0082] In embodiments of this disclosure, the term "convolutional layer" refers to a layer in a convolutional neural network. A convolutional layer is used to perform convolution on an input image to obtain an output image. Optionally, different convolutional kernels are used to perform different convolutions on the same input image. Optionally, different convolutional kernels are used to perform convolutions on different parts of the same input image. Optionally, different convolutional kernels are used to perform convolutions on different input images; for example, multiple images are input into the convolutional layer, and corresponding convolutional kernels are used to perform convolutions on images from the multiple images. Optionally, different convolutional kernels are used depending on the different characteristics of the input image.
[0083] As used herein, the term "activation layer" refers to a layer in a convolutional neural network. An activation layer performs a nonlinear mapping on the output signal from a convolutional layer. Various functions can be used in activation layers. Examples of functions suitable for use in activation layers include, but are not limited to, the rectified linear unit (ReLU) function, the sigmoid function, and the hyperbolic tangent function (e.g., the tanh function). In one example, the activation layer is not included within a convolutional layer.
[0084] In some embodiments, convolutional layers are the core layers of a convolutional neural network. In a convolutional layer, neurons are connected to neurons in directly adjacent convolutional layers. Optionally, a convolutional layer applies multiple convolutional kernels to the input image to extract multiple features from the input image. The convolutional layer can extract one type of feature from the input image. Optionally, the initialized convolutional kernel is a random fractional matrix. During the pre-training process of the convolutional neural network, the convolutional kernel learns to obtain reasonable values.
[0085] Optionally, the result obtained by applying convolutional kernels to the input image is called a feature map. The number of feature maps is equal to the number of convolutional kernels. Each feature map corresponds to a specific convolutional kernel.
[0086] Optionally, a corresponding neuron in one of the multiple feature maps is formed by a rectangular arrangement of neurons. The neurons of a corresponding neuron in the multiple feature maps share a corresponding one of the multiple convolutional kernels.
[0087] Optionally, the convolutional neural network has multiple convolutional layers. A feature map from a corresponding output of one of the multiple convolutional layers is input into a downstream convolutional layer. The downstream convolutional layer processes the feature map output from the corresponding convolutional layer and outputs a downstream feature map from the multiple feature maps.
[0088] In embodiments of this disclosure, the size of an image (e.g., feature map, data, signal) refers to the width and height of the image, for example, width × height.
[0089] In some embodiments, the term "downsampling" refers to the process of extracting features from an input image and outputting an image with a smaller size. As used herein, the term "upsampling" refers to the process of adding more information to an input image and outputting an image with a larger size.
[0090] This disclosure provides an image processing method for use with a convolutional neural network. This image processing method is particularly suitable for processing license plate images. Figure 1 Here are flowcharts of image processing methods in some embodiments of this disclosure, such as... Figure 1 As shown, the image processing methods include:
[0091] S1. Obtain the input image to be processed and the weight coefficients. The input image is, for example, a license plate image.
[0092] S2. The first trained neural network is used to process the input image to obtain the target output image. The clarity of the target output image is greater than that of the input image.
[0093] In this disclosure, "clarity" refers to the contrast between image objects, such as the clarity of image outlines and associated fine shadow patterns within the image. Higher image clarity results in higher accuracy when the recognition system identifies the image. For example, if the input image is a license plate image, and the target output image is obtained by processing the input image using the image processing methods provided in this disclosure (e.g., denoising and / or deblurring or resolution enhancement processing), the accuracy of the license plate recognition system on the target output image is higher than the recognition rate of the license plate recognition system on the input image. It is understood that clarity and blurriness are relative concepts; higher image clarity corresponds to lower blurriness.
[0094] Figure 2 This is a flowchart illustrating the processing of an input image using a trained first neural network in some embodiments of this disclosure, such as... Figure 2 As shown, step S2 specifically includes:
[0095] S21. Obtain the residual feature image of the input image, wherein the number of channels of the residual feature image is the same as the number of channels of the input image. The residual feature image is an image of the edge information features of the input image. The residual feature image can be obtained using the residual extraction module of the trained first neural network. The residual extraction module can include at least one residual block (ResNet).
[0096] In this embodiment, the input image, residual feature image, and the residual adjusted image, first intermediate image, first residual image, second residual image, upsampled image, and downsampled image mentioned below all have multiple channels, and each channel's image is a feature map. During image processing, each feature map is represented in matrix form, where the height of the feature map is the number of rows in the matrix corresponding to the feature map, and the width is the number of columns in the matrix corresponding to the feature map. It should be noted that in this embodiment, "image size" refers to the size of the feature map for each channel of the image. If the image contains multiple channels, the size of the feature map for each channel is the same.
[0097] For example, the input image includes an image with a red channel, an image with a green channel, and an image with a blue channel; that is, the input image has three channels.
[0098] S22. Generate a residual adjusted image based on the residual feature image and weight coefficients. The number of channels in the residual adjusted image is the same as the number of channels in the input image.
[0099] For example, the image of each channel of the residual feature image is multiplied by a weight coefficient to obtain the image of each channel of the residual adjusted image. Specifically, multiplying the image of a particular channel by the weight coefficient involves multiplying the value of each pixel in the image of that channel by the weight coefficient.
[0100] It's important to note that the weighting coefficients are not fixed values; they can be set according to actual needs. For example, when the input image has low resolution, the weighting coefficient can be set to a larger value; when the input image has high resolution, the weighting coefficient can be set to a smaller value. In one example, the weighting coefficient can be set by the user; in another example, the value of the weighting coefficient can be determined based on the resolution of the input image to be processed and the preset correspondence between resolution and weighting coefficients.
[0101] S23. Fuse the residual adjusted image with the input image to obtain the target output image.
[0102] In some examples, fusing the residual adjusted image with the input image may include: superimposing the residual adjusted image with the input image, that is, superimposing the values of pixels at corresponding positions in the residual adjusted image with those in the input image. A target output image is then generated based on the superimposed image. When both the residual adjusted image and the input image contain multiple channels, the images of each channel in the residual adjusted image and the images of each channel in the input image are in one-to-one correspondence. Superimposing the residual adjusted image with the input image mainly involves superimposing the images of corresponding channels from both images separately.
[0103] In this embodiment, the trained first neural network, when processing the input image, first acquires the residual feature image (i.e., edge information image) of the input image. Then, based on the residual feature image and weight coefficients, it generates a residual adjustment image. Finally, it fuses the residual adjustment image with the input image to obtain the target output image. This is equivalent to enhancing the edge information of the input image to obtain the target output image; and the larger the weight coefficients, the greater the enhancement of the edge information; the smaller the weight coefficients, the smaller the enhancement of the edge information. Therefore, the value of the weight coefficients can be determined based on the ambiguity of the input image, thereby enabling the sharpening of input images with different levels of ambiguity. This improves the accuracy of image recognition after processing by the image processing method of this disclosure.
[0104] Figure 3 This is a schematic diagram of the specific architecture of the trained first neural network in some embodiments of this disclosure, such as... Figure 3 As shown, the trained first neural network includes a residual extraction module, which may specifically include: convolutional layer 10, first residual blocks 21-22, downsampling layers 31-32, second residual block 40, third residual block 50, upsampling layers 61-62, fourth residual blocks 71-72, and convolutional layer 80.
[0105] Figure 4 This is a flowchart illustrating optional implementations of step S21 provided in some embodiments of this disclosure. For example... Figure 4 As shown, step S21 above may specifically include the following steps S211 to 215:
[0106] S211. The input image is convolved using convolutional layer 10 to obtain a first intermediate image, wherein the number of channels in the first intermediate image is greater than the number of channels in the input image. For example, convolutional layer 10 includes multiple 3×3 convolutional kernels, the input image has 3 channels, and the first intermediate image has 128 channels. The size of the input image is not adjusted when convolutional layer 10 performs convolution processing on the input image.
[0107] S212. Perform m first feature transformations based on the first intermediate image. Where m is a positive integer, each first feature transformation includes: extracting residual features from the first image to be processed to obtain a first residual image; then downsampling the first residual image to obtain a downsampled image.
[0108] Wherein, when m=1, the first image to be processed by the first feature transformation is the first intermediate image; when m>1, the first image to be processed by the first feature transformation is the first intermediate image; and the first image to be processed by each subsequent first feature transformation is the downsampled image generated by the previous first feature transformation.
[0109] In some examples, the first residual image can be downsampled using downsampling layers 31 / 32. These downsampling layers 31 / 32 may include: inverted Muxout layers, strided convolutions, maxpooling layers, or standard per-channel downsamplers (such as bicubic interpolation). The kernel size in downsampling layers 31 / 32 is, for example, 3×3, and the sampling rate of downsampling layers 31 / 32 is, for example, 2x. That is, when the size of the first residual image is 100×100, the size of the downsampled image obtained after downsampling layers 31 / 32 is 500×500. It should be understood that the downsampling process only changes the size of the image (i.e., width and height), and does not change the number of channels of the image.
[0110] In some examples, residual feature extraction is performed on the first image to be processed, which may specifically include: using the first residual block 21 / 22 of the trained first neural network to extract residual features from the first image to be processed. Figure 5 This is a schematic diagram of a first residual block provided in some embodiments of this disclosure, such as... Figure 5 As shown, the first residual block 21 / 22 includes at least two first convolutional layers 211, each followed by an activation layer 212. In the first residual block 21 / 22, its input is superimposed on the output of the last activation layer 212, thus serving as the output of the first residual block 21 / 22 (i.e., the first residual image). The first residual image output by the first residual block 21 / 22 has the same number of channels as the first image to be processed input to the first residual block 21 / 22.
[0111] In some examples, each first convolutional layer 211 includes multiple 3×3 convolutional kernels, and the activation layer 212 includes an activation function, which may include a linear correction unit (ReLU) function, a leaky correction linear unit (leaky ReLU) function, a sigmoid function, or a hyperbolic tangent function (tanh function), etc.
[0112] The number of first convolutional layers 211 in the first residual block 21 / 22 can be 2, 3, or other numbers. As mentioned above, if the input image is a license plate image, then the first convolutional layer 211 in the first residual block 21 / 22 can be set to 2 layers to reduce the computational load. Compared to other types of images (e.g., face images), license plate images do not contain much information; therefore, even if the first convolutional layer 211 is set to 2 layers, it will not affect the image processing effect.
[0113] The value of m can be determined based on the size of the input image; in some examples disclosed herein, m is 2. For example... Figure 3As shown, the trained first convolutional network includes two first residual blocks 21-22, each followed by a downsampling layer 31-32. The first residual block 21 receives the image output by the convolutional layer 10; the second residual block 22 receives the image output by the first downsampling layer 31. Specifically, when m=2, in the two first feature transformations based on the first intermediate image, the first feature transformation includes: extracting residual features from the first intermediate image using the first residual block 21 to output a first residual image, and downsampling the first residual image output by the first residual block 21 using the downsampling layer 31 to obtain the downsampled image of the first feature transformation. The second feature transformation includes: extracting residual features from the downsampled image of the first feature transformation using the first residual block 22 to obtain a first residual image; and downsampling the first residual image output by the first residual block 22 using the downsampling layer 32 to obtain the downsampled image of the second feature transformation.
[0114] S213. Extract residual features from the downsampled image generated by the last first feature transformation to generate a second residual image. The number of channels in the second residual image is greater than the number of channels in the input image.
[0115] In some embodiments, step S213 includes: extracting residual features from the downsampled image generated by the last first feature transformation using two residual blocks. For example, step S213 includes: extracting residual features from the downsampled image generated by the last first feature transformation using a second residual block 40 to generate an intermediate residual image; then, extracting residual features from the intermediate residual image using a third residual block 50 to generate a second residual image.
[0116] For example, the structures of the second residual block 40 and the third residual block 50 can both be the same as those of the first residual blocks 21 / 22. That is, the second residual block 40 includes two second convolutional layers, each followed by an activation layer. The input of the second residual block 40 is superimposed with the output of the last activation layer in the second residual block 40 to obtain the output of the second residual block 40. Each second convolutional layer can include multiple 3×3 convolutional kernels. The third residual block 50 includes two third convolutional layers, each followed by an activation layer. The input of the third residual block 50 is superimposed with the output of the last activation layer in the third residual block to obtain the output of the third residual block 50. Each third convolutional layer includes multiple 3×3 convolutional kernels. Of course, the number of second convolutional layers in the second residual block 40 and the number of third convolutional layers in the third residual block 50 can also be greater than 2.
[0117] S214. Perform m second feature transformations sequentially based on the second residual image. The i-th second feature transformation includes: upsampling the second image to be processed to obtain an upsampled image; obtaining a superimposed image of the upsampled image and the first residual image generated by the (m-i+1)-th (i.e., the m-th from the end) first feature transformation; and extracting residual features from the superimposed image to obtain a third residual image. Here, i is a positive integer less than or equal to m. When m = 1, the second image to be processed in the second feature transformation is the second residual image; when m > 1, the second image to be processed in the first second feature transformation is the second residual image; and the second image to be processed in each subsequent second feature transformation is the third residual image generated by the previous second feature transformation.
[0118] The residual feature extraction of the superimposed image specifically includes: using the fourth residual block 71 / 72 of the first neural network to extract residual features from the superimposed image, generating a third residual image. The third residual image has a greater number of channels than the input image.
[0119] The fourth residual block 71 / 72 includes at least two fourth convolutional layers. In some examples, the fourth residual block 71 / 72 adopts the same structure as the first residual block 21 / 22. Specifically, the fourth residual block 71 / 72 includes two fourth convolutional layers, each followed by an activation layer. The input of the fourth residual block 71 / 72 is superimposed with the output of the last activation layer in the fourth residual block 71 / 72 to form the output of the fourth residual block.
[0120] As mentioned above, m is 2, meaning that two second feature transformations are performed based on the second residual image. The first second feature transformation includes: upsampling the second residual image using the first upsampling layer 61; then, superimposing the output of the first upsampling layer 61 with the output of the second first residual block 22, and providing the superimposed image to the first fourth residual block 71; using the first fourth residual block 71 to extract residual features from the superimposed image, obtaining the third residual image of the first second feature transformation, which is then provided to the second upsampling layer 62. The second second feature transformation includes: upsampling the third residual image of the first second feature transformation using the upsampling layer 62, obtaining an upsampled image; then, superimposing the upsampled image output by the upsampling layer 62 with the output of the first first residual block 21, and providing the superimposed image to the second fourth residual block 72; using the second fourth residual block 72 to extract residual features from the received superimposed image, obtaining the third residual image of the second second feature transformation. This method enhances the flow of feature information between different scales of the image, thereby improving the clarity of the target output image.
[0121] S215. Use convolutional layer 80 to perform convolution processing on the third residual image generated by the last second feature transformation to obtain the residual feature image.
[0122] In some examples, convolutional layer 80 includes multiple 3×3 convolutions. In some examples, the residual feature image output by convolutional layer 80 has 3 channels, while the remaining images (e.g., the images output by convolutional layer 10, the upsampled images output by upsampling layers 61-62, the images output by the first residual block 21-22, the images output by the second residual block 40, the images output by the third residual block 50, the images output by the fourth residual block 71-72, and the downsampled images output by downsampling layers 31-32) all have 128 channels.
[0123] In some embodiments, the weighting coefficient is between 0 and 4. Figure 6 Here is a flowchart of optional implementations of step S23 provided in some embodiments of this disclosure, such as... Figure 6 As shown, step S23 may specifically include:
[0124] S231. The residual adjusted image is superimposed on the input image to obtain an initial output image. When both the residual adjusted image and the input image contain multiple channels, the images of each channel in the residual adjusted image and the images of each channel in the input image are in one-to-one correspondence. Superimposing the residual adjusted image on the input image mainly involves superimposing the images of the corresponding channels of the two images separately. Each channel image includes multiple pixels. When superimposing the pixels of two channels, specifically, the pixel values of the pixels in the two channels can be superimposed.
[0125] The pixel value can be a grayscale level.
[0126] S232. Determine whether each pixel value in the initial output image is within a preset range. If each pixel value in the initial output image is within the preset range, then the initial output image is used as the target output image. If at least one pixel value is not within the preset range, adjust the corresponding pixel value to the preset range according to a preset rule, and the adjusted initial output image is used as the target output image.
[0127] Optionally, the preset rules include: for any pixel value in the initial output image, if the pixel value is less than the minimum value of a preset range, the pixel value is adjusted to the minimum value of the preset range; if the pixel value is greater than the maximum value of the preset range, the pixel value is adjusted to the maximum value of the preset range. For example, the preset range is 0 to 255. If the grayscale of a pixel in the initial output image is less than 0, the grayscale of that pixel is adjusted to 0; if the grayscale of a pixel is greater than 255, the grayscale of that pixel is adjusted to 255.
[0128] In some embodiments, the image processing method further includes: before step S1, training the first neural network to be trained and the discriminant network to be trained alternately by means of generative adversarial network (GAN) training, so as to obtain the trained first neural network and the trained discriminant network.
[0129] It should be noted that the structure of the first neural network to be trained and the first neural network after training is the same (i.e., the number of convolutional layers and the number of convolutional kernels in the convolutional layers), the difference lies in the weights of the convolutional layers; the structure of the discriminant network to be trained and the discriminant network after training is the same, the difference lies in the weights of the convolutional layers.
[0130] The discriminant network is used to determine whether the image it receives is a "real" sample (i.e., the original sample image) or a "fake" sample (i.e., the image generated by the first neural network). For example, when the output of the discriminant network is 0, it means that the discriminant network determines that the image it receives is the image generated by the first neural network; when the output of the discriminant network is 1, it means that the discriminant network determines that the image it receives is the original sample image.
[0131] Figure 7 This is a schematic diagram of the discriminant network provided in some embodiments of this disclosure, such as... Figure 7 As shown, the discriminant network adopts a stacked convolutional neural network structure, which includes: a downsampling layer 3a, a convolutional layer 3b, a downsampling layer 3c, a convolutional layer 3d, a downsampling layer 3e, a fully connected layer 3f, and an output layer 3g. Each downsampling layer 3a / 3c / 3e and each convolutional layer 3b / 3d is followed by an activation layer 3h. For example, the downsampling layer 3a includes 32 3×3 convolutional kernels with a downsampling factor of 2; the convolutional layer 3b includes 64 3×3 convolutional kernels; the downsampling layer 3c includes 128 3×3 convolutional kernels with a downsampling factor of 2; the convolutional layer 3d includes 192 3×3 convolutional kernels; the downsampling layer 3e includes 96 3×3 convolutional kernels with a downsampling factor of 2; the fully connected layer 3f converts the 96-channel image into a one-dimensional vector; and the final output layer 3g outputs a probability value of 0 to 1 from the one-dimensional vector.
[0132] Figure 8 This is a schematic diagram illustrating the process of training a generative adversarial network provided in some embodiments of this disclosure, such as... Figure 8 As shown, the training process of a generative adversarial network includes alternating between steps S01a and S01b until a preset training condition is met. The preset training condition is, for example, that the number of alternations between steps S01a and S01b reaches a preset number.
[0133] S01a. The first sample image and sample weight coefficients are provided to the current first neural network so that the current first neural network outputs a first sharpness-enhanced image. The first sharpness-enhanced image and the original sample image corresponding to the first sample image are provided to the current discriminant network, and the parameters of the current discriminant network are adjusted according to the first loss function so that the output of the parameter-tuned discriminant network can characterize whether the input of the discriminant network is the input of the first neural network or the discrimination result of the original sample image.
[0134] S01b: The second sample image and sample weight coefficients are provided to the current first neural network to generate a second sharpness-enhanced image. The second sharpness-enhanced image is then input into the parameter-tuned discriminant network to generate a second discrimination result based on the second sharpness-enhanced image. Based on the second loss function, the parameters of the current first neural network are adjusted to obtain an updated first neural network.
[0135] It should be noted that the sharpness of the first sample image is less than that of the corresponding original sample image. In some examples, the original sample image can be blurred to obtain either the first or second sample image. In some examples, the original sample set can be the CCPC high-definition license plate dataset (200K frames of images), with an image size of 192×96.
[0136] It should also be noted that if we consider the nth step S01a and the nth step S01b as a training round, then in the first training round, the current first neural network is the first neural network to be trained; in each subsequent training round, the current first neural network is the first neural network updated in step S01b of the previous training round. In the first training round, the current discriminator network is the discriminator network to be trained; in each subsequent training round, the current discriminator network is the discriminator network with parameters tuned in step S01a of the previous training round.
[0137] During the alternating processes of steps S01a and S01b, the sample weight coefficient is a fixed value. For example, the sample weight coefficient is 1.
[0138] In some embodiments, the first loss function includes the cross-entropy loss function. Specifically, the first loss function Loss1 is shown in equation (1) below:
[0139]
[0140] Where x is the original sample image corresponding to the first sample image; D(x) is the discrimination result of the current discriminant network based on the original sample image; z is the first sample image; G(z) is the generation result of the current first neural network based on the first sample image; D(G(z)) is the discrimination result of the current discriminant network based on G(z); P data (x) represents the probability distribution of the original image set data; x ~ P data (x) represents the probability distribution of the original sample image x following the original image set data; P z (z) represents the probability distribution of the first sample set; z ~ P z (z) indicates that the first sample image z follows the probability distribution of the sample image set; E[] indicates taking the expectation of the values within the brackets.
[0141] In some embodiments, the second loss function includes a superposition of a first loss term, a second loss term, and a third loss term. The first loss term is obtained based on the adversarial loss of the second sample image. The second loss term is obtained based on the perceptual loss between the second sample image and its corresponding original sample image. The third loss term is obtained based on the L1 loss between the second sharpness-enhanced image and its corresponding original sample image.
[0142] Specifically, the second loss function Loss2 is shown in equation (2) below:
[0143]
[0144] in, This is the first loss item; γ‖yG(s)‖1 is the second loss term; γ‖yG(s)‖1 is the third loss term. s is the second sample image; G(s) is the generation result of the current first generator network based on the second sample image; D(G(s)) is the discrimination result of the current discriminator network based on G(s); y is the original sample image corresponding to the second sample image, that is, the original sample image corresponding to the second sharpness-enhanced image; s~P s (s) indicates that the second sample image follows the probability distribution of the second sample set; E[] indicates taking the expectation of the values within the brackets; α, β, and γ are preset weights.
[0145] in, Let j be the preset network layer in the preset optimization network, j be the number of layers in the preset optimization network, C be the number of channels in the output image of the preset network layer, H be the height of the output image of the preset network layer, and W be the width of the output image of the preset network layer. The output result of the preset network layer when the original sample image corresponding to the second sharpness-enhanced image is input into the preset optimization network. The output of the preset network layer when the second enhanced image is input into the preset optimization network. In some examples, both the first and second sample images are license plate images; in this case, the preset optimization network uses an LPR license plate recognition network.
[0146] It should be noted that the first and second sample sets can be the same dataset. During the same training round, the first and second sample images can be the same image, but in different training rounds, different images can be used for the first sample image. Furthermore, in each training round, the training steps for the discriminative network can be performed first, or the training steps for the first neural network can be performed first.
[0147] In some examples, the original sample set can be the CCPD high-definition license plate dataset (200K frames of images), with an image size of 192×96, a training learning rate of 1e-4, and 100 iterations.
[0148] Figure 9A This is a schematic diagram of a license plate image to be processed. When processing the license plate image using the image processing method of this embodiment, the weighting coefficient is set to 1.0. The processed image is as follows. Figure 9B As shown. Typically, the accuracy rate of license plate recognition systems for license plate images taken at a distance of 10 meters is around 40%. After improving the clarity of license plate images taken at a distance of 10 meters using the image processing method provided in this embodiment, the accuracy rate of license plate recognition systems can reach over 70%.
[0149] This disclosure also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, implements the above-described image processing method.
[0150] This disclosure also provides a non-transient computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described image processing method.
[0151] The aforementioned memory and the non-transitory computer-readable storage medium include, but are not limited to, the following readable media: such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, magnetic disks or magnetic tapes, optical storage media such as optical discs (CDs) or DVDs (Digital Universal Discs), and other non-transitory media. Examples of processors include, but are not limited to, general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc.
[0152] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.
Claims
1. An image processing method, characterized in that, include: Obtain the input image to be processed and the weight coefficients; The input image is processed using a trained first neural network to obtain a target output image, wherein the sharpness of the target output image is greater than that of the input image; The process of processing the input image using a trained first neural network to obtain the target output image includes: Obtain the residual feature image of the input image, wherein the number of channels in the residual feature image is the same as the number of channels in the input image; A residual adjusted image is generated based on the residual feature image and the weight coefficients; the number of channels in the residual adjusted image is the same as the number of channels in the input image. The residual adjusted image is fused with the input image to obtain the target output image; Obtaining the residual feature image of the input image specifically includes: The input image is convolved to obtain a first intermediate image, wherein the number of channels in the first intermediate image is greater than the number of channels in the input image; Based on the first intermediate image, perform m first feature transformations; where m is a positive integer, each first feature transformation includes: extracting residual features from the first image to be processed to obtain a first residual image; and downsampling the first residual image to obtain a downsampled image; wherein... When m=1, the first image to be processed by the first feature transformation is the first intermediate image; when m>1, the first image to be processed by the first feature transformation is the first intermediate image; and the first image to be processed by each subsequent first feature transformation is the downsampled image generated by the previous first feature transformation. Residual features are extracted from the downsampled image generated by the last first feature transformation to generate a second residual image; Based on the second residual image, m second feature transformations are performed; wherein, the i-th second feature transformation includes: upsampling the second image to be processed to obtain an upsampled image; then obtaining a superimposed image of the upsampled image and the first residual image generated by the (m-i+1)-th first feature transformation, and extracting residual features from the superimposed image to obtain a third residual image; i is a positive integer less than or equal to m, where, When m=1, the second image to be processed in the second feature transformation is the second residual image; when m>1, the second image to be processed in the first second feature transformation is the second residual image; the second image to be processed in each subsequent second feature transformation is the third residual image generated by the previous second feature transformation. The third residual image generated by the last second feature transformation is convolved to obtain the residual feature image; The number of channels in the first residual image, the second residual image, and the third residual image are all greater than the number of channels in the input image.
2. The method according to claim 1, characterized in that, m=2。 3. The method according to claim 1, characterized in that, Residual feature extraction is performed on the first image to be processed, specifically including: The first residual block of the trained first neural network is used to extract residual features from the first image to be processed. The first residual block includes at least two first convolutional layers.
4. The method according to claim 1, characterized in that, Residual feature extraction is performed on the downsampled image generated by the last first feature transformation, specifically including: The second residual block of the trained first neural network is used to extract residual features from the downsampled image generated by the last first feature transformation to generate an intermediate residual image. The third residual block of the trained first neural network is used to extract residual features from the intermediate residual image to generate the second residual image; The second residual block includes at least two second convolutional layers, and the third residual block includes at least two third convolutional layers.
5. The method according to claim 1, characterized in that, Residual feature extraction is performed on the superimposed image of the upsampled image and the first residual image generated by the (m-i+1)th first feature transformation, specifically including: The third residual image is generated by extracting residual features from the superimposed image using the fourth residual block of the trained first neural network. The fourth residual block includes at least two fourth convolutional layers.
6. The method according to any one of claims 1 to 5, characterized in that, The weighting coefficient is between 0 and 4; The residual adjusted image is fused with the input image to obtain the target output image, specifically including: The residual adjusted image is superimposed on the input image to obtain the initial output image; Determine whether each pixel value in the initial output image is within a preset range. If all pixel values in the initial output image are within the preset range, then the initial output image is used as the target output image. If at least one pixel value is not within the preset range, adjust the corresponding pixel value to the preset range according to a preset rule, and the adjusted initial output image is used as the target output image.
7. The method according to claim 6, characterized in that, The preset rules include: For any pixel value in the initial output image, if the pixel value is less than the minimum value of the preset range, the pixel value is adjusted to the minimum value of the preset range; if the pixel value is greater than the maximum value of the preset range, the pixel value is adjusted to the maximum value of the preset range.
8. The method according to any one of claims 1 to 5, characterized in that, The method further includes: alternatingly training the first neural network to be trained and the discriminant network to be trained to obtain the trained first neural network and the trained discriminant network; The training of the discriminant network to be trained includes: The first sample image and sample weight coefficients are provided to the current first neural network so that the current first neural network outputs a first image with improved clarity. The first enhanced image and the original sample image corresponding to the first sample image are provided to the current discrimination network, and the parameters of the current discrimination network are adjusted according to the first loss function of the discrimination network so that the output of the parameter-tuned discrimination network can characterize whether the input of the discrimination network is the input of the first neural network or the discrimination result of the original sample image. The first neural network to be trained is trained, including: The second sample image and the sample weight coefficients are provided to the current first neural network to enable the first neural network to generate a second sharpness-enhanced image. The second sharpness-enhanced image is input into the parameter-tuned discriminant network so that the parameter-tuned discriminant network generates a second discrimination result based on the second sharpness-enhanced image; Based on the second loss function, the parameters of the current first neural network are adjusted to obtain the updated first neural network; In the process of alternating training of the first neural network to be trained and the discriminant network to be trained, the sample weight coefficient is a fixed value.
9. The method according to claim 8, characterized in that, The sample weight coefficient is 1.
10. The method according to claim 8, characterized in that, The first loss function includes the cross-entropy loss function.
11. The method according to claim 8, characterized in that, The second loss function includes the superposition of the first loss term, the second loss term, and the third loss term; The first loss term is obtained based on the adversarial loss of the second sample image. The second loss term is obtained based on the perceptual loss between the second sample image and its corresponding original sample image; The third loss term is obtained based on the L1 loss between the second sharpness-enhanced image and its corresponding original sample image.
12. The method according to claim 11, characterized in that, The second loss item is: ; in, The preset weights, This refers to the original sample image corresponding to the second enhanced image. The image is enhanced to the second level of clarity; ; To optimize the preset network layer in the preset network, The preset network layer is the number of layers in the preset optimized network. The number of channels in the output image of the preset network layer. The height of the output image of the preset network layer. The width of the output image of the preset network layer; When the original sample image corresponding to the second sharpness-enhanced image is input into the preset optimization network, the output result of the preset network layer; The output result of the preset network layer when the second enhanced image is input to the preset optimization network; both the first sample image and the second sample image are license plate images, and the preset optimization network adopts the LPR license plate recognition network.
13. An image processing apparatus, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer program is executed by the processor, it implements the image processing method according to any one of claims 1 to 12.
14. A non-transient computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the image processing method according to any one of claims 1 to 12.