A card image reflection correction method based on lightweight neural network

Abstract

This application provides a card image reflection correction method based on a lightweight neural network. The method constructs a card image reflection correction model, realizes adaptive correction of the reflective image through the encoding and decoding neural network structure in the card image reflection correction model, and uses a deep double convolution residual module in the encoding part to fully mine the contextual semantic information of the card image, so that the algorithm has good robustness while maintaining lightweight; in the decoding part, the low-level and high-level semantic information are effectively integrated through the cascade method of pixel-by-pixel addition, which not only ensures the algorithm's inference speed but also effectively improves the restoration quality of the card image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a card image reflection correction method based on a lightweight neural network. Background Art

[0002] ID cards, driver's licenses, vehicle registration certificates, and other documents are commonly used in daily life. Many businesses require the use of digital image analysis technology to quickly extract and recognize text on cards, facilitating the entry of key document information required for government approvals. Card optical character recognition (OCR) is a commonly used image recognition technology in card-related transactions. In practice, due to various complex environmental and camera factors, captured card images may exhibit distortion, deformation, and reflections. Specular reflections caused by the transparent plastic film on the card are particularly common. If these issues are not addressed, the subsequent accurate reading of the card information will be significantly affected. Therefore, card image correction technology has emerged. It can help standardize the captured card images and pre-process card images affected by lighting interference, camera angle distortion, noise pollution, and other factors, thereby enhancing the accuracy of subsequent card information recognition.

[0003] Existing methods for processing card image reflections mostly rely on scene statistics, sparse priors, and gradient smoothing to correct for specular reflections caused by interference from the transparent plastic film on the card. These traditional methods are simple to implement, but the correction effect is limited. Different thresholds often require manual setting for different card image types, and the algorithm's generalization performance is insufficient. To address this issue, researchers have proposed correction methods based on deep learning. Through large-sample training, deep learning-based correction methods have shown significant correction results and good robustness. However, these methods suffer from inference speed. Summary of the Invention

[0004] In order to solve the problem that existing methods for processing card image reflections cannot strike a balance between accuracy and speed, the present invention provides a card image reflection correction method based on a lightweight neural network, which can accurately extract the features of the reflective area and combine the global illumination of the card to perform image restoration, while taking into account both accuracy and processing speed.

[0005] The technical solution of the present invention is as follows: a card image reflection correction method based on a lightweight neural network, characterized in that it includes the following steps:

[0006] S1: Collect non-reflective card image samples, synthesize reflective images, and construct the reflective card artificial dataset D1;

[0007] S2: Build a card image reflection correction model;

[0008] The card image reflection correction model includes: an encoding network designed for card size and a decoding network based on feature cascade;

[0009] The encoding network achieves feature size scaling through convolution with a stride of 2, aggregating card contextual semantic information and rectifying the image during the scaling process;

[0010] The decoding network uses deconvolution with a stride of 2 to achieve feature size amplification. During the amplification process, the image details are gradually restored through feature cascading. The final model outputs a de-reflected card image.

[0011] The encoding network uses four identical dual convolution residual modules for feature extraction at the lowest layer, and the input and output feature sizes and number of channels of each dual convolution residual module are the same; the feature size of the encoding network output is 1 / 8 of the original image; the feature size of the final output of the decoding network is the same as the original image;

[0012] S3: Pre-training the card image reflectivity correction model using the reflective card artificial dataset D1 to obtain pre-trained model weights and the pre-trained card image reflectivity correction model;

[0013] S4: Collect real card reflective image samples and build a real reflective card dataset D2;

[0014] S5: fine-tuning the pre-trained card image reflection correction model using the D2 dataset to obtain model weights adapted to the real reflective card dataset and the trained card image reflection correction model;

[0015] S6: Convert the trained card image reflection correction model into a static image model for engineering deployment to perform reflection correction on card images.

[0016] It is further characterized by:

[0017] Step S1 specifically includes the following steps:

[0018] a1: Collect non-reflective card image samples I old and a reflective background image F;

[0019] a2: Combine the two according to formula I new =αI old +(1-α)F to synthesize and obtain the artificially synthesized reflection image pair (I old ,I new );

[0020] Where α is the synthesis effect adjustment threshold;

[0021] The encoding network includes: two 3x3 convolution modules with a stride of 1, three 3x3 convolution modules with a stride of 2, and four double convolution residual modules connected in sequence;

[0022] The decoding network includes: three 3x3 deconvolution modules with a stride of 2 connected in sequence and finally two 3x3 convolution modules with a stride of 1; wherein each deconvolution module is cascaded with the output features of the same-level module in the encoding network in a pixel-by-pixel cumulative manner;

[0023] In the double residual convolution module, the input feature F1 is extracted to feature F2 through a 3*3 convolution layer with a step size of 1. After F2 is spliced ​​with the input feature F1, feature F3 is extracted through a 3*3 convolution layer with a step size of 1. F3 is then spliced ​​with the input feature F1 to obtain the output feature of the double convolution residual module;

[0024] In step S3, the objective function F used in the training model is:

[0025] F=F p +λF C

[0026] Among them, F p is the structural loss, F c is the content loss, λ is the weight parameter;

[0027] Structural loss F p is defined as follows:

[0028]

[0029] Among them, S represents the true value image without reflection, An image representing the model's predicted output, Indicates the calculation of the gradient map in the x-axis direction, Indicates the calculation of the gradient map in the y-axis direction, Indicates the calculation of the 2-norm difference between the true image and the model prediction image, It represents the 1-norm difference between the true image and the model prediction image, and β is the weight parameter;

[0030] F c is the content loss, which is defined as follows:

[0031]

[0032] ψ l Represents the output features of the lth layer of the VGG19 network;

[0033] In step S1, the number of image samples of the reflective card artificial dataset D1 is greater than 3000;

[0034] In step S5, the number of image samples of the reflective card real data set D2 is greater than 200;

[0035] In step S6, when converting to a static graph model, a quantization operation is used to quantize the model parameters to 16 bits.

[0036] This application provides a card image reflection correction method based on a lightweight neural network. The method constructs a card image reflection correction model and implements adaptive correction of the reflection image through the encoding and decoding neural network structure within the card image reflection correction model. In the encoding phase, a deep dual-convolution residual module is used to fully exploit the contextual semantic information of the card image, enabling the algorithm to maintain lightweight while maintaining good robustness. In the decoding phase, a cascaded pixel-by-pixel addition method effectively fuses low-level and high-level semantic information, ensuring both algorithm inference speed and effectively improving the restored card image quality. The card image reflection correction method provided in this application can restore the brightness and color of the reflective area based on the global semantic information of the image to achieve overall visual balance. The correction effect is significant, the speed is fast, and the hardware requirements are low. The method can be embedded as an independent software module in card image acquisition devices such as high-definition cameras. It is particularly suitable as a preprocessing step for automated card image analysis to eliminate or reduce reflection interference caused by plastic film. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a schematic diagram of the structure of the card image reflection correction model in this application;

[0038] Figure 2 This is the structure diagram of the double convolution residual module. DETAILED DESCRIPTION

[0039] The present invention includes a card image reflection correction method based on a lightweight neural network, which includes the following steps.

[0040] S1: Collect non-reflective card image samples, synthesize reflective images, and construct the reflective card artificial dataset D1.

[0041] The specific steps include:

[0042] a1: Collect non-reflective card image samples I old and a reflective background image F;

[0043] a2: Combine the two according to formula I new =αI old +(1-α)F to synthesize and obtain the artificially synthesized reflection image pair (I old ,I new );

[0044] Here, α is the synthesis effect adjustment threshold, which is used to control the reflective effect. For example, based on experimental experience, α can be set to 0.8, or it can be set to a range of [0.6, 0.9]. In specific implementation, various non-reflective card image samples and various reflective background images of varying degrees and effects are collected and synthesized according to the synthesis formula. The resulting artificial reflective card dataset D1 contains more than 3,000 image samples. In this application, an artificially synthesized training dataset is used to adjust the threshold α based on the synthesis effect to control the range of image effects in the dataset. This ensures that a sufficient training dataset can be quickly obtained, thereby ensuring the effectiveness of the subsequent training of the card image reflective correction model.

[0045] S2: Build a card image reflection correction model;

[0046] like Figure 1 As shown in the figure, the card image reflection correction model includes: an encoding network designed for the card size and a decoding network based on feature cascade.

[0047] The encoding network achieves feature size scaling through convolution with a stride of 2. During the scaling process, it aggregates the card context semantic information and corrects the image. The decoding network uses deconvolution with a stride of 2 to achieve feature size magnification. During the magnification process, the image detail information is gradually restored through feature cascading. Finally, the model outputs the de-reflected card image.

[0048] The encoding network uses four identical double convolution residual modules for feature extraction at the lowest layer. The input and output features of each double convolution residual module have the same size and number of channels. The feature size of the encoding network output is 1 / 8 of the original image. The feature size of the decoding network's final output is the same as the original image.

[0049] The encoding network consists of two sequentially connected 3x3 convolutional modules with a stride of 1, three 3x3 convolutional modules with a stride of 2, and four double convolutional residual modules. An input image of size 512x800x3 is first processed through two 3x3 convolutional modules with a stride of 1 to obtain a 512x800x16 feature map. These two convolutional modules filter out some image noise. The image is then processed through three 3x3 convolutions with a stride of 2 to obtain a 64x100x128 feature map. These three convolutional modules aggregate image context and help extract different high-level semantic features of the image. Finally, four double convolutional residual modules are used to obtain the final 64x100x128 encoded feature map. These four double convolutional residual modules are used to repair reflective areas within a high-scale range.

[0050] The decoding network consists of three sequentially connected 3x3 deconvolution modules with a stride of 2, followed by two 3x3 convolution modules with a stride of 1. Each deconvolution module is concatenated pixel by pixel with the output features of the module at the same level in the encoding network. A 64x100x128 feature map is passed through three 3x3 deconvolution modules with a stride of 2 to produce a 512x800x16 feature map. Each deconvolution module is concatenated pixel by pixel with the output features of the module at the same level in the encoding network, gradually recovering image detail information. Finally, two 3x3 convolution modules with a stride of 1 are passed through to produce a 512x800x3 output feature map.

[0051] like Figure 2 As shown in the figure, in the double residual convolution module, the input feature F1 is extracted through a 3*3 convolution layer with a stride of 1 to obtain a feature F2 of size 64x100x128. F2 is concatenated with the input feature F1, and then the feature F3 of size 64x100x128 is extracted through a 3*3 convolution layer with a stride of 1. F3 is then concatenated with the input feature F1 to obtain the output feature of the double convolution residual module.

[0052] This application achieves adaptive correction of reflective images through an encoder-decoder neural network structure. In the encoding phase, a deep dual-convolution residual module is used to fully exploit the contextual semantic information of the card image, making the algorithm lightweight while maintaining good robustness. In the decoding phase, a cascaded pixel-by-pixel addition method effectively integrates low-level and high-level semantic information, ensuring both algorithm inference speed and effectively improving the quality of card image restoration.

[0053] S3: Use the reflective card artificial dataset D1 to pre-train the card image reflectivity correction model to obtain the pre-trained model weights and the pre-trained card image reflectivity correction model.

[0054] The objective function F used in the training model is:

[0055] F=F p +λF C

[0056] Among them, F p is the structural loss, F c is the content loss, λ is the weight parameter, and in this embodiment, the value is 4 based on the experimental results.

[0057] Structural loss F p is defined as follows:

[0058]

[0059] Among them, S represents the true value image without reflection, An image representing the model's predicted output, Indicates the calculation of the gradient map in the x-axis direction, Indicates the calculation of the gradient map in the y-axis direction, Indicates the calculation of the 2-norm difference between the true image and the model prediction image, It represents the 1-norm difference between the true image and the model-predicted image, and β is a weight parameter. In this embodiment, the value is 2 based on experimental experience.

[0060] F c is the content loss, which is defined as follows:

[0061]

[0062] ψ l Represents the output features of layer 1 of the VGG19 network. In practice, the VGG19 network is pre-trained on the ImageNet dataset, and its model parameters are obtained for use in the perceptual feature calculations of the present invention. The VGG19 network is not part of the model structure of the present invention, and its model weights do not need to be updated during training.

[0063] S4: Collect real card reflective image samples to construct a real reflective card dataset D2; the number of image samples in the real reflective card dataset D2 is greater than 200.

[0064] S5: Use the D2 dataset to fine-tune the pre-trained card image reflection correction model to obtain the model weights adapted to the real data set of reflective cards and the trained card image reflection correction model.

[0065] S6: The trained card image reflection correction model is a neural network dynamic graph model. During implementation, the card image reflection correction model is converted into a static graph model for engineering deployment to achieve reflection correction of card images.

[0066] When converting to a static graph model, the quantization operation is used to quantize the model parameter accuracy to 16 bits, thereby reducing the model size and speeding up the model inference.

[0067] After using the technical solution of the present application, it can quickly repair the image reflection problem caused by transparent materials such as plastic film during card collection, with good correction effect and strong robustness. The card image reflection correction model uses an encoding and decoding network structure for feature extraction, in which the encoding network is used to extract semantic information at different levels, which can fully explore the global semantic features of the image, and the decoding network is used to restore image details layer by layer; the entire model uses carefully defined structural loss and content loss functions, and uses image data before and after reflection correction for targeted training. It can accurately repair reflective areas at the pixel level and has strong robustness to shooting angles and noise interference. This method has a fast correction speed and supports fast inference under the CPU. In order to improve the model inference speed, the card image reflection correction model structure does not use a pre-trained basic model, but is implemented using a lightweight encoding and decoding network. The convolution modules all use small-sized 3x3 convolution kernels and a small number of channels. 16-bit quantization is used when finally converted into a static image, further reducing the number of model parameters. At the same time, this method has low requirements for the hardware deployment environment. Because the card image reflection correction model in this method is lightweight, it can be used even without graphics accelerators such as GPUs, NPUs, or TPUs. This facilitates the development of small, standardized card acquisition and recognition devices, thereby reducing overall system costs. The card image reflection correction method provided by this invention can be embedded as a standalone software module in card image acquisition devices such as high-definition scanners. It is particularly suitable as a preprocessing step for automated card image analysis, providing an efficient and reliable solution for standardized card image acquisition, and possesses significant value and significance in practical applications.

Claims

1. A card image reflection correction method based on a lightweight neural network, characterized in that: It includes the following steps: S1: Collect non-reflective card image samples, synthesize reflective images, and construct the reflective card artificial dataset D1; S2: Build a card image reflection correction model; The card image reflection correction model includes: an encoding network designed for card size and a decoding network based on feature cascade; The encoding network achieves feature size scaling through convolution with a stride of 2, aggregating card contextual semantic information and rectifying the image during the scaling process; The decoding network uses deconvolution with a stride of 2 to achieve feature size amplification. During the amplification process, the image details are gradually restored through feature cascading. The final model outputs a de-reflected card image. The encoding network uses four identical dual convolution residual modules for feature extraction at the lowest layer, and the input and output feature sizes and number of channels of each dual convolution residual module are the same; the feature size of the encoding network output is 1 / 8 of the original image; the feature size of the final output of the decoding network is the same as the original image; S3: Pre-training the card image reflectivity correction model using the reflective card artificial dataset D1 to obtain pre-trained model weights and the pre-trained card image reflectivity correction model; S4: Collect real card reflective image samples and build a real reflective card dataset D2; S5: fine-tuning the pre-trained card image reflection correction model using the D2 dataset to obtain model weights adapted to the real reflective card dataset and the trained card image reflection correction model; S6: Convert the trained card image reflection correction model into a static image model for engineering deployment to perform reflection correction on card images.

2. The card image reflection correction method based on a lightweight neural network according to claim 1, characterized in that: In step S1, specifically The following steps are involved: a1: Collect non-reflective card image samples I old and a reflective background image F; a2: Combine the two according to formula I new =αI old +(1-α)F to synthesize and obtain the artificially synthesized reflection image pair (I old ,I new ); Where α is the synthesis effect adjustment threshold.

3. The card image reflection correction method based on a lightweight neural network according to claim 1, characterized in that: The encoding network includes: two 3x3 convolution modules with a step size of 1, three 3x3 convolution modules with a step size of 2, and four double convolution residual modules connected in sequence.

4. The card image reflection correction method based on a lightweight neural network according to claim 1, characterized in that: The decoding network includes: three 3x3 deconvolution modules with a step size of 2 connected in sequence and finally two 3x3 convolution modules with a step size of 1; wherein, each deconvolution module is cumulatively cascaded with the output features of the same-level module in the encoding network in a pixel-by-pixel manner.

5. The card image reflection correction method based on a lightweight neural network according to claim 1, characterized in that: In the double residual convolution module, the input feature F1 is extracted to feature F2 through a 3*3 convolution layer with a step size of 1. F2 is concatenated with the input feature F1, and then the feature F3 is extracted through a 3*3 convolution layer with a step size of 1. F3 is then concatenated with the input feature F1 to obtain the output feature of the double convolution residual module.

6. The card image reflection correction method based on a lightweight neural network according to claim 1, characterized in that: In step S3, the objective function F used in the training model is: F=F p +λF C Among them, F p is the structural loss, F c is the content loss, λ is the weight parameter; Structural loss F p is defined as follows: Among them, S represents the true value image without reflection, An image representing the model's predicted output, Indicates the calculation of the gradient map in the x-axis direction, Indicates the calculation of the gradient map in the y-axis direction, Indicates the calculation of the 2-norm difference between the true image and the model prediction image, It represents the 1-norm difference between the true image and the model prediction image, and β is the weight parameter; F c is the content loss, which is defined as follows: ψ l Represents the output features of the lth layer of the VGG19 network.

7. The card image reflection correction method based on a lightweight neural network according to claim 1, characterized in that: In step S1, the number of image samples of the reflective card artificial dataset D1 is greater than 3,000.

8. The card image reflection correction method based on a lightweight neural network according to claim 1, characterized in that: In step S5, the number of image samples of the reflective card real data set D2 is greater than 200.

9. The card image reflection correction method based on a lightweight neural network according to claim 1, characterized in that: In step S6, when converting to a static graph model, a quantization operation is used to quantize the model parameters to 16 bits.

Images