A method, system, apparatus and medium for processing card and document images.
By using a lightweight backbone network and multi-task joint learning, the problems of complex processes and error accumulation in card and document image processing are solved, achieving efficient and accurate image processing that is suitable for mobile and embedded devices.
Patent Information
- Application Number
- CN202511463237.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-14
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-10-14
AI Technical Summary
Existing card and document image processing technologies suffer from problems such as complex processes, scattered tasks, error accumulation, complex network structures that are difficult to scale up and enable multi-task joint learning, resulting in long processing times, low accuracy, and difficulty in efficient deployment in diverse application scenarios.
We employ a lightweight backbone network, LCENet, and construct corner regression and orientation classification network branches through a multi-task joint learning mechanism. We use a joint loss function for optimization to achieve corner regression and orientation classification for card and document images, supporting simultaneous operations within a single network.
It significantly reduces inference latency, improves geometric correction accuracy and recognition robustness, supports real-time operation on mobile and embedded devices, and can accurately process various card types and multi-angle images to meet the diverse application needs of government affairs, finance and other sectors.
Smart Images

Figure CN121366421B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, specifically to a method, system, apparatus, and medium for processing image data of cards and documents. Background Technology
[0002] With the rapid digitalization of various online and offline businesses, it has become commonplace for users to take pictures of ID cards, business licenses, invoices, and other documents using mobile phones, scanning devices, and other means. However, due to the complex shooting environment, document images often suffer from perspective distortion, tilting, rotation, and misalignment, which seriously affects the accuracy of subsequent text recognition and information extraction.
[0003] To address the aforementioned issues, academia and industry have proposed various technical solutions for geometric correction and orientation determination of card and document images. However, most current solutions still rely on a multi-stage sequential process of "corner detection—perspective correction—independent orientation classification." While current technologies have made some progress in single tasks such as corner detection, perspective correction, or orientation classification in the field of card and document image processing, they still suffer from the following shortcomings: First, the process is complex and the tasks are scattered, often requiring multiple independent models to be processed in series, resulting in a long overall inference time. Second, there is a lack of feature sharing between corner detection and orientation determination, which easily leads to error accumulation and affects subsequent recognition accuracy. Third, many existing methods have complex network structures and lack support for lightweight and multi-task joint learning, making it difficult to deploy efficiently under different computing power environments. Fourth, the granularity of orientation determination is limited, making it difficult to accurately address the problem of card and document rotation from multiple angles and in multiple formats in actual shooting scenarios.
[0004] Therefore, there is an urgent need to propose a technical solution based on a lightweight backbone network, which can simultaneously achieve corner regression and orientation classification of card and document images within a single network through a multi-task joint learning mechanism, and has fast and accurate reasoning capabilities, so as to meet the high-precision and high-efficiency requirements of diverse application scenarios. Summary of the Invention
[0005] This invention addresses the needs and shortcomings of current technological development by providing a method, system, device, and medium for processing card and document images.
[0006] In a first aspect, the present invention provides a method for processing image data of cards and documents, and the technical solution adopted to solve the above-mentioned technical problems is as follows:
[0007] A method for processing card and document images, comprising the following steps:
[0008] S1. Obtain the image of the card or document, and preprocess the obtained image to generate a standard image;
[0009] S2. Construct a lightweight multi-task model, including: a lightweight convolutional neural network backbone based on LCENet, used to extract multi-scale shared feature maps from standard images; a corner regression network branch, used to regress the two-dimensional coordinates of the four corner points of the standard image based on the shared feature maps; and an orientation classification network branch, used to predict the orientation category of the standard image based on the shared feature maps.
[0010] S3. Train a lightweight multi-task model using a pre-prepared dataset. During training: a) Introduce perspective perturbation and random rotation strategies to simulate geometric distortions in real-world scenes and enhance the anti-interference ability of the orientation classification network branch against complex poses; b) Employ a joint loss function, using smooth L1 loss for the corner regression branch and cross-entropy loss for the orientation classification branch; c) The joint loss function is a weighted combination of corner regression loss and orientation classification loss, with the weight coefficients dynamically adjusted based on task errors during training to achieve collaborative optimization of the two tasks.
[0011] S4. Load the lightweight multi-task model trained in step S3, and input the standard image generated in step S1 into the model for inference: extract multi-scale shared feature maps through the lightweight convolutional neural network backbone based on LCENet, and then output the two-dimensional coordinates of the four corner points by the corner regression network branch and the direction classification network branch outputs the image direction category.
[0012] S5. Construct a perspective transformation matrix using the two-dimensional coordinates of the four corner points from the corner regression network branch regression, and correct perspective distortion in the original image; rotate and output the image after perspective distortion correction based on the output of the orientation classification network branch.
[0013] Optionally, the preprocessing operations performed on the acquired image may specifically include:
[0014] First, the image resolution is uniformly adjusted to the preset size through size normalization;
[0015] Subsequently, mean and variance normalization is performed to eliminate differences in pixel value magnitudes and adjust the pixel distribution to a standard distribution that meets the requirements of subsequent input.
[0016] The final result is a standard image.
[0017] Optionally, the pre-prepared dataset includes images of different types of cards and documents, covering different shooting angles, lighting, and background scenes. Each image is labeled with the real two-dimensional coordinates of the four corner points and the real direction category.
[0018] We perform random rotation, scaling, and noise augmentation on the images in the dataset and use them to train a lightweight multi-task model.
[0019] Optionally, the corner regression network branch involved includes convolutional layers and fully connected layers, which are used to take the shared feature map as input, extract local features and map coordinate information, and output the two-dimensional coordinates of the four corner points of the corresponding standard image, so as to provide the positional basis for subsequent image perspective correction.
[0020] The orientation classification network involved includes convolutional layers, global average pooling layers, and fully connected layers. These layers take shared feature maps as input and, through feature abstraction, global information aggregation, and category mapping, output the orientation category of the corresponding standard image, including four categories: 0°, 90°, 180°, and 270°. This provides a basis for subsequent image rotation adjustments.
[0021] Secondly, the present invention provides a card and document image processing system, and the technical solution adopted to solve the above-mentioned technical problems is as follows:
[0022] A card and document image processing system, used to implement the card and document image processing method as described in the first aspect, specifically includes:
[0023] Image acquisition module, used to acquire images of cards or documents;
[0024] The image preprocessing module is used to preprocess the acquired images and generate standard images;
[0025] The model building module is used to build lightweight multi-task models, including: a lightweight convolutional neural network backbone based on LCENet, used to extract multi-scale shared feature maps from standard images; a corner regression network branch, used to regress the two-dimensional coordinates of the four corner points of the standard image based on the shared feature maps; and an orientation classification network branch, used to predict the orientation category of the standard image based on the shared feature maps.
[0026] The model training module is used to train a lightweight multi-task model using a pre-prepared dataset. During training: a) perspective perturbation and random rotation strategies are introduced to simulate geometric distortions in real-world scenes, enhancing the anti-interference ability of the orientation classification network branch against complex poses; b) a joint loss function is adopted, using smooth L1 loss for the corner regression branch and cross-entropy loss for the orientation classification branch; c) the joint loss function is a weighted combination of corner regression loss and orientation classification loss, and the weight coefficients are dynamically adjusted based on task errors during training to achieve collaborative optimization of the two tasks.
[0027] The loading and calling module is used to load the trained lightweight multi-task model and input the standard image into the model for inference: multi-scale shared feature maps are extracted through the lightweight convolutional neural network backbone based on LCENet, and then the corner regression network branch outputs the two-dimensional coordinates of the four corner points, and the direction classification network branch outputs the image direction category.
[0028] The image processing module first constructs a perspective transformation matrix using the two-dimensional coordinates of the four corner points regressed by the corner regression network branch, corrects perspective distortion in the original image, and then rotates the perspective-distortion-corrected image based on the output of the orientation classification network branch.
[0029] The image output module is used to output images of cards or documents that have been corrected for perspective distortion and are oriented correctly.
[0030] Optionally, the image preprocessing modules involved specifically include:
[0031] Normalization unit one is used to normalize the size of the acquired image and adjust the image resolution to a preset size.
[0032] Normalization unit two is used to normalize the mean and variance of the image after size normalization, eliminate the difference in pixel value magnitude, adjust the pixel distribution to a standard distribution that meets the requirements of subsequent input, and obtain a standard image.
[0033] Optionally, the pre-prepared dataset includes images of different types of cards and documents, covering different shooting angles, lighting, and background scenes. Each image is labeled with the real two-dimensional coordinates of the four corner points and the real direction category.
[0034] We perform random rotation, scaling, and noise augmentation on the images in the dataset and use them to train a lightweight multi-task model.
[0035] Optionally, the corner regression network branch involved includes convolutional layers and fully connected layers, which are used to take the shared feature map as input, extract local features and map coordinate information, and output the two-dimensional coordinates of the four corner points of the corresponding standard image, so as to provide the positional basis for subsequent image perspective correction.
[0036] The orientation classification network involved includes convolutional layers, global average pooling layers, and fully connected layers. These layers take shared feature maps as input and, through feature abstraction, global information aggregation, and category mapping, output the orientation category of the corresponding standard image, including four categories: 0°, 90°, 180°, and 270°. This provides a basis for subsequent image rotation adjustments.
[0037] Thirdly, the present invention provides a card and document image processing device, characterized in that it includes: a memory and at least one processor;
[0038] The memory contains computer programs;
[0039] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the card and document image processing method as described in the first aspect.
[0040] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that can be executed by a processor to implement the card and document image processing method as described in the first aspect.
[0041] The card and document image processing method, system, apparatus, and medium of the present invention have the following advantages compared with the prior art:
[0042] 1. This invention can realize the processing of card and certificate images from shooting input to perspective correction and orientation determination. During the processing, a lightweight convolutional neural network based on LCENet is used to extract multi-scale shared feature maps. At the same time, a corner regression network branch is used to regress the two-dimensional coordinates of the four corner points of the standard image, and an orientation classification network branch is used to predict the orientation category of the standard image. This can significantly reduce the inference latency of the lightweight multi-task model and support real-time operation on mobile or embedded devices.
[0043] 2. This invention reduces the negative impact of corner point errors on direction determination through multi-task joint learning and feature sharing, effectively suppresses error cascading, and improves geometric correction accuracy and overall recognition robustness. It supports rapid processing of various card and certificate types and multi-angle images, and can accurately output the four corner coordinates and direction labels, providing a high-quality preprocessing foundation for subsequent OCR and structured recognition, meeting the diversified application needs of government affairs, finance, self-service and other fields.
[0044] 3. This invention achieves corner regression and orientation classification within a single model through a lightweight backbone network, multi-task joint learning, and a dynamic loss balancing mechanism. Compared with the existing "corner detection—perspective correction—independent orientation classification" cascade method, it has the following advantages: significantly reduces inference time by an average of about 30%; improves orientation classification accuracy by more than 6%; reduces corner localization error by nearly 20% and suppresses error cascading; has adaptability to mobile and embedded edge devices, enabling real-time inference in low-power environments; and provides a high-quality preprocessing foundation for subsequent OCR and structured recognition, meeting the diverse needs of government, finance, and self-service terminals. Attached Figure Description
[0045] Appendix Figure 1 This is a flowchart of the method according to Embodiment 1 of the present invention;
[0046] Appendix Figure 2 This is a module connection block diagram of Embodiment 2 of the present invention.
[0047] Appendix Figure 3 This is a flowchart illustrating the training process of the lightweight multi-task model in this embodiment of the invention. Detailed Implementation
[0048] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.
[0049] Example 1:
[0050] Combined with appendix Figure 1 This embodiment proposes a card and document image processing method, which includes the following steps:
[0051] S1. Obtain the image of the card or document, and preprocess the obtained image to generate a standard image.
[0052] The card or document images obtained in this embodiment are not limited to the original images of ID cards, business licenses, and invoices obtained through a mobile terminal camera or a high-speed scanner.
[0053] The preprocessing operations performed on the acquired images specifically include:
[0054] First, the image resolution is uniformly adjusted to the preset size through size normalization;
[0055] Subsequently, mean and variance normalization is performed to eliminate differences in pixel value magnitudes and adjust the pixel distribution to a standard distribution that meets the requirements of subsequent input.
[0056] The final result is a standard image that can be adapted to the subsequent lightweight convolutional neural network backbone.
[0057] S2. Construct a lightweight multi-task model, including: a lightweight convolutional neural network backbone based on LCENet, used to extract multi-scale shared feature maps from standard images; a corner regression network branch, used to regress the two-dimensional coordinates of the four corner points of the standard image based on the shared feature maps; and an orientation classification network branch, used to predict the orientation category of the standard image based on the shared feature maps.
[0058] Reference Appendix Figure 3 The lightweight convolutional neural network backbone based on LCENet in this embodiment includes a stem convolutional layer, Stage 1 (low-level features), Stage 2 (mid-level features), and Stage 3 (high-level semantic features), and its output is a shared feature map. The shared feature map serves as the input to both the corner regression network branch and the orientation classification network branch.
[0059] The corner regression network branch in this embodiment includes convolutional layers (Conv) and fully connected layers (FC), which are used to take the shared feature map as input, extract local features and map coordinate information, and output the two-dimensional coordinates of the four corner points of the corresponding standard image, providing a positional basis for subsequent image perspective correction.
[0060] The orientation classification network branch in this embodiment includes a convolutional layer (Conv), a global average pooling layer (GAP), and a fully connected layer (FC). It takes a shared feature map as input and outputs the orientation category of the corresponding standard image through feature abstraction, global information aggregation, and category mapping. The categories include 0°, 90°, 180°, and 270°, providing a basis for subsequent image rotation adjustment.
[0061] S3. Train a lightweight multi-task model using a pre-prepared dataset. During training: a) Introduce enhancement operations such as random rotation (±30°), perspective perturbation (±15%), and noise addition to simulate geometric distortions in real-world scenes and enhance the anti-interference ability of the orientation classification network branch against complex poses; b) Employ a joint loss function, using smoothed L1 loss for the corner regression branch and cross-entropy loss for the orientation classification branch; c) The joint loss function is a weighted combination of corner regression loss and orientation classification loss, with the weight coefficients dynamically adjusted based on task errors during training to achieve collaborative optimization of the two tasks.
[0062] The joint loss function is expressed as follows:
[0063] L_total=Loss_reg+λ*Loss_cls;
[0064] In the formula, Loss_reg represents the corner regression loss, and Loss_cls represents the orientation classification loss;
[0065] During training, the weights λ are dynamically adjusted based on real-time task errors to balance corner location accuracy and orientation classification accuracy, thereby improving the overall robustness of the model.
[0066] S4. Load the lightweight multi-task model trained in step S3, and input the standard image generated in step S1 into the model for inference: extract multi-scale shared feature maps through the lightweight convolutional neural network backbone based on LCENet, and then output the two-dimensional coordinates of the four corner points by the corner regression network branch and the direction classification network branch outputs the image direction category.
[0067] S5. Construct a perspective transformation matrix using the two-dimensional coordinates of the four corner points regressed by the corner regression network branch to correct perspective distortion in the original image; based on the output of the orientation classification network branch, rotate and output the perspective-distortion-corrected image for subsequent OCR, information extraction, or risk control algorithms.
[0068] Experimental results show that, compared with existing single-task concatenation methods, the method in this embodiment improves the orientation classification accuracy by more than 6%, reduces the average corner positioning error by nearly 20%, and shortens the inference time per image by about 30%, which can meet the real-time processing requirements of mobile and embedded devices.
[0069] Example 2:
[0070] Combined with appendix Figure 2 This embodiment proposes a card and document image processing system, which is suitable for deployment on mobile or embedded edge devices and enables low-latency inference in computing-constrained environments. It is used to implement the card and document image processing method as described in the first aspect, specifically including:
[0071] Image acquisition module, used to acquire images of cards or documents;
[0072] The image preprocessing module is used to preprocess the acquired images and generate standard images;
[0073] The model building module is used to build lightweight multi-task models, including: a lightweight convolutional neural network backbone based on LCENet, used to extract multi-scale shared feature maps from standard images; a corner regression network branch, used to regress the two-dimensional coordinates of the four corner points of the standard image based on the shared feature maps; and an orientation classification network branch, used to predict the orientation category of the standard image based on the shared feature maps.
[0074] The model training module is used to train a lightweight multi-task model using a pre-prepared dataset. During training: a) perspective perturbation and random rotation strategies are introduced to simulate geometric distortions in real-world scenes, enhancing the anti-interference ability of the orientation classification network branch against complex poses; b) a joint loss function is adopted, using smooth L1 loss for the corner regression branch and cross-entropy loss for the orientation classification branch; c) the joint loss function is a weighted combination of corner regression loss and orientation classification loss, and the weight coefficients are dynamically adjusted based on task errors during training to achieve collaborative optimization of the two tasks.
[0075] The loading and calling module is used to load the trained lightweight multi-task model and input the standard image into the model for inference: multi-scale shared feature maps are extracted through the lightweight convolutional neural network backbone based on LCENet, and then the corner regression network branch outputs the two-dimensional coordinates of the four corner points, and the direction classification network branch outputs the image direction category.
[0076] The image processing module first constructs a perspective transformation matrix using the two-dimensional coordinates of the four corner points regressed by the corner regression network branch, corrects perspective distortion in the original image, and then rotates the perspective-distortion-corrected image based on the output of the orientation classification network branch.
[0077] The image output module is used to output images of cards or documents that have been corrected for perspective distortion and are oriented correctly.
[0078] In this embodiment, the image preprocessing module specifically includes:
[0079] Normalization unit one is used to normalize the size of the acquired image and adjust the image resolution to a preset size.
[0080] Normalization unit two is used to normalize the mean and variance of the image after size normalization, eliminate the difference in pixel value magnitude, adjust the pixel distribution to a standard distribution that meets the requirements of subsequent input, and obtain a standard image.
[0081] Reference Appendix Figure 3 The lightweight convolutional neural network backbone based on LCENet in this embodiment includes a stem convolutional layer, Stage 1 (low-level features), Stage 2 (mid-level features), and Stage 3 (high-level semantic features), and its output is a shared feature map. The shared feature map serves as the input to both the corner regression network branch and the orientation classification network branch.
[0082] The corner regression network branch in this embodiment includes convolutional layers (Conv) and fully connected layers (FC), which are used to take the shared feature map as input, extract local features and map coordinate information, and output the two-dimensional coordinates of the four corner points of the corresponding standard image, providing a positional basis for subsequent image perspective correction.
[0083] The orientation classification network branch in this embodiment includes a convolutional layer (Conv), a global average pooling layer (GAP), and a fully connected layer (FC). It takes a shared feature map as input and outputs the orientation category of the corresponding standard image through feature abstraction, global information aggregation, and category mapping. The categories include 0°, 90°, 180°, and 270°, providing a basis for subsequent image rotation adjustment.
[0084] In this embodiment, the pre-prepared dataset includes over 100,000 images of different types of cards and documents, covering various shooting angles, lighting conditions, and background scenes. Each image is labeled with the true 2D coordinates of its four corner points and its true orientation category. The images in the dataset undergo random rotation, scaling, and noise-adding data augmentation operations, and are then used to train a lightweight multi-task model.
[0085] The joint loss function is expressed as follows:
[0086] L_total=Loss_reg+λ*Loss_cls;
[0087] In the formula, Loss_reg represents the corner regression loss, and Loss_cls represents the orientation classification loss;
[0088] During training, the weights λ are dynamically adjusted based on real-time task errors to balance corner location accuracy and orientation classification accuracy, thereby improving the overall robustness of the model.
[0089] The system described in this embodiment can be deployed and run on mobile or embedded edge devices. Even in environments with limited computing power, it can still achieve an inference time of less than 25ms for a single image, meeting the real-time application needs of government, finance, and self-service terminals.
[0090] Example 3:
[0091] This embodiment proposes a card and document image processing device, which is suitable for deployment on mobile terminals or embedded edge devices and enables low-latency inference in computing-constrained environments. Specifically, it includes: a memory and at least one processor.
[0092] The memory contains computer programs;
[0093] The processor can be a CPU, GPU, FPGA, NPU, etc., adapted to mobile SoCs and edge AI modules; the memory can be eMMC, UFS, or other non-volatile storage media, supporting lightweight model calls. This device can run on embedded systems such as Android and Linux, facilitating deployment on self-service terminals or mobile devices.
[0094] Example 4:
[0095] This embodiment proposes a computer-readable storage medium storing a computer program that can be executed by a processor to implement the card and document image processing method as described in Embodiment 1. The program can be provided through local installation or downloaded from the cloud, and supports OTA online updates to optimize model parameters according to business scenarios. The program can run on lightweight inference frameworks (such as TensorRT Lite and NCNN), ensuring low power consumption and real-time performance.
[0096] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.
[0097] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.
[0098] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0099] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.
[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for processing image data of cards and documents, characterized in that, Includes the following steps: S1. Obtain the image of the card or document, and preprocess the obtained image to generate a standard image; S2. Construct a lightweight multi-task model, including: a lightweight convolutional neural network backbone based on LCENet, used to extract multi-scale shared feature maps from standard images; a corner regression network branch, used to regress the two-dimensional coordinates of the four corners of the standard image based on the shared feature maps; and an orientation classification network branch, used to predict the orientation category of the standard image based on the shared feature maps. The lightweight convolutional neural network backbone based on LCENet includes a stem convolutional layer, Stage 1 for extracting low-level features, Stage 2 for extracting mid-level features, and Stage 3 for extracting high-level semantic features. Its output is a shared feature map, which is also used as the input to the corner regression network branch and the orientation classification network branch. S3. Train a lightweight multi-task model using a pre-prepared dataset. During training: a) Introduce perspective perturbation and random rotation strategies to simulate geometric distortions in real-world scenes and enhance the anti-interference ability of the orientation classification network branch against complex poses; b) Employ a joint loss function, using smooth L1 loss for the corner regression branch and cross-entropy loss for the orientation classification branch; c) The joint loss function is a weighted combination of corner regression loss and orientation classification loss, with the weight coefficients dynamically adjusted based on task errors during training to achieve collaborative optimization of the two tasks. S4. Load the lightweight multi-task model trained in step S3, and input the standard image generated in step S1 into the model for inference: extract multi-scale shared feature maps through the lightweight convolutional neural network backbone based on LCENet, and then output the two-dimensional coordinates of the four corner points by the corner regression network branch and the direction classification network branch outputs the image direction category. S5. Construct a perspective transformation matrix using the two-dimensional coordinates of the four corner points from the corner regression network branch regression, and correct perspective distortion in the original image; rotate and output the image after perspective distortion correction based on the output of the orientation classification network branch.
2. The card and document image processing method according to claim 1, characterized in that, The preprocessing operations performed on the acquired images specifically include: First, the image resolution is uniformly adjusted to the preset size through size normalization; Subsequently, mean and variance normalization is performed to eliminate differences in pixel value magnitudes and adjust the pixel distribution to a standard distribution that meets the requirements of subsequent input. The final standard image is obtained.
3. The card and document image processing method according to claim 1, characterized in that, The pre-prepared dataset includes images of different types of cards and documents, covering different shooting angles, lighting, and background scenes. Each image is labeled with the real two-dimensional coordinates of the four corner points and the real direction category. We perform random rotation, scaling, and noise augmentation on the images in the dataset and use them to train a lightweight multi-task model.
4. The card and document image processing method according to claim 1, characterized in that, The corner regression network branch includes convolutional layers and fully connected layers, which are used to take the shared feature map as input, extract local features and map coordinate information, and output the two-dimensional coordinates of the four corner points of the corresponding standard image, so as to provide a positional basis for subsequent image perspective correction. The orientation classification network branch includes convolutional layers, global average pooling layers, and fully connected layers. It takes a shared feature map as input and outputs the orientation category of the corresponding standard image through feature abstraction, global information aggregation, and category mapping. The categories include 0°, 90°, 180°, and 270°, providing a basis for subsequent image rotation adjustments.
5. A card and document image processing system, characterized in that, It is used to implement the card and document image processing method as described in any one of claims 1-4, specifically including: Image acquisition module, used to acquire images of cards or documents; The image preprocessing module is used to preprocess the acquired images and generate standard images. The model building module is used to construct a lightweight multi-task model, including: a lightweight convolutional neural network backbone based on LCENet, used to extract multi-scale shared feature maps from standard images; a corner regression network branch, used to regress the two-dimensional coordinates of the four corners of the standard image based on the shared feature maps; and an orientation classification network branch, used to predict the orientation category of the standard image based on the shared feature maps. The lightweight convolutional neural network backbone based on LCENet includes a stem convolutional layer, Stage 1 for extracting low-level features, Stage 2 for extracting mid-level features, and Stage 3 for extracting high-level semantic features. Its output is a shared feature map, which is also used as the input to the corner regression network branch and the orientation classification network branch. The model training module is used to train a lightweight multi-task model using a pre-prepared dataset. During training: a) perspective perturbation and random rotation strategies are introduced to simulate geometric distortions in real-world scenes, enhancing the anti-interference ability of the orientation classification network branch against complex poses; b) a joint loss function is adopted, using smooth L1 loss for the corner regression branch and cross-entropy loss for the orientation classification branch; c) the joint loss function is a weighted combination of corner regression loss and orientation classification loss, with the weight coefficients dynamically adjusted based on task errors during training to achieve collaborative optimization of the two tasks. The loading and calling module is used to load the trained lightweight multi-task model and input the standard image into the model for inference: multi-scale shared feature maps are extracted through the lightweight convolutional neural network backbone based on LCENet, and then the corner regression network branch outputs the two-dimensional coordinates of the four corner points, and the direction classification network branch outputs the image direction category. The image processing module first constructs a perspective transformation matrix using the two-dimensional coordinates of the four corner points regressed by the corner regression network branch, corrects perspective distortion in the original image, and then rotates the perspective-distortion-corrected image based on the output of the orientation classification network branch. The image output module is used to output images of cards or documents that have been corrected for perspective distortion and are oriented correctly.
6. The card and document image processing system according to claim 5, characterized in that, The image preprocessing module specifically includes: Normalization unit one is used to normalize the size of the acquired image and adjust the image resolution to a preset size. Normalization unit two is used to normalize the mean and variance of the image after size normalization, eliminate the difference in pixel value magnitude, adjust the pixel distribution to a standard distribution that meets the requirements of subsequent input, and obtain a standard image.
7. The card and document image processing system according to claim 5, characterized in that, The pre-prepared dataset includes images of different types of cards and documents, covering different shooting angles, lighting, and background scenes. Each image is labeled with the real two-dimensional coordinates of the four corner points and the real direction category. We perform random rotation, scaling, and noise augmentation on the images in the dataset and use them to train a lightweight multi-task model.
8. The card and document image processing system according to claim 5, characterized in that, The corner regression network branch includes convolutional layers and fully connected layers, which are used to take the shared feature map as input, extract local features and map coordinate information, and output the two-dimensional coordinates of the four corner points of the corresponding standard image, so as to provide a positional basis for subsequent image perspective correction. The orientation classification network branch includes convolutional layers, global average pooling layers, and fully connected layers. It takes a shared feature map as input and outputs the orientation category of the corresponding standard image through feature abstraction, global information aggregation, and category mapping. The categories include 0°, 90°, 180°, and 270°, providing a basis for subsequent image rotation adjustments.
9. A card and document image processing device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the card and document image processing method as described in any one of claims 1-4.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the card and document image processing method as described in any one of claims 1-4.
Citation Information
Patent Citations
Multi-task detection identity card image correction method and system
CN113962906A
Document image correction method and device based on multi-task learning
CN117727039A
