Card identification method, card identification model training method and electronic device

Abstract

The application provides a card identification method, a card identification model training method and an electronic device. Specifically, the method can include: using the corner point position of a card in an image to eliminate distortion of the card to obtain a processed image; using the features of the processed image to determine a card template corresponding to the card; using the card template to determine key information in the processed image; and using the key information as the identification result of the card. According to the embodiments of the application, the accuracy of card identification can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a card recognition method, a card recognition model training method, and an electronic device. Background Technology

[0002] In many scenarios, there is a need to automatically extract structured information from cards and certificates for further data entry. These scenarios often present several issues that make it difficult to guarantee recognition accuracy. For example, the accuracy may be reduced due to the limitations of card and certificate photography. Another example is the constraint of the computing power of the card and certificate recognition algorithm, which can also lead to decreased accuracy. Summary of the Invention

[0003] This application provides a card recognition method, a card recognition model training method, a card recognition device, a card recognition model training device, an electronic device, and a storage medium. Through a universal card recognition template, different types of cards can be recognized.

[0004] In a first aspect, embodiments of this application provide a card identification method, which may include the following steps:

[0005] By utilizing the corner points of the card in the image, distortion removal processing is performed on the card to obtain the processed image;

[0006] Using the overall features of the processed image, the card template corresponding to the card is determined;

[0007] Using the card template, key information in the processed image is determined; the key information is used as the recognition result of the card.

[0008] Secondly, embodiments of this application provide a method for training a card recognition model, which may include the following steps:

[0009] Card samples are generated using materials from a resource library; the materials in the resource library are generated through open-source code or obtained through public channels.

[0010] The card sample is processed into an image to obtain an image sample containing the card sample;

[0011] The card recognition model is trained using the image sample containing the card sample; the card recognition model is used to perform distortion removal processing on the card sample using the corner positions of the card sample in the image sample containing the card sample, to obtain a processed image sample; the features of the processed image sample are used to determine the card template corresponding to the card sample; the card template is used to determine the key information in the processed image sample; the key information is used as the recognition result of the card sample.

[0012] Thirdly, embodiments of this application provide a card identification device, which may include:

[0013] The distortion correction module is used to correct the distortion of the card by utilizing the corner points of the card in the image, and obtain the processed image.

[0014] The card template determination module is used to determine the card template corresponding to the card by utilizing the overall features of the processed image;

[0015] The key information determination module is used to determine key information in the processed image using the card template; the key information is used as the recognition result of the card.

[0016] Fourthly, embodiments of this application provide a training apparatus for a card recognition model, which may include:

[0017] The card sample generation module is used to generate card samples using materials in the material library; the materials in the material library are generated through open-source code or obtained through public channels.

[0018] An image sample generation module is used to perform image processing on the card sample to obtain an image sample containing the card sample.

[0019] The training module is used to train a card recognition model using the image samples containing the card samples; the card recognition model is used to perform distortion removal processing on the card samples using the corner positions of the card samples in the image samples containing the card samples, to obtain processed image samples; the features of the processed image samples are used to determine the card template corresponding to the card samples; the card template is used to determine the key information in the processed image samples; the key information is used as the recognition result of the card samples.

[0020] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method described in any of the above-mentioned embodiments.

[0021] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the preceding claims.

[0022] Compared with the prior art, this application has the following advantages:

[0023] According to the embodiments of this application, for images containing cards and certificates, the cards and certificates can be detected and corrected first, thereby reducing the recognition error rate caused by card tilting or flipping that may occur during card and certificate shooting. Furthermore, feature extraction can be performed on the processed entire image to obtain multi-dimensional features of the cards and certificates, which can then be used to determine the card and certificate template. The card and certificate template can be used to classify cards and certificates and locate key information, thereby reducing the uncertainty of single-dimensional features (such as text features) and achieving better accuracy in card and certificate recognition.

[0024] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0025] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0026] Figure 1 A schematic diagram illustrating a scenario for the card recognition method provided in this application;

[0027] Figure 2 This is a flowchart of a card recognition method according to an embodiment of this application;

[0028] Figure 3 This is a flowchart of a training method for a card recognition model according to an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of materials in a material library according to an embodiment of this application;

[0030] Figure 5 This is a schematic diagram of a generated card sample according to an embodiment of this application;

[0031] Figure 6 This is a schematic diagram illustrating the image processing of a card sample according to an embodiment of this application;

[0032] Figure 7 This is a structural block diagram of a card recognition device according to an embodiment of this application;

[0033] Figure 8 This is a structural block diagram of a training device for a card recognition model according to an embodiment of this application; and

[0034] Figure 9This is a block diagram of an electronic device used to implement embodiments of this application. Detailed Implementation

[0035] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0036] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0037] First, the terms used in this application will be explained.

[0038] Corner detection is a method used in computer vision to obtain image features, widely applied in motion detection, image matching, video tracking, 3D modeling, and object recognition. A corner is typically defined as the intersection of two edges. In practice, most corner detection algorithms detect image points with specific features. These feature points have specific coordinates in the image.

[0039] Feature vectors (Embeddings): These map high-dimensional raw data such as images and text to low-dimensional features. These low-dimensional features represent the characteristic representation of the high-dimensional raw data. Feature vectors can be used for similarity comparison, retrieval, and other applications.

[0040] Optical Character Recognition (OCR) refers to the process by which electronic devices examine the shape of characters and then translate those shapes into computer text using character recognition methods. In other words, it involves scanning text and image data, then analyzing and processing the image files to obtain text and layout information.

[0041] Figure 1 This is a schematic diagram illustrating an application scenario for implementing the method of this application embodiment. For card and document recognition, it can be applied to scenarios such as rapid entry or filing of card and document information. The acquired image is detected to confirm whether it contains any cards or documents. These cards and documents can include ID cards, driver's licenses, passports, medical insurance cards, shopping cards, etc. For images containing cards or documents, distortion correction processing can be performed on the cards or documents in the image. For example, Figure 1In the example shown, the card / passport in the image is held in a hand and is tilted at a certain angle. Based on this, distortion correction processing can be performed on the card / passport in the image to obtain an image of the card / passport without distortion from a normal viewing angle. By extracting features from the entire image of the card / passport, a holistic feature representation of the image can be obtained. Using this holistic feature representation, matching can be performed in a pre-established feature library to determine the category of the card / passport. For example, Chinese driver's licenses, Turkish passports, etc. Correspondingly, after determining the category of the card / passport, a corresponding preset template can be determined based on the category. The preset template can be used to determine the location of key information in the card / passport, as well as the language of the key information. Taking a driver's license as an example, key information can include the types of vehicles that the driver's license allows to drive, as well as the driver's license holder's photo, name, etc. Based on the language of the key information, a corresponding text recognition model can be selected to extract the key information.

[0042] This application provides a method for identifying cards and certificates, such as... Figure 2 The diagram shown is a flowchart of a card recognition method according to an embodiment of this application, which may include:

[0043] Step S201: Using the corner positions of the card in the image, perform distortion removal processing on the card to obtain the processed image.

[0044] The executing entity in this application embodiment can be an information input device. Images can be acquired through methods such as data collection or user input. Images can include images containing cards or documents, or images where cards or documents do not exist. The method involved in this application embodiment can identify cards or documents in images containing them. A pre-trained card detection model can be used to distinguish between images containing cards or documents and images where cards or documents do not exist. For example, the determination principle of the card detection model can include extracting image features. The extracted features are then subjected to dimensionality upscaling or dimensionality reduction processing to obtain features at multiple scales. After fusing the features at multiple scales, foreground and background classification, card / document border prediction, and corner point position prediction are performed based on the fusion processing result. Foreground and background classification determines whether an image contains a card or document.

[0045] By using the positions of the corner points, the aspect ratio of the card in the image can be determined. If the aspect ratio is the same as the pre-stored aspect ratio, it means that the angle of the card in the image is conducive to recognition. Conversely, if the aspect ratio is different from the pre-stored aspect ratio, distortion correction can be performed on the card in the image based on the pre-stored aspect ratio. Alternatively, a pre-trained 3D perspective transformation matrix can be used to correct the distortion of the card in the image. In the image after distortion correction, the card is displayed with a frontal view and no distortion.

[0046] Before performing card and document recognition, distortion correction processing on the card and document images can reduce recognition interference caused by the image acquisition angle to a certain extent, thus providing a foundation for accurate subsequent recognition.

[0047] Step S202: Utilize the overall features of the processed image to determine the card template corresponding to the card.

[0048] When performing feature extraction, the object of extraction can be the entire processed image. The overall features of the processed image can reflect the content contained in the image as a whole from multiple dimensions, such as text dimension, image dimension, and card layout dimension. The feature can be represented in the form of feature vectors.

[0049] On the other hand, feature databases can be pre-built for different categories of cards and certificates. The categories in the feature database can be divided into vertical categories such as driver's licenses, passports, health insurance cards, and shopping cards. Each vertical category can be further subdivided. Taking driver's licenses as an example, it can be further subdivided according to the permitted vehicle types, the issuing country or region, etc. Permitted vehicle types can include car driver's licenses, motorcycle driver's licenses, etc. Furthermore, car driver's licenses can be subdivided into large passenger vehicles, sedans, etc. In the feature database, each category of card or certificate stores a corresponding feature vector.

[0050] The overall features of the processed image are used to perform a search in a feature database. Based on the search results, the card template corresponding to the card in the processed image can be determined. In one card recognition method, determining the card category requires OCR information extraction from the card to obtain short text data. This text data is then used for card category recognition. Since cards come in various colors and patterns, and text printed on different patterns can interfere with text recognition, this affects the accuracy of category determination. The proposed solution, however, utilizes the overall features of the image to determine the card template, which is essentially a card category recognition process. By using the overall features of the image for classification, interference factors are less significant, resulting in relatively higher accuracy.

[0051] Step S203: Using the card template, determine the key information in the processed image; the key information is used as the recognition result of the card.

[0052] The card template pre-labels information such as the card type and language of the text. Using this information, key information can be identified in the processed image. For example, in a driver's license template, fixed information might include the driver's number, driver's name, driver's gender, and permitted vehicle type. The content following this fixed information is generally considered key information. For example, Driver's Number: 1234567. Driver's Name: XXX. Driver's Gender: Male. Permitted Vehicle Type: C1. In the above example, "1234567", "XXX", "Male", and "C1" all correspond to key information. That is, key information is what distinguishes the current card from other cards. Based on this identified key information, the card content can be recognized.

[0053] Through the above process, for images containing cards and certificates, the cards and certificates can be detected and corrected first, thereby reducing the recognition error rate caused by tilting or flipping during card and certificate shooting. Furthermore, feature extraction can be performed on the processed entire image to obtain multi-dimensional features of the cards and certificates, which can then be used to determine the card and certificate template. The card and certificate template can be used to classify cards and certificates and locate key information, thereby reducing the uncertainty of single-dimensional features (such as text features) and achieving better accuracy in card and certificate recognition.

[0054] In one possible implementation, step S201, which involves using the corner positions of the card in the image to perform distortion correction processing on the card, may include:

[0055] Step S2011: Determine the corner positions of the card in the image.

[0056] Once the presence of a card or document in the image is confirmed, corner detection technology can be used to determine the corners of the card or document and their positions. Ideally, there are four corners. However, it's possible for a card or document to be present in the image, but its content may be missing. For example, the card or document in the image may only have three corners, meaning part of its content was not captured. Based on this, the card or document in the image can be automatically supplemented. For instance, based on the positional relationships between adjacent and opposite corners of the card or document in the image, the position of the missing corner can be calculated, thus obtaining the virtual position of that corner.

[0057] Step S2012: Combine the corner point positions with the corner points of the pre-stored card image template to form corner point pairs.

[0058] Based on the position of the card's corner points, the aspect ratio of the card can be calculated. Then, according to the card's aspect ratio, the pre-stored set of card image templates is searched to find the card image template with the closest aspect ratio. Subsequently, corner points of the card in the image are paired with the corresponding corner points of the card image template.

[0059] Step S2013: Determine the distortion elimination parameters by utilizing the positional differences between the corner points and the centering corner points.

[0060] The perspective transformation matrix can be solved by utilizing the difference in the positions of the corner points. Perspective transformation is the process of projecting an image onto a new view plane. This process includes converting the coordinates of the corner points in a first two-dimensional coordinate system to coordinates in a three-dimensional coordinate system, and then projecting the coordinates in the three-dimensional coordinate system onto a second two-dimensional coordinate system. The perspective transformation matrix can be used as a distortion correction parameter.

[0061] Step S2014: Use distortion correction parameters to perform distortion correction processing on the card in the image.

[0062] By utilizing the corner positions of the cards in the image and solving the perspective transformation matrix, distortion correction can be performed on the cards in the image. After distortion correction, an image of the cards from a positive perspective can be obtained.

[0063] Compared to methods that require using a card vertex detection model to obtain the positions of all vertices of the card, then using the card detection model to obtain a fitted bounding box for the card, combining the bounding box to assign vertex coordinates to each card group, determining whether the card is horizontal based on the coordinates, and then using an additional classification network to determine whether the card is upside down, the above methods are complex in terms of judgment process and can only correct simple cases such as horizontal / upside-down placement. The method used in this embodiment can eliminate distortion for cards in any state, making it more versatile.

[0064] One possible implementation may also include:

[0065] The processed image is then optimized and adjusted; the image optimization and adjustment includes at least one of card scaling adjustment and background removal adjustment.

[0066] After distortion correction processing of the card in the image, an image of the card from a frontal view can be obtained. Furthermore, the image containing the card from a frontal view can be optimized and adjusted.

[0067] For example, based on the detection results of the foreground and background of the image, background removal processing can be performed on the image, thereby retaining only the part of the image corresponding to the card content. Another example is that the size of the card from a frontal view can be scaled based on the size of a pre-stored card image template, so that the size of the scaled card image is the same as or differs from the size of the card image template within an acceptable range.

[0068] In one possible implementation, step S202, which involves using the overall features of the processed image to determine the card template corresponding to the card, may include:

[0069] Step S2021: Determine the overall features of the processed image, which are used to reflect the card category from at least one dimension, including the card's background texture, the layout of elements in the card, and the icons contained in the card.

[0070] Generally speaking, the background of cards and certificates is not a blank sheet of paper, but rather has a background pattern. For example, my country's ID card has the Great Wall pattern on one side and a curved grid pattern on the other. Similarly, my country's medical insurance card has a radial grid pattern on one side and the Temple of Heaven pattern on the other. Utilizing these differences in background pattern can aid in the identification of cards and certificates.

[0071] For example, different cards have different layouts. Let's take my country's ID card and medical insurance card as examples. On the side of the medical insurance card that includes the image, the elements are arranged in a left-center-right layout. The left side contains the electronic chip, the center contains the user's personal information, and the right side contains the photo. On the side of the ID card that includes the image, the elements are arranged in a left-right layout, with the left side containing the user's personal information and the right side containing the user's photo.

[0072] For example, some cards may have their own unique logos printed on them. For instance, a membership card from a car dealership or shopping mall might display the brand's logo or the mall's logo. Similarly, a medical insurance card might display the official seal of the social security bureau. These logos typically serve as distinguishing features from other types of cards.

[0073] By utilizing at least one of the following information—the background texture of the card, the layout of its elements, and the icons it contains—the accuracy of card recognition can be enhanced. Based on this, a trained feature extraction model can be used to extract features from the processed image, ensuring that the extracted features reflect the card's category from at least one dimension: the background texture, the layout of its elements, and the icons it contains.

[0074] Step S2022: Using the overall features of the processed image, search in the pre-built template library to determine the card template corresponding to the card.

[0075] The overall features of the processed image can be in the form of feature vectors. Alternatively, a feature library can be pre-built for different categories of cards and certificates. After obtaining the overall features of the processed image, vector retrieval can be used to match card and certificate templates that match the cards and certificates in the processed image from the pre-built feature library. The card and certificate templates can be marked with the location of key information and the language. Utilizing the location of key information can help in the accurate extraction of that information. Utilizing the language can help in selecting a suitable text extraction tool, which could be a text extraction model specific to the corresponding language.

[0076] In one possible implementation, step S203, which involves using a card template to determine key information in the processed image, may include:

[0077] Step S2031: Using the card template, determine the location of key information in the card image and the language of the text in the card.

[0078] The language information marked in the card template can help in selecting an appropriate text extraction tool. For example, if the language is English, an English text recognition model can be used to identify key information. If the language is French, a French text recognition model can be used to identify key information. Key information may be related to the cardholder's identification.

[0079] Step S2032: Based on the location of the key information, use a text extraction tool corresponding to the language of the text to determine the key information.

[0080] Taking a driver's license template as an example, fixed information may include driver's number, driver's name, driver's gender, and permitted vehicle type. The content following this fixed information is generally considered key information. For example, Driver's Number: 1234567. Driver's Name: XXX. Driver's Gender: Male. Permitted Vehicle Type: C1. In the above example, 1234567, XXX, Male, and C1 correspond to the key information. Using a driver's license template, key information can be accurately and quickly located. Furthermore, using corresponding text extraction tools, the text can be accurately recognized. This is especially true for less common languages, where the accuracy of text extraction can be improved.

[0081] This application provides a method for training a card recognition model, such as... Figure 3 The flowchart shown is a training method for a card recognition model according to an embodiment of this application, which may include:

[0082] Step S301: Generate a card sample using materials from the material library; the materials in the material library are generated through open-source code or obtained through public channels.

[0083] The resource library can be pre-built. Materials in the resource library can be stored according to material categories. Combined with... Figure 4 As shown, exemplarily, the material categories in the material library may include at least one of the following: backgrounds, icons, portraits, backgrounds, and languages. These materials can be stored in sub-libraries; for example, backgrounds can be stored in a background library. Backgrounds may correspond to the backgrounds of cards or certificates, and can be generated using background generation software or obtained from existing card or certificate images. Icons may correspond to trademarks of different brands or seals used to represent different companies or individuals, and can be obtained from existing icons or open-source icon communities. Portraits can be generated as virtual portraits using generative AI algorithms. For example, the AI ​​algorithm may be a Style Generative Adversarial Network (StyleGAN). Backgrounds may correspond to the shooting environment of the card or certificate, serving as the background. Backgrounds can be obtained from open-source community datasets. Layouts can be established based on prior knowledge. The materials in the material library can be dynamically updated.

[0084] Using materials from the resource library, virtual samples of various cards and certificates can be generated. Since these samples are generated automatically, they do not require real information or involve sensitive data. Sample acquisition is faster and more secure.

[0085] Step S302: Perform image processing on the card sample to obtain an image sample containing the card sample.

[0086] The purpose of image processing for card and document samples is to simulate the user's experience of photographing the card or document. For example, in some scenarios, the card or document to be identified is photographed while the user is holding it. This can lead to background interference in the image containing the card or document, and the card or document may not be viewed from a direct angle while the user is holding it. Based on this, the card or document sample can be combined with background materials from a material library for image processing to obtain an image sample containing the card or document sample.

[0087] Step S303: Train the card recognition model using image samples containing card samples; the card recognition model is used to eliminate distortion of card samples by using the corner positions of card samples in the image samples containing card samples, and obtain processed image samples; use the features of the processed image samples to determine the card template corresponding to the card samples; use the card template to determine the key information in the processed image samples; the key information is used as the recognition result of the card samples; wherein, the card recognition model is a neural network model.

[0088] Image samples containing card / ID card samples are labeled, including foreground / background classification, border positions, corner positions, card / ID card offset, and card / ID card sample type. The labeled image samples containing and without card / ID card samples are then input into the card / ID card recognition model to be trained. The model can determine whether an image contains a card / ID card sample, perform distortion correction on offset card / ID card samples, and identify the type of card / ID card sample. In other words, the working principle of the card / ID card recognition model is the same as the steps in the aforementioned card / ID card recognition method.

[0089] Card recognition models, as neural network models, are a type of feedforward neural network that incorporates convolutional computations. Designed based on the translation invariance of image tasks, they excel in image processing and other similar applications. Convolutional neural networks utilize convolutional and pooling layers to efficiently extract image features. The image information is then compressed by fully connected layers following the pooling layers, ultimately producing the output. The output can include whether the image contains a card or ID. If the image does contain a card or ID, the output can also include the location of key information within the card or ID.

[0090] Through the above steps, the card recognition model can be trained using materials generated from open-source data to produce card samples, reducing reliance on real card data. Furthermore, since the content in the generated card samples is all virtual, privacy data leakage is eliminated. This achieves the training of a general card recognition model capable of recognizing different types of cards while ensuring data security.

[0091] In one possible implementation, the step S301 involving generating a card sample using materials from the material library may include:

[0092] Step S3011: Cut the background pattern of the card sample randomly selected from the material library to the specified aspect ratio.

[0093] Generally, the background of cards and certificates is not a blank sheet of paper, but rather has a background pattern. For example, my country's ID card has the Great Wall pattern on one side and a curved grid pattern on the other. Similarly, my country's medical insurance card has a radial grid pattern on one side and the Temple of Heaven pattern on the other. Various existing or randomly generated background patterns are pre-stored in the resource library. Combined with... Figure 5 As shown, background patterns can be randomly selected from the texture library in the material library, and the patterns can be detected according to a specified aspect ratio. The cards can include both horizontal and vertical versions. The aspect ratio of horizontal cards is generally between 1.4 and 1.62, while that of vertical cards is generally between 0.7 and 0.79. Furthermore, the corner radii of the four ends of the background pattern can be randomly set.

[0094] Step S3012: Based on the layout of the card, set multiple areas on the cut-out background pattern, including at least one of the card title area, card body area, card annotation area, and portrait area.

[0095] Based on regional divisions, cards and certificates generally include a title area, a body area, annotations, and a photo area. The title area represents the card's content, such as an ID card or medical insurance card. The body area contains key information, such as the username, date of birth, and serial number. The annotations area provides explanations of the card's content; for example, a driver's license annotation might include an explanation of the permitted vehicle type code. For instance, A1 might be annotated as "large passenger vehicle," A2 as "tractor-trailer," and C1 as "small car." The photo area displays the cardholder's image.

[0096] The card layout can be random, or it can be generated by adjusting the positions of each area based on an existing card layout. Appropriate adjustments can include moving existing areas in various directions, with the movement distance within a pre-defined range.

[0097] Step S3013: Fill the area with text content to obtain a card sample.

[0098] In each area, corresponding text content is filled in to generate a complete card sample with virtual information. For example, icons can be randomly selected from the icon library in the material library and filled in the designated positions on the card. Portraits can be randomly selected from the portrait library and filled in the headshot area. Languages ​​can be randomly selected from the font corpus to generate the filled text. The filled text can be randomly generated based on objective facts. For example, for birth dates, dates from the previous 100 years can be used. For the card title, it can be a randomly generated name such as a shopping mall or brand.

[0099] In one possible implementation, the image processing of the card sample involved in step S302 to obtain an image sample containing the card sample includes:

[0100] Step S3021: Randomly select a background image from the material library.

[0101] Combination Figure 6 As shown, the background image can be used to simulate the photographic environment. For example, the background image can be an image of a hand, a table, the sky, or green plants. Background images can be selected from... Figure 5The background image is randomly selected from the background image library shown. If the background image is a hand image, it can be used to simulate a photo environment where a user is holding a card or ID. If the background image is a table image, it can be used to simulate a photo environment where the card or ID is placed on a table.

[0102] Step S3022: According to the specified rules, combine the card sample with the background image, and use the combined image as an image sample containing the card sample; the specified rules are determined based on at least one of the following: the size, position, and occlusion of the card sample in the background image.

[0103] correspond Figure 6 The concept of "combining" refers to the process of combining a card sample with a background image. This can be achieved by using both the card sample and the background image as top layers, resulting in a combined image. Furthermore, certain rules must be followed during the combination process. For example, the cards can be rotated and scaled during the combination. The scaling rules can be set within a specified range to prevent the cards from being reduced too much and becoming difficult to identify. Also, during the combination process, the card may not be positioned in the center of the background image, but could be at any position, or even extend beyond the background image's boundaries. Therefore, rules can constrain the size of the area extending beyond the background image's boundaries to prevent excessively large areas from causing information loss. Additionally, multiple card samples may be combined with a single background image. This can lead to situations where multiple cards obscure each other. Similar to the situation where card samples extend beyond the background image's boundaries, rules can constrain the extent of obscuring to prevent excessively large obscured areas from causing information loss.

[0104] In one possible implementation, the process of performing data augmentation on an image sample containing a card sample;

[0105] Data augmentation processing may include at least one of perspective transformation, saturation transformation, brightness transformation, contrast transformation, and motion blur transformation.

[0106] correspond Figure 6 The "data augmentation" mentioned above involves performing at least one of the following transformations on image samples containing card samples: perspective transformation, saturation transformation, brightness transformation, contrast transformation, and motion blur transformation. This can both increase the number of image samples and enhance the recognition ability of the trained model.

[0107] In one possible implementation, step S203, which involves training the card recognition model using image samples containing card samples, may include:

[0108] Step S2031: The first sub-model of the card recognition model is trained using the background annotation results of the image sample containing the card sample, the border annotation results of the card sample, and the corner position annotation results of the card sample. The first sub-model is used to predict whether the image sample contains the card sample, and to predict the corner position of the card sample based on the border when the prediction result is that the card sample is contained.

[0109] The card recognition model can be an end-to-end model, which can include different sub-models. Each sub-model can perform its own task.

[0110] The first sub-model can be used to identify image samples to detect whether an image contains a card or ID card sample. Specifically, after extracting features from the image sample using a backbone network, features at different scales are fused using a feature fusion network (Neck), and then a classification network (Head) is used to obtain the detection result. If the detection result indicates the presence of a card or ID card sample, the corner positions of the card or ID card sample can be predicted based on the card or ID card sample's bounding box. That is, the first sub-model can perform three tasks: the first task is to use foreground and background classification to determine which pixels belong to the card or ID card; the second task is to determine whether a pixel belongs to the card or ID card by detecting the card or ID card's bounding box; and the third task is to detect the positions of corner points to determine which pixels belong to the card or ID card.

[0111] The labeled image samples are input into the first sub-model to be trained. The first sub-model can determine whether the image sample contains a card, and the location of the card's corners, based on the overall features of the image, using foreground / background classification dimensions, card bounding box dimensions, and card corner dimensions. During training, the output results of the first sub-model and... Figure 6 As shown in the annotation, the weight parameters among the three sub-tasks are adjusted. Specifically, increasing the weight parameter for foreground / background classification allows for greater reliance on the foreground / background classification results during recognition. By adjusting the weight parameters among the three sub-tasks, different image conditions can be addressed. For example, some image samples have clearly defined foreground and background, while the borders or corners of card / ID card samples are not obvious. Conversely, some card / ID card samples have clearly defined borders but indistinct foreground and background. By adjusting the weight parameters among the three sub-tasks of the first sub-model according to the image sample conditions, different shooting scenarios can be addressed, thereby improving the accuracy of card / ID card detection.

[0112] Step S2032: Use the type labeling results of the card and certificate samples to train the second sub-model of the card and certificate recognition model; the second sub-model is used to determine the category corresponding to the card and certificate sample based on the features of the image sample.

[0113] For image samples containing card / ID card samples, distortion correction can be performed first. For example, during image processing, the card / ID card sample might be reduced to 1 / 2 or 1 / 3 of its original size. Additionally, perspective transformation might be applied during image processing, such as rotating it 45° along the X-axis or 15° along the Z-axis. By solving for the distortion correction parameters, the card / ID card image with scaling and / or perspective transformation can be restored to its standard size, thus completing the distortion correction process.

[0114] The primary task of the second sub-model is to perform category identification on the card samples. Based on the difference between the category identification results and the category labeling results, the parameters in the backbone network are adjusted to ensure that the features extracted by the backbone network satisfy a compact feature distribution within categories and a dispersed feature distribution between categories. The second sub-model can be implemented using an image recognition algorithm (Margin-Based). For category identification processing, the second sub-model can be trained using angular margin loss (Arcface).

[0115] The first and second sub-models described above can be trained simultaneously, meaning that the input to both sub-models is the labeled image sample. The two sub-models iteratively adjust their parameters based on the difference between the predicted and labeled results until the difference is within acceptable limits.

[0116] The aforementioned card recognition method and card recognition model training method can be a local application (APP) on the user terminal, or a functional module of an APP, or a service provided by the cloud. The user calls the corresponding call interface of the service, uploads the image containing the card or the material in the material library to the cloud, and receives the results fed back by the cloud, such as the card recognition result or the trained card recognition model.

[0117] For example, several distributed computing nodes can be deployed in the cloud, each with processing resources such as computing and storage. In the cloud, multiple computing nodes can be organized to provide one or more services within a card recognition method; alternatively, multiple computing nodes can be organized to provide one or more services within a card recognition model training method. Taking a card recognition method as an example, services may include using the corner positions of the card in an image to eliminate distortion and obtain a processed image; using the overall features of the processed image to determine the card template corresponding to the card; and using the card template to determine key information in the processed image, etc. Of course, a single computing node can also provide one or more services. The cloud can provide these services by providing an external service interface, which users can call to use the corresponding services.

[0118] According to the solution provided in this embodiment of the invention, the cloud can provide a service interface for information recognition services, referred to as the target service interface. When a user needs to recognize an image containing a card or document, or when a user needs to train a card or document recognition model, the user device calls the target service interface to trigger a request to the cloud to call the target service interface. This request carries an image of the target object to be recognized. The cloud determines the computing node that responds to the request and uses the processing resources in that computing node to execute the steps provided in this embodiment of the application.

[0119] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a card recognition device. For example... Figure 7 The diagram shown is a structural block diagram of a card recognition device according to an embodiment of this application. The card recognition device may include:

[0120] The distortion correction module 701 is used to perform distortion correction on the card by utilizing the corner positions of the card in the image, and obtain the processed image.

[0121] The card template determination module 702 is used to determine the card template corresponding to the card by utilizing the overall features of the processed image.

[0122] The key information determination module 703 is used to determine key information in the processed image using the card template; the key information is used as the recognition result of the card.

[0123] In one possible implementation, the distortion correction module 701 may include:

[0124] The corner point location determination submodule is used to determine the corner point location of the card in the image;

[0125] The corner point pair assembly submodule is used to assemble corner point pairs by combining the corner point positions with the corner points of the pre-stored card image templates;

[0126] The distortion elimination parameter determination submodule is used to determine the distortion elimination parameters by utilizing the positional differences between the corner points and the centering corner points;

[0127] The distortion correction processing submodule is used to perform distortion correction processing on the cards in the image using distortion correction parameters.

[0128] In one possible implementation, an optimization adjustment module may also be included, which can be used to: perform image optimization adjustments on the processed image; the image optimization adjustments include at least one of card scaling adjustment and background removal adjustment.

[0129] In one possible implementation, the card template determination module 702 may include:

[0130] The feature determination submodule is used to determine the overall features of the processed image. The features are used to reflect the card category from at least one dimension, including the background texture of the card, the layout of the elements in the card, and the icons contained in the card.

[0131] The retrieval submodule is used to search within a pre-built template library using the overall features of the processed image to determine the card template corresponding to the card.

[0132] In one possible implementation, the key information determination module 703 may include:

[0133] The location and language determination submodule is used to determine the location of key information in the card image and the language of the text in the card using the card template;

[0134] The key information determination execution submodule is used to determine key information based on its location using a text extraction tool corresponding to the language of the text.

[0135] Corresponding to the application scenarios and methods provided in the embodiments of this application, the embodiments of this application also provide a training device for a card recognition model. For example... Figure 8 The diagram shown is a structural block diagram of a training device for a card recognition model according to an embodiment of this application, which may include:

[0136] Card sample generation module 801 is used to generate card samples using materials in the material library; the materials in the material library are generated through open source code or obtained through public channels.

[0137] The image sample generation module 802 is used to perform image processing on the card sample to obtain an image sample containing the card sample.

[0138] Training module 803 is used to train a card recognition model using image samples containing card samples. The card recognition model is used to perform distortion removal processing on the card samples using the corner positions of the card samples in the image samples containing card samples, to obtain processed image samples. The features of the processed image samples are used to determine the card template corresponding to the card samples. The card template is used to determine the key information in the processed image samples. The key information is used as the recognition result of the card samples. The card recognition model is a neural network model.

[0139] In one possible implementation, the card sample generation module 801 may include:

[0140] The cropping submodule is used to crop the background pattern of a card sample randomly selected from the material library to a specified aspect ratio;

[0141] The layout submodule is used to set multiple areas on the cut-out background pattern according to the layout of the card. The areas include at least one of the following: card title area, card body area, card annotation area, and portrait area.

[0142] The card sample generation execution submodule is used to fill in text content in the area to obtain a card sample.

[0143] In one possible implementation, the image sample generation module 802 may include:

[0144] The background image selection submodule is used to randomly select a background image from the media library;

[0145] The combination submodule is used to combine the card sample with the background image according to specified rules, and use the combined image as an image sample containing the card sample; the specified rules are determined based on at least one of the following: the size, position, and occlusion of the card sample in the background image.

[0146] One possible implementation also includes a data augmentation processing module;

[0147] The data augmentation module is used to perform data augmentation processing on image samples containing card samples; the data augmentation processing includes at least one of perspective transformation, saturation transformation, brightness transformation, contrast transformation, and motion blur transformation.

[0148] In one possible implementation, the training module 803 may include:

[0149] The first training submodule is used to train the first sub-model of the card recognition model using the background annotation results of the image sample containing the card sample, the border annotation results of the card sample, and the corner position annotation results of the card sample. The first sub-model is used to predict whether the image sample contains the card sample, and predict the corner position of the card sample based on the border when the prediction result is that the card sample is contained.

[0150] The second training submodule is used to train the second sub-model of the card recognition model using the type labeling results of the card samples; the second sub-model is used to determine the category corresponding to the card sample based on the features of the image sample.

[0151] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0152] Figure 9 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 9As shown, the electronic device includes a memory 910 and a processor 920. The memory 910 stores a computer program that can run on the processor 920. When the processor 920 executes the computer program, it implements the method described in the above embodiments. The number of memories 910 and processors 920 can be one or more.

[0153] The electronic device also includes:

[0154] The communication interface 930 is used to communicate with external devices and exchange and transmit data.

[0155] If the memory 910, processor 920, and communication interface 930 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0156] Optionally, in a specific implementation, if the memory 910, processor 920, and communication interface 930 are integrated on a single chip, the memory 910, processor 920, and communication interface 930 can communicate with each other through an internal interface.

[0157] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0158] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0159] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0160] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0161] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0162] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0163] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0164] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0165] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0166] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0167] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0168] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0169] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for identifying cards and certificates, characterized in that, include: By utilizing the corner points of the card in the image, distortion removal processing is performed on the card to obtain the processed image; Using the overall features of the processed image, the card template corresponding to the card is determined. The overall features are used to reflect the category of the card from at least one dimension, including the background of the card, the layout of the elements in the card, and the icons contained in the card. Using the card template, key information in the processed image is determined; The key information is used as the text recognition result of the card; The step of using the card template to determine key information in the processed image includes: Using the card template, determine the location of key information in the card and the language of the text in the card; Based on the location of the key information, the key information is determined using a text extraction tool corresponding to the language of the text.

2. The method according to claim 1, characterized in that, The process of eliminating distortion of the card by utilizing the corner positions of the card in the image includes: Determine the corner positions of the card in the image; The corner point positions are paired with the corner points of the pre-stored card image template; The distortion elimination parameters are determined by using the positional differences between the corner points and the center corner points; The distortion correction parameters are used to correct distortion in the image of the card.

3. The method according to claim 1 or 2, characterized in that, Also includes: The processed image is then subjected to image optimization adjustments; these adjustments include at least one of card scaling adjustments and background removal adjustments.

4. The method according to claim 1, characterized in that, The step of determining the card template corresponding to the card using the overall features of the processed image includes: Determine the overall features of the processed image; Using the overall features of the processed image, a search is performed in a pre-built template library to determine the card template corresponding to the card.

5. A training method for a card recognition model, characterized in that, include: Card samples are generated using materials from a resource library; the materials in the resource library are generated through open-source code or obtained through public channels. The card sample is processed into an image to obtain an image sample containing the card sample; The card recognition model is trained using the image samples containing the card samples. The card recognition model uses the corner positions of the card samples in the image samples to perform distortion correction on the card samples, obtaining processed image samples. Using the features of the processed image samples, the card template corresponding to the card samples is determined. These features reflect the card category from at least one dimension: the card's background texture, the layout of elements in the card, and the icons contained in the card. The card template is used to determine key information in the processed image samples. This key information is used as the text recognition result of the card samples. The step of using the card template to determine key information in the processed image sample includes: Using the card template, determine the location of key information in the card and the language of the text in the card; Based on the location of the key information, the key information is determined using a text extraction tool corresponding to the language of the text. The card recognition model is a neural network model.

6. The method according to claim 5, characterized in that, The process of generating card samples using materials from the material library includes: The background pattern of the card sample randomly selected from the material library will be cut out to a specified aspect ratio; Based on the layout of the card, multiple areas are set on the cut-out background pattern, including at least one of the card title area, card body area, card annotation area, and portrait area; Fill the area with text content to obtain the card sample.

7. The method according to claim 5, characterized in that, The step of image processing the card sample to obtain an image sample containing the card sample includes: Randomly select a background image from the resource library; According to a specified rule, the card sample is combined with the background image, and the combined image is used as an image sample containing the card sample; the specified rule is used to indicate at least one of the following: size, position, and occlusion of the card sample in the background image.

8. The method according to claim 5, characterized in that, It also includes data augmentation processing on the image samples containing the card samples; The data augmentation process includes at least one of perspective transformation, saturation transformation, brightness transformation, contrast transformation, and motion blur transformation.

9. The method according to claim 5, characterized in that, The step of training the card recognition model using the image samples containing card samples includes: The first sub-model of the card recognition model is trained using the background annotation results, card border annotation results, and card corner position annotation results of the image sample containing the card sample. The first sub-model is used to predict whether the image sample contains a card sample, and to predict the corner position of the card sample based on the border when the prediction result is that the image sample contains a card sample. The second sub-model of the card recognition model is trained using the type labeling results of the card samples; the second sub-model is used to determine the category corresponding to the card samples based on the features of the image samples.

10. A card / certificate identification device, characterized in that, include: The distortion correction module is used to correct the distortion of the card by utilizing the corner points of the card in the image, and obtain the processed image. The card template determination module is used to determine the card template corresponding to the card by utilizing the overall features of the processed image. The overall features are used to reflect the category of the card from at least one dimension, including the background of the card, the layout of the elements in the card, and the icons contained in the card. The key information determination module is used to determine key information in the processed image using the card template; The key information is used as the text recognition result of the card; The key information determination module includes: The location and language determination submodule is used to determine the location of key information in the card image and the language of the text in the card using the card template; The key information determination execution submodule is used to determine key information based on its location using a text extraction tool corresponding to the language of the text.

11. A training device for a card recognition model, characterized in that, include: The card sample generation module is used to generate card samples using materials in the material library; the materials in the material library are generated through open-source code or obtained through public channels. An image sample generation module is used to perform image processing on the card sample to obtain an image sample containing the card sample. The training module is used to train a card recognition model using image samples containing card samples. The card recognition model is used to perform distortion correction processing on the card samples using the corner positions of the card samples in the image samples containing the card samples, obtaining processed image samples. Using the features of the processed image samples, the card template corresponding to the card sample is determined. These features reflect the card category from at least one dimension, including the card's background texture, the layout of elements in the card, and the icons contained in the card. The card template is used to determine key information in the processed image samples. The key information is used as the text recognition result of the card sample; The training module includes: The location and language determination submodule is used to determine the location of key information in the card image and the language of the text in the card using the card template; The key information determination and execution submodule is used to determine the key information based on its location using a text extraction tool corresponding to the language of the text. The card recognition model is a neural network model.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1-9.

13. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images