Identity card information extraction method and device, computer device, and storage medium

Abstract

The application relates to the technical field of financial technology and artificial intelligence, and discloses an identity card information extraction method and device, computer equipment and a storage medium, wherein the obtained target identity card image is input into an identity information extraction model, the image encoder of the identity information extraction model is used for performing feature extraction and coding on the target identity card image, and an image vector representation is obtained; the decoder of the trained identity information extraction model is used for decoding the image vector representation, and an image sequence is obtained; target format conversion is performed on the image sequence, and the identity card information of the target identity card image is obtained. According to the application, the identity card image is subjected to image feature extraction and text prediction by using an OCR-free identity information extraction model, so that an image sequence which can be converted into a target format is obtained, accurate and efficient document understanding and information extraction of the identity card information are realized without depending on an OCR technology, manpower and material resources are saved, and the digital development of the field of financial technology is promoted.

Description

Technical Field

[0001] This application relates to the fields of financial technology and artificial intelligence technology, and in particular to a method, apparatus, computer equipment and storage medium for extracting ID card information. Background Technology

[0002] An ID card is an important identification credential for every citizen. In the financial sector, business operations also rely heavily on ID card verification. After verification, staff often need to input the ID card information. To improve efficiency, ID card recognition has gradually moved towards automated data entry. A common method for extracting ID card information in the industry is to first extract all information from the ID card using OCR (Optical Character Recognition) technology, and then use post-processing methods such as regularization to output a structured version of the information.

[0003] Although the aforementioned method has been proven to be the most effective extraction method and has been successfully applied in fields such as the internet and finance, this OCR-based extraction method often suffers from unavoidable problems such as OCR errors and the propagation of these errors, leading to poor results in the extraction of ID card information. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device, computer equipment and storage medium for extracting ID card information, which can solve the technical problems such as the poor effect of using OCR technology to read image text in the prior art.

[0005] To achieve the above objectives, the first aspect of this application provides a method for extracting identity card information, the method comprising:

[0006] Obtain the target ID card image;

[0007] The target ID card image is input into the trained identity information extraction model. The image encoder of the trained identity information extraction model extracts and encodes the target ID card image to obtain an image vector representation.

[0008] The image vector representation is decoded by the decoder of the trained identity information extraction model to obtain the image sequence;

[0009] The image sequence is converted to the target format to obtain the ID card information of the target ID card image.

[0010] To achieve the above objectives, a second aspect of this application provides an ID card information extraction device, the device comprising:

[0011] The first data acquisition module is used to acquire the target ID card image;

[0012] The encoding module is used to input the target ID card image into the trained identity information extraction model. The image encoder of the trained identity information extraction model extracts and encodes the target ID card image to obtain an image vector representation.

[0013] The decoding module is used to decode the image vector representation through the decoder of the trained identity information extraction model to obtain the image sequence;

[0014] The conversion module is used to convert the image sequence to the target format to obtain the ID card information of the target ID card image.

[0015] To achieve the above objectives, a third aspect of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0016] Obtain the target ID card image;

[0017] The target ID card image is input into the trained identity information extraction model. The image encoder of the trained identity information extraction model extracts and encodes the target ID card image to obtain an image vector representation.

[0018] The image vector representation is decoded by the decoder of the trained identity information extraction model to obtain the image sequence;

[0019] The image sequence is converted to the target format to obtain the ID card information of the target ID card image.

[0020] To achieve the above objectives, a fourth aspect of this application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0021] Obtain the target ID card image;

[0022] The target ID card image is input into the trained identity information extraction model. The image encoder of the trained identity information extraction model extracts and encodes the target ID card image to obtain an image vector representation.

[0023] The image vector representation is decoded by the decoder of the trained identity information extraction model to obtain the image sequence;

[0024] The image sequence is converted to the target format to obtain the ID card information of the target ID card image.

[0025] The embodiments of this application have the following beneficial effects:

[0026] This application utilizes an OCR-free identity information extraction model to extract image features and predict text from ID card images, resulting in image sequences that can be converted to the target format. This achieves accurate and efficient identification and extraction of ID card information without relying on OCR technology. Compared to OCR technology, this embodiment breaks free from dependence on OCR, is unaffected by the negative impact of OCR error propagation, achieves efficient and accurate document understanding, and reduces computational and time costs. The solution in this embodiment automates most of the document image digitization work, saving manpower and resources. This promotes the digital development of financial services such as banking and insurance. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] in:

[0029] Figure 1 This is a diagram illustrating the application environment of the ID card information extraction method in this application embodiment;

[0030] Figure 2 This is a flowchart of the method for extracting ID card information in the embodiments of this application;

[0031] Figure 3 This is a structural block diagram of the ID card information extraction device in the embodiments of this application;

[0032] Figure 4 This is a structural block diagram of the computer device in the embodiments of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0034] Existing technologies for document recognition in images require the use of OCR technology followed by document understanding. OCR is computationally expensive and time-consuming, leading to resource consumption, poor robustness, and limitations in language and document type handling. Furthermore, OCR models are inflexible and lack flexibility; OCR errors are passed to the document understanding model (VDU), negatively impacting subsequent processes and causing inaccurate document recognition. Therefore, this application provides a solution for extracting ID card information without OCR.

[0035] Figure 1 This is a diagram illustrating the application environment of a method for extracting ID card information in one embodiment. (Refer to...) Figure 1 This method for extracting ID card information is applied to an ID card information extraction system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. The terminal 110 inputs a target ID card image to the server 120, and the server 120 acquires the target ID card image. The target ID card image is then input into a trained ID card information extraction model. The image encoder of the trained ID card information extraction model extracts and encodes features from the target ID card image to obtain an image vector representation. The decoder of the trained ID card information extraction model decodes the image vector representation to obtain an image sequence. The image sequence is then converted to a target format to obtain the ID card information of the target ID card image.

[0036] like Figure 2 As shown, in one embodiment, a method for extracting ID card information is provided. This method specifically includes the following steps:

[0037] S100: Obtain the target ID card image.

[0038] Specifically, the target ID card image is a document image containing text such as name, date of birth, ID number, home address, etc.

[0039] S200: Input the target ID card image into the trained identity information extraction model. The image encoder of the trained identity information extraction model extracts and encodes the target ID card image to obtain an image vector representation.

[0040] Specifically, the trained identity information extraction model is an OCR-free (i.e., OCR-free) visual document understanding (VDU) model that can be used for document information extraction, breaking the dependence on OCR in existing technologies for ID card information extraction.

[0041] An image encoder, also known as a visual encoder, is used to extract and encode features from a target ID card image to map the image into an embedding vector. The resulting image vector representation is a set of image vectors that form the embedding vectors. The visual encoder can be built using a CNN-based model or a Transformer-based model, such as the Swin Transformer or Vision Transformer (ViT). This application does not impose any restrictions on this approach.

[0042] The Swin Transformer includes multiple Stage modules, which are used to downsample and encode the image patches sequentially to obtain an image vector representation.

[0043] The Vision Transformer first divides the image into multiple patches. Each patch is then input into the transformer as a token. The patches are ordered, and the transformer performs embedding and position encoding on each patch to obtain its patch embedding and position embedding. The transformer also performs attention between patches based on their patch and position embeddings, ultimately resulting in an image vector representation. Patch embedding specifically involves a linear mapping that maps each patch to a one-dimensional vector, with each patch corresponding to a one-dimensional vector.

[0044] S300: The image vector representation is decoded by the decoder of the trained identity information extraction model to obtain the image sequence.

[0045] Specifically, the decoder in the identity information extraction model is a text decoder. The decoder maps the image vector representation of the target ID card image to an image sequence. The image sequence is a labeled sequence that can be converted into structured information of the target type.

[0046] The decoder is a language decoder for NLP, which can be built using a text generation model, such as the BART model, the BERT model, etc. This application does not impose any restrictions on it.

[0047] Preferably, both the image encoder and decoder are built based on Transformer.

[0048] S400: Perform target format conversion on the image sequence to obtain the ID card information of the target ID card image.

[0049] Specifically, the image sequence contains the token corresponding to the text in the target ID card image. Therefore, by performing target format conversion on the image sequence, the image sequence can be converted into target type structured information, which contains ID card information, thereby realizing the extraction of ID card information.

[0050] The target format identity information is structured data, so it can be more conveniently and accurately entered into the business system automatically, reducing manpower and material resources.

[0051] This embodiment utilizes an OCR-free identity information extraction model to extract image features and predict text from ID card images, resulting in an image sequence that can be converted to the target format. This achieves accurate and efficient recognition and extraction of ID card information without relying on OCR technology. Compared to OCR technology, this embodiment breaks free from dependence on OCR, is unaffected by the negative impact of OCR error propagation, achieves efficient and accurate document understanding, and reduces computational and time costs. The solution automates most of the document image digitization work, saving manpower and resources. This promotes the digital development of financial services such as banking and insurance.

[0052] In one embodiment, the image encoder is constructed using the Swing Transformer model;

[0053] And / or,

[0054] The decoder is built using the BART model.

[0055] Specifically, the Swin Transformer model includes four different network frameworks, which, from smallest to largest, are Swin-T, Swin-S, Swin-B, and Swin-L. Any of these four network frameworks can be chosen to build an image encoder.

[0056] The BART (Bidirectional and Auto-Regressive Transformers) model uses a standard Transformer-based sequence-to-sequence structure. The main difference lies in replacing the ReLU in the original structure with the GeLU (Gaussian Error LinerarUnits) activation function, and initializing the parameters according to a normal distribution. BART combines a bidirectional Transformer encoder with a unidirectional autoregressive Transformer decoder, and is a typical denoising autoencoder because it is pre-trained on denoised and reconstructed input text containing noise.

[0057] BART can be used for text generation. The pre-training process of BART can be summarized in the following two stages. First, noise is introduced into the input text to scramble the original text, and the scrambled text is encoded using a bidirectional encoder. Then, the original text is reconstructed using a unidirectional autoregressive decoder.

[0058] The BART model can be directly used for conditional text generation tasks, such as abstract question answering and abstract summarization. In these tasks, the encoder takes input text as conditions, and the decoder generates the corresponding target text in an autoregressive manner.

[0059] Furthermore, if the image encoder is built using the Swing Transformer model and the decoder is built using the BART model, then the constructed identity information extraction model is the Donut model, which implements a method for extracting ID card information based on the Donut model. The Donut model is an end-to-end VDU (Visual Document Understanding) model used to understand document images. The Donut model consists of a Transformer-based visual encoder and a text decoder.

[0060] The Donut model uses a visual encoder to extract features from a given document image to obtain an image vector representation, that is, to map a given document image into an embedding vector. The Donut model uses a text decoder to map the derived image vector representation into a token sequence, which includes a series of sub-word tokens. This token sequence can be used to construct the desired structured data, such as JSON text, but is not limited to this.

[0061] The Donut model directly maps input document images to the desired structured output. Built on the Transformer architecture, it is known as the document understanding transformer. Donut does not rely on OCR and can be easily trained end-to-end. The Donut model can also be trained using synthetic document images to reduce its dependence on large-scale real-world document images, and it can easily scale to multilingual environments.

[0062] This embodiment achieves efficient extraction of document information (Document IE). This embodiment can also be applied to other fields, such as extracting corresponding target text from document images like receipts, invoices, and business licenses. This application can automate most of the document image digitization work, saving manpower and resources.

[0063] In one embodiment, if the image encoder is constructed using the Swing Transformer model, the image encoder includes an image segmentation module and multiple Stage modules connected in sequence.

[0064] In step S200, the image encoder of the trained identity information extraction model extracts and encodes features from the target ID card image to obtain an image vector representation, including:

[0065] The image encoder of the trained identity information extraction model segments the target ID card image into multiple non-overlapping patches.

[0066] The image is downsampled using the Stage modules of the image encoder to obtain an image vector representation.

[0067] Specifically, the image segmentation module of the Swin Transformer model includes a Patch Partition layer, which can segment the target ID card image into multiple non-overlapping patches, which are called patches.

[0068] The Swin Transformer model is a multi-stage network framework, with each stage consisting of a Stage module. The output of a Stage module is a set of Feature Maps. The input to the first Stage module is the segmented feature map, and the input to subsequent Stage modules is the feature map output from the previous Stage module. Stage modules are used to downsample the input feature maps or patches. Each Stage module reduces the resolution of the input feature map, progressively expanding the receptive field. As the network depth increases, the receptive field of each node also expands.

[0069] The number of Stage modules can be configured according to the actual application scenario. For example, the number of Stage modules can be 3, 4, 5, 6, etc. This application does not limit this. The downsampling of the patch or feature map can be performed by 4 times, 8 times, 16 times, 32 times, etc. This application does not limit this.

[0070] The first Stage module includes a Linear Embedding layer and a Swing Transformer Block layer. The Linear Embedding layer is connected to the Patch Partition layer, and the Swing Transformer Block layer is connected to the next Stage module. Except for the first Stage module, all other Stage modules include a PatchMerging layer and a Swing Transformer Block layer connected in sequence.

[0071] The Patch Merging layer downsamples the image by sliding a window. The parameters of the Patch Merging layer include the window size and the stride of the sliding window.

[0072] For example, if the downsampling ratio is 4, then after downsampling, we can obtain H / 4*W / 4 feature vectors with a length of 4*4*n, where n is the number of channels of the feature map input to the Stage module, and H and W are the length and width of the feature map input to the Stage module.

[0073] The Swin Transformer Block layer includes LayerNorm (LN layer), MLP layer, window multi-head self-attention (W-MSA) layer, and shifted-window multi-head self-attention (SW-MSA) layer. The MLP layer is a "Multilayer Perceptron" and is also known as a feedforward neural network layer.

[0074] The Swin transformer divides the feature map into different windows during downsampling, and Multi-Head Self-Attention is performed only within each window (Windows Multi-Head Self-Attention (W-MSA)).

[0075] The image vector is the output obtained after downsampling through the last Stage module.

[0076] Each Stage module includes several basic hyperparameters, such as the number of hidden layer nodes, the number of network layers, the number of multi-head self-attention heads, and the downsampling scale.

[0077] This embodiment uses the Swin-Transformer model to encode the ID card image and obtain an image vector representation.

[0078] In one embodiment, if the decoder is constructed using a BART model, then in step S300, the image vector representation is decoded using the decoder of the trained identity information extraction model to obtain an image sequence, including:

[0079] The decoder of the trained identity information extraction model maps the image vector representation to a token sequence, and then uses the token sequence as the image sequence.

[0080] Specifically, the image vector representation serves as the input to the attention layer of the decoder, which learns the understanding of the text.

[0081] The decoder is used to generate a token sequence that can be converted into structured information of the target type. This token sequence can be a token sequence. The token sequence includes a vector representation of at least one token. The vector representation of the token can be a one-hot vector or other types of vector representation; this application does not impose any limitations on this.

[0082] The process of converting the image sequence to its target format to obtain the ID card information of the target ID card image specifically includes: converting the token sequence into ID card information in JSON format and outputting it.

[0083] This embodiment utilizes the Swing Transformer as the backbone network, which can better extract the hidden features behind the text in the ID card image, thereby greatly improving the accuracy of identity information recognition and extraction.

[0084] In one embodiment, the trained identity information extraction model is obtained through the following steps:

[0085] In the first training phase, a first training dataset is obtained, which includes sample text.

[0086] With the goal of optimizing the reconstruction loss between the reconstructed text and the sample text, the text destruction and reconstruction functions of the decoder to be trained are trained using the first training dataset to obtain the pre-trained decoder.

[0087] In the second training phase, a second training dataset is obtained, which includes multiple document images labeled with real text labels;

[0088] The document image is input into the identity information extraction model to be trained. The image encoder of the identity information extraction model to be trained encodes the input document image to obtain the sample image vector representation.

[0089] The sample image sequence is obtained by decoding the vector representation of the sample image through the pre-trained decoder of the identity information extraction model to be trained.

[0090] Calculate the loss function value based on the sample image sequence and the corresponding real text label;

[0091] Based on the loss function value, the model parameters of the identity information extraction model to be trained are iteratively updated using the gradient descent method, and the second training dataset is used to continue iteratively training the identity information extraction model to be trained until the training stopping condition is met, thus obtaining the trained identity information extraction model.

[0092] Specifically, the identity information extraction model to be trained includes an image encoder and a decoder to be trained. The identity information extraction model employs a pre-training and fine-tuning approach during training.

[0093] In the first training phase, the decoder to be trained is independent and not part of the identity information extraction model. The original text is input into the decoder to be trained and noise is used to corrupt the original text. The decoder predicts and reconstructs the corrupted text. The reconstruction loss is calculated based on the original text and the reconstructed text, and the model parameters are iteratively updated.

[0094] Taking a decoder built on the BART model as an example, the BART model is a pre-trained language model using the overall structure of the Transformer model. It consists of multiple encoder and decoder layers, each composed of a multi-head self-attention mechanism and a feedforward neural network. During pre-training, the BART model first corrupts the original text with various types of noise, and then reconstructs the original text using a seq2seq model. In the decoding stage, the BART model uses autoregressive techniques to progressively generate the output sequence. The model predicts the next token at each time step and generates continuous text based on previously generated tokens.

[0095] The pre-training of the BART model includes: decomposing the original text using any noise function, reconstructing the text based on the decomposed text, calculating the reconstruction loss function based on the reconstructed text and the original text, updating the model parameters of the decoder based on the reconstruction loss function, and repeating the iterative training until the pre-training stopping condition is met.

[0096] The reconstruction loss of the BART model is the cross-entropy CELoss between the decoder's model output and the original text. CE loss (Cross-Entropy loss) is a commonly used loss function in classification tasks. It is a measure of the cross-entropy between the predicted probability distribution and the true label probability distribution.

[0097] In the second training phase, the identity extraction model to be trained includes an image encoder to be trained and a pre-trained decoder trained in the first training phase. The second training phase is used to jointly train the image encoder and decoder.

[0098] The second training dataset uses document images and their text annotations as training samples. Each document image is labeled with its actual text, meaning it is annotated with its real text. Document images can be either synthetic or non-synthetic. Document images can be ID card images or other images containing text. Preferably, the document images are ID card images.

[0099] In the second training phase, the identity extraction model being trained considers both the image and the preceding text context to predict the next word, thereby learning how to read text. Domain and language flexibility can be achieved by using synthetic data for training.

[0100] Training can stop when the number of iterations reaches a threshold, or when the loss function value is less than a threshold, or when the model converges, etc., but it is not limited to these.

[0101] This embodiment trains the identity information extraction model through individual training and joint training.

[0102] In another specific embodiment, a third training phase, or fine-tuning phase, is also included. This phase involves acquiring a third training dataset, which includes multiple ID card images and real identity information labels. The identity information extraction model trained in the second training phase is then fine-tuned using this third training data. This allows the model to learn how to perform well in the downstream task of ID card information extraction, correctly understanding and extracting ID card information from the ID card images. Of course, the real identity information labels and the extracted ID card information can be partial information from the ID card itself, such as name, ID card number, and possibly one or more of the following: birth date, address, and expiration date.

[0103] In one embodiment, the trained identity information extraction model is obtained through the following steps:

[0104] Obtain the third training dataset, which includes sample ID card images, real identity information labels, and the real identity information in the real identity information labels in the sample ID card images.

[0105] The sample ID card image is input into the identity information extraction model to be trained. The image encoder of the identity information extraction model to be trained encodes the input sample ID card image to obtain the vector representation of the sample ID card image.

[0106] The decoder of the identity information extraction model to be trained decodes the vector representation of the sample ID card image and predicts the text box position to obtain the sample ID card image sequence and the predicted text box position.

[0107] The loss function value is calculated based on the sample ID card image sequence and the corresponding real identity information label, the real text box position and the predicted text box position;

[0108] Based on the loss function value, the model parameters of the identity information extraction model to be trained are iteratively updated using the gradient descent method, and the third training dataset is used to continue iteratively training the identity information extraction model to be trained until the training stopping condition is met, thus obtaining the trained identity information extraction model.

[0109] Specifically, an ID card contains multiple lines of text data, each corresponding to a text box. By extracting the required ID card information, the true identity information of the sample ID card image and the location of the corresponding text box can be determined.

[0110] The number of real text box positions is determined based on the number of real ID card information entries. For example, if the real ID card information is distributed across 8 different rows, then the number of real text box positions is 8.

[0111] The actual text box positions on the sample ID card image can be provided to the model by selecting text boxes on the sample ID card image, or by providing the text box position coordinates to the model as labels. This application does not impose any restrictions on this.

[0112] If the text box position coordinates are provided to the model, they can be represented as (x, y, w, h), where x and y represent the x-coordinate and y-coordinate of the center point of the text box, respectively, and w and h represent the width and height of the text box, respectively. Alternatively, the text box position coordinates can be represented in polar coordinates, specifically as ((x1, y1), (x2, y2), (x3, y3), (x4, y4)), where (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are the coordinates of the four points within the same text box.

[0113] A text box position prediction branch is added to the identity information extraction model, and position-learnable parameters are introduced into the decoder to enable the model to predict the position of the text box for identity information in the input ID card image.

[0114] The text box position prediction branch includes a feature transformation layer and an MLP layer. The feature transformation layer includes a fully connected layer, an activation layer, and a pooling layer. The feature transformation layer is used to convert high-dimensional features into low-dimensional features. The output of the last stage module of the image encoder is input to the feature transformation layer and converted into an image vector representation, which is then input to the decoder. At the same time, the output of the activation layer of the feature transformation layer is passed through the MLP layer to predict the text box position.

[0115] The text box position coordinates are essentially position prompts. The Prompt method is a language prompting-based approach. Its principle is to provide the computer with a prompt, helping it better understand the context of the current task and thus better comprehend the text. Prompt refers to adding extra prompt information as input, transforming downstream tasks such as prediction into language modeling tasks. Prompts provide the language model with a clue / hint, helping it better understand human questions. In this embodiment, by providing the model with the text box position for the ID card information as a prompt, it helps the model better understand which parts of the ID card image to extract the information from, thereby accurately extracting the required ID card information.

[0116] The Prompt method improves model performance by adding prompts to the model, helping it better understand the task's context. In many natural language processing tasks, the Prompt method has demonstrated superior performance compared to traditional models, such as text classification, question answering systems, and machine translation. The Prompt method reduces uncertainty in text processing because the prompts help the model better understand the text and reduce contextual guessing. This is crucial for applications requiring high accuracy, such as financial forecasting and document understanding and extraction of specific areas from ID cards. The Prompt method also helps the model better understand the task's context and improves its generalization ability. This is particularly important for handling data from new domains, which often have different contexts and vocabulary.

[0117] In this embodiment, the loss function value is calculated as an overall loss function value based on the first loss function value between the sample ID card image sequence and the corresponding real identity information label, and the second loss function value between the real text box position and the predicted text box position. Specifically, the overall loss function value can be obtained by weighting the first and second loss function values.

[0118] The first loss function value can be CELoss (Cross-Entropy loss), and the second loss function value can be MSE Loss. MSE Loss measures the squared average difference between the model's predicted output and the true label.

[0119] In one specific embodiment, before training the identity information extraction model using the third training data, the text destruction and reconstruction functions of the decoder to be trained are pre-trained using the first training dataset to obtain a pre-trained decoder. The pre-trained decoder and the image encoder are then used to construct the identity information extraction model to be trained. Pre-training can improve the model's generalization ability, and in the subsequent fine-tuning stage, a small number of samples for specific tasks can be used to train the model to enable it to perform specific tasks.

[0120] This embodiment, based on an OCR-free identity information extraction model, improves the model structure and training method by utilizing location prompts, taking into account the characteristics of scenarios like ID cards. The prompt method adds hints to the model, helping it better understand the context of the ID card information extraction task. Specifically, this embodiment uses location prompts to help the identity information extraction model better understand the identity information to be extracted, thereby improving model performance and the accuracy of identity information extraction. Ultimately, this achieves efficient ID card information recognition, thus promoting the digital development of financial services.

[0121] In one embodiment, the real identity information label is a label that has the same or similar structure as the sample ID card image sequence output by the pre-trained identity information extraction model.

[0122] In one embodiment, a fixed task_token is input along with the target ID card image into the identity information extraction model. This fixed task_token is set according to the specific task and is a string consisting of the task name, converted through the embedding layer of the decoder. Different tasks have different task_tokens.

[0123] For example, the tasks include text generation tasks, automatic question answering tasks, text extraction tasks, etc., and this application does not limit this. The task_token will be fused with other tokens in the image vector representation in the model to form an image vector representation to specify that the model implements a specific function. For example, the task of this application is a text extraction task of identity information. Inputting the specific task_token corresponding to this text extraction task can guide the identity information extraction model to extract identity information. The identity extraction model of this application can also implement other functions, such as automatic question answering, text classification, etc. Of course, different specific task_tokens need to be input to achieve different functions to realize the multi-functional switching of the model.

[0124] In one embodiment, the real text label is a label having the same or similar structure as the sample image sequence output by the pre-trained identity information extraction model.

[0125] Specifically, the real text is converted into a structured label. For example, the structured real text label is: <s_idcard><s_attribute>xxx< / s_attribute>…< / s_idcard>.

[0126] Among them, <s_idcard> is the start flag bit, < / s_idcard> is the end flag bit; <s_attribute> is an attribute bit or attribute field (such as name, gender, etc.), <s_attrbute> is the end bit of the attribute, and xxx is the field value of the attribute field of s_attribute.

[0127] This embodiment is used to extract ID card information. Suppose the extracted ID card information includes name, gender, ethnicity, date of birth, address, ID card number, and expiration date. Then a real text label can be represented as follows:

[0128] <s_idcard><Name>Zhang San< / Name><Gender>Male< / Gender><Ethnicity>Han< / Ethnicity><Birth>August 10, 2002< / Birth><Address>Building D, Community C, Street B, Area A, Beijing< / Address><ID Card Number>110101200208101121< / ID Card Number><Expiration Date>2023.6.10 - 2033.6.10< / Expiration Date>< / s_idcard>.

[0129] Converting real text labels into labels with the same or similar structure as the sample image sequence facilitates the pre-trained identity information extraction model to learn the true difference between the predicted and real values ​​during the training process. This reduces the technical problem of inaccurate loss function calculation caused by inconsistent data formats between predicted and real values, which leads to model training deviation. This embodiment can speed up model training and ensure the accuracy of model training.

[0130] In one embodiment, step S400 involves converting the image sequence to a target format to obtain the ID card information of the target ID card image, including:

[0131] The image sequence is converted into structured data in the target format using regular expressions, and the resulting structured data is used as the ID card information of the target ID card image.

[0132] Specifically, for example, if the target format is JSON, the image sequence of the target ID card image is as follows:

[0133] <parsing><s_idcard><Name>Li Si< / Name><Gender>Male< / Gender><Ethnicity>Han< / Ethnicity><Date of Birth>September 12, 2001< / Date of Birth><Address>Building Q, Community K, Street F, District E, Beijing< / Address><ID Number>110101200109124523< / ID Number><Valid Period>August 10, 2022 - August 10, 2032< / Valid Period>< / s_idcard> <end>.

[0134] The image sequence of the target ID card image is transformed into structured data in JSON format using regular expressions to perform a specific format conversion:

[0135]

[0136] This embodiment uses regular expressions to achieve target format conversion of image sequences. Regular expressions are applicable to the conversion of data in various formats and have a wide range of applications.

[0137] This application achieves end-to-end ID card information extraction, eliminating the influence of text recognition on subsequent processing. The identity information extraction model in this application uses high-dimensional features for information extraction, exhibiting stronger transfer capabilities and greater robustness compared to direct post-processing. This application introduces position-learning variables, which improves the model's recognition performance. The single-stage model simplifies the process, increases automation, and better serves the development of financial services.

[0138] refer to Figure 3 This application also provides a device for extracting ID card information, characterized in that the device includes:

[0139] The first data acquisition module 100 is used to acquire the target ID card image;

[0140] The encoding module 200 is used to input the target ID card image into the trained identity information extraction model, and to extract and encode the target ID card image through the image encoder of the trained identity information extraction model to obtain an image vector representation;

[0141] The decoding module 300 is used to decode the image vector representation through the decoder of the trained identity information extraction model to obtain the image sequence;

[0142] The conversion module 400 is used to convert the image sequence to a target format to obtain the ID card information of the target ID card image.

[0143] This embodiment utilizes an OCR-free identity information extraction model to extract image features and predict text from ID card images, resulting in an image sequence that can be converted to the target format. This achieves accurate and efficient recognition and extraction of ID card information without relying on OCR technology. Compared to OCR technology, this embodiment breaks free from dependence on OCR, is unaffected by the negative impact of OCR error propagation, achieves efficient and accurate document understanding, and reduces computational and time costs. The solution automates most of the document image digitization work, saving manpower and resources. This promotes the digital development of financial services such as banking and insurance.

[0144] In one embodiment, the image encoder is constructed using the Swing Transformer model;

[0145] And / or,

[0146] The decoder is built using the BART model.

[0147] In one embodiment, if the image encoder is constructed using the Swing Transformer model, the image encoder includes an image segmentation module and multiple Stage modules connected in sequence.

[0148] The encoding module 200 specifically includes:

[0149] The segmentation module is used to segment the target ID card image into multiple non-overlapping patches using the image encoder of the trained identity information extraction model;

[0150] The downsampling module is used to downsample the image patches using the various Stage modules of the image encoder to obtain an image vector representation.

[0151] In one embodiment, if the decoder is constructed using the BART model, the decoding module 300 is specifically used to map the image vector representation into a token sequence through the decoder of the trained identity information extraction model, and use the token sequence as an image sequence.

[0152] In one embodiment, the device further includes:

[0153] The second data acquisition module is used to acquire the first training dataset during the first training phase, wherein the first training dataset includes sample text.

[0154] The first training module is used to train the text destruction and reconstruction function of the decoder to be trained using the first training dataset with the goal of optimizing the reconstruction loss between the reconstructed text and the sample text, so as to obtain the pre-trained decoder.

[0155] The third data acquisition module is used to acquire the second training dataset in the second training phase. The second training dataset includes multiple document images labeled with real text labels.

[0156] The second training module is used to input document images into the identity extraction model to be trained. The image encoder of the identity extraction model encodes the input document images to obtain sample image vector representations. The pre-trained decoder of the identity extraction model decodes the sample image vector representations to obtain a sample image sequence. The loss function value is calculated based on the sample image sequence and the corresponding real text labels. Based on the loss function value, the model parameters of the identity extraction model to be trained are iteratively updated using the gradient descent method. The second training dataset is used to continue iteratively training the identity extraction model to be trained until the training stopping condition is met, resulting in the trained identity extraction model.

[0157] In one embodiment, the device further includes:

[0158] The fourth data acquisition module is used to acquire the third training dataset, which includes sample ID card images, real identity information labels, and the real text box positions of the real identity information in the sample ID card images.

[0159] The third training module is used to input sample ID card images into the identity information extraction model to be trained. The image encoder of the identity information extraction model encodes the input sample ID card images to obtain a vector representation of the sample ID card images. The decoder of the identity information extraction model decodes the vector representation of the sample ID card images and predicts the text box positions to obtain a sequence of sample ID card images and the predicted text box positions. Based on the sample ID card image sequence and the corresponding real identity information labels, the real text box positions and the predicted text box positions, the loss function value is calculated. Based on the loss function value, the model parameters of the identity information extraction model to be trained are iteratively updated using the gradient descent method, and the identity information extraction model to be trained is continued to be trained iteratively using the third training dataset until the training stopping condition is met, resulting in the trained identity information extraction model.

[0160] In one embodiment, the real text label is a label that has the same or similar structure as the sample image sequence output by the pre-trained identity information extraction model.

[0161] In one embodiment, the real identity information label is a label that has the same or similar structure as the sample ID card image sequence output by the pre-trained identity information extraction model.

[0162] Figure 4 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 4 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program causes the processor to perform the steps in the above-described method embodiments. The internal memory may also store a computer program, which, when executed by the processor, causes the processor to perform the steps in the above-described method embodiments. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0163] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps:

[0164] Obtain the target ID card image;

[0165] The target ID card image is input into the trained identity information extraction model. The image encoder of the trained identity information extraction model extracts and encodes the target ID card image to obtain an image vector representation.

[0166] The image vector representation is decoded by the decoder of the trained identity information extraction model to obtain the image sequence;

[0167] The image sequence is converted to the target format to obtain the ID card information of the target ID card image.

[0168] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the following steps:

[0169] Obtain the target ID card image;

[0170] The target ID card image is input into the trained identity information extraction model. The image encoder of the trained identity information extraction model extracts and encodes the target ID card image to obtain an image vector representation.

[0171] The image vector representation is decoded by the decoder of the trained identity information extraction model to obtain the image sequence;

[0172] The image sequence is converted to the target format to obtain the ID card information of the target ID card image.

[0173] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0174] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.< / end> < / parsing>

Claims

1. A method for extracting ID card information, characterized in that, The method includes: Obtain the target ID card image; The target ID card image is input into a trained identity information extraction model. The image encoder of the trained identity information extraction model extracts and encodes the target ID card image to obtain an image vector representation. The image vector representation is decoded by the decoder of the trained identity information extraction model to obtain an image sequence; The image sequence is converted to a target format to obtain the ID card information of the target ID card image; The step of decoding the image vector representation using the decoder of the trained identity information extraction model includes decoding the image vector representation using position-learnable parameters introduced into the decoder. The trained identity information extraction model is obtained through the following steps: In the first training phase, a first training dataset is obtained, wherein the first training dataset includes sample text; With the goal of optimizing the reconstruction loss between the reconstructed text and the sample text, the text destruction and reconstruction functions of the decoder to be trained are trained using the first training dataset to obtain the pre-trained decoder. In the second training phase, a second training dataset is obtained, which includes multiple document images labeled with real text tags; The document image is input into the identity information extraction model to be trained, and the image encoder of the identity information extraction model to be trained encodes the input document image to obtain a sample image vector representation. The sample image vector representation is decoded by the pre-trained decoder of the identity information extraction model to be trained, and a sample image sequence is obtained. Calculate the loss function value based on the sample image sequence and the corresponding real text labels; Based on the loss function value, the model parameters of the identity information extraction model to be trained are iteratively updated using the gradient descent method, and the identity information extraction model to be trained is further iteratively trained using the second training dataset until the training stopping condition is met, thus obtaining the trained identity information extraction model. The trained identity information extraction model is obtained through the following steps: Obtain a third training dataset, wherein the third training dataset includes sample ID card images, real identity information labels, and the real identity information in the real identity information labels in the sample ID card images. The sample ID card image is input into the identity information extraction model to be trained. The image encoder of the identity information extraction model to be trained encodes the input sample ID card image to obtain the vector representation of the sample ID card image. The decoder of the identity information extraction model to be trained decodes the vector representation of the sample ID card image and predicts the text box position to obtain the sample ID card image sequence and the predicted text box position. Based on the sample ID card image sequence and the corresponding real identity information label, the real text box position and the predicted text box position, calculate the loss function value; Based on the loss function value, the model parameters of the identity information extraction model to be trained are iteratively updated using the gradient descent method, and the identity information extraction model to be trained is further iteratively trained using the third training dataset until the training stopping condition is met, thus obtaining the trained identity information extraction model. The real identity information label is a label that has the same or similar structure as the sample ID card image sequence output by the pre-trained identity information extraction model.

2. The method according to claim 1, characterized in that, The image encoder is constructed using the Swing Transformer model. And / or, The decoder is constructed using the BART model.

3. The method according to claim 1, characterized in that, If the image encoder is constructed using the Swing Transformer model, then the image encoder includes an image segmentation module and multiple Stage modules connected in sequence; The step of extracting and encoding features from the target ID card image using the image encoder of the trained identity information extraction model to obtain an image vector representation includes: The image encoder of the trained identity information extraction model divides the target ID card image into multiple non-overlapping patches. The image patch is downsampled using each Stage module of the image encoder to obtain an image vector representation.

4. The method according to claim 1, characterized in that, If the decoder is constructed using the BART model, then the decoder, which uses the trained identity information extraction model, decodes the image vector representation to obtain an image sequence, including: The decoder of the trained identity information extraction model maps the image vector representation into a token sequence, and uses the token sequence as an image sequence.

5. A device for extracting ID card information, characterized in that, The method of any one of claims 1-4 is applied to the apparatus, the apparatus comprising: The first data acquisition module is used to acquire the target ID card image; The encoding module is used to input the target ID card image into the trained identity information extraction model, and to extract and encode the target ID card image through the image encoder of the trained identity information extraction model to obtain an image vector representation; A decoding module is used to decode the image vector representation through the decoder of the trained identity information extraction model to obtain an image sequence; wherein, the decoding of the image vector representation through the decoder of the trained identity information extraction model includes decoding the image vector representation using position-learnable parameters introduced into the decoder; The conversion module is used to convert the image sequence to a target format to obtain the ID card information of the target ID card image.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 4.

7. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 4.

Images