A method and system for extracting structured data from credentials

Abstract

This invention provides a method and system for extracting structured data from certificates and licenses, applied in the field of data processing technology. This application determines the layout by using the first 5 characters of the first line in the machine-readable area and features of the visually readable area. It dynamically records any unconfigured new layouts, generates new layout configurations and annotations, completes the extraction of structured data for the new layout, and updates the training dataset in real time. A three-level positioning algorithm fusion strategy is employed, using machine-readable area information as a reference to calculate the consistency score of surname and given name groups, the accuracy score of date data, the proportion of structured data acquisition, and the positioning point deviation value. A preset priority ranking is used, and Hough transform and SIFT feature matching are applied to process image deviations, generating a comprehensive positioning processing result. Using verified accurate data from the machine-readable and visually readable areas as a reference, abnormal certificate data is automatically corrected and completed to generate data to be detected. Based on a target dual-channel OCR model, the data to be detected is processed to generate target certificate and license structured data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for extracting structured data from certificates and licenses. Background Technology

[0002] In the context of globalization, documents (such as passports, visas, and ID cards) serve as crucial credentials for cross-border circulation and identity verification. Accurate extraction of their information is vital for scenarios such as immigration management, financial risk control, and government services. With increasing international exchange, document formats are becoming more diverse. Documents from different countries / regions exhibit significant differences in language type (such as Chinese, Arabic, and Thai), layout (such as the position of machine-readable and visually readable areas), and encoding rules (such as the MRZ code format for machine-readable areas), posing challenges to automated information extraction.

[0003] Existing technologies for extracting document information suffer from the following shortcomings: Insufficient accuracy in recognizing special printed fonts; traditional OCR technology has poor adaptability to handwritten fonts and artistic fonts in the visually readable area of ​​documents, as well as OCR-B-specific fonts in the machine-readable area, easily leading to recognition errors due to font differences (e.g., misrecognizing "ZHANG" as "ZAHNG"). Lack of adaptability to multiple country formats; existing methods largely rely on fixed format templates, failing to automatically adapt to documents from emerging countries / regions or new formats (e.g., passports with minor format adjustments), requiring manual intervention to update templates, resulting in low efficiency. Weak handling of printing and photographic deviations; documents may experience positional shifts or rotations during printing, and uneven lighting, stains, or obstructions during photography can degrade image quality, making it difficult for existing positioning algorithms to accurately locate information elements (such as name and date fields), leading to omissions or misalignments in extraction. The lack of multi-source information cross-verification mechanisms: As the dual carriers of certificate information, the machine-readable area and the visually readable area do not fully utilize the correlation between the two for cross-verification (such as the matching verification of "CHN" in the machine-readable area and "China" in the visually readable area), which can easily lead to the distortion of the overall result due to single-area recognition errors.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore includes information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for extracting structured data from certificates and licenses, which at least to some extent overcomes the problems existing in the prior art. It achieves efficient and accurate extraction of structured data from certificates and licenses through multi-step collaborative processes. First, it acquires certificate and license images containing complex situations, along with machine-readable and visually readable features, and constructs a training dataset after preprocessing. Next, it determines the format and extracts structured data, using a three-level localization algorithm combined with Hough transform to generate localization results. It corrects and completes abnormal data to generate data to be detected, and finally generates the target structured data through a dual-channel OCR model. This method is capable of adapting to multiple national formats, resisting complex interference, and integrating multi-source information to improve the accuracy and robustness of information extraction, meeting the high-efficiency requirements of automated verification scenarios.

[0006] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0007] According to one aspect of this application, a method for extracting structured data of certificates and licenses is provided, comprising: acquiring certificate and license image data, including various certificate and license images with uneven lighting, stains, and occlusion, as well as the first 5 characters of the first line of the machine-readable area and the feature images / text of the visually readable area; preprocessing the images by using black border removal, noise reduction, Gaussian blurring, sharpening, and watermark removal to improve image quality, and constructing training datasets for the machine-readable area and the visually readable area; determining the layout based on the first 5 characters of the first line of the machine-readable area and the features of the visually readable area, dynamically recording the new layout that has not been configured, generating the new layout configuration and annotation, completing the extraction of structured data for the new layout, and updating the training dataset in real time; and employing a three-level localization algorithm fusion strategy, using the machine-readable area information as a reference to calculate... The system uses a pre-defined priority ranking for the consistency score of surname and given name groups, the accuracy score of date data, the proportion of structured data acquisition, and the value of location point deviation. Simultaneously, it employs Hough transform and SIFT feature matching to process image deviations, generating a comprehensive location processing result. Using the verified accurate data from the machine-readable and visually readable areas as a reference, it automatically corrects and completes abnormal document data, generating data to be detected. Finally, it processes the data to be detected using a dual-channel OCR model consisting of a ResNet-Transformer hybrid architecture recognition model for visually readable areas and a CRNN network recognition model for machine-readable areas, designed with character set constraints based on the MRZ encoding standard, generating structured data for the target document.

[0008] Another aspect of this application discloses a certificate structured data extraction device, comprising: an acquisition module for acquiring certificate image data, including various certificate images with uneven lighting, stains, and occlusion, as well as the first 5 characters of the machine-readable area and the image / text features of the visually readable area; a processing module for preprocessing the images, using methods such as black border removal, noise reduction, Gaussian blurring, sharpening, and watermark removal to improve image quality, and constructing training datasets for the machine-readable and visually readable areas; determining the layout based on the first 5 characters of the machine-readable area and the features of the visually readable area, dynamically recording the new layout without configuration, generating new layout configurations and annotations, completing the extraction of structured data for the new layout, and updating the training dataset in real time; and employing a three-level positioning algorithm fusion strategy, using machine-readable area information... The system calculates the consistency score of surname and given name groups, the accuracy score of date data, the proportion of structured data acquisition, and the value of location point deviation based on the information. It adopts a preset priority sorting and uses Hough transform and SIFT feature matching to process image deviations, generating a comprehensive location processing result. Using the verified accurate data of machine-readable and visually readable areas as a reference, abnormal document data is automatically corrected and completed to generate data to be detected. The target dual-channel OCR model, which consists of a ResNet-Transformer hybrid architecture recognition model for visually readable areas and a CRNN network recognition model for machine-readable areas based on the character set constraints designed according to the MRZ encoding standard, processes the data to be detected to generate target document structured data.

[0009] According to another aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a second processor, implements the above-described method for extracting structured data of certificates.

[0010] This application provides a method and system for extracting structured data from certificates and licenses. The server achieves efficient and accurate extraction of structured data through multi-step collaborative processes. First, it acquires certificate images with complex conditions such as uneven lighting and stains, along with the first 5 characters of the machine-readable area and features of the visually readable area. Preprocessing, including black border removal and noise reduction, improves quality and constructs a training dataset. The layout is determined by combining the first 5 characters of the machine-readable area and the visually readable area features. New layouts are dynamically recorded and configured to complete structured extraction. A three-level localization algorithm is used to calculate and sort indicators such as name consistency. Image deviations are addressed using Hough transform and SIFT to generate localization results. Using the verified data as a reference, abnormal data is corrected and supplemented through dynamic mapping verification and Transformer repair networks to generate data to be detected. Finally, a dual-channel OCR model is used to process the data, fusing the recognition results to generate the target structured data. This supports multiple national layouts, achieving a recognition accuracy of 99.2% and strong resistance to printing deviations.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] Figure 1 This document illustrates a flowchart of a method for extracting structured data from certificates and licenses according to an embodiment of this application.

[0013] Figure 2 A schematic diagram of a certificate structure data extraction device provided in one embodiment of this application is shown. Detailed Implementation

[0014] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0015] The following is combined with Figure 1 This application describes a method for extracting structured data from certificates and licenses according to exemplary embodiments. In one embodiment, this application also proposes a method and system for extracting structured data from certificates and licenses. Figure 1 As shown, the method includes:

[0016] S101, acquire certificate image data, including various certificate images with uneven lighting, stains, and obstructions, as well as the first 5 characters of the machine-readable area and the feature images / text of the visual reading area.

[0017] In one implementation, document images are collected under different lighting conditions (such as direct sunlight, low light, backlight, etc.) to ensure coverage of uneven brightness and reflections caused by lighting differences. Specifically, passport images taken outdoors in sunlight are acquired, where direct light causes the photo area to be too bright and the machine-readable text to be obscured by shadows; or visa images taken indoors in low light conditions, where the overall image is dark and the character edges are blurred. Document samples with stains (such as ink, oil, water stains, etc.) are collected to preserve the impact of stains on image clarity and character recognition. Specifically, an image of a driver's license slightly stained with ink is acquired, where some characters in the machine-readable area are covered by ink, forming irregular black patches; or an image of an ID card stained with coffee, where yellowish-brown stains appear in the photo area, causing the edges of the portrait to be blurred.

[0018] Collect license image with partial occlusion, where the occluders include fingers, other objects, or folding and damage of the license itself. Specifically, obtain an image of a passport with the first 5 characters in the machine-readable area partially occluded by a finger when holding the passport, with the first 5 characters in the first row covered by the finger by half; or an image of a Hong Kong and Macao permit with missing text in the visual reading area due to long-term folding, where the issue date area is occluded by a crease and the characters are incompletely displayed. Extract the first 5 characters in the first row from the machine-readable area of the license (such as the MRZ code area at the bottom of the passport), and this information is used for layout determination and subsequent positioning processing. Specifically, for a certain passport, the first row characters in the machine-readable area are "P<CHN123456789", and the first 5 characters extracted are "P<CHN", where "P" represents the passport type and "CHN" is the country code. Through these 5 characters, it can be preliminarily determined that the license belongs to the layout of a Chinese passport.

[0019] Collect key features in the visual reading area, including text information such as portrait photos, names, dates of birth, as well as visual features such as patterns and borders with layout differentiability. Specifically, obtain the portrait photo, text fields such as "name", "gender", "ethnicity" in the visual reading area of the ID card, and the pattern of the border of the certificate; or obtain the picture of the national emblem of the issuing country and the text description of the visa type in the visual reading area of the visa. These features can be used for cross-verification with the information in the machine-readable area to determine the specific layout of the license.

[0020] S102, preprocess the image, use operations such as removing black edges, denoising, Gaussian blur, sharpening, and removing watermarks to improve the image quality, and construct a training data set for the machine-readable area and the visual reading area.

[0021] In one implementation, remove the black border or redundant blank area around the license image caused by the shooting angle or equipment, making the main body area of the license more prominent. Process a scanned image of a passport with an obvious black border. Identify the range of the black border through an edge detection algorithm and crop it off, retaining the main body part of the passport to avoid interference of the black border on subsequent feature extraction. Use a filtering algorithm to remove noise points in the image, such as salt-and-pepper noise, Gaussian noise, etc., to improve the clarity of the image. For an ID card image covered with fine noise due to poor scanning quality, use a median filtering algorithm to process it, making the text and photos on the ID card clearer and reducing the impact of noise on character recognition.

[0022] Smooth the image using a Gaussian kernel function to reduce high-frequency noise in the image while preserving the main edges and contour information of the image. When processing a driver's license image with slight scratches, the Gaussian blur can weaken the impact of the scratches, making the text and patterns on the driver's license smoother and facilitating subsequent feature extraction. Enhance the clarity of object edges and details in the image, making the strokes of the text more distinct and improving the accuracy of OCR recognition. Sharpen a visa image with blurred text edges due to inaccurate focusing during shooting. By increasing the pixel contrast at the edges, the text on the visa becomes clearer and more distinguishable. Remove the watermark pattern or text present in the image to avoid the watermark obscuring the key information of the certificate. For a Hong Kong and Macao permit image with a semi-transparent watermark, use an image restoration algorithm to remove the watermark and fully display the key information such as personal information and issue date on the permit.

[0023] Extract the text area of the machine-readable zone from the preprocessed image, classify and label it according to different character types (such as letters, numbers, special symbols), and construct a dataset for training the machine-readable zone text recognition model. Extract the MRZ code area of the machine-readable zone from a large number of preprocessed passport images, label the letters, numbers, and special symbols (such as "<", "-", etc.) in it. For example, label "P<CHN" as the passport type and country code, and form a machine-readable zone training dataset for training the CRNN network.

[0024] Extract the text and picture areas of the visual-readable zone from the preprocessed image, classify them according to different countries and certificate types, label the text content and visual features in them, and construct a dataset for training the visual-readable zone recognition model. For the visual-readable zone of a Chinese ID card, extract text fields such as portrait photos, names, genders, ethnicities, and visual features such as border patterns, and label them as "Chinese ID card - Name - Zhang San", "Chinese ID card - Portrait Photo", etc., to form a visual-readable zone training dataset for training the ResNet-Transformer hybrid architecture model.

[0025] S103, Determine the format through the first 5 characters in the first row of the machine-readable zone and the features of the visual-readable zone, dynamically record the unconfigured new format, generate new format configuration and annotation, complete the extraction of new format structured data, and update the training dataset in real time.

[0026] In one implementation,版式 determination processing is performed on the first 5 character features of the first line of the machine-readable area and the visual reading area features to generate a版式 determination result. By analyzing the encoding rules of the first 5 characters of the first line of the machine-readable area (such as document type, country code) and the visual features of the visual reading area (such as text layout, pattern style), the preset版式 template is matched to determine the country or region版式 to which the certificate belongs. The first 5 characters of the first line of a certain passport machine-readable area are "P<USA", where "P" represents the passport type and "USA" is the US country code. At the same time, the visual reading area contains the US国徽图案 and the text "UNITED STATES OF AMERICA", from which it is determined that this certificate is of the US passport版式.

[0027] Dynamic recording processing is performed on the unconfigured new版式 to generate an unconfigured new版式 record. When the certificate features input cannot match any existing版式 template, the system automatically records information such as the character features of the machine-readable area and the visual features of the visual reading area of the certificate to form an unconfigured new版式 record. A certificate from a certain emerging country is received. The first 5 characters of the first line of its machine-readable area are "N<XYZ" ("XYZ" is the new country code), and there are unique text layouts and local特色图案 in the visual reading area. There is no corresponding版式 in the existing templates, and the system automatically records the feature information of this certificate to generate an unconfigured new版式 record.

[0028] Configuration and annotation processing are performed on the newly added版式 to generate newly added版式 configuration and annotation information. For the unconfigured new版式 confirmed by人工, detailed annotation of the版式 features (such as the meaning of the characters in the machine-readable area, the field positions in the visual reading area) is performed, and it is configured into the system's版式 template library to generate the configuration file and annotation information of the newly added版式. For the above-mentioned certificate of the emerging country, after人工 confirms that it is a valid new版式, the "N" in the machine-readable area is marked as the document type code of this country, "XYZ" as the country code, and the specific areas in the visual reading area as the name and date of birth fields, and the coordinate mapping relationship of the版式 template is configured to generate newly added版式 configuration and annotation information.

[0029] Structured data extraction processing is performed based on the版式 determination result, unconfigured new版式 record, and newly added版式 configuration and annotation information to generate a new版式 structured data extraction result. Among them, the new版式 structured data extraction result is used to complete the new版式 structured data extraction and update the training data set in real time. Combining the determined版式 result, unconfigured版式 record, and newly added版式 configuration information, key information (such as name, date, document number, etc.) is extracted from the certificate image and output in a structured format, and at the same time, the new版式 data is updated to the training data set in real time.

[0030] For documents determined to be in the format of a US passport, extract the MRZ code information in the machine-readable zone (such as name "SMITH<JOHN", passport number "123456789") and the portrait, issuance date, etc. information in the visual reading zone according to the format template, and generate structured data. If a new format is discovered during the processing (such as minor format changes in the new version of the US passport), update the characteristics of this new format to the training dataset for subsequent learning by the model.

[0031] S104, adopt a three-level positioning algorithm fusion strategy. Taking the information in the machine-readable zone as a reference, calculate the consistency score of surname and name grouping, the accuracy score of date data, the acquisition ratio of structured data, and the deviation value of positioning points. Use a preset priority sorting, and at the same time apply the Hough transform and SIFT feature matching to process image deviation, and generate a comprehensive result of positioning processing.

[0032] In one implementation, based on the three-level positioning algorithm fusion strategy, take the information in the machine-readable zone as a reference to calculate and process the consistency score of surname and name grouping, the accuracy score of date data, the acquisition ratio of structured data, and the deviation value of positioning points, and generate a calculation result. The framework of the three-level positioning algorithm is as follows. The benchmark positioning method, based on a pre-constructed format template, directly locates the information elements of the certificate through the coordinate mapping relationship (such as the standard positions of name and date fields). For the format of a Chinese passport, when the first 5 characters in the first line of the MRZ code in the machine-readable zone are "P<CHN", the benchmark positioning method presets that the surname and name fields are located at the 6th - 13th characters in the machine-readable zone (such as "ZHANG<SAN"), and maps them to the corresponding area in the visual reading zone (such as the coordinate position of "张三").

[0033] The SVM detection method uses a support vector machine (SVM) model to train a key feature detector, and identifies the information element area through feature vector matching (such as distinguishing the name from other fields through features such as character spacing and font size). When training the SVM model to identify the surname and name fields in the visual reading zone of the passport, taking the position of the two characters "姓名" as an anchor point, combined with feature vectors such as character height (such as 12 pixels) and field spacing (such as 5 pixels), detect whether the position of "张三" conforms to the standard format. The auxiliary information positioning method assists in positioning missing or blurred fields through associated information (such as the logical relationship between name and date of birth, and the coding rules of the certificate number). If the surname and name fields in the visual reading zone are partially blocked, but the name in the machine-readable zone is "LI<HUA", and the date of birth is "1990010"1", through the logical association of "name - date of birth", it can be inferred that the "李华" field in the visual reading zone should be adjacent to the date of birth field, thus assisting in positioning.

[0034] The calculation formula for the consistency score of surname and name grouping is ; Compare the name splitting results (such as surname and given name grouping) in the machine-readable area and the visually-readable area, and count the proportion of matching quantities. Specifically, the name in the machine-readable area is "ZHANG<SAN" (surname "ZHANG", given name "SAN", total number of fields 2); the name in the visually-readable area is "张三" (surname "张", given name "三", total number of fields 2); the number of matching fields = 2, and the final score is .

[0035] The calculation formula for the accuracy score of date data is

[0036] . Compare the actual date with the standard date (such as the reasonable range of the issue date), and the smaller the error, the higher the score. Specifically, the actual issue date is 20230501 (converted to the ordinal date is 121); the standard date takes the mid-value of 2023, which is 20230615 (ordinal date is 166); the time range is 365 days throughout 2023; the final score is

[0037] .

[0038] The calculation formula for the acquisition ratio of structured data is

[0039] . Count the proportion of the extracted key information (such as name, date, ID number) in the total fields to be extracted. Specifically, the fields to be obtained are name, gender, date of birth, ID number, issue date (a total of five items); the fields that have been obtained are name, date of birth, ID number (a total of three items); the final proportion is .

[0040] The calculation formula for the deviation value of the positioning point:

[0041] . Quantify the deviation between the positioning coordinates and the standard coordinates through the distance formula between two points in the plane rectangular coordinate system.

[0042] Specifically, the upper left corner coordinates of the standard name area: (100, 200); the actual positioning coordinates: (102, 203);

[0043] The final deviation value is pixels.

[0044] In another implementation, the benchmark positioning method is preferably used for standard format certificates (such as common national passports) to directly generate initial positioning coordinates; if the benchmark positioning method fails to match (such as abnormal format), switch to the SVM detection method to reposition through feature matching; when there is still ambiguity in the SVM detection method, enable the auxiliary information positioning method to correct the positioning result through logical association (such as the character length rule of the name and the ID number).

[0045] Due to the tilted shooting angle of a Thai visa image, the name area generated by the reference positioning method has a large deviation (deviation value of 5 pixels). However, the SVM detection method matches through the national emblem feature (similarity of 90%) and the auxiliary information positioning method (logical association between visa type and name format). Finally, it is calculated that the name consistency score is 85% (partial character matching between "CHAI<THONG" in the machine-readable area and " " in the visual reading area); the date accuracy score is 95% (the actual date 20231001 is within the standard range); the structured data acquisition ratio is 70% (4 / 5 fields have been acquired); after comprehensive scoring, the positioning result of the SVM detection method is selected. After correcting the tilt through the Hough transform, the final deviation value is reduced to 2 pixels.

[0046] Sort the calculation results according to the preset priority to generate the target sorting result. According to the preset priority (name grouping consistency Score1 > date accuracy Score2 > structured data acquisition ratio Score3 > positioning point deviation Score4), sort the results of different positioning algorithms for Thai visas and select the optimal comprehensive scoring plan. A Thai visa image obtains the following scores through the reference positioning method (A) and the SVM detection method (B): Algorithm A, Score1 (name consistency): The name "CHAI<THONG" in the machine-readable area partially matches " " in the visual reading area, getting 0.85 (full score 1); Score2 (date accuracy): The issuance date "20231001" is within the standard range, getting 1; Score3 (data acquisition ratio): 4 / 5 fields have been extracted, getting 0.8; Score4 (positioning deviation): 5 pixels, converted to 0.2 according to the threshold (the smaller the value, the higher the score).

[0047] Algorithm B, Score1: Through national emblem feature matching and layout logic, the name completely matches, getting 1; Score2: The date score is 0.95; Score3: 4 / 5 fields are obtained, getting 0.8; Score4: The initial deviation is 3 pixels, converted to 0.4. Since Score1 has the highest priority and the Score1 (1) of Algorithm B is higher than that of Algorithm A (0.85), the positioning result of Algorithm B is selected as the target sorting result.

[0048] Process the image deviation using the Hough transform and SIFT feature matching to generate the image deviation processing result. The Hough transform corrects the tilt deviation to detect the edge straight line of the Thai visa, calculates the tilt angle and then rotates for correction. For example, due to a 15° tilt in the shooting of the visa image, the MRZ code in the machine-readable area is slanted. The Hough transform detects that the angle between the edge straight line and the horizontal axis is 15°, and rotates the image clockwise by 15° to make the MRZ code horizontal again, reducing the positioning deviation from 5 pixels to 2 pixels.

[0049] SIFT feature matching is used to process scaling deviations and extract SIFT feature points from the national emblem pattern in the reading area. These points are then matched with a standard template, and the image size is restored after calculating the scaling ratio. For example, in this visa image, due to the close shooting distance, a portion is magnified by 1.2 times, and the spacing between the feature points of the national emblem pattern is 20% larger than that of the standard template. SIFT matching yields a scaling ratio of 1.2, reducing the image size to 0.83 times (1 / 1.2), thus reducing the name area positioning deviation from 3 pixels to 1 pixel.

[0050] Based on the target sorting results and image deviation processing results, a comprehensive positioning processing result is generated, which is used to achieve precise positioning. The optimal algorithm result after sorting (SVM detection method) and the deviation correction result are fused to generate the final positioning coordinates, ensuring that the deviation of key information (name, date) is ≤2 pixels. Initial positioning and sorting: The SVM detection method initially positions the name area coordinates as (120, 180), but due to image tilt and scaling, the actual display deviation is 3 pixels. Deviation correction: After a 15° rotation using Hough transform, the coordinates are adjusted to (125, 185); after SIFT scaling, the coordinates are further adjusted to (122, 183). The deviation between the final positioning coordinates and the standard layout is... The image size was 2 pixels, but after correction, the visual deviation was less than 2 pixels (OCR recognition threshold), which meets the requirements for accurate positioning.

[0051] When a Thai visa has both layout anomalies (non-standard positioning) and geometric deviations: the three-level algorithm prioritizes SVM detection (with a higher Score1) to determine the positioning coordinates; Hough transform and SIFT matching only correct geometric deviations without changing the logical priority of the algorithm's ranking results, ensuring that the scoring logic of "name consistency > date accuracy" is maintained throughout the entire process. If the corrected coordinates conflict with the logical positioning of Algorithm B (e.g., the name area exceeds the image boundary), the system will revert to Algorithm A and re-correct to ensure the result is reasonable (e.g., in this example, the corrected coordinates are still within the valid area of ​​the image, and no revert is triggered).

[0052] S105 uses the verified accurate data of the machine-readable and visually readable areas as a reference to automatically correct and complete abnormal document data, generating data to be tested.

[0053] In one implementation, a dynamic mapping verification mechanism is constructed. The first 5 characteristic characters in the machine-readable area and the key information in the visually-readable area are cross-verified through a two-way hashing verification algorithm. The semantically structured accurate data in the machine-readable area and the visual feature vector matrix in the visually-readable area after verification are extracted to generate a data reference benchmark with confidence weights. The cross-verification logic between the machine-readable area and the visually-readable area is as follows. The input is the first 5 characters in the machine-readable area (such as "V<VNM") and the key features in the visually-readable area (name, national emblem pattern, etc.). Semantic parsing of the machine-readable area, the encoding rules of the machine-readable area are matched through regular expressions. For example, in "V<VNM", "V" represents the visa type, and "VNM" is the country code of Vietnam (ISO3166-1 code). Feature extraction in the visually-readable area, the visual feature vector (such as V1) of the national emblem pattern in the visually-readable area is extracted using CNN (such as ResNet-18), and the name in the visually-readable area is recognized through OCR and its format is parsed (such as "NGUYEN<VANA" conforms to the format of the Vietnamese name "surname<given name"). Two-way hashing mapping, the hash values are respectively generated for the semantic label in the machine-readable area ("Vietnamese visa") and the feature vector in the visually-readable area (V1), and the hash matching degree between the two is compared (such as calculating the similarity using the Hamming distance).

[0054] The verification threshold and confidence are calculated as follows. The matching degree formula is

[0055] ; where (the weight of the machine-readable area is higher), , both the semantic matching degree (country code, document type) and the visual feature matching degree (national emblem, layout) are normalized to [0, 1]. The matching degree between the Vietnamese country code "VNM" and the national emblem in the visually-readable area belonging to Vietnam is 1; the cosine similarity between the national emblem feature vector V1 and the standard Vietnamese national emblem vector V_std is 0.95; finally , it is converted to a confidence of 0.95 (mapped through the Sigmoid function). The benchmark generation process is as follows. Machine-readable area parsing: The machine-readable area "V<VNM" of the Vietnamese visa is parsed into the visa type V and the country code VNM; Visually-readable area feature extraction: The name "NGUYEN<VANA" is recognized through CRNN, and the coordinates (100, 150) are determined by the positioning algorithm; The national emblem pattern is extracted through HOG features + SVM classification to obtain the feature vector V1; The hash matching degree between "VNM" in the machine-readable area and the national emblem V1 in the visually-readable area is 95%, triggering the "correct layout" mark; The process of generating the benchmark is to integrate semantic, visual, and coordinate information, attach a confidence of 0.95, and store it in the dynamic template library.

[0056] Secondary verification of low-confidence data. If the matching degree between the machine-readable area "T<THA" of a Thai visa and the features in the visually-readable area is only 0.7 (lower than the threshold of 0.8), the system automatically starts: multi-feature supplementary verification, and additional extraction of the text layout in the visually-readable area (such as unique Thai characters ), features such as border patterns; historical data retrieval, query the similar feature vectors of Thai visas in the dynamic template library, and match the three closest templates through the KNN algorithm, and weight the average confidence to 0.85.

[0057] When the data in the machine-readable area conflicts with the data in the visual-readable area (such as "C<CHN" in the machine-readable area corresponds to China, but the national emblem in the visual-readable area is the chrysanthemum pattern of Japan): trust the data in the machine-readable area first (confidence 0.9 vs 0.7 in the visual-readable area); mark the features in the visual-readable area as abnormal, trigger the manual review process, and update the template library after the review is passed (such as the new version of the Japanese visa misusing the machine-readable area code of China). Anti-forgery ability, prevent single-modal forgery of the machine-readable area / visual-readable area through two-way verification (such as when forging the machine-readable area code but the features in the visual-readable area do not match, the confidence is <0.5); dynamic adaptability, new layouts (such as the subtle adjustment of the national emblem of the new version of the Vietnamese visa in 2024) can be updated through the reference data, and the subsequent recognition accuracy can be automatically improved.

[0058] Design an anomaly semantic repair network based on Transformer, take the semantic feature vector of the data reference benchmark as the conditional input, perform context-related repair on the semantic missing or conflicting fields in the abnormal certificate data, dynamically match the feature distribution of similar certificate samples through the attention mechanism, generate the corrected result of the certificate data with the corrected confidence, and introduce adversarial training in the correction process to ensure the consistency of the feature distribution with the benchmark data. Taking the semantic features of the data benchmark as the condition, repair the semantic missing fields through the Transformer network, and introduce adversarial training to ensure consistency with the benchmark feature distribution. The name "SCHMIDT<" in the visual-readable area of a German passport is missing the last character due to stains, and the corresponding field in the machine-readable area is "SCHMIDT<J". Input the semantic features of the German layout in the data benchmark (such as the name format is "surname<first name");

[0059] Transformer matches similar samples through the attention mechanism (such as "SCHMIDT<JOHANNES" in historical German passports), predicts that the missing character is "J"; adversarial training compares the corrected result with the benchmark feature distribution to ensure that the character spacing and encoding rules of the corrected name "SCHMIDT<J" are consistent with the benchmark, and the confidence is 0.92.

[0060] A multimodal feature completion model is established to address scenarios where the visually readable area is occluded or the machine-readable area is incomplete. It utilizes a generative adversarial network combined with Hough transform geometric constraints to extract prior knowledge of the document layout from a data reference benchmark. This knowledge is then used to structurally complete the missing data, generating a document data completion result that includes spatial location constraints. The completion accuracy is optimized using a dual loss function of cross-entropy and structural similarity. The multimodal fusion logic is as follows: input data consists of occluded / incomplete document images (e.g., the visually readable area of ​​an Indian visa "199?0101"); and a data reference benchmark (including prior knowledge of the Indian visa layout, such as the birth date format "YYYYMMDD").

[0061] The model also includes the following modules: a Hough transform module, used to detect geometric features in the image (such as fold lines and borders) and locate incomplete regions; a generative adversarial network (GAN), which generates reasonable completion results for missing data based on the layout prior; and a spatial constraint module, which adds coordinate position constraints to the completed data based on the localization algorithm results.

[0062] The formatting rules analyzed from the data benchmark are as follows: Indian visa birth date format: 4-digit year + 2-digit month + 2-digit day (e.g., "19950101"); spatial distribution characteristics: the birth date field is usually located in the lower right corner of the reading area, with a font size of 10-12 pixels and uniform character spacing.

[0063] The Hough transform was used to locate occluded regions. The Indian visa image was converted to grayscale, and contours were extracted using Canny edge detection. The Hough transform was applied to detect straight line features and identify folded line parameters (such as the equation of a straight line in polar coordinates). ); Calculate the coordinates of the occluded area, and based on the position of the fold line, determine the pixel range of the last digit "?" in the year as (150, 200-180, 220). The fold line results in the area containing "?" in "199?0101" being a rectangle 30 pixels wide and 20 pixels high, with the top-left corner at coordinates (150, 200).

[0064] Based on GAN-based data completion, the specific conditional GAN ​​architecture is as follows: Generator (G): Input is the format prior from the data benchmark (e.g., "Indian visa-birth date-4-digit year") and occlusion region features, outputting the completed character image; Discriminator (D): Distinguishes the difference between the generated completed characters and the actual Indian visa date characters, improving generation quality through adversarial training. The distribution of Indian visa years (90% from 1980-2020) is obtained from the data benchmark. Combined with the prior information of the machine-readable region "19950101", the missing character is determined to be "5". GAN generates an image of "5" and matches it with the Indian visa numeral font (e.g., OCR-B font, tilt angle 5°, stroke thickness 1.5 pixels).

[0065] The coordinates of the completed character "5" are fixed at (175, 200) (center of the occluded area) to ensure consistent font spacing with the original date field "199?0101" (e.g., 5 pixels between characters). A dual verification mechanism is used: Cross-entropy verification: Calculate the error between the generated year "1995" and the baseline year distribution (e.g., 85% from 1990-2000). In this example, the error is 0.03 < 0.05. Structural similarity (SSIM) verification: Compare the pixel structure of the completed date "19950101" with that of a standard Indian visa date. SSIM = 0.92 > 0.9.

[0066] In another implementation, the logic for completing missing data in the machine-readable area is as follows: If the machine-readable area of ​​a Brazilian passport is damaged, and the birth date in the visually readable area only displays "19??", and the birth date format of the Brazilian passport in the data benchmark is "YYYY": A GAN is used to generate possible years (e.g., estimating 1970-2000 based on the holder's age); Hough transform is used to detect the line spacing of the text in the visually readable area to ensure that the character height (12 pixels) of the generated year "1985" is consistent with the original field; Monte Carlo sampling is used to generate 10 possible years, and SSIM is used to filter the result "1985" which is closest to the benchmark format.

[0067] Multiple rounds of completion were performed for complex occlusion in a Saudi Arabian visa reading area where oil stains obscured multiple fields, including name and date of birth. The first round of completion involved locating the oil stain area using Hough transform and then using GAN to complete the first letter "M" of the name. The second round of completion used the completed name as a condition, combined with the Saudi name format (such as "surname + first name") in the data baseline, to complete the remaining characters "OHAMMED". Each round of completion was validated using cross-entropy and SSIM to ensure that the cumulative error was <0.1.

[0068] In another implementation, a generative adversarial network (GAN) and Hough transform are combined to extract the format prior from the data baseline, complete the occluded / incomplete data, and add spatial constraints. For example, the birth date on a certain Indian visa is displayed as "199?0101" due to folding in the visual reading area, while the corresponding field in the machine-readable area is "19950101". The Hough transform detects the fold line position and determines the coordinates of the occluded area (150, 200-180, 220). The GAN extracts the format of the Indian visa birth date (4-digit year + 2-digit month + 2-digit day) from the data baseline; completes "19950101", and verifies it through cross-entropy (the error in the distribution of the predicted year compared to the baseline year is <0.05) and structural similarity (SSIM>0.9), generating a complete result with spatial constraints (coordinates (150, 200)).

[0069] An heterogeneous data fusion decision-making mechanism is proposed. A graph neural network is used to construct an association graph of the data in the machine-readable area and the visual-readable area. The correction results and completion results of the certificate data are mapped to the graph nodes. The node features are fused through attention weights to generate the integrated certificate data with cross-modal consistency verification. The consistency verification calculates the matching degree of the feature sequences between modalities through the dynamic time warping algorithm. Use a graph neural network to construct a data graph of the machine-readable area and the visual-readable area, fuse the correction and completion results, and verify the cross-modal consistency through dynamic time warping (DTW). There is a conflict between the name "DUPONT<JEAN" in the machine-readable area and the name "DUPONTJEAN" (with extra spaces) in the visual-readable area of a French visa.

[0070] Construct graph nodes: the name in the machine-readable area (node A), the name in the visual-readable area (node B), and the layout rule (node C); fuse the node features through attention weights and find that the extra spaces in the visual-readable area are in a format allowed by the French layout (weight of node C is 0.8); calculate the sequence matching degree of "DUPONT<JEAN" in the machine-readable area and "DUPONTJEAN" in the visual-readable area by DTW as 90%, confirm the consistency, and generate the integrated name "DUPONTJEAN".

[0071] Based on the feature complexity of the integrated certificate data, dynamically adjust the input dimension of the ResNet-Transformer hybrid architecture to generate the data to be detected with preprocessing strategy labels. The strategy labels are automatically optimized according to the historical OCR recognition errors through a reinforcement learning algorithm, realizing the intelligent adaptation of the input data for the subsequent dual-channel OCR model. According to the feature complexity of the integrated data, dynamically adjust the ResNet-Transformer input dimension and optimize the preprocessing strategy through reinforcement learning. The integrated data of a Russian passport contains complex national emblem patterns and encrypted characters in the machine-readable area, with high feature complexity. Dynamically adjust the input dimension of ResNet-50 from

[0072] to to enhance the extraction of pattern features; the reinforcement learning automatically selects the preprocessing strategy of "sharpening + color enhancement" according to the historical errors (such as the national emblem recognition error rate of 0.15); generate the data to be detected with strategy labels (preprocessing level: high complexity - enhanced processing), and improve the dual-channel OCR recognition accuracy to 98.5%.

[0073] Conflict avoidance mechanism. If the "J<JPN" in the machine-readable area of a Japanese visa fails to match the cherry blossom pattern in the visual reading area (normally it should be a chrysanthemum pattern), the dynamic mapping verification will mark it as abnormal, and the visual reading area features will be corrected based on the data in the machine-readable area first to avoid misjudgment of the layout. Priority linkage. When there is a conflict between the supplemented data and the repaired data (such as the date "20231231" in the machine-readable area and the supplemented date "20230101" in the visual reading area), based on the fact that the confidence level (0.98) of the machine-readable area in the data benchmark is higher than the confidence level (0.85) of the supplemented data in the visual reading area, the data in the machine-readable area is selected as the final result.

[0074] S106, process the data to be detected based on the target dual-channel OCR model composed of a ResNet-Transformer hybrid architecture recognition model dedicated to the visual reading area and a CRNN network recognition model dedicated to the machine-readable area and designed with a character set constraint based on the MRZ coding specification, and generate the structured data of the target certificate.

[0075] In one implementation, based on the target dual-channel OCR model composed of a ResNet-Transformer hybrid architecture recognition model dedicated to the visual reading area and a CRNN network recognition model dedicated to the machine-readable area and designed with a character set constraint based on the MRZ coding specification, input processing is performed on the data to be detected to generate model input data. The dual-channel OCR model is composed as follows. The visual reading area model is a ResNet-Transformer hybrid architecture. Specifically, the hierarchical structure: ResNet-50 is used as the feature extraction backbone (16 residual blocks, outputting 2048-dimensional features), followed by 2 layers of Transformer encoders (8 heads, hidden layer dimension 512), which are used to process the visual semantic features of the image text. The total number of parameters is about 38 million, which is suitable for extracting visual features of fonts in multiple countries (such as Chinese, Thai, Arabic).

[0076] The machine-readable area model is a CRNN network based on MRZ coding constraints. The hierarchical structure: The CNN layer (3 layers of convolution + max pooling) extracts character features, the BiLSTM layer (256 hidden units) captures sequence dependencies, and the CTC loss layer constrains the output character set (only including letters, numbers and special symbols such as "<", "-" in the OCR-B font). According to the MRZ coding specification, the character set in the machine-readable area is limited to , a total of 64 characters.

[0077] The data to be detected for a certain Chinese passport is as follows. The image in the visual inspection area contains the image areas of the name "Zhang San" and the date of birth "19900101". The text in the machine-readable area is the MRZ code "P<CHNZHANG<SAN<1990010>1<CHN…". The image in the visual inspection area is normalized to 224×224 pixels and input into ResNet-Transformer. The text in the machine-readable area is converted into a sequence vector, and the length is standardized to 44 (the standard length of MRZ), and then input into CRNN.

[0078] Use the ResNet-Transformer hybrid architecture recognition model dedicated to the visual inspection area to perform feature extraction and recognition processing on the visual inspection area data in the model input data, and generate the recognition result of the visual inspection area. At the same time, use the CRNN network recognition model dedicated to the machine-readable area and designed with character set constraints based on the MRZ coding specification to perform feature extraction and recognition processing on the machine-readable area data in the model input data, and generate the recognition result of the machine-readable area. The recognition process of the ResNet-Transformer hybrid architecture in the visual inspection area is as follows. The ResNet-50 feature extraction layer contains 5 stages (stage1-stage5) and a total of 16 residual blocks. Specifically: stage1:

[0079] Max pooling, and the output feature map size : It contains 3, 4, 6, and 3 residual blocks respectively, and each residual block adopts the "bottleneck structure"

[0080] , and finally outputs a feature map of dimension. For the image of "Zhang San", the residual block in stage3 of ResNet-50 extracts the features of the left and right strokes of the character "Zhang" (such as the curvature of the strokes and the position of the intersection points), and stage5 captures the global features such as the horizontal stroke spacing of the character "San", forming a 2048-dimensional feature vector.

[0081] The Transformer encoder layer includes 2 layers of encoders, and each layer contains: multi-head self-attention mechanism (8 heads, each head has a dimension of 64, and the total hidden layer dimension is 512); feed-forward neural network

[0082] , with LayerNorm normalization. Through the attention mechanism, the model learns the positional association between "Zhang" and "San" (such as the name layout rule that "Zhang" is on the left and "San" is on the right), and combines the knowledge of the pre-trained language model (such as Chinese names are usually 2-3 characters) to strengthen the semantic consistency of "Zhang San" as a complete name. The name image in the visual reading area is normalized to 224×224 pixels. After extracting features by ResNet-50, it is flattened into a 7×7×2048 = 100352-dimensional vector, and then converted into a 512-dimensional input for the Transformer through linear projection. The output sequence of the Transformer passes through the softmax layer to generate the character probability distribution at each position, and combines the language model (such as n-gram) to correct logical errors (such as avoiding non-name characters following "Zhang"). Finally, the text "Zhang San" is generated, and the confidence is calculated by the mean of the softmax probabilities (0.98 for "Zhang" + 0.95 for "San") / 2 = 0.965.

[0083] The CRNN network in the machine-readable area is recognized as follows. The CRNN hierarchical structure includes the CNN layer: 3 layers of convolution + max pooling, specifically: Convolution layer 1: 3×3×64 (stride 1, padding 1) → ReLU → 2×2 max pooling; Convolution layer 2: 3×3×128 (stride 1, padding 1) → ReLU → 2×2 max pooling; Convolution layer 3: 3×3×256 (stride 1, padding 1) → ReLU → 1×1 max pooling, and the output feature map size is 1×1×256. The BiLSTM layer, 2 layers of bidirectional LSTM, with 256 hidden units in each layer, captures the forward and backward dependencies of the character sequence (such as the position rule of "<" as a separator in the MRZ code). The CTC loss layer: The output dimension is 64 (corresponding to the MRZ encoding character set: ), forcing the model to output legal characters.

[0084] The MRZ encoding rules are constrained as follows. Character set limitation: Only 64 characters are allowed, and illegal characters such as "@" and "#" are prohibited from being output; Format constraint: For example, a "<" separator must follow the name segment (such as "ZHANG<SAN<"), and this rule is learned through the sequence modeling ability of CRNN. Sequence decoding and error elimination. The MRZ code image in the machine-readable area is normalized to 128×32 pixels. After the CNN extracts features, a feature sequence of length 32 (1×256 dimensions at each time step) is formed and input into the BiLSTM. The CTC decoding example is as follows. Input the image of "ZHANG<SAN", the hidden state of the output sequence of the BiLSTM, and the CTC loss function merges consecutive repeated characters and removes blank characters, and finally decodes to "ZHANG<SAN". Due to the character set constraint, the model will not mis-recognize "<" as "," or other symbols, ensuring that the output conforms to the MRZ standard.

[0085] Regarding complex font recognition, when the visual reading area processes the Arabic name " ", ResNet-50 extracts the curve features of Arabic letters, and Transformer captures the letter joining rules through the attention mechanism, with an identification accuracy rate of 98.7%. Regarding the repair of blurred characters in the machine-readable area, when some characters in the machine-readable area "ZHANG<SAN" are blurred due to stains, CRNN, through the sequence memory ability of BiLSTM, combined with the context of the front and back characters "ZHANG" and "SAN", can still correctly output the complete field, and the accuracy rate is increased by 15% compared with the single CNN model. The visual reading area model fuses visual features and semantic understanding through ResNet-Transformer to solve the recognition problem of multi-language mixed layout (such as Chinese-English bilingual visas), and the semantic error rate is reduced by 40% compared with the pure CNN model. The machine-readable area model embeds the MRZ encoding rules into the CRNN structure and realizes end-to-end sequence recognition through the CTC loss, without manual alignment, and adapts to more than 100 MRZ layout changes.

[0086] Perform cross-modal fusion processing on the recognition results of the visual reading area and the machine-readable area, and generate a fusion recognition result through feature association and consistency verification. Construct the mapping relationship between the machine-readable area and the visual reading area (such as the name "ZHANG<SAN" in the machine-readable area corresponds to "张三" in the visual reading area). The consistency verification algorithm is When the number of matching fields ≥ 3 and the average confidence level ≥ 0.9, it is considered that the fusion is effective. The specific fusion example is as follows. The recognition result in the machine-readable area is the name "ZHANG<SAN" and the date of birth "19900101"; the recognition result in the visual reading area is the name "张三" and the date of birth "January 1, 1990"; the name mapping "ZHANG<SAN" and "张三" match, getting 1 point; "19900101" is converted to "January 1, 1990", and the match gets 1 point; the average confidence level is 0.95 in the machine-readable area, 0.92 in the visual reading area, and the average is 0.93; Consistency_score=(2 / 2)×0.6+(0.93 / 1)×0.4=0.6+0.372=0.972, and the fusion result "Name: 张三, Date of Birth: January 1, 1990" is generated.

[0087] Based on the fusion recognition result, generate the structured data of the target license certificate, where the structured data of the target license certificate is used to realize the structured extraction and output of the license certificate information. The structured data of the target license certificate is If the "Date of Birth" in the visual reading area is identified as "January 1, 199?" due to occlusion, the corresponding field in the machine-readable area is "19900101": Based on the machine-readable area data, complete the date in the visual reading area to "January 1, 1990"; label the recognition confidence level of the visual reading area as 0.8, the machine-readable area as 0.95, and the confidence level of the date field in the final structured data as 0.9 (weighted by α=0.7 for the machine-readable area and β=0.3 for the visual reading area).

[0088] This application achieves efficient and accurate extraction through multi-step collaborative processing. First, it acquires image data of certificates with complex conditions, along with features of the machine-readable and visually readable areas. Preprocessing improves quality and a training dataset is constructed. Next, the layout is determined by combining the first five characters of the first line in the machine-readable area with features of the visually readable area. New layouts are dynamically recorded and configured, completing structured extraction. A three-level localization algorithm is employed, calculating and ranking multiple indicators. Hough transform and SIFT feature matching are used to handle image deviations and generate localization results. Using the verified data as a reference, abnormal data is corrected and supplemented through dynamic mapping verification and a Transformer repair network, generating data to be detected. Finally, a dual-channel OCR model is used to process the data to be detected, fusing the recognition results to generate target structured data. This supports multiple national layouts, achieving a recognition accuracy of 99.2% and strong resistance to printing deviations.

[0089] In one implementation, such as Figure 2 As shown, this application also provides a certificate structured data extraction device, comprising:

[0090] The acquisition module 201 is used to acquire certificate image data, including various certificate images with uneven lighting, stains, and occlusion, as well as the first 5 characters of the first line of the machine-readable area and the feature images / text of the visual reading area;

[0091] Processing module 202 is used for image preprocessing, employing techniques such as black border removal, noise reduction, Gaussian blurring, sharpening, and watermark removal to improve image quality and construct training datasets for machine-readable and visually readable areas. It determines the layout based on the first 5 characters of the first line in the machine-readable area and features of the visually readable area, dynamically records any unconfigured new layouts, generates new layout configurations and annotations, completes the extraction of structured data for the new layout, and updates the training dataset in real time. A three-level localization algorithm fusion strategy is used, employing machine-readable area information as a reference to calculate the consistency score of surname and given name groups, the accuracy score of date data, the proportion of structured data acquisition, and the value of localization point deviation. The system employs a preset priority ranking and utilizes Hough transform and SIFT feature matching to process image deviations, generating a comprehensive localization processing result. Using verified accurate data from both machine-readable and visually readable areas as a reference, it automatically corrects and completes abnormal document data, generating data to be detected. A target dual-channel OCR model, consisting of a ResNet-Transformer hybrid architecture recognition model for visually readable areas and a CRNN network recognition model for machine-readable areas with character set constraints based on MRZ encoding standards, processes the data to be detected, generating structured data of the target document.

[0092] The various embodiments in this application are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for the method for extracting structured data of certificates and documents, the electronic device, the electronic device, and the readable storage medium are basically similar to the above-described embodiments for extracting structured data of certificates and documents, so the description is relatively simple. Relevant parts can be referred to in the descriptions of the above-described embodiments for extracting structured data of certificates and documents.

Claims

1. A method for extracting structured data from certificates and licenses, characterized in that, include: Acquire certificate image data, including various certificate images with uneven lighting, stains, and obstructions, as well as the first 5 characters of the first line of the machine-readable area and feature images / text of the visual reading area; The images are preprocessed using techniques such as black border removal, noise reduction, Gaussian blurring, sharpening, and watermark removal to improve image quality and construct training datasets for machine-readable and visually readable regions. The layout is determined by the first 5 characters of the first line in the machine-readable area and the features of the visual reading area. The layout is dynamically recorded when no new layout is configured, and the new layout configuration and annotation are generated. The structured data of the new layout is extracted and the training dataset is updated in real time. A three-level positioning algorithm fusion strategy is adopted, using machine-readable area information as a reference to calculate the consistency score of surname and given name groups, the accuracy score of date data, the proportion of structured data acquisition, and the value of positioning point deviation. A preset priority ranking is used, and Hough transform and SIFT feature matching are used to process image deviations to generate a comprehensive positioning processing result. The three-level positioning algorithms include the benchmark positioning method, which directly locates certificate information elements based on a pre-constructed template and coordinate mapping relationship; and the SVM detection method, which uses a support vector machine model to train a key feature detector and identifies information element regions through feature vector matching. The auxiliary information location method uses related information to help locate missing or ambiguous fields; among them, the formula for calculating the consistency score of surname and given name groups is as follows: The formula for calculating the date data accuracy score is as follows: The formula for calculating the proportion of structured data acquisition is as follows: ; Formula for calculating the value of the positioning point deviation: ; Using the verified accurate data of the machine-readable and visually readable areas as a reference, the abnormal document data is automatically corrected and completed to generate data to be tested. The target dual-channel OCR model, consisting of a ResNet-Transformer hybrid architecture recognition model for visual reading and a CRNN network recognition model for machine reading based on the MRZ encoding specification character set constraints, processes the data to be detected and generates structured data of the target certificate.

2. The method as described in claim 1, characterized in that, The layout is determined by the first 5 characters of the first line in the machine-readable area and the characteristics of the visually readable area. Unconfigured new layouts are dynamically recorded, and new layout configurations and annotations are generated. This completes the extraction of structured data for the new layout, including: The layout determination process is performed on the features of the first 5 characters of the first line in the machine-readable area and the features of the visually readable area to generate the layout determination result; Dynamically record instances where a new layout is not configured, generating a record of instances where a new layout is not configured. Configure and annotate the new layout, and generate configuration and annotation information for the new layout. Based on the layout determination results, records without new layout configuration, and newly added layout configuration and annotation information, structured data extraction processing is performed to generate new layout structured data extraction results. These results are used to complete the new layout structured data extraction and update the training dataset in real time.

3. The method as described in claim 2, characterized in that, A three-level localization algorithm fusion strategy is adopted, using machine-readable area information as a reference to calculate the consistency score of surname and given name groups, the accuracy score of date data, the proportion of structured data acquisition, and the value of localization point deviation. A preset priority ranking is used, and Hough transform and SIFT feature matching are applied to process image deviations, generating a comprehensive localization processing result, including: Based on a three-level positioning algorithm fusion strategy, the consistency score of surname and given name groups, the accuracy score of date data, the proportion of structured data acquisition, and the value of positioning point deviation are calculated and processed with reference to machine-readable area information to generate calculation results; The calculation results are sorted according to a preset priority to generate the target sorting result; Image aberrations are processed using Hough transform and SIFT feature matching to generate image aberration processing results; Based on the target sorting results and image deviation processing results, a comprehensive positioning processing result is generated, which is used to achieve accurate positioning.

4. The method as described in claim 1, characterized in that, Using the verified accurate data from the machine-readable and visually readable areas as a reference, the abnormal document data is automatically corrected and completed to generate data to be tested, including: A dynamic mapping verification mechanism is constructed. The first 5 characteristic characters of the machine-readable area are cross-verified with the key information of the visual reading area through a two-way hash verification algorithm. The verified semantic structured accurate data of the machine-readable area and the visual feature vector matrix of the visual reading area are extracted to generate a data reference benchmark with confidence weight. We design an anomaly semantic repair network based on Transformer. Taking the semantic feature vector of the data reference benchmark as conditional input, we perform context-related repair on semantically missing or conflicting fields in abnormal document data. We dynamically match the feature distribution of similar document samples through an attention mechanism to generate document data correction results with corrected confidence. The correction process introduces adversarial training to ensure consistency with the feature distribution of the benchmark data. A multimodal feature completion model is established. For scenarios where the visual reading area is occluded or the machine-readable area is incomplete, a generative adversarial network is used in combination with Hough transform geometric constraints to extract prior knowledge of the layout from the data reference benchmark. The missing data is then structurally completed to generate a document data completion result containing spatial location constraints. The completion accuracy is optimized by a dual loss function of cross-entropy and structural similarity. A heterogeneous data fusion decision mechanism is proposed. A graph neural network is used to construct a correlation graph between machine-readable and visually readable data. The correction and completion results of the document data are mapped to graph nodes. The node features are fused through attention weights to generate integrated document data with cross-modal consistency verification. The consistency verification is performed by calculating the matching degree of feature sequences between modalities through a dynamic time warping algorithm. Based on the feature complexity of the integrated document data, the input dimension of the ResNet-Transformer hybrid architecture is dynamically adjusted to generate detection data with preprocessed strategy labels. The strategy labels are automatically optimized based on historical OCR recognition errors through reinforcement learning algorithms, thereby achieving intelligent adaptation to the input data of subsequent dual-channel OCR models.

5. The method as described in claim 4, characterized in that, A dual-channel OCR model, consisting of a ResNet-Transformer hybrid architecture recognition model for visual reading and a CRNN network recognition model based on MRZ encoding specifications for machine reading, processes the data to be detected to generate structured data of the target document, including: A target dual-channel OCR model, consisting of a ResNet-Transformer hybrid architecture recognition model for visual reading areas and a CRNN network recognition model for machine reading areas based on MRZ encoding specification character set constraints, is used to process the data to be detected and generate model input data. The ResNet-Transformer hybrid architecture recognition model, which is dedicated to visual reading areas, is used to extract and recognize features from the visual reading area data in the model input data to generate visual reading area recognition results. At the same time, the CRNN network recognition model, which is dedicated to machine-readable areas and designed with character set constraints based on the MRZ encoding standard, is used to extract and recognize features from the machine-readable area data in the model input data to generate machine-readable area recognition results. Cross-modal fusion processing is performed on the visual reading area recognition results and the machine reading area recognition results. Through feature association and consistency verification, a fused recognition result is generated. Based on the fusion recognition results, target certificate structured data is generated, which is used to realize the structured extraction and output of certificate information.

6. A device for extracting structured data from certificates and licenses, characterized in that, The device includes: The acquisition module is used to acquire certificate image data, including various certificate images with uneven lighting, stains, and occlusion, as well as the first 5 characters of the first line of the machine-readable area and the feature images / text of the visual reading area; The processing module is used for image preprocessing, employing techniques such as black border removal, noise reduction, Gaussian blurring, sharpening, and watermark removal to improve image quality and construct training datasets for machine-readable and visually readable areas. It determines the layout based on the first 5 characters of the first line in the machine-readable area and features of the visually readable area, dynamically records unconfigured new layouts, generates new layout configurations and annotations, completes the extraction of structured data for the new layout, and updates the training dataset in real time. A three-level positioning algorithm fusion strategy is employed, using machine-readable area information as a reference to calculate the consistency score of surname and given name groups, the accuracy score of date data, the proportion of structured data acquisition, and the positioning point deviation value. A preset priority ranking is used, and Hough transform and SIFT feature matching are applied to process image deviations, generating a comprehensive positioning processing result. This includes the three-level positioning algorithm: a baseline positioning method, which directly locates certificate information elements based on a pre-constructed layout template through coordinate mapping; an SVM detection method, which uses a support vector machine model to train a key feature detector and identifies information element regions through feature vector matching; and an auxiliary information positioning method, which uses associated information to assist in locating missing or ambiguous fields. The formula for calculating the consistency score of surname and given name groups is as follows: The formula for calculating the date data accuracy score is as follows: The formula for calculating the proportion of structured data acquisition is as follows: ; Formula for calculating the value of the positioning point deviation: Using the verified accurate data of the machine-readable and visually readable areas as a reference, the abnormal document data is automatically corrected and completed to generate the data to be detected. The target dual-channel OCR model, which consists of a ResNet-Transformer hybrid architecture recognition model for visually readable areas and a CRNN network recognition model for machine-readable areas with character set constraints based on the MRZ encoding standard, processes the data to be detected to generate the target document structured data.

7. An electronic device, characterized in that, include: First processor; and memory for storing executable instructions of the first processor; The first processor is configured to execute the certificate structured data extraction method according to any one of claims 1 to 5 by executing the executable instructions.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the second processor, it implements the certificate structured data extraction method according to any one of claims 1 to 5.

Images