File image character key information extraction method and system based on deep learning

Abstract

The invention discloses an archive image character key information extraction method and system based on deep learning, and belongs to the technical field of machine vision and deep learning. The method comprises the following steps: constructing a data set containing a real archive image and a synthetic archive image; according to the data set, adopting a YOLOv5 network to train an identification photo detection model, and adopting the trained identification photo detection model to position an image from a to-be-processed file and cut out an identification photo; based on an optical character recognition (OCR) technology and a target detection model, extracting key information of a printed form and a handwritten form in the file to be processed; wherein printing body characters are directly recognized through OCR, and handwritten characters are firstly subjected to OCR assisted positioning and then extracted through the target detection model; and carrying out structured processing on the extracted information and outputting the processed information. According to the invention, the working efficiency of management personnel can be improved, the maintenance cost can be reduced, and the digital archive information is fully utilized to automatically extract key information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision and deep learning, in particular to a deep learning-based archive image person key information extraction method and system, which is suitable for digital management of archives. BACKGROUND

[0002] At present, with the development of information technology, the management of archives tends to be digital and paperless. The existing technology lacks a person key information extraction scheme that can simultaneously process multiple archive types, such as printed and handwritten bodies. In the archive review, the verification of information such as "three ages and two histories" is crucial, but manual processing is low in efficiency and prone to errors.

[0003] Deep learning technologies such as YOLOv5 and PP-OCR perform well in target detection and character recognition, but have not been effectively applied to the whole process of archive image extraction.

[0004] Therefore, how to more efficiently process key information extraction of multiple archive types to improve the utilization efficiency of archives is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] In view of the above problems, the present application is proposed in order to provide a deep learning-based archive image person key information extraction method and system which overcomes the above problems or at least partially solves the above problems. A solution is provided for the deep utilization of archive images, which will reduce the workload of auditors and improve efficiency.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a deep learning-based archive image person key information extraction method, comprising the following steps: Constructing a data set containing real archive images and synthetic archive images; According to the data set, training a passport photo detection model using a YOLOv5 network, and using the trained passport photo detection model to locate images and crop out passport photos from the archives to be processed; Based on optical character recognition (OCR) technology and a target detection model, extracting printed and handwritten key information in the archives to be processed; wherein for printed text, directly recognizing through OCR, and for handwritten text, first positioning through OCR and then extracting through the target detection model; Structuring and outputting the extracted information.

[0007] In one embodiment, constructing a data set containing real archive images and synthetic archive images comprises: Obtaining passport photo images and collecting and organizing real archive images through a web crawler, and preprocessing; The pre-processed ID photo is combined with the real archive image without an ID photo to generate a composite archive image containing the ID photo; The synthetic and real images are labeled to generate the label files needed to train the ID photo detection model.

[0008] In one embodiment, the preprocessing includes: OpenCV was used to process the color space of the ID photo to obtain a yellowish effect and a binarized black and white effect. Then, noise was added to the image and smoothing was performed to obtain a photo with the effect of multiple copies.

[0009] In one embodiment, directly recognizing printed text via OCR includes: Based on OpenCV, the table structure of the archive image is recognized and the archive image is segmented into multiple sub-images; Use a lightweight OCR model to perform text recognition on the sub-image; Text understanding technology is used to classify the identified content and output it as a structured file.

[0010] In one embodiment, for handwritten text, OCR-assisted localization is first used, followed by extraction using the target detection model, including: Use OCR technology to locate the position of key fields in the printed text of the document; Based on the positioning results, the target area containing the handwritten content is cropped out; The system identifies and classifies handwritten content based on an object detection model. The object detection model is a key information detection model for individuals trained using the YOLOv5 network. Through model inference prediction and image cropping functions, it is used to classify, crop, and output the name, gender, date of birth, and date of employment in the file, and then perform text content recognition and extraction.

[0011] In one embodiment, before using OCR technology to locate the position of key fields in the printed text of the document, the method further includes: adjusting the exposure of the document image, enhancing the image, and correcting edge clipping to improve recognition accuracy.

[0012] Secondly, embodiments of the present invention also provide a system for extracting key information about individuals from archival images based on deep learning, comprising: The dataset building module is used to build datasets containing real and synthetic archival images; The ID photo extraction module trains an ID photo detection model using the YOLOv5 network based on the dataset, and then uses the trained ID photo detection model to locate the image and crop out the ID photo from the file to be processed. The key information extraction module, based on optical character recognition (OCR) technology and a target detection model, extracts key information from printed and handwritten text in the file to be processed. Printed text is directly recognized by OCR, while handwritten text is first located using OCR and then extracted by the target detection model. The structured processing output module is used to process and output the extracted information in a structured manner.

[0013] In one embodiment, the key information extraction module includes: Printed text processing unit, used for table segmentation and OCR recognition; Handwriting processing unit for OCR-assisted localization and target detection and recognition. As can be seen from the above technical solution, compared with the prior art, the present invention has the following technical effects: This invention can improve the work efficiency of managers and reduce maintenance costs. It makes full use of digitized archival information and automatically extracts key information such as age to help archival reviewers quickly retrieve and match key information, thereby greatly reducing the workload of archival staff and improving review efficiency. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0015] Figure 1 This is a flowchart of a method for extracting key information about people from archival images based on deep learning, provided in an embodiment of the present invention. Figure 2 This is a rendering of the self-made ID photo for the present invention; Figure 3 This is a composite image of an ID photo archive based on real archives, as presented in this invention. Figure 4 This is a visualization of the training of the self-made ID photo archive model of this invention. Figure 5 This is a prediction effect diagram of the self-made ID photo archive model of the present invention; Figure 6 This is a visualization of the training of the fusion ID photo archive model of the present invention; Figure 7 This is a screenshot showing the operation of the ID photo extraction function of this invention. Figure 8 This is a diagram illustrating the handwritten fill data of the present invention; Figure 9 This is a diagram illustrating the self-made cadre appointment and dismissal approval form of the present invention. Figure 10 This is a diagram showing the effect of exposure adjustment in this invention; Figure 11 This is an image enhancement effect diagram of the present invention; Figure 12 This is an image correction effect diagram of the present invention; Figure 13 This is a diagram showing the effect of Chinese lightweight OCR document recognition used in this invention; Figure 14 This is a diagram showing the effect of cutting the file sub-images according to the present invention; Figure 15 This is a diagram illustrating the text recognition effect of the file sub-image of the present invention. Figure 16 This is a diagram showing the structured output of the approval form for this invention. Figure 17 This is an image showing the OCR-assisted positioning and cropping effect used in this invention; Figure 18 This is a diagram illustrating the implementation effect of extracting key information about individuals in this invention. Figure 19 This is a visualization of the training effect of the fusion archive key information extraction model for individuals in this invention.

[0016] Figure 20 A block diagram of a deep learning-based system for extracting key information about people from archival images, provided by this invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] With the development of the information age, digitization and paperless processes have become the mainstream in the field of archival image management. After thorough research and analysis of the problems existing in the technology for extracting key information about individuals from archival images, the inventors clarified two main tasks: extracting identity photos and extracting key information about individuals from archival images. To achieve the function of extracting document photos from archival images, this invention uses a document photo detection model trained based on a YOLOv5 object detection network. Simultaneously, it employs a collaborative processing approach of Optical Character Recognition (OCR) technology and object detection technology to extract key information from archival images, including name, gender, date of birth, and work date. Specifically, printed text is directly recognized using OCR technology; for handwritten text, OCR technology is first used to locate its associated key printed fields, and then the handwritten content is cropped and recognized based on the object detection model.

[0019] Furthermore, to overcome the problem of limited real-world data and to ensure the confidentiality of such data, this invention creates a dataset of simulated certificate photo files. The resulting archival ID photo model demonstrates excellent performance. This method provides a solution for the in-depth utilization of archival images, reducing the workload of auditors and improving efficiency.

[0020] Reference Figure 1 As shown in the figure, this invention discloses a method for extracting key information about people from archival images based on deep learning, including the following steps: S1. Construct a dataset containing real archival images and synthetic archival images; S2. Based on the dataset, a YOLOv5 network is used to train an ID photo detection model. The trained ID photo detection model is then used to locate images in the files to be processed and crop out the ID photos. YOLOv5 is an improvement on YOLOv4 and is an end-to-end object detection algorithm. It combines various deep learning techniques, including residual networks, multi-scale detection, multi-scale feature pyramid networks, multi-modal detection, prediction branches, multi-scale cross-network fusion, and multi-scale joint loss, making the model simpler, lighter, and more effective.

[0021] S3. Based on optical character recognition (OCR) technology and target detection model, extract key information of printed and handwritten text from the file to be processed; wherein printed text is directly recognized by OCR, and handwritten text is first located with OCR assistance and then extracted by the target detection model. S4. The extracted information is structured and then output.

[0022] Embodiments of the present invention include training a document image ID photo detection model using a YOLOv5 object detection network, employing a combination of OCR text recognition and object detection techniques to extract key information about individuals from the document image, and then detecting, extracting, and processing these elements in the document image. Wherein: In step S1, the dataset can be divided into two parts: real archival images and synthetic images. Specifically, color ID photos are crawled from the web, while real archival images are collected and organized. OpenCV is used to process the ID photos in color space, resulting in a yellowish effect and a binarized black and white effect. Further processing, such as adding noise and smoothing, yields photos with the effect of multiple photocopies. The simulated ID photos are then synthesized with archival images without ID photos to construct a self-made dataset based on real archival images. Finally, the archival images containing ID photos are annotated with stamp images using the open-source image annotation tool LabelImg, generating YOLO format annotation files, thus forming the ID photo detection dataset.

[0023] In step S2, the document photo extraction part is implemented: Using the PyTorch framework, YOLOv5 is used to train the pre-created dataset. Model inference is performed on the obtained model, and based on the output, the location coordinates of the document photo are obtained and cropped onto the document image. Finally, the result is output, and the document photo is placed in the specified folder.

[0024] In practice, a YOLOv5 network is used to train an object detection model for processing. In actual application, the images of the documents to be processed are placed in a designated folder. After running the inference detection module, all ID photos in the documents are displayed. Finally, based on the detection results, OpenCV is used to crop the ID photos from the documents and output them to a designated folder for centralized storage.

[0025] Combining steps S1 and S2 above, in specific implementation, the extraction of archival ID photos is achieved through model training and application reasoning. The complete process specifically includes: Step S21: Use web crawlers to crawl color ID photos from the web. At the same time, collect and organize real archive images for future expansion of the dataset. Step S22: Use OpenCV to process the color space of the ID photo to obtain a yellowish effect and a binarized black and white effect. Then, by adding noise and smoothing the image, a photo with the effect of multiple copies is obtained. Step S23: The simulated ID photo and the archive image without an identity photo were combined to construct a self-made dataset based on real archive images; Step S24: Use the open-source image annotation tool LabelImg to annotate the document with stamp images, generate YOLO format txt tags, and save them; Step S25: In the PyTorch framework, use YOLOv5s to train the already created dataset; Step S26: Using the model obtained in step S25, perform model reasoning, and based on the output results, obtain the location coordinates of the ID photo and crop it onto the file image. Step S27: Output the results and put the document photos into the specified folder.

[0026] In step S3, the key information of the archival personnel is extracted. This involves a large amount of information, presented in both printed and handwritten formats. Therefore, during dataset construction, the content is first divided according to the reporting requirements fields. Then, based on the different reporting contents, two parts of data are constructed: printed and handwritten. Table borders are created, and the fields for both reporting requirements and content are filled in separately. During data preparation, the focus is on analyzing and preparing data for information such as name, gender, date of employment, and date of birth, ensuring diversity in the table content and facilitating broad applicability for subsequent information extraction. Specifically: (1) Full-printed archives: First, the table is detected by region detection. Then, the table structure of the archive image is identified based on OpenCV. The image is cut into sub-images according to the table structure. The sub-images are then input into a pre-trained Chinese lightweight OCR model. The text in the image is identified and classified by OCR technology. The text in the sub-images is identified by optical character recognition technology such as PP-OCR. The prediction results of each archive are written into a text file. After conversion, the text is finally saved as a CSV file that is easy to statistically analyze, so as to realize the automatic extraction and structured processing of the archive content.

[0027] (2) Handwritten Archives: A method combining OCR and object detection was proposed. Before object detection training, OCR text recognition was used for auxiliary localization detection. The scope of the approval form was located by recognizing the key printed text of the fields requiring the information to be filled in. The entire archive image was cropped into small images centered on the key information, thus narrowing the feature extraction range during object detection. A key information detection model based on the YOLOv5 network structure was then trained. Through model inference prediction and image cropping functions, the name, gender, date of birth, and date of employment in the archive can be classified, cropped, and output to a specified folder with the information source displayed.

[0028] Combining steps S3 and S4 above, the extraction of key information about a person in specific implementation includes the following steps: Step 31: Based on the characteristics that the fields in the reporting requirements are all printed and the format is relatively fixed, we further summarize the formats of important information such as name, gender, date of birth, and date of employment. Step 32: For the printed data set of fields, analyze the actual content filled in the cadre appointment and removal approval form, summarize and subdivide the types of names, genders, and dates, and randomly generate several types of information to match with key information. Step 33: For the printable data set, in addition to the key elements, other elements in the table are also simulated. The names of ethnic groups, provinces, cities and counties, university majors, and higher education institutions are collected, randomly arranged and combined and stored in an Excel spreadsheet. Subsequently, based on the content in the table, the table file is automatically generated and the corresponding fields are filled in. Step 34: For the handwritten field dataset, crawl some handwritten signatures and artistic signatures, download some online handwritten fonts, scan and crop them as data expansion, so that the distribution of the key information dataset is more extensive and authentic. Step 35: Use Python-docx to write a script to complete the construction of the dataset; Step 36: Optimize the real archive dataset by processing the images to make them clear and easy to distinguish, which will facilitate the subsequent extraction of key information. Step 37: Based on OpenCV, recognize the table structure of the archive image, cut the image into sub-images according to the table structure, and then input the sub-images into the pre-trained Chinese lightweight OCR model to recognize and classify the text in the image through OCR technology. Step 38: Use PP-OCR technology to recognize the identified text content, write the prediction results of each document into a text file, convert it, and finally save it as a CSV file that is easy to statistically analyze, so as to realize the automatic extraction and structured processing of the document content.

[0029] The technical solution of the present invention will be explained in detail below through specific examples: The two main tasks are extracting identity photos from archival images and extracting key information about individuals: I. An example of extracting identity photos is explained below: (1) Expand the ID photo dataset.

[0030] The main difficulties in creating a dataset are as follows: 1) Limited availability of authentic archival data. Project organizations typically provide a limited number of archival images, with a small proportion containing ID photos, which is insufficient to meet the requirements for training, testing, and validation sets in deep learning.

[0031] 2) Diverse types of ID photos. Among real archival images, there are many types of archives containing ID photos, but the number of each type is small and the distribution of types is uneven. Color and black and white photos are the main types, while old photos and images that have lost some facial features due to multiple photocopyings are too few to meet the training needs.

[0032] 3) Data confidentiality protection. The archival images provided by the project unit are all genuine personnel files containing sensitive information, requiring archival management personnel to handle their contents with confidentiality. Therefore, the requirements and approval process for exchanging relevant information and training models on internal and external networks are extremely cumbersome, making it impossible to meet the frequent daily training and testing needs.

[0033] Therefore, due to the scarcity of authentic archival data, the variety of ID photo types, and data confidentiality protection requirements, this invention proposes a semi-automatic method for generating archival images containing ID photos, based on the effects of authentic archival images, to construct a self-made ID photo archival dataset. Based on a large number of ID photo images searched online, different ID photo effects are categorized, and image processing techniques are used to transform these effects. The resulting images are then synthesized with authentic archival backgrounds that do not contain ID photos, essentially covering all situations encountered in reality.

[0034] (2) ID Photo Simulation. After crawling relevant color ID photos from the internet, OpenCV was used to process the photos in color space, resulting in a yellowish effect and a binarized black and white effect. Further processing, such as adding noise and smoothing, was then applied to achieve the effect of multiple photocopies. See the appendix for specific results. Figure 2 The top part is the web crawler, and the bottom part is the processed result.

[0035] (3) Dataset Construction. Real personal archives contain some files without identification photos, such as table of contents, personal information statements, and social relationship forms. These forms do not contain the required identification photos, totaling 698 images. To make the extraction of archival image information more authentic and reliable, the composite images in the self-made dataset are based on real archival images. While striving to reproduce the real archival conditions as accurately as possible, simulated identification photos were combined with archival images without identification photos to construct a self-made dataset based on real archival images.

[0036] In the subsequent dataset construction, self-made ID photos were synthesized with real archival backgrounds to achieve an effect close to that of real archives. The aspect ratio of 1-inch photos is 5:7, and that of 2-inch photos is 3:4. The ID photos were proportionally reduced, and through ROI region image overlay processing, the processed ID photo image was embedded into random positions on real archival backgrounds that did not contain ID photos, using image synthesis techniques. First, the ROIs of the original image were obtained. The grayscale values ​​of the original image were binarized to obtain an ROI region mask. The masked regions were then reversed, and the masked background of the archival background was overlaid with the masked foreground of the ID photo to obtain the synthesized archival image containing the ID photo. This principle is used to increase the diversity of the training set. See the appendix for specific results. Figure 3 .

[0037] (3) Test result analysis. The test results are analyzed from two aspects: model test results and final prediction results.

[0038] (4) Data preparation and label configuration. In this embodiment, the open-source image annotation tool LabelImg is used to annotate the stamp images on the archive. After all the data is annotated, it is divided into training set, validation set, and test set in a ratio of 7:2:1, and the images and their corresponding labels are placed in the images and labels folders respectively. A data.yaml file is created in the idphoto folder as a configuration file. The paths of training and validation are entered, and the label type is set to 1. According to the label configuration file, the label name is entered as names:[idphoto].

[0039] (5) Install the experimental environment. Install the GPU version of PyTorch. Download the source code provided on the official website and install Python tools and libraries according to its requirements.

[0040] (6) Model Training. Since the target to be detected and extracted is relatively large, and to maximize speed in practical applications, YOLOv5s, the YOLOv5 series with the smallest depth and feature map width, was selected for training within the PyTorch framework. The training code train.py is configured as follows.

[0041] '--weights' is a pre-trained model, which can effectively improve training efficiency. Select 'YOLOv5s.pt'.

[0042] '--cfg' is the configuration file for the model architecture; select 'YOLOv5s.yaml'.

[0043] '--data' is the configuration file containing the images and labels needed to train the model; select 'data.yaml'.

[0044] '--batch-size' is the number of training samples per batch. Considering the strong computing power of GPUs, it is set to 16.

[0045] Run train.py, train for 300 epochs, and finally get the ID photo.

[0046] The visualization of the self-made ID photo archive model training is attached. Figure 4 This corresponds to a visualization of the training process of the YOLOv5s ID photo detection model trained solely on a "self-made ID photo archive dataset." It includes the following three typical curve types: (1) Loss Curves This includes training loss and validation loss.

[0047] As the number of training epochs increases, the loss should gradually decrease and stabilize, indicating that the model is learning effective features.

[0048] (2) Precision & Recall Curves It reflects the model's classification performance in detection tasks.

[0049] High precision means fewer false positives, and high recall means fewer false negatives.

[0050] (3) mAP curve (mean Average Precision) A comprehensive evaluation index for the performance of detection models, especially suitable for target detection tasks.

[0051] mAP@0.5 and mAP@0.5:0.95 are common evaluation metrics; the higher the value, the more accurate the model's detection.

[0052] The dataset was constructed as follows: ① Use web crawlers to obtain color ID photos, and process them with OpenCV (yellowing, binarization, noise addition, smoothing) to simulate the effect of aging and multiple photocopying; ② Combine these processed ID photos with real archive images without ID photos to generate diverse and realistic training samples.

[0053] Appendix Figure 4 The training effect reflected is explained as follows: This chart shows the convergence of the model after training it on a homemade dataset for 300 epochs; If the loss curve decreases smoothly and the accuracy and recall gradually increase and stabilize, it indicates that the model training is effective and there is no overfitting or underfitting. A high mAP value (e.g., close to or above 0.95) indicates that the model has good generalization ability and recognition accuracy in the ID photo detection task; The figure also shows examples of the model's detection on the validation set, such as successfully locating and cropping the ID photo area in the file.

[0054] The predicted results of the self-made ID photo archive model are attached. Figure 5 .

[0055] The visualization of the training results for the fusion ID photo archive model is attached. Figure 6 Appendix Figure 6 This is a visualization of the training process of an ID photo detection model trained on a fusion dataset. (Attached) Figure 4 (Compared to training with synthetic data only), Figure 6 This demonstrates the performance evolution of a YOLOv5s model trained on a mixed dataset of real archival images (scanned images from actual archives, including real ID photos, forms, handwritten content, etc.) and synthetic archival images (images synthesized from ID photos and undocumented archives generated through simulation processing (yellowing, binarization, noise addition, etc.)). The figure illustrates the following characteristics: I. Losses decline more smoothly: Compared to training with purely synthetic data, the loss curve of training with fused data enters the stable range earlier and has less fluctuation.

[0056] II. Higher accuracy and recall with faster convergence: Because the training data is closer to the real distribution, the model can learn effective features faster, and the Precision and Recall curves reach high levels (such as above 0.95) earlier.

[0057] III. mAP value significantly improved: Comparison of mAP@0.5 and mAP@0.5:0.95 Figure 4 The model trained using only synthetic data has higher accuracy, indicating that the model is more accurate in detecting complex real-world scenarios.

[0058] IV. Enhanced model generalization ability: The model's performance on the validation set is similar to that on the training set, indicating that it is not overfitting and is suitable for practical archival image processing.

[0059] (7) Model Inference. Using the trained model, perform model inference and, based on the output, obtain the location coordinates of the ID photo and crop it onto the archive image. See the appendix for specific results. Figure 7 .

[0060] (8) Output Results. Based on the output results, obtain the location coordinates of the ID photo, crop it on the file image, and put the file ID photo into the specified folder.

[0061] II. Examples of extracting key information about a person are illustrated below: PP-OCR is a practical, ultra-lightweight Chinese-English OCR system, primarily designed for Chinese-English OCR problems. It provides two pre-trained models: a lightweight Chinese OCR model and a general Chinese OCR model. Through quantitative comparison, we selected the lightweight Chinese OCR model for text extraction. Using the prediction confidence of all bounding boxes in the personnel appointment and dismissal approval forms of the sample set, we calculated the average accuracy. The average accuracy of the ultra-lightweight Chinese OCR was over 98.5%, while the average accuracy of the general Chinese OCR was over 98%. Combining this with a comparison table of the number of bounding boxes, we concluded that the general Chinese OCR model has more bounding boxes than the ultra-lightweight Chinese OCR model, but its accuracy in predicting personnel appointment and dismissal approval forms is lower than that of the ultra-lightweight Chinese OCR model. Therefore, we chose the ultra-lightweight Chinese OCR model for document recognition.

[0062] (1) Subdivide the content and format of the information to be filled in. Based on the fact that the fields required for filling in the information are all printed and the format is relatively fixed, we have subdivided and summarized the formats of important information such as name, gender, date of birth, and date of joining the workforce.

[0063] For the printed data set of fields, we analyze the content filled in the actual personnel appointment and dismissal approval forms, categorize and subdivide the types of names, genders, and dates, and randomly generate several types of information to match with key information.

[0064] The form content can be divided into two types: printed and handwritten. To extract key information about the individuals in the archives, the forms for both printed and handwritten forms are processed separately.

[0065] 1. For the printout field dataset, analysis was conducted based on the content of actual cadre appointment and removal approval forms. Names were categorized into three types: two-character, three-character, and four-character names; gender was categorized into male and female; and several sets of information were randomly generated and matched with key information. In addition to key elements, other elements in the tables were also simulated. Names from 56 ethnic groups, 34 provinces, 690 cities and 1420 counties / districts, 506 university majors, and 1270 higher education institutions were collected, randomly arranged, and stored in an Excel spreadsheet. Subsequently, based on the content of the spreadsheet, table files were automatically generated and the corresponding fields were filled in.

[0066] 2. Creating the handwritten form was quite tedious. First, we scraped handwritten signatures and artistic signatures from the internet to use as data for the name field. Second, we downloaded various online handwritten fonts and replaced the prepared data with different fonts, saving them accordingly. During the matching process, we also considered font variations. To ensure font diversity, we recruited ten students to write 200 key fields each, following the above classification format. These were then scanned, cropped, and used to expand the data, making the distribution of the key information dataset more extensive and authentic. See the attached image for the handwritten data filling effect. Figure 8 .

[0067] (2) Creating a dataset. A script written in Python can be used to obtain the format of all borders in the cadre appointment and removal approval form. Then, the text of the required fields can be entered into the table using the `table.cell.text()` function. Reading the generated table data and image data, a cadre appointment and removal approval form can be generated. See the attached image for the result. Figure 9 .

[0068] (3) Optimization of archival datasets. In real archives, many old paper archives are yellowed, damaged, or stained due to their age. During the scanning process of paper archives, the clarity of the archive images will also decrease, such as blurry edges and curves. The text content may also be distorted to varying degrees, and the digitized images may have problems such as angular offset. Therefore, for real archives with issues such as yellowed, damaged, dirty paper or scanning marks caused by unevenness, it is necessary to perform high-definition processing on the images to make the archive images clear and easy to distinguish, which will facilitate the subsequent extraction of key information.

[0069] 1. Image Exposure Adjustment. Issues affecting image quality in genuine old archives mainly include: content on the back showing through the paper to the front, uneven yellowing of the paper, and marks caused by uneven paper during scanning. For example... Figure 10 The image is shown in the left half. To address this issue, an adaptive binarization method is typically used to remove lighter areas such as yellowing and background content, fill the background with a single-color background, and then fill with an RGB (197, 246, 255) background image. See the attached image for the exposure adjustment effect. Figure 10 As shown in the right half of the middle section.

[0070] 2. Image Enhancement. To increase image visibility and make it easier to observe, analyze, and process, the Laplace operator is used for sharpening. See the appendix for image enhancement results. Figure 11 .

[0071] 3. Image Skew Correction and Cropping. When scanning paper documents into electronic documents, tilting or background distortion is unavoidable. Therefore, we need to perform rotation and cropping on the document image. After rotation, edge gaps will appear, which also need to be cropped. First, the tilted image is filled with edges to prevent gaps after rotation, facilitating subsequent cropping. Determining the rotation amount requires skew correction techniques. Skew correction is mainly used to calculate the X, Y, and R positional errors of the measured image and to transmit deviation information. The collected images are preprocessed, then the angle is measured using Hough transform, and the positional error is calculated using interpolation and image registration difference calculations in phase relationships. Finally, the rotation angle is determined to complete the skew correction operation on the tilted document image. Then, the Canny algorithm is used for edge detection and cropping, and black borders are removed, resulting in the desired processed image. See the appendix for the image skew correction and cropping results. Figure 12 .

[0072] 4. Structured Extraction of Fully Printed Archives. First, region detection is performed on the tables. Then, based on OpenCV, the table structure of the archive image is recognized, and the image is segmented into sub-images according to the table structure. These sub-images are then input into a pre-trained lightweight Chinese OCR model. OCR technology is used to recognize and classify the text in the images. PP-OCR and other optical character recognition technologies are used to recognize the text in the sub-images. The prediction results for each archive are written to a text file, converted, and finally saved as a CSV file for easy statistical analysis, achieving automatic extraction and structured processing of archive content. See the appendix for the results of the lightweight Chinese OCR archive recognition. Figure 13 See attached image for the effect of cropping the file sub-image. Figure 14 The results of text recognition for the file sub-images are attached. Figure 15 The structured output of the cadre appointment and removal approval form is attached. Figure 16 .

[0073] 5. Detection of Key Information in Handwritten Documents. When extracting element blocks containing key handwritten information, OCR technology alone cannot solve the problem of handwritten text recognition. The detection rate and accuracy of OCR for handwritten text are insufficient for information verification requirements. Therefore, a method based on ID photo extraction was proposed, using a deep learning-trained object detection model to extract the required key information. However, in actual experiments, the similarity of Chinese character strokes and the similarity of table cell borders meant that a single object detection model could not recognize similar-looking characters or tables with similar proportions when extracting information.

[0074] To address the above issues, a method combining OCR and object detection is proposed. Before object detection training, OCR text recognition is used for assisted localization detection. This involves identifying key printed text in the fields requiring information to pinpoint the scope of the approval form, cropping the entire document image into smaller images centered on the key information, thus narrowing the feature extraction range during object detection. A key information detection model based on the YOLOv5 network structure is then trained. Through model inference and image cropping, key information such as name, gender, date of birth, and date of employment in the document can be categorized, cropped, and output to a designated folder with the information source displayed. The OCR-assisted localization and cropping results are shown in the appendix. Figure 17 .

[0075] 6. Experimental Results and Analysis. After labeling all images, the model was trained using the same experimental environment and configuration as the ID photo model. After running the model, key information was cropped based on the inference results to obtain text block images. The results of extracting key information about people are shown in the appendix. Figure 18 Previous experiments showed that the model trained on a fused dataset containing both real and custom datasets performed best. Therefore, the key information model for individuals was also trained on the fused dataset. The visualization results of the trained model are attached. Figure 19 This is a visualization of the training process of a key information extraction model for individuals on a fused dataset. The model, based on the YOLOv5 network architecture, is specifically designed to detect and extract key handwritten information, such as name, gender, date of birth, and date of employment, from archival images.

[0076] The following are typical training monitoring curves: i. Loss Curves Training loss and validation loss; It reflects the model's fit as the number of training epochs increases.

[0077] ii. Precision & Recall Curves Evaluate the completeness of the model's classification and detection.

[0078] iii. Mean Accuracy Curve (mAP) mAP@0.5: Average accuracy when the cross-union ratio threshold is 0.5; mAP@0.5:0.95: Overall accuracy under multiple thresholds; The dataset used by this model is a fused dataset, including: Authentic archival images: 399; Composite archive images: 600; To train the handwriting information detection model, the aforementioned archival images were used to locate key fields using OCR technology, and then the local areas containing handwritten content were cropped out to form a set of 5324 auxiliary training images. The data fusion strategy aims to improve the model's ability to generalize handwritten information and adapt to real archives.

[0079] The training process exhibits the following characteristics: i.) The loss function steadily decreases: The losses on both the training and test sets showed an overall decreasing trend, indicating that the model was learning effectively and that overfitting did not occur.

[0080] ii.) Precision and recall converge rapidly: The Precision and Recall curves converge rapidly, and their final values ​​are close to 1, indicating that the model has a low false negative rate and a low false positive rate in detecting handwritten information.

[0081] iii.) Excellent mAP value: The mAP@0.5 reaches 0.99, and the mAP@0.5:0.95 reaches 0.98, indicating that the model has extremely high detection accuracy and robustness at different thresholds.

[0082] iv.) The training process was smooth: The curves of each indicator converge smoothly, reflecting the high quality of the fused dataset and the reasonable training strategy.

[0083] Appendix Figure 19 It not only records the training process, but also demonstrates a technological breakthrough in handwritten information extraction: (i.) A hybrid strategy of OCR-assisted localization and object detection: First, key fields of the printed text are located using OCR, and then the handwritten area is cropped out, which greatly reduces the detection range and improves the accuracy of handwritten text recognition.

[0084] (ii.) Training of a dedicated model for handwritten characters Traditional OCR has a low handwriting recognition rate. This invention significantly improves the ability to extract structured handwriting information by constructing a dedicated handwriting dataset and a YOLOv5 detection model.

[0085] (iii.) High precision and high availability The mAP@0.5 score reaches 0.99, indicating that the model has strong practicality and reliability in real-world archival scenarios.

[0086] like Figure 20 As shown, this embodiment of the invention also provides a system for extracting key information about people in archival images based on deep learning, characterized in that it includes: The dataset building module is used to build datasets containing real and synthetic archival images; The ID photo extraction module trains an ID photo detection model using the YOLOv5 network based on the dataset, and then uses the trained ID photo detection model to locate the image and crop out the ID photo from the file to be processed. The key information extraction module, based on optical character recognition (OCR) technology and a target detection model, extracts key information from printed and handwritten text in the file to be processed. Printed text is directly recognized by OCR, while handwritten text is first located using OCR and then extracted by the target detection model. The structured processing output module is used to process and output the extracted information in a structured manner.

[0087] The key information extraction module includes: Printed text processing unit, used for table segmentation and OCR recognition; The handwriting processing unit is used for OCR-assisted localization and target detection and recognition.

[0088] This invention can automatically extract ID photos and key information from archives, reducing the workload of manual review, improving efficiency, reducing maintenance costs, and promoting the development of digital archives.

[0089] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0090] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting key information about people from archival images based on deep learning, characterized in that, Includes the following steps: Construct a dataset containing both real and synthetic archival images; Based on the dataset, a YOLOv5 network was used to train an ID photo detection model. The trained ID photo detection model was then used to locate images from the files to be processed and crop out the ID photos. Based on optical character recognition (OCR) technology and target detection model, key information of printed and handwritten characters in the file to be processed is extracted. Printed text is directly recognized by OCR, while handwritten text is first located using OCR and then extracted using the target detection model. The extracted information is structured and then output.

2. The method according to claim 1, characterized in that, Construct a dataset containing both real and synthetic archival images, including: We obtained ID photo images through web crawlers and collected and organized real archival images, and preprocessed them; The pre-processed ID photo is combined with the real archive image without an ID photo to generate a composite archive image containing the ID photo; The synthetic and real images are labeled to generate the label files needed to train the ID photo detection model.

3. The method according to claim 2, characterized in that, The preprocessing includes: OpenCV was used to process the color space of the ID photo to obtain a yellowish effect and a binarized black and white effect. Then, noise was added to the image and smoothing was performed to obtain a photo with the effect of multiple copies.

4. The method according to claim 1, characterized in that, For printed text, direct OCR recognition is used, including: Based on OpenCV, the table structure of the archive image is recognized and the archive image is segmented into multiple sub-images; Use a lightweight OCR model to perform text recognition on the sub-image; Text understanding technology is used to classify the identified content and output it as a structured file.

5. The method according to claim 1, characterized in that, For handwritten text, OCR is first used to locate it, and then the target detection model is used to extract it, including: Use OCR technology to locate the position of key fields in the printed text of the document; Based on the positioning results, the target area containing the handwritten content is cropped out; The system identifies and classifies handwritten content based on an object detection model. The object detection model is a key information detection model for individuals trained using the YOLOv5 network. Through model inference prediction and image cropping functions, it is used to classify, crop, and output the name, gender, date of birth, and date of employment in the file, and then perform text content recognition and extraction.

6. The method according to claim 5, characterized in that, Before using OCR technology to locate the key fields in the printed text of the document, the process also includes: adjusting the exposure of the document image, enhancing the image, and correcting edge cropping to improve recognition accuracy.

7. A system for extracting key information about people from archival images based on deep learning, characterized in that, include: The dataset building module is used to build datasets containing real and synthetic archival images; The ID photo extraction module trains an ID photo detection model using the YOLOv5 network based on the dataset, and then uses the trained ID photo detection model to locate the image and crop out the ID photo from the file to be processed. The key information extraction module, based on optical character recognition (OCR) technology and a target detection model, extracts key information from printed and handwritten text in the file to be processed. Printed text is directly recognized by OCR, while handwritten text is first located using OCR and then extracted by the target detection model. The structured processing output module is used to process and output the extracted information in a structured manner.

8. The system according to claim 7, characterized in that, The key information extraction module includes: Printed text processing unit, used for table segmentation and OCR recognition; The handwriting processing unit is used for OCR-assisted localization and target detection and recognition.

Images