Method for extracting multi-modal data of multi-source heterogeneous document based on human-in-the-loop

Abstract

The invention relates to a multi-source heterogeneous document multi-modal data extraction method based on a human-in-the-loop. The method comprises the steps that S1, an original PDF file is preprocessed; s2, sorting and cleaning the PDF file metadata obtained in the step S1; s3, the text data obtained through analysis are cleaned and sorted; s4, cleaning and sorting table data obtained through analysis; s5, cleaning and sorting the picture data obtained by analysis; s6, inputting the text data processed in the step S3 into a model for fine tuning; s7, inputting the table data processed in the step S4 into a model for fine tuning; s8, designing a table structure recognition algorithm to restore the table structure; s9, inputting the processed picture data in the step S5 into the model for fine tuning, and ensuring the precision and robustness of automatic data extraction by introducing a'human-in-the-loop 'mechanism, thereby realizing rapid and efficient target data collection and improving the database construction efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal data recognition, and particularly relates to a multi-source heterogeneous document multi-modal data extraction method based on human-in-the-loop. BACKGROUND

[0002] In recent years, the rapid development and popularization of information technology have profoundly changed the way of information recording and data management. With the advancement of digitization, a large number of complex, multi-modal data sources have emerged in production and life, such as electronic documents, web articles and multimedia content. These data are no longer limited to traditional pure text records, but are integrated with pictures, tables and other data forms, forming multi-source heterogeneous data. Multi-source heterogeneous data, with its complex structure and rich content, brings new challenges and opportunities for information analysis and knowledge extraction.

[0003] In the era of big data, the scale and complexity of multi-source heterogeneous data are constantly rising, and the amount of information and potential knowledge contained in them is growing at an unprecedented rate. How to efficiently extract valuable information from them has become an important issue in current research and application fields. The fusion processing and recognition of different data modalities not only need to consider their respective characteristics, but also require the integration of their relationships to achieve comprehensive and in-depth knowledge mining and utilization.

[0004] Against this background, multi-source heterogeneous data pattern recognition technology has emerged to cope with the challenges of automated processing and analysis. Pattern recognition aims to identify meaningful patterns and features through structured analysis of data, such as semantic information in images, structured data in tables, and language patterns in text. The ultimate goal of this technology is to extract the essential characteristics of data in order to classify, analyze and predict data, thereby providing more accurate and efficient support for scientific research, industrial practice and data-driven decision-making. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art, and the present application provides a multi-source heterogeneous document multi-modal data extraction method based on human-in-the-loop, which aims to quickly and accurately extract target data from a large number of documents, thereby significantly improving the efficiency of database construction.

[0006] The present application discloses a multi-source heterogeneous document multi-modal data extraction method based on human-in-the-loop, characterized in that: specifically, the method comprises the following steps:

[0007] Step S1: Preprocessing the original PDF file, parsing the file to obtain text, tables, pictures and metadata;

[0008] Step S2: Organizing and cleaning the PDF file metadata obtained in step S1 to ensure the standardization and integrity of the data;

[0009] Step S3: cleaning and arranging the parsed text data, and dividing it into a training set and a test set for training the NER model for named entity recognition;

[0010] Step S4: cleaning and arranging the parsed table data, and dividing it into a training set and a test set for training the table detection and table structure recognition model;

[0011] Step S5: cleaning and arranging the parsed picture data, and dividing it into a training set and a test set for training the target picture detection and recognition model;

[0012] Step S6: inputting the text data processed in step S3 into the UIE model for fine-tuning to train the NER model;

[0013] The UIE model is very suitable for extracting structured information from unstructured natural language text, especially in recognizing entity names, location information and other related entities. Under this sequence labeling framework, the task is to predict the optimal arrangement of label sequences by maximizing the conditional probability shown in the following formula:

[0014]

[0015] This probabilistic formalization method enables the present application to systematically infer the most likely label configuration of the mark sequence, and improves the accuracy of entity recognition by utilizing the inherent relevance between labels;

[0016] Step S7: inputting the table data processed in step S4 into the YOLOV3 model for fine-tuning to train the table detection and table structure recognition model. This model is widely praised for its accuracy and ease of deployment. The present application uses a standard YOLOv3 loss function:

[0017] L=L bbox +L obj +L class

[0018] wherein L bbox represents the boundary box regression loss, L obj represents the object existence loss, and L class represents the class prediction loss. The boundary box regression loss L bbox is calculated using mean square error when an object is detected, mainly for the x, y coordinates of the center and the width and height of the boundary box. YOLOv3 adjusts the offset of the predefined anchor box and applies the loss to these offsets as follows:

[0019]

[0020] wherein Lobj represents the existence of an object x, y, w, h in a cell i and are actual and predicted bounding box coordinates and dimensions, respectively, and the object existence loss L obj is defined by penalizing incorrect object existence scores, as follows:

[0021]

[0022] where C represents the confidence score, and λ obj is a weight factor for balancing object and non-object detection, and the class prediction loss L class uses a cross-entropy loss, aiming to accurately classify detected objects, as follows:

[0023]

[0024] where p(c) and represent the actual and predicted probabilities of the class present in the bounding box, respectively, and this comprehensive loss function ensures that YOLOv3 can efficiently locate and classify objects, enhancing its adaptability in real-time object detection tasks;

[0025] Step S7: input the processed picture data in step S5 into the model for fine-tuning to train the target picture detection and recognition model;

[0026] Step S8: reconstruct the table structure according to the table line recognition algorithm;

[0027] Step S9: input the processed picture data in step S5 into the model for fine-tuning to train the target picture detection and recognition model;

[0028] Step S10: integrate and apply the text, table and picture recognition models trained in steps S6, S7 and S8, input the processed file in step S1 into the model, and automatically extract multi-modal target data; Step S11: manually check and label the multi-modal data extracted in step S9, save the labeled data and correction content, and feed back to steps S6, S7, S8 and S9 for continuous iteration and optimization of the model, while saving the labeled data;

[0029] Step S12: normalize the multi-modal data manually labeled in step S11 and the metadata arranged in step S2;

[0030] Step S13: fuse and save the normalized multi-modal data to construct an efficient data storage system.

[0031] The application further provides that the step S1 comprises:

[0032] Step S101: The GROBID tool is used to automatically parse the PDF document, GROBID optimizes document parsing by using a high-level sequence labeling model, GROBID classifies and labels the text area in the PDF document systematically, then integrates the content into the structured label generated by the model, and converts it into an XML document;

[0033] Step S102: The data packets obtained by parsing in step S101 are stored as PDF metadata, text data in the PDF, table data in the PDF, and picture data in the PDF.

[0034] The application further provides that the step S2 comprises:

[0035] Step S201: The PDF file metadata obtained by parsing in step S1 is sorted and cleaned, and the specific data cleaning rule is: removing data documents whose metadata cannot be obtained by parsing, and removing data documents with PDF parsing errors.

[0036] The application further provides that the step S3 comprises:

[0037] Step S301: The PDF text obtained by parsing in step S1 is sorted and cleaned, and the specific data cleaning rule is: removing punctuation marks and spaces from the parsed text data.

[0038] The application further provides that the step S4 comprises:

[0039] Step S401: The PDF table data obtained by parsing in step S1 is sorted and cleaned to ensure the integrity and accuracy of the data, and the specific data cleaning rule is: removing document data without position information of the parsed table.

[0040] The application further provides that the step S5 comprises:

[0041] Step S501: The PDF picture data obtained by parsing in step S1 is sorted and cleaned to ensure the integrity and accuracy of the data, and the document data with failed picture parsing is removed.

[0042] The application further provides that the step S6 comprises:

[0043] Step S601: The named entity of the text data obtained by screening is data labeled, and the labeled data is divided into training set and test set according to the ratio of 7:3 for model training;

[0044] Step S602: Based on the unified information extraction model UIE model in the Baidu Paddle open source model library, the named entity recognition training is performed.

[0045] Step S603: save the trained model parameters for subsequent prediction and fine-tuning, use the UIE model and process the target entity recognition after fine-tuning.

[0046] The application further provides that the step S7 comprises:

[0047] Step S701: data processing is performed on the screened table data, original corresponding page content is converted into a picture and the table picture obtained after picture cropping according to the page position is saved by combining the parsed PDF page code and the position information of the table in the page;

[0048] Step S702: data labeling is performed on the obtained picture, and the labeled data is divided into a training set and a test set in a 7:3 ratio for model training; step S703: the YOLOV3 model is trained on the open source table position detection dataset TableBank, and the trained model is fine-tuned on the labeled data collected from the PDF;

[0049] The trained model parameters are saved for subsequent prediction and fine-tuning.

[0050] The application further provides that the step S8 comprises:

[0051] Step S801: the table picture obtained by screening is converted into a grayscale picture and is subjected to adaptive threshold processing for binarization;

[0052] Step S802: the picture after binarization processing is subjected to morphological operation, i.e., image erosion and expansion, to identify vertical and horizontal lines;

[0053] Step S803: the table picture after morphological processing is subjected to binary classification labeling, and the table picture with an inner frame line is labeled as 1, and the table picture without an inner frame line is labeled as 0;

[0054] Step S803: the table after binary classification labeling is subjected to binary classification classifier training to distinguish whether the table has an internal frame line;

[0055] Step S804: for the picture with internal linear structure, the vertical and horizontal lines are extracted, and the intersection coordinates are calculated;

[0056] Step S805: for the picture without internal linear structure, picture processing is performed, a threshold is set, and lines longer than the threshold are removed;

[0057] Step S806: the picture is subjected to denoising processing, and the blank area and the text area are distinguished by standardizing the pixel value;

[0058] Step S807: Systematically scan from the top to the bottom of the image, and mark the row with pixel value and zero as a potential area containing horizontal internal frame line;

[0059] Step S808: Merge all adjacent horizontal line potential areas, and locate the midpoint of these continuous zero value areas to obtain the horizontal internal frame line position of the table row;

[0060] Step S809: Systematically scan from left to right of the image, and mark the row with pixel value and zero as a potential area containing vertical internal frame line;

[0061] Step S810: Merge the continuous blank areas aligned vertically, and eliminate those areas with a single row height, and the vertical line intersecting the largest cumulative blank area is designated as the vertical internal frame line of the table, and each iteration marks the intersecting area as traversed until all blank areas are considered;

[0062] Step S811: The intersection of the boundary and the vertical internal boundary line divides the main cells of the table, and by judging the vertical line of each cell, the intersection of the text-containing area is determined, and the vertical line segment intersecting the text is removed, thereby saving the final table internal frame line;

[0063] Step S812: Reconstruct the table structure by combining the table internal frame line, the outer frame line position, and the parsed table content, and save the extracted table data.

[0064] The application further provides that the step S9 comprises:

[0065] Step S901: Data labeling is performed on the target entity in the screened picture data for target detection, and the labeled data is divided into a training set and a test set in a 7:3 ratio for model training;

[0066] Step S902: The YOLOV3 model is trained using the labeled data;

[0067] Step S903: Save or obtain the target detection model parameters;

[0068] The step S10 comprises:

[0069] Step S1001: The PDF document requiring data extraction after being parsed in step S1 is subjected to target element recognition,

[0070] The step S11 comprises:

[0071] Step S1101: The domain expert manually checks the extracted naming result obtained from step S10, adds the undetected named entity, and removes the detected incorrect entity;

[0072] Step S1102: The field expert manually checks the table outer frame line result obtained from step S10, adjusts the table outer frame line position, and re-labels the outer frame line for the undetected table;

[0073] Step S1103: The field expert manually checks the table inner frame line result obtained from step S10, adjusts, adds, or deletes the erroneous table inner frame line position; step S1104: The field expert manually checks the picture target detection result obtained from step S10, adjusts, adds, or deletes the erroneous picture target frame line position;

[0074] Step S1105: Continuously fine-tune and iterate the model with the updated labeled text, table, and picture data, and update the target element detection network parameters;

[0075] The step S12 comprises:

[0076] Step S1201: Set a regularization keyword table and perform regularization processing on the data obtained from step S11;

[0077] The step S13 comprises:

[0078] Step S1301: Set a regularization keyword table, perform data fusion on the data obtained from step S12, and save.

[0079] The present application has the following beneficial effects: 1. In view of the problem that the data fusion effect is not good due to the difficulty of cross-modal information association, the present application combines the cross-modal information normalization domain knowledge injection strategy, explores the domain knowledge injection method based on "man in the loop", and based on the algorithm, a mode recognition network for multi-modal data such as tables, pictures and texts is constructed.

[0080] 2. By extracting detailed data from tables and images in PDF documents, the system can achieve comprehensive coverage of data in multiple dimensions, providing convenience for subsequent data analysis and modeling process. Compared with the traditional manual data compilation method, the present application significantly improves the data collection efficiency and reduces the dependence on human resources. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1 The algorithm flowchart of the present application;

[0082] Figure 2 The data processing flow preview;

[0083] Figure 3 PDF metadata extraction example. DETAILED DESCRIPTION

[0084] The application will be described in detail below with specific implementation examples. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be pointed out that, for those skilled in the art of named entity normalization, table structure recognition, table detection, picture detection, and data annotation interaction design, without departing from the concept of the present application, a number of changes and improvements can be made. These are within the scope of the present application.

[0085] The present application designs and implements a method and system for multi-source heterogeneous document multi-modal data extraction based on human-in-the-loop. Specifically, it includes the following steps:

[0086] Step S1: Preprocessing the original PDF file, parsing the file to obtain text, tables, pictures and metadata.

[0087] Step S2: Organize and clean the metadata of the PDF file obtained in step S1 to ensure the normalization and integrity of the data.

[0088] Step S3: Clean and organize the parsed text data, and divide it into training set and test set for training of the named entity recognition (NER) model.

[0089] Step S4: Clean and organize the parsed table data, and divide it into training set and test set for training of the table detection and table structure recognition model.

[0090] Step S5: Clean and organize the parsed picture data, and divide it into training set and test set for training of the target picture detection and recognition model.

[0091] Step S6: Input the text data processed in step S3 into the UIE model for fine-tuning to train the NER model.

[0092] The UIE model is very suitable for extracting structured information from unstructured natural language text, especially in identifying entity names, location information and other related entities. Under this sequence labeling framework, the task is to predict the optimal arrangement of label sequences by maximizing the conditional probability shown in the following formula:

[0093]

[0094] This probabilistic formalization method enables the present application to systematically infer the most likely label configuration of the mark sequence, and improves the accuracy of entity recognition by utilizing the inherent relevance between labels.

[0095] Step S7: input the table data processed in step S4 into the YOLOV3 model for fine-tuning to train the table detection and table structure recognition model. This model is widely praised for its accuracy and ease of deployment. The present application adopts a standard YOLOv3 loss function:

[0096] L = L bbox + L obj + L class .

[0097] where L bbox represents the bounding box regression loss, L obj represents the object existence loss, and L class represents the class prediction loss. The bounding box regression loss L bbox is calculated using mean squared error when an object is detected, mainly for the x, y coordinates of the center and the width and height of the bounding box. YOLOv3 adjusts the offset of the predefined anchor box and applies the loss to these offsets as follows:

[0098]

[0099] where L obj represents the existence of objects x, y, w, h and are the actual and predicted box coordinates and dimensions in cell i. The object existence loss L obj is defined by penalizing the wrong object existence score as follows:

[0100]

[0101] where C represents the confidence score, and λ obj is a weight factor used to balance object and non-object detection. The class prediction loss L class uses cross-entropy loss, aiming to accurately classify the detected objects as follows:

[0102]

[0103] where p(c) and represent the actual probability and predicted probability of the existence of a class in the box, respectively. This comprehensive loss function ensures that YOLOv3 can efficiently locate and classify objects, enhancing its adaptability in real-time object detection tasks.

[0104] Step S7: input the picture data processed in step S5 into the model for fine-tuning to train the target picture detection and recognition model.

[0105] Step S8: reconstruct the table structure according to the table inner frame line recognition algorithm.

[0106] Step S9: input the processed picture data in step S5 into the model for fine-tuning to train the target picture detection and recognition model.

[0107] Step S10: integrate the trained text, table and picture recognition models in steps S6, S7 and S8, input the processed file in step S1 into the model, and perform automatic extraction of multi-modal target data.

[0108] Step S11: manually check and label the multi-modal data extracted in step S9, save the labeled data and correction content, and feed back to steps S6, S7, S8 and S9 for continuous iteration and optimization of the model, while saving the labeled data.

[0109] Step S12: normalize the multi-modal data manually labeled in step S11 and the metadata processed in step S2.

[0110] Step S13: fuse and save the normalized multi-modal data to build an efficient data storage system.

[0111] In the above technical solution, the step S1 comprises:

[0112] Step S101: using GROBID tool to automatically parse PDF document. GROBID uses advanced sequence labeling model to optimize document parsing, and its modular design can adapt to different structural levels of documents to ensure proper adjustment of data, features and text representation. Each model is equipped with a set of specific labels, and by collectively applying these models, detailed and structured results are generated. GROBID systematically classifies and labels text regions in PDF documents, then integrates the content into structured labels generated by the model, and converts it into an XML document.

[0113] Step S102: group and store the data obtained by parsing in step S101 as PDF metadata, text data in PDF, table data in PDF and picture data in PDF.

[0114] In the above technical solution, the step S2 comprises:

[0115] Step S201: arrange and clean the PDF file metadata obtained by parsing in step S1 (specifically, use a computer to clean and filter according to rules) to ensure the integrity and accuracy of the data. The specific data cleaning rules are: remove data documents that cannot obtain metadata by parsing, and remove data documents with PDF parsing errors.

[0116] In the above technical solution, the step S3 comprises:

[0117] Step S301: The PDF text obtained by parsing in step S1 is arranged and cleaned (specifically, computer rule cleaning and screening are performed) to ensure the integrity and accuracy of the data. The specific data cleaning rules are: for the parsed text data, remove punctuation marks and spaces.

[0118] In the above technical solution, the step S4 comprises:

[0119] Step S401: The PDF table data obtained by parsing in step S1 is arranged and cleaned (specifically, computer rule cleaning and screening are performed) to ensure the integrity and accuracy of the data. The specific data cleaning rules are: remove the document data without position information of the parsed table.

[0120] In the above technical solution, the step S5 comprises:

[0121] Step S501: The PDF picture data obtained by parsing in step S1 is arranged and cleaned (specifically, computer rule cleaning and screening are performed) to ensure the integrity and accuracy of the data. Remove the document data that fails to parse the picture.

[0122] In the above technical solution, the step S6 comprises:

[0123] Step S601: Data annotation is performed on the named entity of the text data obtained by screening, and the annotated data is divided into training set and test set according to the ratio of 7:3 for model training.

[0124] To cope with the challenge of limited training sample size in the study, 14,665 samples annotated by domain experts are used for optimization. The annotation content covers key information such as country, region, latitude and longitude, rock type, geological period, time, disaster type, related indicators, loss, disaster scale, influencing factors, prevention measures, and disaster causes. In addition, these annotated data are also used for further fine-tuning of the model to more accurately identify and extract content related to potential target entities.

[0125] Step S602: Based on the unified information extraction model (UIE) in the open source model library of Baidu Paddle, the named entity recognition training is performed.

[0126] Step S603: Save the trained model parameters for subsequent prediction and fine-tuning. The results of target entity recognition after using UIE and fine-tuning are shown in the following table.

[0127]

[0128] In the above technical solution, the step S7 comprises:

[0129] Step S701: Perform data processing on the screened table data, combine the parsed PDF page code where the table is located and the position information of the table in the page, convert the original corresponding page content into a picture and perform picture cropping according to the page position, and save the obtained table picture.

[0130] Step S702: Perform data labeling on the obtained picture, and divide the labeled data into a training set and a test set in a 7:3 ratio for model training.

[0131] Step S703: Train the YOLOV3 model on the open source table position detection dataset TableBank, and fine-tune the trained model on the labeled data collected from the PDF.

[0132] TableBank is a public dataset for table detection and recognition, consisting of a large number of high-quality document images, and is specifically designed to promote research and development in the field of table structure recognition and information extraction. The dataset is composed of scientific documents and other materials from different sources, including PDF format journal papers, conference papers, technical reports, etc., which makes it diverse and valuable for real-world applications. The table data in TableBank is carefully annotated, containing detailed structured information such as table boundaries and cell contents, which is very helpful for training and evaluating table detection models. The core goal of TableBank is to improve the performance of table detection and structured data extraction, especially in complex documents in natural scenes. Researchers and developers can use TableBank to train deep learning models such as YOLO, Mask R-CNN, etc., to achieve more accurate table recognition and content extraction. Due to its extensive coverage and high-quality annotation, TableBank has proven to be of significant value in promoting table detection research and related technology development.

[0133] Step S704: Save the trained model parameters for subsequent prediction and fine-tuning.

[0134] In the above technical solution, the step S8 includes:

[0135] Step S801: Convert the screened table picture to a grayscale image and perform adaptive thresholding for binarization.

[0136] Step S802: Perform morphological operations on the binarized picture, i.e., image erosion and dilation, to identify vertical and horizontal lines.

[0137] Step S803: Perform binary classification labeling on the table picture after morphological processing, labeling the inner frame line where the table exists as 1 and labeling the inner frame line where the table does not exist as 0.

[0138] Step S803: Binary classification classifier training is performed on the table after binary classification annotation to distinguish whether the table has internal frame lines.

[0139] Step S804: For pictures with internal linear structures, vertical and horizontal lines are extracted, and the intersection coordinates are calculated.

[0140] Step S805: For pictures without internal linear structures, picture processing is performed, threshold is set, and lines exceeding the threshold length are removed.

[0141] Step S806: The picture is denoised by standardizing the pixel value to distinguish the blank area and the text area.

[0142] Step S807: Systematic pixel scanning from the top to the bottom of the image. The row with a pixel value of zero is marked as a potential area containing horizontal internal frame lines.

[0143] Step S808: Merge all adjacent horizontal line potential areas, and locate the midpoint of these continuous zero value areas to obtain the horizontal internal frame line position of the table row.

[0144] Step S809: Systematic pixel scanning from left to right. The row with a pixel value of zero is marked as a potential area containing vertical internal frame lines.

[0145] Step S810: Merge the continuous blank areas aligned vertically, and eliminate those with a height of a single row. The vertical line intersecting the largest cumulative blank area is designated as the vertical internal frame line of the table, and each iteration marks the intersection area as traversed until all blank areas are considered.

[0146] Step S811: The intersection of the boundary and the vertical internal boundary line divides the main cells of the table. By judging the vertical line of each cell, the intersection with the text area is determined. The vertical line segment intersecting the text is removed, thereby saving the final table internal frame line.

[0147] Step S812: Combine the table internal frame line, the outer frame line position, and the parsed table content to reconstruct the table structure, and save the extracted table data.

[0148] In order to test the effectiveness of the table detection and table structure recognition algorithm, the present application comprehensively evaluates 100 selected academic articles, a total of 423 tables are recognized, and the recognition results are as follows.

[0149]

[0150] In the above technical solution, the step S9 comprises:

[0151] Step S901: Data labeling is performed on the target entity in the screened picture data for target detection, and the labeled data is divided into a training set and a test set in a 7:3 ratio for model training.

[0152] Step S902: The YOLOV3 model is trained using the labeled data.

[0153] Step S903: Save or obtain the target detection model parameters.

[0154] In order to make the model meet the needs of identifying target images, the research team fine-tuned the model on 422 images carefully labeled by domain experts. The data set was divided into a training set and a test set in a 9:1 ratio to ensure that the model was comprehensively trained and evaluated under diverse environmental conditions. This configuration aims to strictly test the performance of the model under different scenarios and complexities to improve its generalization ability and stability in practical applications. The target picture detection results are shown in the following table.

[0155] IoU 0.5 0.75 0.9 Precision 95.4％ 72.9％ 7％

[0156] In the above technical solution, the step S10 comprises:

[0157] Step S1001: The PDF document requiring data extraction after parsing in step S1 is subjected to target element recognition.

[0158] In the above technical solution, the step S11 comprises:

[0159] Step S1101: The domain expert manually checks the extraction and naming results obtained from step S10, adds undetected named entities, and removes incorrectly detected entities.

[0160] Step S1102: The domain expert manually checks the table outer frame line results obtained from step S10, adjusts the table outer frame line positions, and re-labels the outer frame line for undetected tables.

[0161] Step S1103: The domain expert manually checks the table inner frame line results obtained from step S10, adjusts, adds, or deletes incorrect table inner frame line positions.

[0162] Step S1104: The domain expert manually checks the picture target detection results obtained from step S10, adjusts, adds, or deletes incorrect picture target frame line positions.

[0163] Step S1105: The updated labeled text, table, and picture data are used to continuously fine-tune and iterate the model, and the target element detection network parameters are updated.

[0164] In the technical solution, the step S12 comprises:

[0165] Step S1201: setting a regularization keyword table to perform regularization processing on the data obtained from step S11.

[0166] In the technical solution, the step S13 comprises:

[0167] Step S1301: setting a regularization keyword table to perform data fusion and save the data obtained from step S12.

[0168] In order to comprehensively evaluate the effect of data fusion method, 2650 data points in 100 scientific articles are carefully organized and manually labeled. This meticulous labeling process provides a solid benchmark for evaluating the efficiency of multi-modal target data recognition system. This paper compares the data filling rate achieved by different modal data types in recognition, and shows the significant improvement of system efficiency after multi-modal data fusion, which verifies the importance of data fusion method in improving recognition performance and data completeness.

[0169]

Claims

1. A method for multi-source heterogeneous document multimodal data extraction based on human-in-the-loop, characterized in that: Specifically, the method comprises the following steps: Step S1: preprocessing the original PDF file, parsing the file to obtain text, tables, pictures and metadata; Step S2: collating and cleaning the metadata of the PDF file obtained in step S1 to ensure the standardization and integrity of the data; Step S3: cleaning and collating the parsed text data, dividing it into a training set and a test set for training the NER model; Step S4: cleaning and collating the parsed table data, and dividing it into a training set and a test set for training the table detection and table structure recognition model; Step S5: cleaning and collating the parsed picture data, and dividing it into a training set and a test set for training the target picture detection and recognition model; Step S6: inputting the processed text data in step S3 into the UIE model for fine-tuning to train the NER model; The optimal arrangement of the label sequence is predicted by the conditional probability shown in the following formula: Step S7: inputting the processed table data in step S4 into the YOLOV3 model for fine-tuning to train the table detection and table structure recognition model. This model is widely praised for its accuracy and ease of deployment, and the present application uses the standard YOLOv3 loss function: L = L bbox + · L obj + L class where L bbox represents the bounding box regression loss, L obj represents the object existence loss, L class represents the class prediction loss, the bounding box regression loss L bbox Using mean squared error calculation when detecting objects, mainly for the x, y coordinates of the center and the width and height of the bounding box, YOLOv3 adjusts the offset of the pre-defined anchor box and applies the loss to these offsets, as follows: where L obj represents the presence of object x, y, w, h in cell i and are actual and predicted box coordinates and dimensions, respectively, and object presence loss L obj is defined by penalizing incorrect object presence scores, as follows: where C denotes the confidence score, λ obj is a weight factor for balancing object and non-object detection, the class prediction loss L class uses a cross-entropy loss, aiming to accurately classify the detected objects, as follows: where p(c) and respectively represent the actual probability and the predicted probability of the category existing in the frame, and the comprehensive loss function ensures that YOLOv3 can efficiently locate and classify objects, enhancing its adaptability in real-time object detection tasks. Step S7: inputting the processed picture data in step S5 into the model for fine-tuning to train the target picture detection and recognition model; Step S8: reconstructing the table structure according to the table inner frame line recognition algorithm; Step S9: inputting the processed picture data in step S5 into the model for fine-tuning to train the target picture detection and recognition model; Step S10: integrating the trained text, table and picture recognition models in steps S6, S7 and S8, inputting the processed file in step S1 into the model for automatic extraction of multi-modal target data; Step S11: manually checking and labeling the multi-modal data extracted in step S9, saving the labeled data and corrections, and feeding back to steps S6, S7, S8 and S9 for continuous iteration and optimization of the model, while saving the labeled data; Step S12: normalizing the multi-modal data manually labeled in step S11 and the metadata collated in step S2; Step S13: fusing and saving the normalized multi-modal data to build an efficient data storage system.

2. The method of claim 1, wherein: The step S1 comprises: Step S101: using the GROBID tool to automatically parse the PDF document. GROBID systematically classifies and labels the text areas in the PDF document, then integrates the content into structured labels generated by the model, and converts it into an XML document; Step S102: grouping and storing the data obtained by parsing in step S101 as PDF metadata, text data in the PDF, table data in the PDF and picture data in the PDF.

3. The method of claim 1, wherein: The step S2 comprises: Step S201: The PDF file metadata obtained by parsing in step S1 is arranged and cleaned. The specific data cleaning rules are: removing data documents that cannot obtain metadata by parsing, and removing data documents with PDF parsing errors.

4. The method of claim 1, wherein: The step S3 includes: step S301: The PDF text obtained by parsing in step S1 is arranged and cleaned. The specific data cleaning rules are: for the text data obtained by parsing, remove punctuation marks and spaces.

5. The method of claim 1, wherein: The step S4 includes: Step S401: The PDF table data obtained by parsing in step S1 is arranged and cleaned to ensure the integrity and accuracy of the data. The specific data cleaning rules are: remove document data without position information of the parsed table.

6. The method of claim 1, wherein: The step S5 includes: Step S501: The PDF picture data obtained by parsing in step S1 is arranged and cleaned to ensure the integrity and accuracy of the data, and the document data with failed picture parsing is removed.

7. The method of claim 1, wherein: The step S6 includes: Step S601: The named entity of the text data obtained by screening is data labeled, and the labeled data is divided into training set and test set according to the ratio of 7:3 for model training; Step S602: Based on the unified information extraction model UIE model in the open source model library of Baidu Paddle, the named entity recognition training is carried out; Step S603: Save the trained model parameters for subsequent prediction and fine tuning, and use the UIE model and the processed target entity recognition after fine tuning.

8. The method of claim 1, wherein: The step S7 includes: Step S701: The table data obtained by screening is data processed. The original corresponding page content is converted into a picture and the picture is cropped according to the page position, and the obtained table picture is saved, combined with the PDF page code obtained by parsing and the position information of the table in the page; Step S702: The obtained picture is data labeled, and the labeled data is divided into training set and test set according to the ratio of 7:3 for model training; Step S703: The YOLOV3 model is trained on the open source table position detection data set TableBank, and the trained model is fine tuned on the labeled data collected from PDF; Save the trained model parameters for subsequent prediction and fine tuning.

9. The method of claim 1, wherein: The step S8 includes: Step S801: The table picture obtained by screening is converted into a gray scale picture and is subjected to adaptive threshold processing for binarization; Step S802: After the binarization processing of the picture, the picture is subjected to morphological operation, i.e. image erosion and expansion, to identify vertical and horizontal lines; Step S803: The table picture after morphological processing is subjected to two-classification labeling, and the table picture with internal frame line is labeled as 1, and the table picture without internal frame line is labeled as 0; Step S803: The table after two-classification labeling is subjected to two-classification classifier training to distinguish whether the table has internal frame line; Step S804: For the picture with internal linear structure, the vertical and horizontal lines are extracted, and the intersection coordinates are calculated; Step S805: For the picture without linear structure inside, picture processing is performed, a threshold is set, and lines exceeding the threshold length are removed; Step S806: The picture is denoised by standardizing the pixel values to distinguish the blank area and the text area; Step S807: The image is scanned pixel by pixel from top to bottom, and the rows with pixel values of zero are marked as potential areas containing horizontal internal frame lines; Step S808: All adjacent horizontal line potential areas are merged, and the midpoints of these continuous zero-value areas are located to obtain the horizontal internal frame line positions of the table rows; Step S809: The image is scanned pixel by pixel from left to right, and the rows with pixel values of zero are marked as potential areas containing vertical internal frame lines; Step S810: The vertically aligned continuous blank areas are merged, and those with a height of a single row are eliminated. The vertical lines intersecting the largest cumulative blank area are designated as the vertical internal frame lines of the table, and each iteration marks the intersecting area as traversed until all blank areas are considered; Step S811: The intersection of the boundary and the vertical internal boundary line divides the main cells of the table, and by judging the vertical lines of each cell, the intersection of the vertical lines with the text area is determined. The vertical line segments intersecting the text are removed to save the final table internal frame lines; Step S812: The table structure is reconstructed by combining the table internal frame lines, the outer frame line positions, and the parsed table content, and the extracted table data is saved.

10. The multi-source heterogeneous document multi-modal data extraction method based on human-in-the-loop according to claim 1, characterized in that: the step S9 comprises: Step S901: Data labeling is performed on the target entities in the screened picture data for target detection, and the labeled data is divided into a training set and a test set in a 7:3 ratio for model training; Step S902: The YOLOV3 model is trained using the labeled data; Step S903: The target detection model parameters are saved or obtained; the step S10 comprises: Step S1001: The target elements in the PDF document requiring data extraction parsed in step S1 are identified, the step S11 comprises: Step S1101: The domain experts manually check the extracted named results obtained from step S10, add undetected named entities, and remove incorrectly detected entities; Step S1102: The domain experts manually check the table outer frame line results obtained from step S10, adjust the table outer frame line positions, and re-label the outer frame lines for undetected tables; Step S1103: The domain experts manually check the table internal frame line results obtained from step S10, adjust, add, or delete incorrect table internal frame line positions; Step S1104: The domain experts manually check the picture target detection results obtained from step S10, adjust, add, or delete incorrect picture target frame line positions; Step S1105: The updated labeled text, table, and picture data are used to continuously fine-tune and iterate the model to update the target element detection network parameters; the step S12 comprises: Step S1201: setting a regularization keyword table, and performing regularization processing on the data obtained from step S11; The step S13 comprises: Step S1301: setting a regularization keyword table, and performing data fusion and saving on the data obtained from step S12.

Images