Multi-modal literature data extraction method and device and medium

Abstract

The invention discloses a multi-modal literature data extraction method and device and a medium. The method comprises the following steps: firstly, standardizing literatures to obtain a structured document comprising a literature directory, author information, a paragraph text, a paragraph layout, a table LaTeX code, a formula LaTeX code and / or image description; then, related retrieval of fragments is performed on texts, formulas, tables and images from the structured document according to keywords and / or vectorization retrieval strategies. The retrieved fragments are optimized through correlation sorting, and the number of the fragments needing to enter large model processing is reduced. And finally, performing question and answer configuration on the large language model based on a user extraction demand and a sorting fragment, generating a structured answer through a prompt engineering technology, ensuring that the results can be traced through an original fragment, ensuring the availability of question and answer results, and reminding the user to verify and process a part which cannot be traced when necessary.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and natural language processing, and in particular to a method, device and medium for extracting multimodal document data. Background Art

[0002] With the continuous advancement of the digitalization process, a large amount of literature information is stored in various forms. The content of the literature not only includes traditional text information, but also contains a large amount of multimodal information such as tables, formulas, and images. In the fields of scientific research, technological development, etc., how to efficiently and accurately extract key information from these documents has become a key issue. However, most of the existing literature information extraction methods only process single-modal data, lack comprehensive analysis and integration of multimodal literature content, and cannot meet the increasingly complex needs of modern scientific researchers and technicians.

[0003] The existing technology uses optical character recognition (OCR) to extract text information when extracting document information. This technology cannot effectively process tables, formulas, and images in documents. In addition, users often obtain redundant data when searching for information contained in documents. How to accurately extract data according to user needs is a technical difficulty currently faced. Therefore, there is an urgent need for a method that can realize multimodal data information extraction and intelligent retrieval to meet users' diverse needs for document information extraction. Summary of the invention

[0004] The purpose of the present invention is to address the deficiencies of the prior art and provide a multimodal document data extraction method, device and medium for extracting data in various forms such as text, tables, formulas and images from documents, and supporting applications such as intelligent retrieval.

[0005] The object of the present invention is achieved through the following technical scheme: a multimodal document data extraction method, comprising:

[0006] Standardize the input document to obtain a structured document containing a bibliography, author information, paragraph text, paragraph layout, table LaTeX code, formula LaTeX code and / or image description;

[0007] Based on the extraction target, extracting a plurality of target segments from the structured document;

[0008] Screening several target fragments using a relevance ranking algorithm;

[0009] The extraction target and the filtered target fragments are input into the large language model to generate the extraction results; and the original document fragments corresponding to each extraction result are given through text similarity matching technology.

[0010] Furthermore, the processing of the paragraph layout includes: using a text detection model to process the input document, dividing it into different paragraphs, and obtaining coordinate information of the corresponding paragraphs; using a classification model to determine the paragraph category, the category including introduction, method and conclusion; using a classification-based sorting model, combining the paragraph category and the paragraph coordinate information for identification and sorting, and completing the paragraph layout.

[0011] Furthermore, according to user needs, data extraction is customized to obtain the extraction target; the configuration content includes the definition of fields and the setting of primary keys and subkeys. The primary key is used for the management of data records. The primary key represents the extracted object, and the subkey represents the attribute of the object, which is used to realize the semantic relevance of data records.

[0012] Furthermore, a keyword search and / or vectorized search strategy is adopted to extract a number of target segments from the structured document.

[0013] Furthermore, a multimodal recognition model is used to convert tables, formulas and images in the literature into table LaTeX codes, formula LaTeX codes and image descriptions; the multimodal recognition model is an encoder-decoder architecture; the encoder includes a number of identical first unit modules, the first unit modules include a first regularization layer, a multi-head attention layer, a second regularization layer, a mixed expert layer and a multi-layer perception layer and are connected in sequence; the decoder includes a number of identical second unit modules, the second unit modules include a multi-head attention module, a forward feedback layer and a regularization layer and are connected in sequence.

[0014] Furthermore, the use of a relevance ranking algorithm to screen a number of target fragments includes:

[0015] The relevance ranking algorithm scores and ranks several target fragments based on the keyword hit rate, vectorized similarity and the position of the fragment in the document, and selects the top n target fragments or selects the target fragments by the threshold of the relevance score.

[0016] Furthermore, the original document fragments corresponding to each extraction result are given by the text similarity matching technology, including:

[0017] The prompt project of the large language model is used to guide the generated extraction results to provide traceability information. If there is a source in the extraction result, string matching is used for verification. If the verification is successful, the tracing result is given. If the verification fails, string matching is used to compare the extraction result generated by the large language model with the paragraph text of the input document. If the comparison is successful, the tracing result is given. For the extraction results that cannot be traced, a prompt is issued, suggesting that the user perform further confirmation and processing.

[0018] The present invention also provides a multimodal document data extraction device, comprising:

[0019] A standardization processing module is used to perform standardization processing on the input document to obtain a structured document including a bibliography, author information, paragraph text, paragraph layout, table LaTeX code, formula LaTeX code and / or image description;

[0020] An extraction module, used for extracting a plurality of target segments from the structured document based on an extraction target;

[0021] A sorting module, used to screen several target fragments using a relevance sorting algorithm;

[0022] The answer module is used to input the extraction target and the filtered target fragments into the large language model to generate the extraction results; and through the text similarity matching technology, the original document fragment corresponding to each extraction result is given.

[0023] The present invention also provides a multimodal document data extraction device, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used for the above-mentioned multimodal document data extraction method.

[0024] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned multimodal document data extraction method is implemented.

[0025] The beneficial effects of the present invention are: by integrating technical means such as optical character recognition, table recognition, formula recognition, multimodal image understanding, intelligent retrieval, relevance ranking, and a large language model, a set of document multimodal data extraction methods is constructed, enabling users to extract required information from complex documents; a multimodal recognition model is also constructed for unified processing of formulas, tables, and image descriptions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0027] Figure 1 A schematic flow chart of a multimodal document data extraction method provided in an embodiment of the present invention.

[0028] Figure 2 A schematic diagram of a document standardization processing module provided in an embodiment of the present invention.

[0029] Figure 3 A schematic diagram of a multimodal artificial intelligence model for formula, table and image description recognition provided in an embodiment of the present invention.

[0030] Figure 4 A schematic diagram of the structure of a multimodal recognition model provided by an embodiment of the present invention.

[0031] Figure 5 A schematic diagram of data extraction configuration and result tracing provided in an embodiment of the present invention.

[0032] Figure 6 A flowchart for implementing result tracing provided in an embodiment of the present invention.

[0033] Figure 7 A schematic diagram of a hardware structure provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0036] The method of the present invention first processes the document by standardization, extracts elements such as document directory, author information, paragraph layout, table, formula and image, and processes the text by OCR (optical character recognition) technology, performs specific model recognition on tables and formulas to generate LaTeX format, and performs multimodal understanding on images to obtain relevant text and data information. Subsequently, the association relationship and primary key setting between data fields are defined according to the data extraction model configured according to user needs. Next, the relevant retrieval of fragments is performed for text, formula, table and image from the document structured data according to keywords and vectorized retrieval strategies. The retrieved fragments are optimized by relevance sorting to reduce the number of fragments that need to be processed by the large language model. Finally, based on the user's extraction needs and sorted fragments, the large language model is configured for question and answer, and structured answers are generated through prompt engineering technology, and it is ensured that these results can be traced back to the original fragments to ensure the availability of the question and answer results, and the user is reminded to verify and process the parts that cannot be traced when necessary.

[0037] Figure 1 The following is a flow chart of a multimodal document data extraction method provided by one embodiment of the present invention. Figure 1 As shown, the multimodal document data extraction method in the embodiment of the present invention includes the following steps:

[0038] (1) Standardize the input document to obtain a structured document containing a bibliography, author information, paragraph text, paragraph layout, table LaTeX code, formula LaTeX code and / or image description;

[0039] Figure 2 A schematic diagram of a document standardization process for converting a standard PDF into a structured Markdown file is shown. Document standardization is the first step of the present invention, and its main goal is to convert different modal contents in the document into a format that is easy to structure and process.

[0040] Text processing: Through OCR technology, the text content in the document is recognized and converted into text data. In order to improve the accuracy of OCR, a multi-level language model is used to correct the recognition results and eliminate common character recognition errors. Especially for documents containing multiple languages, the system can automatically identify different language modules and call the corresponding recognition engine to output as text strings.

[0041] Processing of tables and formulas: Tables and mathematical formulas in documents usually have complex structures. For table images, they are converted into standardized LaTeX codes. For mathematical formula images, they are converted into corresponding LaTeX codes.

[0042] Processing of image description information: For non-tabular and formula images in the literature, we first use the image text recognition model to extract the text content in the image. At the same time, we combine the image classification and object detection models to analyze the graphic elements in the image and establish a connection with the text. The multimodal artificial intelligence model is used to process it into a text string.

[0043] The multimodal recognition model is used to uniformly process formulas, tables, and image descriptions. The model structure is a MoE (mixture of experts) visual transformer structure. The specific implementation is as follows: Figure 3 As shown. Figure 4As shown in the figure, the model structure is an Encoder-Decoder architecture. The Encoder is composed of multiple identical first unit modules for visually representing the image. The MoE implements the expert selection capability and implements the selection of the Feed Forward Neural Network (FFNN) under different recognition tasks by expert routing (Router) of the signal. The Decoder is composed of multiple identical second unit modules for generating image recognition results. The first unit module includes a first regularization layer, a multi-head attention layer, a second regularization layer, a hybrid expert layer and a multi-layer perception layer, which are connected in sequence; the decoder includes several identical second unit modules, and the second unit modules include a multi-head attention module, a forward feedback layer and a regularization layer, which are connected in sequence.

[0044] In one embodiment, the network layer of the encoder corresponds to a 20-layer Swin Transformer model (Shifted Window Transformer), which accepts document input images, and the image size is uniformly scaled to 896 and 672 pixels in height and width. After being processed by the encoder, a sequence of encoding vectors of the image blocks is obtained. Compared with the standard Swin Transformer model (the model structure remains consistent with the paper Swin Transformer: Hierarchical Vision Transformer using Shifted Windows), each standard encoding layer is embedded in an expert mixture layer MoE, which is used to autonomously select expert encoders during the reasoning process, and perform multi-task learning on the generation of formulas, tables and image descriptions through adaptive processing of different image similarities by routing units. Since MoE adopts a sparse activation method, only some experts are activated during each training process, and the parameter update is more reference-based, reducing the risk of overfitting in model training and achieving faster convergence of model effects under the same parameter conditions.

[0045] The implementation of MoE is as follows. First, the input tensor X is gated. The signal after passing through the routing unit becomes S = Softmax(WgX + bg), where Wg and bg are the parameters of the routing unit (combination of feedforward neural network and Softmax), and the normalized probability distribution of each expert is obtained. Then, the expert with the highest Top-k score is selected (the implementation here is selected as 1), and the sparse weight W_active_i corresponding to the selected expert is output. Sparsity helps to reduce computational complexity and achieve better convergence of the model. The output of each expert corresponding to the tensor X is E_i = f(W_i, X + b_i), where i is the expert number, W_i is the expert network parameter, and b_i is the expert network bias; the output of the MoE network layer is Y = sum(W_i E_i), where i is the expert number.

[0046] The decoder uses the decoder part of the mBART model. The decoder contains 12 layers (the model structure remains consistent with the corresponding decoder in the paper Multilingual Denoising Pre-training for Neural Machine Translation). The input is the image block encoding vector sequence output by the encoder, and the output is a text token probability vector of length 4096. One of the advantages of mBART over BART is that it supports multiple languages, which is very important for our recognition scenarios because there are many cases where Chinese and English are mixed.

[0047] Author information is obtained by pattern matching and calling the Crossref API to query the literature database; specifically, regular expressions and other pattern matching techniques are used to identify the author information part of the literature. If the input literature is not clearly marked, pattern matching can be performed based on common combinations of names and institution names.

[0048] The paragraph layout is obtained by module detection and paragraph sorting; specifically, first, the document is processed into different paragraphs by page detection model, and the coordinate information of the corresponding paragraphs is obtained. Secondly, the natural language processing (NLP) model is used to identify the topics of different paragraphs, and a classification model (BERT fine-tuning model) is trained to determine whether the paragraph belongs to the "introduction", "method", "conclusion" and other modules according to the content of the paragraph. In view of the multi-column situation in the document, it is necessary to introduce a paragraph sorting function to align the reading and information processing order requirements. The paragraph sorting adopts a classification-based sorting model, which combines the paragraph content of the document and the coordinate information obtained by the classification and detection model to automatically sort different modules (introduction, method, conclusion, etc.). The specific implementation method is to use the previously identified classification labels, combined with the coordinate information of the paragraph, introduce Pairwise Ranking Loss, and train the BERT fine-tuning model to determine the relative order between different modules.

[0049] Through the full-text assembly module, the information extracted from the input documents above is converted into a structured document (such as a Markdown file) containing multimodal information such as a document table of contents, author information, paragraph text, paragraph layout, table LaTeX code, formula LaTeX code, and / or image descriptions.

[0050] In one embodiment, the format of the structured document is as follows: {

[0051] "Title": {"Stratigraphic Dictionary: Cenozoic Sedimentary Strata on Land in a Certain Country"};

[0052] "Author": {"So-and-so"};

[0053] "Abstract": {"Up until now, the stratigraphic knowledge of a certain country has been scattered in hundreds of scientific / geological publications over the past 120 years. In the early 20th century, the construction of canals in a certain country promoted the development of engineering and geological disciplines to understand the tectonics and stratigraphy of a certain country..."}

[0054] "Paragraph 1": {"Introduction: At the beginning of 2020, I planned to retire in a certain country and end my overseas work life..."}

[0055] "Paragraph 2": {"Understanding the dictionary:... If all the information about the unit exists, the format will be exactly the same as the template on this page,..., Figure 2, Schematic diagram of the canal in a certain country, showing all the locks along a certain island, a certain lake, a certain city in a certain place and the canal. ( Image multimodal understanding outputs text strings )}

[0056] …

[0057] "Paragraph N": {"... The name was later changed back to XXXX. In Spanish, this incision is called the XXXX incision and is also known as the Snake Cut. : \(\quad 9^{\circ} 4^{\prime} \quad 52.1394{}^{\prime\prime} \mathrm{N}, \quad 79^{\circ} 40^{\prime}\) 42.5994"W (Formula LaTeX)... Figure 004. The innermost layer of the pericarp which surrounds a seed in a fruit. It may be membranous (as in apples) or woody (as in the stone of a peach or cherry) ( Image multimodal understanding outputs text strings )}

[0058] …

[0059] "Paragraph N": {...\begin{tabular}{|c|l|c|}

[0060] \hline Unit & \multicolumn{1}{|c|}{ Lithology} & \begin{tabular}{c}

[0061] Thickness \\

[0062] \(\mathbf{( m )}\)

[0063] \end{tabular} \\

[0064] \hline \(\mathbf{8}\) & Andesitic lava & 15 \\

[0065] \hline \(\mathbf{7}\) & Agglomerate & 60 \\

[0066] \hline \(\mathbf{6}\) & Sandy Siltstone & 8 \\

[0067] \hline \(\mathbf{5}\) & Hard, dense, buff limestone; Clypeaster

[0068] fragments & 3 \\

[0069] \hline \(\mathbf{4}\) & \begin{tabular}{l}

[0070] Sandy siltstone and calcareous, pebbly sandstone; Clypeaster concavus

[0071] and few mollusks \\

[0072] (locality 60)

[0073] \end{tabular} & 18 \\

[0074] \hline \(\mathbf{3}\) & Hard, dense, buff limestone; Clypeaster

[0075] fragments & 8 \\

[0076] \hline \(\mathbf{2}\) & \begin{tabular}{l}

[0077] Sandy siltstone and poorly sorted, silty sandstone; manyHeterostegina

[0078] israelskyi, also \\

[0079] Lepidocyclina asterodisca (locality 59)

[0080] \end{tabular} & 18 \\

[0081] \hline \(\mathbf{1}\) & \begin{tabular}{l}

[0082] Conglomerate and coarse-grained, poorly sorted conglomeratic

[0083] sandstone; boulders have \\maximum length of 60 cm, but cobbleshaving

[0084] length of 3 to 6 cm more common than \\boulders; dacite porphyry

[0085] conspicuous among clasts

[0086] \end{tabular} & 70 \\

[0087] \hline & \multicolumn{1}{l|l|}{ TOTAL} & 200 \\

[0088] \hline

[0089] \end{tabular} ( Table Latex code )

[0090] Table 1. The section exposed along a tributary of Río Mandinga, as

[0091] recorded by Stewart et al. (1980) and Woodring (1982).

[0092] …}

[0093] "Paragraph K" { Acknowledgements: The author thanks Dr.

[0094] …}

[0095] "Paragraph L" { References:

[0096] Aguilera O., de Aguilera DR (1999). Bathymetric distribution of

[0097] Miocene to Pleistocene Caribbean teleostean fishes from the coast of

[0098] Panama and Costa Rica. Bulletins of American Paleontology, 357, 251-269.

[0099] Aguilera Orangel, A.; Werner, S.; Béarez, P. (2016). Otoliths of the

[0100] Sciaenidae from the Neogene of tropical America. PalaeoIchthyologica, 14, 7-90.

[0101] …}

[0102] }.

[0103] (2) extracting a plurality of target segments from the structured document based on an extraction target;

[0104] Specifically, the extraction target is obtained based on user needs.

[0105] In one embodiment, the user can customize the data extraction according to specific research needs to obtain the extraction target. The configuration content includes the definition of the field and the setting of the primary key and subkey, which is used for the user to specify the field to be extracted, and is used to define the information to be extracted in batches from the literature and combine it into data entries.

[0106] like Figure 5 As shown, by setting the fields to be extracted and setting the primary key for data record management, the primary key represents the object to be extracted. It can be an experiment, a location, a process, etc., and a primary key is defined for a single extraction process. The subkey represents the attribute of the object (primary key), and there can be multiple subkeys in one extraction to achieve the semantic relevance of data records.

[0107] In order to retrieve the fragments related to the user's needs from the structured document, a keyword search and / or vectorized search strategy is adopted. Preferably, a combination strategy of keyword search and vectorized search is adopted.

[0108] Keyword retrieval: Through simple keyword matching, fragments that may contain target data are retrieved from the text content of the document.

[0109] Vectorized retrieval: Use a text semantic vector model that performs well in text retrieval (such as the open source model bge-large-en provided by BAAI) to vectorize the document content and perform retrieval based on semantic similarity. Compared with keyword matching, vectorized retrieval can discover more potential information related to user needs.

[0110] Extension of search objects: In addition to text, the method of the present invention also supports the search of tables, formulas and images. For tables and formulas, the search is based on LaTeX code, and for images, the search is based on the text information (i.e., image description) corresponding to the multimodal recognition of the image. The matching strategies include text matching and similarity matching after text vectorization.

[0111] Combination strategy of keyword search and vectorized search: In order to achieve the recall rate and accuracy of data fragment retrieval, a combination of keyword search and vectorized search is used. Specifically, the first step is to use keyword search to quickly lock related fragments to ensure the recall rate. The second step is to perform vectorized similarity screening on the results of the first step to retain the fragments with the closest semantics and improve the accuracy of the retrieval.

[0112] (3) Screening several target fragments using a relevance ranking algorithm;

[0113] The multiple fragments retrieved often contain redundant information or incompletely relevant content. In order to improve the accuracy of the results, the search results are sorted by relevance. The sorting algorithm is based on multiple factors, including keyword hit rate, vectorization similarity, and the position of the fragment in the document.

[0114] In one embodiment, the number of segments is automatically reduced according to the needs of the user to reduce the computational overhead of subsequent processing. In particular, for large documents, the reduction of segments can significantly improve the processing efficiency of large language models. The number of reduced segments can be controlled by a threshold of the relevance score, or the maximum number of output segments set by the user.

[0115] (4) The extraction target and the filtered target fragments are input into the large language model to generate the extraction results; and the original document fragments corresponding to each extraction result are given through text similarity matching technology.

[0116] In one embodiment, based on the sorted fragments, the user is interacted with through a large language model, and the large language model (such as OpenAI's GPT-4 and Alibaba Cloud's Tongyi Qianwen large model qwen2) is guided by prompt engineering to generate the data results required by the user. In order to ensure the reliability of the question and answer results, the generated data results are first verified using a large language model to ensure that the output data results meet the user's demands. Secondly, through the traceability mechanism of the data results, the generated answers are corresponded to the original fragments in the document. The implementation of traceability helps to improve the transparency and trust of the system, so that users can understand the basis for generating answers. For answers that cannot be traced, a prompt is issued to suggest that the user make further confirmation and processing.

[0117] The traceability mechanism is mainly implemented by combining the big model prompt project with string matching. Based on the answer given by the big model, it is verified and supplemented by automatic string matching. Figure 6 As shown in the figure, firstly, the prompt project of the large language model is used to guide the generated answers to provide traceability information, such as asking the model to try to "cite the basis of the original text" when generating answers. If there is a clear source in the answer, string matching is used for verification. If the verification is successful, the traceability is given; if the verification fails, string matching is used to compare the answer generated by the large language model with the paragraph of the original text. If there is no clear source in the answer, string matching is used to compare the answer generated by the large language model with the paragraph of the original text. If the comparison is successful, the traceability result is given. If it fails, the traceability is not returned to avoid interference to the user.

[0118] Each extracted data result will be attached with a corresponding document fragment, and users can click to view the original content. After clicking, they can directly trace the source and jump to the corresponding fragment in the original document for secondary confirmation by users, thus improving the user experience. Figure 5 shown.

[0119] The present invention also provides a multimodal document data extraction device, comprising:

[0120] A standardization processing module is used to perform standardization processing on the input document to obtain a structured document including a bibliography, author information, paragraph text, paragraph layout, table LaTeX code, formula LaTeX code and / or image description;

[0121] An extraction module, used for extracting a plurality of target segments from the structured document based on an extraction target;

[0122] A sorting module, used to screen several target fragments using a relevance sorting algorithm;

[0123] The answer module is used to input the extraction target and the filtered target fragments into the large language model to generate the extraction results; and through the text similarity matching technology, the original document fragment corresponding to each extraction result is given.

[0124] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, which will not be repeated here.

[0125] Corresponding to the aforementioned embodiment of a multimodal document data extraction method, the present invention also provides an embodiment of a multimodal document data extraction device.

[0126] See also Figure 7 A multimodal document data extraction device provided in an embodiment of the present invention includes one or more processors for implementing a multimodal document data extraction method in the above embodiment.

[0127] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0128] An embodiment of a multimodal document data extraction device of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory and running them. From the hardware level, if Figure 7 As shown, it is a hardware structure diagram of any device with data processing capability where a multimodal document data extraction device of the present invention is located, except Figure 7 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0129] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0130] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0131] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a multimodal document data extraction method in the above embodiment is implemented.

[0132] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0133] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A multimodal document data extraction method, characterized in that: include: Standardize the input document to obtain a structured document containing a bibliography, author information, paragraph text, paragraph layout, table LaTeX code, formula LaTeX code and / or image description; Based on the extraction target, extracting a plurality of target segments from the structured document; Several target fragments are screened using a relevance ranking algorithm; The extraction target and the filtered target fragments are input into the large language model to generate the extraction results; and the original document fragments corresponding to each extraction result are given through text similarity matching technology.

2. A multimodal document data extraction method according to claim 1, characterized in that: The paragraph layout processing includes: Use the text detection model to process the input document, divide it into different paragraphs, and obtain the coordinate information of the corresponding paragraphs; Using a classification model to determine paragraph categories, the categories include introduction, method, and conclusion; A classification-based sorting model is used to identify and sort paragraphs in combination with paragraph categories and paragraph coordinate information to complete paragraph layout.

3. A multimodal document data extraction method according to claim 1, characterized in that: According to user needs, data extraction is customized to obtain the extraction target; the configuration content includes the definition of fields and the setting of primary keys and subkeys. The primary key represents the extracted object and is used for the management of data records. The subkey represents the attribute of the object and is used to realize the semantic relevance of data records.

4. A multimodal document data extraction method according to claim 1, characterized in that: A keyword search and / or vectorized search strategy is adopted to extract a number of target segments from the structured document.

5. A multimodal document data extraction method according to claim 1, characterized in that: Using a multimodal recognition model, tables, formulas and images in the literature are converted into table LaTeX codes, formula LaTeX codes and image descriptions; the multimodal recognition model is an encoder-decoder architecture; the encoder includes a plurality of identical first unit modules, the first unit modules include a first regularization layer, a multi-head attention layer, a second regularization layer, a mixed expert layer and a multi-layer perception layer and are connected in sequence; the decoder includes a plurality of identical second unit modules, the second unit modules include a multi-head attention module, a forward feedback layer and a regularization layer and are connected in sequence.

6. A multimodal document data extraction method according to claim 1, characterized in that: The method of using a correlation ranking algorithm to screen a plurality of target fragments includes: The relevance ranking algorithm scores and ranks several target fragments based on the keyword hit rate, vectorized similarity and the position of the fragment in the document, and selects the top n target fragments or selects the target fragments by the threshold of the relevance score.

7. A multimodal document data extraction method according to claim 1, characterized in that: The text similarity matching technology is used to provide the original document fragments corresponding to each extraction result, including: Prompt engineering of large language models is used to guide the generated extraction results to provide traceability information; In the case where there is a source in the extraction results, string matching is used for verification; if the verification is successful, the tracing result is given; if the verification fails, string matching is used to compare the extraction results generated by the large language model with the paragraph text of the input document. If the comparison is successful, the tracing result is given; for extraction results that cannot be traced, a prompt is issued, suggesting that the user perform further confirmation and processing.

8. A multimodal document data extraction device, characterized in that: include: A standardization processing module is used to perform standardization processing on the input document to obtain a structured document including a bibliography, author information, paragraph text, paragraph layout, table LaTeX code, formula LaTeX code and / or image description; An extraction module, used for extracting a plurality of target segments from the structured document based on an extraction target; A sorting module, used to screen several target fragments using a relevance sorting algorithm; The answer module is used to input the extraction target and the filtered target fragments into the large language model to generate the extraction result; And through text similarity matching technology, the original document fragment corresponding to each extracted result is given.

9. A multimodal document data extraction device, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement a multimodal document data extraction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a multimodal document data extraction method according to any one of claims 1 to 7 is implemented.

Images