A structured analysis method for PDF paper metadata based on a multimodal large model

Abstract

The present invention discloses a structured analysis method for PDF paper metadata based on a multimodal large model, and relates to the technical field related to academic paper management. The method comprises: collecting a first PDF paper for visual analysis, generating metadata extraction prompts; constructing an initial multimodal large model, and outputting first initial metadata; defining multiple data texts to generate a metadata extraction prompt set; outputting N initial metadata of N PDF papers; generating a multimodal large model; verifying the first initial metadata, and outputting the first metadata of the first PDF paper. The method solves the technical problem that the key and detailed information of the document is lost during the OCR text conversion process in the existing PDF paper metadata analysis, and all the texts in the PDF paper cannot be accurately identified, which leads to poor accuracy and integrity of metadata extraction and identification, and achieves the technical effect of improving the accuracy and integrity of the PDF paper metadata extraction results.

Description

Technical Field

[0001] The present application relates to the technical field related to academic paper management, and specifically to a structured parsing method for PDF paper metadata based on a multimodal large model. Background Art

[0002] With the increasing number of scientific research activities, the number of academic papers has exploded. Among them, PDF format is the main carrier of academic documents. Metadata extraction of PDF papers is an important cornerstone of information integration and knowledge management. The existing PDF paper metadata extraction mainly relies on OCR (optical character recognition) technology. OCR scans the text in the image and converts it into an editable text format, and then extracts metadata based on the text results. However, with the diversification and complexity of academic documents, the limitations of PDF paper metadata extraction based on OCR technology are becoming increasingly prominent. First, when OCR processes PDF documents, the font size and text in the PDF document are not well understood. Visual features such as arrangement and color depth are often ignored in the process of converting to plain text, and this information is crucial for accurately identifying key data such as paper titles, authors, and abstracts. Secondly, when faced with PDF documents containing complex layouts, tables, charts, and multi-column text, the recognition ability of OCR technology seems to be inadequate, and misreading and omissions often occur, resulting in errors in metadata extraction results. In addition, OCR is not robust enough in actual application. Since a large number of PDF documents come from various scanning devices, their quality varies. For example, they may contain defects such as noise, blur, or distortion. These defects will seriously affect the OCR recognition accuracy of PDF documents.

[0003] Therefore, in the current PDF paper metadata parsing related technologies, there is a technical problem that the key and detailed information of the document is lost during the OCR text conversion process, and all the text in the PDF paper cannot be accurately identified, which in turn leads to poor accuracy and integrity of metadata extraction and recognition. Summary of the invention

[0004] This application provides a structured parsing method for PDF paper metadata based on a multimodal large model, which solves the technical problem that the key and detailed information of the document is lost during the OCR text conversion process of the existing PDF paper metadata parsing, and all the text in the PDF paper cannot be accurately identified, which leads to poor accuracy and integrity of metadata extraction and identification, thereby achieving the technical effect of improving the accuracy and integrity of PDF paper metadata extraction results.

[0005] The present application provides a structured analysis method for PDF paper metadata based on a multimodal large model, the method comprising: collecting a first PDF paper for visual analysis, generating first PDF image information, marking the first PDF image information according to text extraction requirements, and generating metadata extraction prompts; constructing an initial multimodal large model, synchronizing the first PDF image information and the metadata extraction prompts to the initial multimodal large model, and outputting first initial metadata of the first PDF paper; collecting N PDF papers for visual analysis based on multiple information sources, generating N PDF image information, defining multiple data texts according to the text extraction requirements to generate metadata extraction prompt sets, N is an integer greater than 1; synchronizing the N PDF image information and the metadata extraction prompt set to the initial multimodal large model, outputting N initial metadata of the N PDF papers, and there is a corresponding relationship between the N PDF papers and the N initial metadata; adjusting the initial multimodal large model based on the N PDF papers and the N initial metadata to generate a multimodal large model; using the multimodal large model in combination with the first PDF paper to verify the first initial metadata, and outputting the first metadata of the first PDF paper.

[0006] In a possible implementation, the first PDF paper further performs the following processing: traversing the multiple information sources for random selection to generate multiple PDF documents; performing virtual generation according to the multiple PDF documents to generate multiple virtual metadata and multiple home page texts, wherein the multiple virtual metadata and the multiple home page texts correspond to the multiple PDF documents; aligning the multiple virtual metadata and the multiple home page texts according to the multiple PDF documents, traversing the alignment results for random extraction to generate the first PDF paper.

[0007] In a possible implementation, a first PDF paper is collected for visual analysis to generate first PDF image information, the first PDF image information is marked according to text extraction requirements, metadata extraction prompts are generated, and the following processing is performed: the document homepage of the first PDF paper is page converted to generate a document homepage image; the document homepage image is image normalized to generate first PDF image information; computer vision technology is used to extract features of the first PDF image information and obtain text extraction requirements; the first PDF image information is traversed and annotated according to the text extraction requirements to generate tags to be extracted; and the metadata extraction prompts are determined according to the tags to be extracted.

[0008] In a possible implementation, the construction of the initial multimodal large model performs the following processing: constructing a visual encoder architecture using a convolutional neural network, and using the visual encoder architecture to train in combination with the first PDF image information to determine the visual encoder; preprocessing is performed based on the metadata extraction prompts to generate a word embedding sequence; constructing a language encoder architecture using a recurrent neural network, and using the language encoder architecture to train in combination with the word embedding sequence to determine the language encoder; collaboratively connecting the visual encoder and the language encoder to construct a model component; calling the LLaVA architecture processing, and constructing the initial multimodal large model in combination with the model component.

[0009] In a possible implementation, based on the synchronization of the first PDF image information and the metadata extraction prompt to the initial multimodal large model, the first initial metadata of the first PDF paper is output, and the following processing is performed: the first PDF image information is input into the visual encoder for feature extraction to generate a visual feature vector; the metadata extraction prompt is input into the language encoder for feature extraction to generate a text feature vector; the visual feature vector and the text feature vector are jointly encoded to generate a fusion feature; and a decoding process is performed based on the fusion feature to generate the first initial metadata.

[0010] In a possible implementation, before the visual feature vector and the text feature vector are jointly encoded, the following processing is also performed: before the data extraction prompt is generated, the visual feature vector and the text feature vector need to be data aligned to generate an image-text pair; a contrast loss function is introduced to calculate the image-text pair to determine the vector distance data; it is determined whether the vector distance data is in a vector distance threshold interval, and if so, a matching signal is generated, and the visual feature vector and the text feature vector are compared based on the matching signal; when the visual feature vector is consistent with the text feature vector, the matching signal is enhanced, and when the visual feature vector is inconsistent with the text feature vector, the matching signal is reduced.

[0011] In a possible implementation, the visual feature vector and the text feature vector are jointly encoded to generate a fusion feature, and the following processing is also performed: dividing a plurality of visual feature sub-regions based on the visual feature vector, wherein the plurality of visual feature sub-regions contain visual tokens; performing attention calculation based on the visual tokens to generate a plurality of visual attention weights; dividing a plurality of text feature sub-regions based on the text feature vector, wherein the plurality of text feature sub-regions contain text tokens; performing attention calculation based on the visual tokens to generate a plurality of text attention weights; based on a matching signal, constructing a plurality of association relationships between the visual feature vector and the text feature vector according to the plurality of visual attention weights and the plurality of text attention weights; and jointly encoding the visual feature vector and the text feature vector based on the plurality of association relationships to generate the fusion feature.

[0012] In a possible implementation, the initial multimodal large model is adjusted based on the N PDF papers and the N initial metadata to generate a multimodal large model, and the following processing is also performed: data enhancement is performed on the N PDF papers and the N initial metadata to construct a training data set and a verification data set; the metadata extraction mode is trained using the training data set to generate multiple training results; the multiple training results are cross-validated using the verification data set; the training results that have passed the verification are extracted to dynamically adjust the initial multimodal large model to generate an adjustment result; an evaluation indicator is introduced to evaluate the adjustment result, and the multimodal large model is generated according to the adjustment evaluation information.

[0013] In a possible implementation, an evaluation indicator is introduced to evaluate the adjustment result, the multimodal large model is generated according to the adjustment evaluation information, and the following processing is performed: a metadata extraction target is set based on the N PDF papers and the N initial metadata; extraction training is performed based on the metadata extraction target, and testing is performed according to the training results to calculate the test pass rate; an extraction quality score is set according to the test pass rate, and the evaluation indicator is formulated according to the quality score value; based on the evaluation indicator, it is determined whether there is adjustment data in the adjustment result that is less than the evaluation indicator; if there is adjustment data in the adjustment result that is less than the evaluation indicator, the adjustment data that is less than the evaluation indicator is marked and recorded, and added to the data group to be verified; the adjustment result is screened based on the data group to be verified to generate the adjustment evaluation information.

[0014] It is intended to use the structured analysis method for PDF paper metadata based on a multimodal large model proposed in this application to collect a first PDF paper for visual analysis, generate a first PDF image information, mark the first PDF image information according to the text extraction requirement, generate a metadata extraction prompt, and then build an initial multimodal large model, synchronize the first PDF image information and the metadata extraction prompt to the initial multimodal large model, output the first initial metadata of the first PDF paper, and then collect N PDF papers based on multiple information sources for visual analysis to generate N PDF image information, define multiple data texts according to the text extraction requirement to generate a metadata extraction prompt set, and then combine the N PDF image information with the metadata extraction prompt. The demonstration set is synchronized to the initial multimodal big model, N initial metadata of N PDF papers are output, the initial multimodal big model is adjusted based on the N PDF papers and the N initial metadata, and the multimodal big model is generated. Finally, the first initial metadata is verified by using the multimodal big model in combination with the first PDF paper, and the first metadata of the first PDF paper is output. The technical problem that the key and detail information of the document is lost during the OCR text conversion process in the existing PDF paper metadata parsing and all the text in the PDF paper cannot be accurately identified, which leads to poor accuracy and integrity of metadata extraction and recognition is solved, and the technical effect of improving the accuracy and integrity of PDF paper metadata extraction results is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the accompanying drawings of the embodiment of the present disclosure will be briefly introduced below. A flow chart is used in the present application to illustrate the operations performed by the system according to the embodiment of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps can be processed in reverse order or simultaneously as needed. At the same time, other operations can also be added to these processes, or one or more operations can be removed from these processes.

[0016] Figure 1 A schematic diagram of a PDF paper metadata structured parsing method based on a multimodal large model provided in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of a process for generating metadata extraction prompts in a PDF paper metadata structured parsing method based on a multimodal large model provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of the process of comparative learning in the structured parsing method of PDF paper metadata based on a multimodal large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below.

[0020] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0021] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict, and the terms "first\second" involved are merely to distinguish similar objects and do not represent a specific ordering of objects. The terms "including" and "having" and any variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by technicians in the technical field of this application. The terms used herein are for the purpose of describing the embodiments of the present application only.

[0022] The present application embodiment provides a PDF paper metadata structured parsing method based on a multimodal large model, such as Figure 1 As shown, the method includes:

[0023] Step S100, collect the first PDF paper for visual analysis, generate the first PDF image information, mark the first PDF image information according to the text extraction requirement, and generate the metadata extraction prompt. Collect a target PDF paper or randomly obtain a PDF paper (for example, including multiple elements such as text, graphics, tables, formulas, etc.) as the first PDF paper, and then use computer technology to perform visual analysis on the first PDF paper. Specifically, the text, graphics, tables and other contents on the first page of the first PDF paper are identified and analyzed to generate a first PDF image, wherein the first PDF image is the first page image of the first PDF paper, and then it is marked according to the text extraction requirement, wherein the text extraction requirement includes extracting the title, author, abstract, keywords, font size of the chapter title, relative position of the text and layout of the page, that is, according to the specific text extraction requirement, mark the corresponding text in the first PDF image information, for example, use annotation tools (such as rectangular box annotation, polygon annotation, etc.) to mark the text area to be extracted, and finally generate the metadata extraction prompt, the paper metadata describes the structured information of the paper data, and is usually used to provide additional information about the data. In academic papers, metadata include title, author, abstract, keywords, etc. Generating metadata extraction prompts means automatically generating or suggesting which metadata should be extracted based on the identified text areas. These extraction prompts can help the model extract corresponding data and detailed information from PDF papers more effectively.

[0024] In a possible implementation, step S100 further includes step S101, traversing the multiple information sources to perform random selection to generate multiple PDF documents; performing virtual generation according to the multiple PDF documents to generate multiple virtual metadata and multiple home page texts, and the multiple virtual metadata and the multiple home page texts correspond to the multiple PDF documents; aligning the multiple virtual metadata and the multiple home page texts according to the multiple PDF documents, traversing the alignment results to perform random extraction, and generating the first PDF paper.

[0025] Preferably, the first PDF paper is generated based on the metadata of massive PDF data synthesized by latex style of a small number of publishers, that is, a small number of publishers are determined by random selection from multiple information sources, latex styles are retrieved by a small number of publishers, and the data of the PDF document is synthesized by the retrieved latex style, which means that each PDF document is virtually extracted, and the PDF document may include various academic papers, reports, etc.; that is, a large number of PDFs with a small number of styles and their corresponding metadata are simulated at low cost by large models such as GPT4, and virtual metadata and front page texts of multiple PDF documents are generated. The virtual metadata includes the title, author, abstract, keywords and publishing information of the paper, etc., wherein each PDF document uniquely corresponds to a virtual metadata and front page text, and then the multiple virtual metadata and multiple front page texts are aligned according to multiple PDF documents, and the corresponding virtual PDF documents are generated by using the LATEX style text provided by the publisher to obtain multiple PDF papers as the alignment result, and finally the multiple PDF papers are traversed and one is randomly extracted from them as the first PDF paper.

[0026] In one possible implementation, Figure 2 As shown, step S100 further includes step S110, performing page conversion on the document homepage of the first PDF paper to generate a document homepage image. Performing page conversion on the document homepage of the first PDF paper refers to converting the document homepage in PDF format into an image format (such as JPEG, PNG, etc.) to generate a document homepage image. Step S120, performing image standardization processing on the document homepage image to generate first PDF image information. Image standardization processing generally includes adjusting the size, resolution, color space, etc. of the document homepage image to ensure the consistency and comparability of the image to obtain a first PDF image. Step S130, using computer vision technology to extract features of the first PDF image information, determine multiple image features, and obtain text extraction requirements. Use computer vision technology (such as deep learning, image processing algorithms, etc.) to identify and analyze key features in images. These features may include font size features (such as the font size of key information such as titles, subtitles, author names, abstracts, etc. is often different from other main text), font relative position features (such as the relative position of elements such as author name, title, abstract on the page, the author name may be located below the title, the abstract may be located below the author name and usually maintain a certain vertical distance from the title and author name), typesetting layout features (such as paragraph distribution, column layout, table position, image embedding method, etc.); by analyzing the above image features, it is possible to infer which parts of the text are the focus of metadata extraction, that is, to obtain text extraction requirements. For example, the title is usually located at the top of the page and has a larger font size, and the author name and unit may be located below the title and have a smaller font size but a fixed position.

[0027] Step S140, traverse the first PDF image information according to the text extraction requirements to mark and generate tags to be extracted. Use image processing technology to convert the text content in the PDF image into a processable format (such as a text string or a text block), traverse the first PDF image information according to the text extraction requirements to find and identify potential text information, and generate tags to be extracted to determine which text is extracted, such as "title", "author", "abstract" and other tags. Step S150, determine the metadata extraction prompt according to the tags to be extracted. The generated tags to be extracted are used to determine metadata extraction hints, which are used to prompt how to effectively extract corresponding metadata from the original document (such as PDF file). Specifically, a metadata extraction strategy is formulated, including using text position features (such as font relative position features) to determine the specific position of metadata in the document; identifying and extracting metadata through text content matching (such as keyword search, regular expression matching); for structured documents (such as PDF with HTML tags), metadata can be extracted by parsing the document structure, etc. Metadata extraction hints are generated and determined according to the metadata extraction strategy, including position hints (page positions or text areas where metadata may appear), format hints (text size, format, etc.), content hints (keywords or phrases that metadata may contain), etc.

[0028] Step S200, constructing an initial multimodal large model, synchronizing the first PDF image information and the metadata extraction prompt to the initial multimodal large model, and outputting the first initial metadata of the first PDF paper. Based on the convolutional neural network, the visual encoder architecture is constructed using the first PDF image information. Specifically, the visual encoder architecture is trained using the first PDF image information as a training set. During the training process, the visual encoder is learned how to automatically extract useful visual information from the PDF image, thereby obtaining the visual encoder. Based on the recurrent neural network, the language encoder architecture is constructed using metadata prompts. The language encoder architecture is trained using metadata extraction prompts as training data. During the training process, the language encoder is learned how to extract useful language information from the metadata extraction prompt sequence, thereby obtaining the language encoder. Then, the LLaVA architecture is used to integrate the constructed visual encoder, language encoder and the connection mechanism between them, so as to construct an initial multimodal large model. The multimodal large model can process multiple modal (such as image, text, etc.) data at the same time, that is, by integrating information of different modalities, a more comprehensive understanding and processing capability can be achieved. The multimodal large model for PDF metadata extraction can simultaneously process the image of the PDF document and the metadata extraction prompt, and output metadata. Specifically, the first PDF image information and the metadata extraction prompt are synchronized to the constructed initial multimodal large model, and after processing, the first initial metadata corresponding to the first PDF paper is output, including the title, author, abstract, keywords, etc. of the first PDF paper.

[0029] In a possible implementation, step S200 further includes step S210, using a convolutional neural network to construct a visual encoder architecture, using the visual encoder architecture to train in combination with the first PDF image information, and determining a visual encoder. A convolutional neural network (CNN) is used to construct a visual encoder capable of processing image data, usually only including a convolution layer and a pooling layer (or a convolution layer with a step size greater than 1) in a feature extraction network, and using the first PDF image as training data to train it. During the training process, it will learn how to identify and extract visual features useful for metadata extraction, and is used to extract high-level features of PDF image information (which may have been converted into an image format suitable for processing), such as font size, typesetting layout, etc. Step S220, preprocessing is performed based on the metadata extraction prompt to generate a word embedding sequence. Preprocessing the text data according to the metadata extraction prompt may include text cleaning, word segmentation, and removal of stop words, and then using word embedding technology (such as Word2Vec, GloVe, or BERT's embedding layer) to convert the text into a vector representation, that is, a word embedding sequence, which can capture the semantic relationship between words.

[0030] Step S230, a language encoder architecture is constructed using a recurrent neural network, and the language encoder architecture is used in combination with the word embedding sequence for training to determine a language encoder. A recurrent neural network is used to construct an encoder that can receive a word embedding sequence as input and output the encoded language features. The word embedding technology can capture the semantic relationship between words, so that similar words are close in the vector space. The text data (such as text extracted from PDF) is preprocessed (such as word segmentation, stop word removal, etc.), and each word is converted into a corresponding vector representation using the word embedding technology, arranged in the order in the text to form a word embedding sequence, and then the word embedding sequence is used to train the language encoder architecture. The language encoder architecture receives a word embedding vector at each time step and updates its internal state to capture the information in the sequence. After sufficient training iterations, the language encoder is determined and will learn how to convert the word embedding sequence into a useful language feature representation.

[0031] Step S240, the visual encoder and the language encoder are collaboratively connected to construct a model component. The visual encoder that processes visual information and the language encoder that processes language information are combined through a specific mechanism to achieve interaction and fusion of visual and language information, such as a dual-stream attention mechanism. The VisualBERT model adopts a dual-stream attention mechanism, and the image and text are input into the visual and language encoders respectively, and interact through a multi-layer attention mechanism, so that the image and text are preliminarily processed in their respective encoders, and then interact through a cross-modal attention layer; specifically, the input image and text are embedded and represented respectively. For images, a pre-trained convolutional neural network is usually used to extract features and convert them into embedded vectors; for text, word embedding and position encoding are used to represent text sequences, and then the embedded representations are input into the visual encoder and language encoder respectively, and preliminary feature extraction and encoding are performed, and the interaction and information transfer between images and texts are realized through a cross-modal attention mechanism.

[0032] Step S250, calling LLaVA architecture processing, and building the initial multimodal large model in combination with the model components. LLaVA is an open source multimodal large model, which is designed to connect the visual encoder and the large language model (LLM) to achieve general visual and language understanding, can process the visual information (images) of PDF documents, and can effectively generate metadata text accurately. Specifically, the visual encoder and the language encoder are integrated together through a specific connection mechanism to form a unified multimodal model, and the model is trained end-to-end using a multimodal instruction following dataset (such as a dataset containing images and corresponding text descriptions). During the training process, the model will learn how to align and fuse visual features and language features to better complete multimodal tasks, and finally obtain an initial multimodal large model.

[0033] In a possible implementation, step S200 further includes step S260, inputting the first PDF image information into the visual encoder for feature extraction and generating a visual feature vector. The first PDF image information is input into the visual encoder, and the visual encoder extracts features from the image through a multi-layer convolutional neural network, including using a convolutional layer to extract local features of the image layer by layer, such as edges, corners, textures, etc., using a pooling layer to downsample the feature map, reduce the dimension, retain the main features, and use a fully connected layer to flatten the pooled feature map and map it to a high-dimensional feature space to generate a visual feature vector, which represents a point in a high-dimensional space and can represent the key information in the image. Step S270, inputting the metadata extraction prompt into the language encoder for feature extraction and generating a text feature vector. The metadata extraction prompt is input into the language encoder, and the language encoder performs word segmentation, embedding, and other processing on the text, and applies structures such as Transformer to capture the semantics and contextual information in the text. Finally, this information is converted into a text feature vector, which is also a point in a high-dimensional space and can represent the key information in the text.

[0034] It also includes step S280, jointly encoding the visual feature vector and the text feature vector to generate a fused feature. Jointly encoding the visual feature vector and the text feature vector usually includes splicing the two vectors together, or fusing them through some form of cross-modal attention mechanism. The purpose of the joint encoding is to generate a fused feature that contains key information in both the image and the text, and this information is interrelated in the vector space. Step S290, decoding is performed based on the fused feature to generate the first initial metadata. Based on the fused feature, the decoding process is performed, and the decoder receives the fused feature as input and applies a series of transformations (such as linear layers, softmax, etc.) to map and generate the final text content output, generating off-site initial metadata, that is, structured initial metadata is generated, such as author name, title, abstract, etc.

[0035] In one possible implementation, Figure 3As shown, step S200 further includes step S201, aligning the visual feature vector with the text feature vector to generate an image-text pair; step S202, introducing a contrast loss function to calculate the image-text pair to determine vector distance data; step S203, judging whether the vector distance data is in a vector distance threshold interval, and if so, generating a matching signal, and comparing the visual feature vector with the text feature vector according to the matching signal; step S204, when the visual feature vector is consistent with the text feature vector, enhancing the matching signal, and when the visual feature vector is inconsistent with the text feature vector, reducing the matching signal.

[0036] Preferably, the visual feature vector is usually a high-dimensional vector extracted from the image by image processing technology (such as convolutional neural network CNN), while the text feature vector is a vector extracted from the text by natural language processing (NLP) technology (such as word embedding, BERT, etc.). The image and text of each PDF document are aligned to form an image-text pair of data, ensuring that each pair of data comes from the first page of the same PDF document (but not necessarily accurate, that is, it may not be certain that they are from the same document, so subsequent comparative training is required. If the vectors are consistent, they are considered to be the same document); the contrast loss function is a loss function commonly used to learn the similarity or difference between samples, which is used to calculate the distance of each pair of image-text vectors (such as Euclidean distance, cosine similarity, etc.) to quantify the similarity between them. The smaller the vector distance data, the closer the representation of the image and text in the feature space is. That is, the more similar they are, the image and text feature vectors in the same PDF document are as similar as possible (the distance is smaller), while the image and text feature vectors in different PDF documents are as different as possible (the distance is larger); a vector distance threshold interval is set to determine when the image and text can be considered to be semantically matched. If the calculated vector distance data falls within the set vector distance threshold interval, the visual feature vector and the text feature vector are considered to be consistent, that is, the image and text are similar. At this time, the matching signal is positive, and the matching signal is enhanced. This means that when the consistency of the visual feature vector and the text feature vector is stronger, the matching signal is positively enhanced. When the visual feature vector and the text feature vector do not fall within the set vector distance threshold interval, the matching signal is negative at this time, and the matching signal is reduced. This means that when the consistency of the visual feature vector and the text feature vector is weaker, the matching signal is negatively weakened.

[0037] In a possible implementation, step S280 further includes step S285, dividing the visual feature vector into multiple visual feature sub-regions, wherein the multiple visual feature sub-regions include visual tokens; step S286, performing attention calculation based on the visual tokens to generate multiple visual attention weights; step S287, dividing the text feature vector into multiple text feature sub-regions, wherein the multiple text feature sub-regions include text tokens; step S288, performing attention calculation based on the text tokens to generate multiple text attention weights; step S289, based on the matching signal, constructing multiple association relationships between the visual feature vector and the text feature vector according to the multiple visual attention weights and the multiple text attention weights; step S2810, jointly encoding the visual feature vector and the text feature vector based on the multiple association relationships to generate the fusion feature.

[0038] Preferably, the original visual feature vector (usually a high-dimensional, continuous vector) is divided into smaller, easier-to-process sub-regions, each of which contains a set of visual tokens. Specifically, in models such as visual Transformer, visual tokens usually refer to a series of image blocks into which an image is divided. Each image block is regarded as a token, which is mapped to the input dimension of the Transformer through linear projection to obtain a series of image block vectors. Specifically, if the resolution of an image is H×W and the size of the divided image blocks is P×P, then (H / P)×(W / P) image blocks, i.e., (HW) / (P2) tokens, can be obtained. These tokens serve as the basic units for the model to process images and carry local information and features of the image. Attention calculation is performed on each visual token to determine their importance in the overall visual representation, i.e., the correlation between each token and other tokens or the entire visual representation is calculated, and a set of visual attention weights are generated accordingly, which reflects the importance of the model to different visual tokens during the fusion process.

[0039] Preferably, similar to the processing of visual feature vectors, text feature vectors are also divided into multiple sub-regions, each of which contains a set of text tokens, which represent different words or phrases in the text and are the basic units of text information. They can be a word, a phrase, a punctuation mark, a subword or a character. In NLP tasks, tokens, as an indispensable basic unit of text, carry semantic and structural information and are the cornerstone of the model for language understanding and generation. During the training process, the model dynamically adjusts parameters to continuously optimize the representation of tokens, thereby achieving understanding and generation of text; attention calculation is performed on each text token to generate a set of text attention weights, which reflects the importance of different parts of the text in the overall text representation.

[0040] Preferably, based on the obtained matching signal, according to the calculated visual attention weights and intersection attention weights, multiple association relationships between the visual feature vector and the intersection feature vector are constructed. These association relationships can be direct (such as based on spatial position correspondence), indirect (such as based on semantic content similarity) or a combination of both, which helps the model to better understand the interaction and relationship between visual and intersection information in the joint encoding process; based on the constructed multiple association relationships, the visual feature vector and the intersection feature vector are jointly encoded, that is, the visual and intersection information are fused in the feature space to generate a fused feature containing the comprehensive information of the two, such as the concatenation of feature vectors, weighted summation, and application of attention mechanism. The final generated fused feature is a high-dimensional vector that contains the key features of both visual and intersection information.

[0041] Step S300, based on multiple information sources, N PDF papers are collected for visual analysis to generate N PDF image information, and multiple data texts are defined according to the text extraction requirements to generate a metadata extraction prompt set, where N is an integer greater than 1. N PDF papers are collected from multiple information sources (such as academic databases, library resources, online archives, etc.), including a large number of PDF papers from different publishers, and then the N PDF papers are visually analyzed to generate N PDF image information corresponding to the N PDF papers (i.e., the front page images of the N papers), and then according to the preset text extraction requirements, multiple data texts of each PDF paper image information are identified, including the titles, authors, abstracts, keywords, etc. of the N PDF papers, to obtain multiple metadata extraction prompts, and these metadata extraction prompts constitute a metadata extraction prompt set.

[0042] Step S400, synchronizing the N PDF image information and the metadata extraction prompt set to the initial multimodal large model, outputting the N initial metadata of the N PDF papers, and the N PDF papers and the N initial metadata have a corresponding relationship. The N PDF image information and the metadata extraction prompt set are used as input data and synchronized to the initial multimodal large model, and the initial multimodal large model uses the internal visual encoder and language encoder to analyze and process the PDF image information and the metadata extraction prompt set, respectively, and extracts the N initial metadata corresponding to the N PDF papers, wherein the N metadata actually refers to the N initial metadata sets, because the metadata of each PDF paper itself contains multiple text data such as the title, author, abstract, keywords, etc. of the paper, and is not a single data, and there is a one-to-one correspondence between the N PDF papers and the generated N initial metadata.

[0043] Step S500, based on the N PDF papers and the N initial metadata, the initial multimodal large model is adjusted to generate a multimodal large model. Data enhancement operation is performed on the N PDF papers and the corresponding metadata, and the N PDF papers obtained by enhancement are used as a test data set, which are input into the initial multimodal large model. The model uses the training data set to train the metadata extraction mode to obtain multiple training results, and then data enhancement is performed on the N initial metadata to obtain a verification data set. The verification data set is used to verify the multiple training results, and then the initial multimodal large model is dynamically adjusted with the verified training results, for example, adding or modifying specific processing modules, adjusting the extraction strategy of visual data and language data, correcting errors in extracting metadata, etc., and finally generating a multimodal large model so that it can adapt to PDF paper documents of different formats and styles, can better understand and process the document layout of complex PDF papers, and achieve high-precision and efficient metadata extraction.

[0044] In a possible implementation, step S500 further includes step S510, performing data enhancement on the N PDF papers and the N initial metadata to construct a training data set and a verification data set. Data enhancement is performed on the N PDF papers and the N initial metadata. Specifically, data enhancement is a technology that improves the generalization ability of the model by increasing the diversity and quantity of data samples. For PDF papers, this may include extracting text content, performing text cleaning, converting formats, adding noise, replacing synonyms, etc.; for metadata, it may include modifying the order of fields, adding or deleting non-key fields, generating new metadata descriptions, etc., and dividing the enhanced data into a training data set and a verification data set. Step S520, using the training data set to train the metadata extraction model and generate multiple training results. The metadata extraction model is trained using the training data set to identify key information in PDF papers and extract it as metadata. The training process may include multiple iterations, and multiple training results are generated during the training process.

[0045] Step S530, cross-validate the multiple training results using the validation data set. Cross-validate the multiple training results using the validation data set, by repeatedly segmenting the data into training sets and test sets, each time using different segmentations to train the model and evaluate its performance, which helps to reduce the risk of overfitting and provide a more reliable estimate of model performance. Step S540, extract the verified training results to dynamically adjust the initial multimodal large model to generate an adjustment result. Select the training result with the best performance from the cross-validation as a reference, and dynamically adjust the initial multimodal large model based on the verified training results, for example, adjust the model's architecture, parameters, optimization algorithms, etc., to improve the model's performance on specific tasks. Step S550, introduce evaluation indicators to evaluate the adjustment results, and generate the multimodal large model based on the adjusted evaluation information. Multiple evaluation indicators are introduced to evaluate the adjustment results and find the model or configuration that performs best on the validation dataset. Specifically, the candidate model is used to predict the validation dataset and generate prediction results. The values ​​of various evaluation indicators are calculated based on the prediction results and the true labels of the validation dataset. The evaluation indicator values ​​of different candidate models are compared to determine which model performs best on the validation dataset. The final multimodal large model is generated based on the configuration and parameters of the model, which directly processes the images of PDF documents and combines diverse training data to improve the accuracy and robustness of metadata extraction.

[0046] In a possible implementation, step S550 further includes step S551, setting a metadata extraction target based on the N PDF papers and the N initial metadata. The metadata extraction target is set based on the N PDF papers and the N initial metadata, and the goal of metadata extraction is to automatically extract metadata that matches the initial metadata or is richer from these PDF papers, such as the title, author, abstract, keywords, etc. of the paper. Step S552, performing extraction training based on the metadata extraction target, performing testing based on the training results, and calculating the test pass rate. The PDF papers are trained for metadata extraction according to the metadata extraction goal. Specifically, PDF papers containing the required metadata are collected as training data sets, and corresponding labels are given to each metadata field. Then, an information extraction model is constructed and trained using the training data. During the training process, the perplexity of the model is calculated regularly, and the training strategy is adjusted according to its changing trend. Specifically, as the training progresses, if the perplexity of the model gradually decreases, it can be considered that the model is learning how to effectively extract metadata from PDF papers. The perplexity is an indicator to measure the performance of the language model, which reflects the model's predictive ability for the test data. Lower perplexity means that the model can better predict the next word or character. A set of PDF papers that are different from the training data but have similar distribution and characteristics are prepared as test data sets. The trained model is applied to the test data set, and the model tries to extract metadata from the test papers. The metadata extracted by the model is compared with the true labels of the test data, and the test pass rate is calculated, that is, the ratio of the number of metadata fields correctly extracted by the model to the total number of metadata fields in the test data is calculated, which intuitively reflects the overall performance of the model in the metadata extraction task.

[0047] Step S553, set the extraction quality score according to the test pass rate, and formulate the evaluation index according to the quality score value. Set the extraction quality score according to the specific numerical range of the test pass rate, for example, set a pass rate threshold, when the test pass rate exceeds the threshold, give a higher quality score; when the test pass rate is lower than the threshold, give a lower quality score, and formulate detailed evaluation criteria based on the quality score result (quality score value), that is, evaluation indicators (such as accuracy, the ratio of the number of correctly extracted metadata to the total number of extractions), which are used to determine whether the extraction quality of the metadata extraction model meets specific quality requirements. Step S554, determine whether there is adjustment data less than the evaluation index in the adjustment result according to the evaluation index. Use the evaluation index to evaluate the adjustment result of the model, and determine whether there is adjustment data less than the evaluation index in the adjustment result.

[0048] Step S555, if the adjustment data that is less than the evaluation index exists in the adjustment result, the adjustment data that is less than the evaluation index is marked and recorded, and added to the data group to be verified. If the adjustment data that is less than the evaluation index exists in the adjustment result, the adjustment data that does not meet the evaluation index is marked and recorded, and added to the data group to be verified. This data group will be used for subsequent verification and possible further adjustment. Step S556, based on the data group to be verified, the adjustment results are screened to generate the adjustment evaluation information. Using the data group to be verified to further verify and adjust the performance of the model, it can be found that some adjustment data is effective in specific situations. Based on these effective situations, the adjustment results are screened and the adjustment evaluation information is finally generated.

[0049] Step S600: Verify the first initial metadata using the multimodal macro model in combination with the first PDF paper, and output the first metadata of the first PDF paper. The first PDF paper is input as training data into the trained multimodal macro model to extract metadata, and then the first initial metadata is verified based on the metadata extraction result, that is, the metadata extracted from the PDF paper by the comparison model is consistent with the initial metadata, and the accuracy and completeness of the initial metadata are checked. After the verification is passed, it is used as the first metadata corresponding to the first PDF paper and output.

[0050] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A structured analysis method for PDF paper metadata based on a multimodal large model, characterized by: The method comprises: Collecting the first PDF paper for visual analysis, generating first PDF image information, marking the first PDF image information according to text extraction requirements, and generating metadata extraction prompts; Constructing an initial multimodal large model, synchronizing the first PDF image information and the metadata extraction prompt to the initial multimodal large model, and outputting first initial metadata of the first PDF paper; Based on multiple information sources, N PDF papers are collected for visual analysis to generate N PDF image information, and multiple data texts are defined according to the text extraction requirements to generate metadata extraction prompt sets, where N is an integer greater than 1; Synchronizing the N PDF image information and the metadata extraction prompt set to the initial multimodal large model, and outputting N initial metadata of the N PDF papers, wherein the N PDF papers correspond to the N initial metadata; Adjusting the initial multimodal large model based on the N PDF papers and the N initial metadata to generate a multimodal large model; The first initial metadata is verified by using the multimodal large model in combination with the first PDF paper, and the first metadata of the first PDF paper is output.

2. The PDF paper metadata structured parsing method based on a multimodal large model as claimed in claim 1 is characterized in that: The first PDF paper, the method includes: Traversing the multiple information sources and performing random selection to generate multiple PDF documents; Performing virtual generation according to the plurality of PDF documents to generate a plurality of virtual metadata and a plurality of home page texts, wherein the plurality of virtual metadata and the plurality of home page texts correspond to the plurality of PDF documents; The multiple virtual metadata and the multiple homepage texts are aligned according to the multiple PDF documents, and the alignment results are traversed for random extraction to generate the first PDF paper.

3. The PDF paper metadata structured parsing method based on a multimodal large model as claimed in claim 1 is characterized in that: Collecting a first PDF paper for visual analysis to generate first PDF image information, marking the first PDF image information according to text extraction requirements, and generating metadata extraction prompts, the method includes: Convert the first page of the first PDF paper to generate a document first page image; Performing image standardization processing on the document homepage image to generate first PDF image information; Using computer vision technology to extract features from the first PDF image information, determine multiple image features, and obtain text extraction requirements; Traversing the first PDF image information to perform annotation according to the text extraction requirement, and generating labels to be extracted; The metadata extraction hint is determined according to the tag to be extracted.

4. The PDF paper metadata structured parsing method based on a multimodal large model as claimed in claim 3 is characterized in that: The initial multimodal large model is constructed, the method comprising: Using a convolutional neural network to construct a visual encoder architecture, using the visual encoder architecture combined with the first PDF image information for training, and determining a visual encoder; Perform preprocessing based on the metadata extraction prompts to generate word embedding sequences; A language encoder architecture is constructed using a recurrent neural network, and the language encoder architecture is used in combination with the word embedding sequence for training to determine a language encoder; The visual encoder and the language encoder are collaboratively connected to construct a model component; Call the LLaVA architecture for processing, and combine the model components to build the initial multimodal large model, wherein LLaVA is an open source multimodal large model that aims to connect visual encoders and large language models to achieve general vision and language understanding, and can process the visual information of PDF documents and generate metadata text.

5. The PDF paper metadata structured parsing method based on a multimodal large model as claimed in claim 4 is characterized in that: Based on synchronizing the first PDF image information and the metadata extraction prompt to the initial multimodal macro model, outputting first initial metadata of the first PDF paper, the method includes: Inputting the first PDF image information into the visual encoder for feature extraction to generate a visual feature vector; Inputting the metadata extraction prompt into the language encoder for feature extraction to generate a text feature vector; Jointly encoding the visual feature vector and the text feature vector to generate a fusion feature; Decoding is performed based on the fusion features to generate the first initial metadata.

6. The PDF paper metadata structured parsing method based on a multimodal large model as claimed in claim 5 is characterized in that: Before jointly encoding the visual feature vector and the text feature vector, the method includes: Performing data alignment on the visual feature vector and the text feature vector to generate an image-text pair; Introducing a contrast loss function to calculate the image-text pair to determine vector distance data; Determine whether the vector distance data is within a vector distance threshold interval, and if so, generate a matching signal, and compare the visual feature vector with the text feature vector according to the matching signal; When the visual feature vector is consistent with the text feature vector, the matching signal is enhanced; when the visual feature vector is inconsistent with the text feature vector, the matching signal is reduced.

7. The PDF paper metadata structured parsing method based on a multimodal large model as claimed in claim 6 is characterized in that: The visual feature vector and the text feature vector are jointly encoded to generate a fusion feature, the method comprising: Dividing a plurality of visual feature sub-regions based on the visual feature vector, wherein the plurality of visual feature sub-regions include visual tokens; Performing attention calculation based on the visual token to generate multiple visual attention weights; Dividing a plurality of text feature sub-regions based on the text feature vector, wherein the plurality of text feature sub-regions contain text tokens; Performing attention calculation based on the visual token to generate multiple text attention weights; Based on the matching signal, constructing multiple association relationships between the visual feature vector and the text feature vector according to the multiple visual attention weights and the multiple text attention weights; The visual feature vector and the text feature vector are jointly encoded based on the multiple association relationships to generate the fusion feature.

8. The PDF paper metadata structured parsing method based on a multimodal large model as claimed in claim 1, characterized in that: The initial multimodal large model is adjusted based on the N PDF papers and the N initial metadata to generate a multimodal large model, the method comprising: Performing data enhancement on the N PDF papers and the N initial metadata to construct a training data set and a verification data set; Using the training data set to train the metadata extraction mode, generating multiple training results; Using the validation data set to cross-validate the multiple training results; Extract the verified training results to dynamically adjust the initial multimodal large model and generate adjustment results; An evaluation index is introduced to evaluate the adjustment result, and the multimodal large model is generated according to the adjustment evaluation information.

9. The PDF paper metadata structured parsing method based on a multimodal large model as claimed in claim 8, characterized in that: An evaluation index is introduced to evaluate the adjustment result, and the multimodal large model is generated according to the adjustment evaluation information. The method includes: Setting a metadata extraction target based on the N PDF papers and the N initial metadata; Performing extraction training based on the metadata extraction target, performing testing according to the training results, and calculating the test pass rate; Set the extraction quality score according to the test pass rate, and formulate the evaluation index according to the quality score value; Determining whether there is adjustment data in the adjustment result that is less than the evaluation index according to the evaluation index; If the adjustment result contains the adjustment data that is smaller than the evaluation index, the adjustment data that is smaller than the evaluation index is marked and recorded, and added to the data group to be verified; The adjustment results are screened based on the data group to be verified to generate the adjustment evaluation information.

Images