Knowledge extraction method of scientific paper tables based on multimodal large model

Abstract

The present invention discloses a method for extracting knowledge from tables in scientific papers based on a multimodal large model, and relates to the technical field of multimodal information processing. The method comprises: constructing a knowledge extraction task data set for a task to be extracted; building a general table large model; using the knowledge extraction task data set to fine-tune the general table large model to obtain a paper table large model; extracting knowledge based on the paper table large model to obtain a standard knowledge multi-tuple. The present invention solves the technical problems in the prior art that the traditional scientific paper knowledge extraction method cannot process content in the form of tables, the extracted knowledge may have information missing, and cannot fully reflect the depth and breadth of the research content, and achieves the technical effect of integrating image recognition and data analysis technology, constructing a paper table large model to extract knowledge from scientific papers, and improving the depth and quality of knowledge extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal information processing, and in particular to a method for extracting knowledge from scientific paper tables based on a multimodal large model. Background Art

[0002] As an important carrier of academic research, scientific papers contain not only rich text information, but also a large amount of tabular data. With its intuitive and easy-to-understand characteristics, tables can effectively convey complex data relationships and experimental results, and have become an indispensable part of papers. However, traditional knowledge extraction technology mainly focuses on the analysis of text content, and often fails to fully explore and utilize the rich information resources contained in tables, which to a certain extent limits the integrity and accuracy of knowledge extraction.

[0003] Traditional knowledge extraction methods mainly rely on natural language processing technology, which processes text data through algorithms to extract key information and knowledge. These methods perform well when processing pure text data, but are powerless when faced with content in the form of tables. Since the data in the tables cannot be effectively parsed, the knowledge extracted from scientific and technological papers may be missing information and cannot fully reflect the depth and breadth of the research content. Summary of the invention

[0004] The present application provides a method for extracting knowledge from tables in scientific papers based on a multimodal large model, which is used to solve the technical problems in the prior art that traditional knowledge extraction methods in scientific papers cannot process content in the form of tables, the extracted knowledge may have missing information, and cannot fully reflect the depth and breadth of the research content.

[0005] In view of the above problems, this application provides a method for extracting knowledge from scientific paper tables based on a multimodal large model.

[0006] In the first aspect, the present application provides a method for extracting knowledge from tables in scientific papers based on a multimodal big model, the method comprising: extracting task data for the task to be extracted and constructing a knowledge extraction task data set; building a multimodal big model architecture, the multimodal big model architecture comprising a visual encoder, a text encoder, a feature transformation module and a decoder; initializing the multimodal big model architecture based on a general table data set to obtain a general table big model; partitioning the knowledge extraction task data set to obtain a fine-tuning data set and a knowledge extraction data set, wherein the fine-tuning data set contains multiple fine-tuning data; using the fine-tuning data set to fine-tune the general table big model to obtain a paper table big model; taking the knowledge extraction data set as input, performing knowledge extraction based on the paper table big model to obtain a standard knowledge tuple.

[0007] In a second aspect, a computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the first aspect when executing the computer program.

[0008] According to a third aspect, a computer-readable storage medium stores a computer program, which implements the steps of the method according to the first aspect when executed by a processor.

[0009] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0010] The embodiment of the present application provides a method for extracting knowledge from scientific paper tables based on a multimodal large model, which relates to the technical field of multimodal information processing. By constructing a large model of paper tables, the model can simultaneously process table image information and text information. Through visual encoders, feature transformations, text encoders and decoders, these two types of information are effectively integrated to supplement the understanding of the way information in the table is associated, thereby generating a multi-tuple output that conforms to a predefined knowledge model. This solves the technical problem that traditional scientific paper knowledge extraction methods in the prior art cannot process content in the form of tables, the extracted knowledge may have information missing, and cannot fully reflect the depth and breadth of the research content. This achieves the technical effect of integrating image recognition and data analysis technology to construct a large model of paper tables to extract scientific paper knowledge and improve the depth and quality of knowledge extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flow chart of a method for extracting knowledge from tables in scientific papers based on a multimodal large model is provided for this application;

[0012] Figure 2 A flow chart of obtaining standard knowledge multi-groups in a method for extracting knowledge from tables in scientific papers based on a multimodal large model is provided for this application;

[0013] Figure 3 A flow chart of a sample of input and output of a piece of data in a method for extracting knowledge from a table in a scientific paper based on a multimodal large model is provided for this application;

[0014] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0015] The present application provides a scientific paper table knowledge extraction system based on a multimodal large model to solve the technical problems in the prior art that traditional scientific paper knowledge extraction methods are unable to process content in the form of tables, the extracted knowledge may have missing information, and cannot fully reflect the depth and breadth of the research content.

[0016] Embodiment 1

[0017] like Figure 1 As shown, the present application provides a method for extracting knowledge from scientific paper tables based on a multimodal large model, the method comprising:

[0018] P10: For the tasks to be extracted, extract task data and build a knowledge extraction task dataset.

[0019] Furthermore, step P10 of the embodiment of the present application further includes:

[0020] P11: receiving tasks to be extracted; P12: based on the tasks to be extracted, performing multi-tuple task extraction, constructing a knowledge extraction task dataset, the knowledge extraction task dataset comprising an input part and an output part; wherein the input part comprises a table image, a table title, a context text, a task description and a knowledge model, the knowledge model is constructed according to the tasks to be extracted, and the output part comprises a plurality of multi-tuple lists. The knowledge model comprises a multi-tuple text structure, a multi-tuple entity type, and a multi-tuple entity order.

[0021] It should be understood that for a specific task to be extracted, refined data extraction and processing operations are performed to construct a data set suitable for the knowledge extraction task, laying the foundation for subsequent model training and knowledge extraction. Data such as table images, table titles, and context text are extracted from the scientific papers to be extracted, combined with task instructions and knowledge patterns to form a complete knowledge extraction task data set.

[0022] Specifically, firstly, a description of a task to be extracted is received from a user. The task to be extracted is a specific requirement for knowledge extraction from scientific papers to be extracted, which specifies the type and scope of knowledge to be extracted and the expected output format.

[0023] Furthermore, based on the task to be extracted, multi-tuple task extraction is performed. The core of this process is to convert unstructured text information into structured knowledge representation, i.e., multi-tuples. To achieve this goal, the key information points in the context text are first analyzed, including data, concepts, relationships, etc. that are directly or indirectly related to the table content. Subsequently, according to the requirements of the task description, specific knowledge extraction goals are defined, including the entity types to be extracted, the relationships between entities, and the order between them. On this basis, the knowledge extraction task dataset is constructed. The dataset consists of an input part and an output part. The input part covers all the key information required to perform the knowledge extraction task, including table images (providing visual intuitive information), table titles (summarizing the table subject), context text (providing detailed context information), task descriptions (clarifying extraction goals), and knowledge patterns (defining the structure, entity type, and order of multi-tuples). The output part is a series of multi-tuple lists generated by the knowledge extraction process based on the input information and knowledge patterns. These multi-tuple lists present the key knowledge information extracted from the text and table in a structured form, which is convenient for subsequent analysis and application.

[0024] It is worth noting that when constructing a knowledge model, it is necessary to fully consider factors such as the text structure, entity type, and entity order of the tuple. Among them, the text structure of the tuple defines the basic framework of knowledge representation and determines how to organize and present the extracted knowledge information. The entity type is an important basis for classifying and labeling the concepts or objects involved in the knowledge information, which helps to improve the accuracy and efficiency of knowledge extraction. The entity order reflects key information such as the logical relationship or time sequence between knowledge information, which is crucial for understanding and applying the extracted knowledge.

[0025] In summary, through the coherent execution of the above steps, a dataset suitable for specific knowledge extraction tasks is constructed. This dataset not only provides rich sample data for subsequent model training, but also provides strong support for the smooth execution of knowledge extraction tasks.

[0026] P20: Build a multimodal large model architecture, which includes a visual encoder, a text encoder, a feature transformation module and a decoder.

[0027] Specifically, a multimodal large model architecture is built, which integrates the information processing capabilities of both visual and textual modalities, aiming to extract rich structured knowledge from the table images and related texts of scientific papers. The multimodal large model architecture is composed of a visual encoder, a text encoder, a feature transformation module, and a decoder.

[0028] First, as the core component for processing image information, the visual encoder uses advanced deep learning technologies, such as convolutional neural networks (CNN) or visual transformers (ViT). These technologies can automatically learn feature representations in images and transform each pixel in the table image into a high-dimensional feature vector rich in semantic information. Through the stacking of multiple layers of networks and nonlinear transformations, the visual encoder can capture the structural features, layout information, and potential semantic relationships in the table image, laying the foundation for subsequent multimodal fusion.

[0029] At the same time, the text encoder is responsible for processing text information related to the table. It uses classic models in the field of natural language processing (NLP), such as BERT and GPT, to learn the language rules and contextual relationships in the text through pre-training and fine-tuning. The text encoder converts the contextual text into text embeddings, which are a series of low-dimensional, dense vector representations that can accurately capture the semantic information in the text and the association between words.

[0030] Furthermore, in order to effectively integrate the information of the two different modalities of vision and text, a feature transformation module is introduced. The main function of this module is to adjust and transform the dimension of the feature vector output by the visual encoder so that it is consistent with the output of the text encoder in terms of dimension and semantics. Through this transformation process, visual and text information can interact and fuse in the same space, providing comprehensive input information for subsequent decoding operations.

[0031] Finally, as the output end of the entire multimodal large model architecture, the decoder is responsible for converting the fused multimodal information into a structured knowledge representation, i.e., a tuple. The decoder usually uses mechanisms such as Seq2Seq or Pointer Network to gradually generate a list of tuples that meet the requirements based on the input multimodal feature vector and predefined knowledge patterns. During the decoding process, the decoder will comprehensively consider constraints such as entity type, entity order, and text structure to ensure that the generated tuples are both accurate and logical.

[0032] In summary, the multimodal large model architecture achieves deep understanding and efficient fusion of tabular images and text information through the close collaboration of the visual encoder, text encoder, feature transformation module and decoder, providing strong technical support for subsequent knowledge extraction tasks.

[0033] P30: Based on the general table data set, the multimodal large model architecture is initialized to obtain a general table large model.

[0034] Furthermore, step P30 of the embodiment of the present application further includes:

[0035] P31: Based on the pre-trained model of the general table data set, the parameters of the visual encoder, the text encoder and the decoder are initialized, and the parameters of the feature transformation module are randomly initialized to obtain the general table large model.

[0036] Optionally, the multimodal large model architecture is initialized using a general table data set to ensure that the model has a good starting point and foundation in subsequent tasks. The general table data set contains multivariate table data, including public table data and synthetic table data. The public table data can be obtained based on big data, and the synthetic table data can be adjusted and generated according to the public table data, as well as generated from scratch based on common table styles. These general table data sets usually contain rich image and text information, covering a wide range of topics and fields, and can provide sufficient training samples for the model. Use the general table data set for machine learning training to obtain multiple pre-trained models, and use the pre-trained pre-trained models to initialize the parameters of the visual encoder, text encoder and decoder. Through transfer learning, the knowledge in these pre-trained models is transferred to the multimodal large model architecture to obtain the general table large model. The general table large model has certain visual understanding and text processing capabilities and can handle general table knowledge extraction tasks, but the task adaptability and the accuracy of extracted data are low.

[0037] Exemplarily, for the visual encoder, the model parameters pre-trained on large-scale image recognition datasets such as PubTabNet are used for initialization. After a long period of training and optimization, these parameters have been able to capture the key features and detail information in the image. By loading these parameters into the visual encoder, the model can more keenly capture the structure and content information in the image when processing table images.

[0038] Similarly, the text encoder is initialized using model parameters pre-trained on a large amount of text data, such as the weights of BERT, GPT and other models. These pre-trained models have strong text understanding and generation capabilities by learning the language rules and contextual relationships in massive text data. Applying these parameters to the text encoder enables the model to more accurately capture the semantic information in the text and the association between words when processing text information related to tables.

[0039] However, for the feature transformation module, since it is a key component connecting the visual encoder and the text encoder, and its parameters need to be adjusted according to the specific task and data set, a random initialization method is adopted. Random initialization means giving a random initial value to the parameters of the feature transformation module, and then gradually adjusting the values ​​of these parameters through optimization algorithms such as gradient descent during the subsequent training process. Although this method requires more training time and data to make the model converge, it can ensure that the model will not be constrained by the parameters of the pre-trained model during the training process, so it is more likely to learn feature representations that are more suitable for the current task.

[0040] In summary, the parameters of the visual encoder, text encoder and decoder are initialized based on the model pre-trained on the general table dataset, and the parameters of the feature transformation module are set by random initialization to obtain the general table large model. This initialization process not only improves the generalization ability and convergence speed of the model, but also ensures the flexibility and adaptability of the model on specific tasks.

[0041] P40: Divide the knowledge extraction task data set to obtain a fine-tuning data set and a knowledge extraction data set, wherein the fine-tuning data set includes a plurality of fine-tuning data;

[0042] Optionally, the knowledge extraction task data set is divided into data, one part is used as fine-tuning data for incremental training of the general table large model, and the other part is used as knowledge extraction data for extracting table knowledge data. The part used as fine-tuning data needs to be labeled, namely: input part (including table image, table title, context text, task description and knowledge mode) and output part (multiple tuple lists). Exemplarily, if the number of context texts is 100,000, 5,000 of them are extracted as fine-tuning data, and the remaining texts are used as knowledge extraction data to obtain a fine-tuning data set and a knowledge extraction data set. Among them, the fine-tuning data set contains a small amount of data for the current table knowledge extraction task, which can be used to fine-tune the general table large model to improve the accuracy of the model in performing the current extraction task. In addition, the model fine-tuning data set contains multiple fine-tuning data, each of which is composed of the input part and the output part of the knowledge extraction task data. The knowledge extraction data set is used for knowledge data extraction and can be used as real-time input data for the subsequent knowledge extraction process.

[0043] P50: Use the fine-tuning dataset to fine-tune the general table large model to obtain the paper table large model.

[0044] Furthermore, step P50 of the embodiment of the present application also includes:

[0045] P51: The fine-tuning dataset contains multiple fine-tuning data, and the fine-tuning data is composed of the input part and the output part of each knowledge extraction task data; P52: The fine-tuning dataset is randomly divided into a training set and a test set; P53: Based on the training set, after incremental learning of the general table large model, the test set is used to evaluate the model performance to obtain the paper table large model.

[0046] It should be understood that the use of the fine-tuning dataset to perform fine-tuning training on the initialized general table large model is intended to further improve the performance of the model on specific knowledge extraction tasks. This fine-tuning process not only consolidates the knowledge learned by the model on the general table dataset, but also enables the model to better adapt to and complete specific extraction tasks.

[0047] First, the input part (including table images, table titles, context text, task instructions, and knowledge patterns) and the output part (multiple tuple lists) of a small amount of knowledge extraction task data are concatenated to form a complete fine-tuning data set, and a fine-tuning dataset containing multiple fine-tuning data is constructed to provide sufficient samples for subsequent fine-tuning training. Further, the fine-tuning dataset is partitioned. The dataset is divided into a training set and a test set according to a certain ratio (such as 80% training set and 20% test set) to ensure that the model has enough samples to learn during the training process and can evaluate its performance on the test set.

[0048] Subsequently, incremental learning is performed on the general table large model based on the training set. Incremental learning is a training method that allows the model to continuously absorb new knowledge while maintaining the original knowledge. In the present application, the training set is used to perform incremental learning on the general table large model to optimize its parameters and improve its performance on the knowledge extraction task. During the training process, the model generates prediction results based on the input part in the fine-tuning data, and compares them with the actual results in the output part. The parameters of the model are updated by calculating the loss function and applying the back-propagation algorithm, and the model performance is evaluated through the test set. This process will be repeated until the performance of the model on the validation set reaches stability or a preset stopping condition.

[0049] It is worth noting that since the general table model has been initialized, the model can converge to a better performance level more quickly during the fine-tuning training process. At the same time, since the fine-tuning dataset is built for a specific knowledge extraction task, the model can focus more on learning the feature representation and mapping relationship related to the task during the fine-tuning process, thereby further improving its performance on the task. Finally, a large paper table model that can efficiently complete the specific knowledge extraction task is obtained.

[0050] P60: Take the knowledge extraction dataset as input, perform knowledge extraction based on the paper table model, and obtain a standard knowledge tuple.

[0051] Further, such as Figure 2 As shown, step P60 of the embodiment of the present application also includes:

[0052] P61: Based on the visual encoder, image information is converted to generate a visual Embedding, and the visual Embedding is converted through the feature transformation module to obtain a visual Embedding*; P62: Based on the text encoder, text information is converted to generate a text Embedding; P63: According to the decoder, the visual Embedding* and text Embedding are decoded to generate a decoding sequence; P64: Based on the decoding sequence, a standard knowledge tuple is generated, and the standard knowledge tuple includes multiple tuple lists.

[0053] In a possible embodiment of the present application, a large model of paper tables that has been trained is used to perform actual knowledge extraction operations on a knowledge extraction task dataset to extract structured standard knowledge tuples from complex table images and text information.

[0054] Specifically, first start the visual encoder to process the table image. The visual encoder uses deep learning technology, such as convolutional neural network (CNN) or visual transformer (ViT), to extract and encode features of the image, and convert the pixel information in the image into a high-dimensional, semantically rich visual embedding. These visual embeddings capture the key features and structural information in the image, providing important visual cues for subsequent multimodal fusion. Next, the visual embeddings are transformed by the feature transformation module to obtain visual embeddings that are compatible with text embeddings*. The feature transformation module adjusts the dimension and transforms the space so that the information of the two modalities of vision and text can interact and fuse in the same semantic space. This step is a bridge connecting visual and text information, and is crucial to achieving effective fusion of multimodal information.

[0055] At the same time, the text encoder is started in parallel to process the text information related to the table. The text encoder uses advanced technologies in the field of natural language processing (NLP), such as BERT, GPT and other pre-trained models, to semantically understand and encode the text and generate text embeddings. These text embeddings accurately capture the semantic information in the text and the relationship between words, providing rich text representation for subsequent knowledge extraction.

[0056] Furthermore, the decoder is used to jointly decode the visual embedding* and the text embedding. The decoder gradually generates a decoding sequence based on the information of these two modalities and the predefined knowledge pattern. The decoding sequence is an ordered and structured output that contains the key knowledge information extracted from the input data. Through the decoding process, the model can accurately map the key entities, relationships, and the order and structure between them in the multimodal information into the decoding sequence.

[0057] Finally, a standard knowledge tuple is generated based on the decoded sequence. The standard knowledge tuple is a structured knowledge representation that presents key knowledge information extracted from the input data in the form of a tuple list. Each tuple contains a set of related entities and their relationships, which together constitute an accurate and comprehensive description of the input data. By generating standard knowledge tuples, not only is the effective extraction and structured representation of key knowledge in the input data achieved, but it also facilitates subsequent knowledge application and analysis.

[0058] For example, Figure 3 The following is an example of the input and output of a piece of data, where:

[0059] Table image, table title: The table of knowledge to be extracted in this task. Examples include table titles [Table title: Table 1 Classification method performance comparison] and their corresponding table images.

[0060] Knowledge model: A template that contains the element types that need to be filled in. For example, to represent knowledge such as "the result value of a certain method on a certain indicator", the knowledge model is defined as: technical method [S] data set [S] indicator name [S] result value, where [S] is a defined special token.

[0061] Task description: Describe what kind of output results should be given according to the input of this task. The example is [Extract the relational tuple representing "specific indicator results achieved by specific technical methods on specific data sets". Please output the tuple according to the template shown in "Knowledge Mode". If it does not exist, output "None"; if there are multiple groups, separate the groups with ;.] Similarly, other task descriptions and knowledge modes in the form of tuples can also be defined.

[0062] Context text: It is the context text that mentions the table. The period is used as the dividing point to extract the text fragment of the paper. The sample is [From Table 1, we can see the performance difference between A1 and A2 on dataset C. A1 has a clear advantage over A2 in B1 index, but A2's B2 index is higher than A1]. Used to supplement knowledge: A1 and A2 are "technical methods", C is "dataset", B1 and B2 are "indicators", and the table context text, table title, and table image are combined to extract knowledge as a whole.

[0063] Furthermore, step P61 of the embodiment of the present application also includes:

[0064] P61-1: The image information is converted into visual embeddings through the visual encoder to form a feature matrix nv×dv;

[0065] P61-2: The visual embedding is subjected to feature transformation through the feature transformation module to generate a nv×dt visual embedding*.

[0066] Specifically, the specific process of image information conversion based on the visual encoder can be that, first, the visual encoder receives the input table image and uses deep learning technology (such as a variant of the convolutional neural network CNN or the visual Transformer ViT) to segment and extract features from the image. Specifically, the image is divided into multiple small blocks (tokens), and each block is regarded as an independent visual unit for subsequent feature representation. Then, the visual encoder converts these visual tokens into high-dimensional Embedding vectors through nonlinear transformations of a multi-layer network. These vectors (each vector has a dimension of dv) capture the feature representation of the image block, including key information such as color, texture, and shape. Therefore, the entire image is converted into a feature matrix of nv×dv, in which each row represents the Embedding vector of a visual token.

[0067] Furthermore, the feature transformation module receives the feature matrix and performs a linear or nonlinear transformation on it to generate a visual embedding*. Linear changes achieve dimensional adjustment or spatial transformation of data by multiplying a transformation matrix (usually a learnable parameter matrix) and adding a bias term. Nonlinear transformation is more complicated. A common practice is to introduce a nonlinear activation function after the linear transformation to improve the fitting ability. Here, the role of the feature transformation module is to convert the original visual embedding (nv×dv matrix) into a new form (nv×dt matrix) compatible with the text embedding. Among them, nv remains unchanged, while dv is converted to dt, indicating the new embedding vector dimension of each visual token, which is usually matched with the dimension of the text embedding, so as to perform multimodal information fusion in subsequent steps. Through this feature transformation process, we obtain a visual embedding*, which retains the key feature information of the original image and is closer to the text embedding in terms of dimension and semantics, which facilitates subsequent multimodal decoding.

[0068] In summary, the embodiments of the present application have at least the following technical effects:

[0069] This application constructs a knowledge extraction task dataset through the tasks to be extracted, extracts part of the data in the knowledge extraction task dataset, fine-tunes the initialized multimodal large model architecture, and obtains a paper table large model. This model can process the table image information and text information in the data for the remaining knowledge extraction data at the same time, and effectively integrates these two types of information through visual encoders, feature transformations, text encoders and decoders to supplement the understanding of how the information in the table is associated, thereby generating a multi-tuple output that conforms to the predefined knowledge pattern.

[0070] The technical effect of integrating the visual features and text features of the table to effectively reveal the intricate connections between scientific and technological entities in the table is achieved, thereby significantly improving the depth and quality of knowledge extraction.

[0071] Embodiment 2

[0072] like Figure 4 As shown, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the method in embodiment 1 when executing the computer program.

[0073] Embodiment 3

[0074] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the first embodiment are implemented.

[0075] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0076] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0078] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A method for extracting knowledge from tables in scientific papers based on a multimodal large model, characterized in that: The method comprises: For the tasks to be extracted, extract task data and build a knowledge extraction task data set; Building a multimodal large model architecture, the multimodal large model architecture comprising a visual encoder, a text encoder, a feature transformation module, and a decoder; Initializing the multimodal large model architecture based on a general table data set to obtain a general table large model; Performing data division on the knowledge extraction task data set to obtain a fine-tuning data set and a knowledge extraction data set, wherein the fine-tuning data set includes a plurality of fine-tuning data; Using the fine-tuning data set, fine-tuning the general table large model is performed to obtain the paper table large model; Taking the knowledge extraction data set as input, performing knowledge extraction based on the paper table macromodel, and obtaining a standard knowledge multi-tuple; The knowledge extraction dataset is used as input, and knowledge extraction is performed based on the paper table model to obtain a standard knowledge multi-tuple, including: Perform image information conversion based on the visual encoder to generate a visual Embedding, and convert the visual Embedding through the feature transformation module to obtain a visual Embedding*; Perform text information conversion based on the text encoder to generate text Embedding; According to the decoder, jointly decode the visual Embedding* and the text Embedding to generate a decoding sequence; Based on the decoding sequence, generating a standard knowledge tuple, the standard knowledge tuple comprising a plurality of tuple lists; Image information is converted based on the visual encoder to generate visual embeddings, including: The image information is converted into visual embeddings through the visual encoder to form a feature matrix of nv×dv; The visual embedding is subjected to feature transformation through the feature transformation module to generate a nv×dt visual embedding*.

2. The method for extracting knowledge from scientific paper tables based on a multimodal large model as claimed in claim 1, characterized in that: For the tasks to be extracted, extract task data and build a knowledge extraction task dataset, including: Receive tasks to be extracted; Based on the tasks to be extracted, perform multi-tuple task extraction to construct a knowledge extraction task data set, wherein the knowledge extraction task data set includes an input part and an output part; The input part includes a table image, a table title, context text, a task description and a knowledge model, the knowledge model is constructed according to the task to be extracted, and the output part includes a plurality of tuple lists.

3. The method for extracting knowledge from scientific paper tables based on a multimodal large model as claimed in claim 2 is characterized in that: The knowledge model includes a multi-tuple text structure, a multi-tuple entity type, and a multi-tuple entity order.

4. The method for extracting knowledge from scientific paper tables based on a multimodal large model as claimed in claim 1, characterized in that: Initializing the multimodal large model architecture based on a common tabular data set includes: Based on the model pre-trained with the general table data set, the parameters of the visual encoder, the text encoder and the decoder are initialized, and the parameters of the feature transformation module are randomly initialized to obtain the general table large model.

5. The method for extracting knowledge from scientific paper tables based on a multimodal large model as claimed in claim 1, characterized in that: Using the fine-tuning dataset, fine-tuning the general table large model is performed to obtain the paper table large model, including: The fine-tuning data set includes a plurality of fine-tuning data, and the fine-tuning data is composed of the input part and the output part of each knowledge extraction task data; Randomly divide the fine-tuning dataset into a training set and a test set; Based on the training set, after incremental learning of the general table large model, the test set is used to evaluate the model performance to obtain the paper table large model.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images