Scientific and technical literature flow chart entity and relation extraction method based on retrieval enhancement

By employing a retrieval-enhanced approach, combined with graph-text alignment training and vector library retrieval techniques, the problem of low efficiency in extracting entities and relationships from flowcharts in traditional methods is solved. This achieves efficient and accurate information extraction, supporting knowledge graph construction and intelligent information processing.

CN119003788AInactive Publication Date: 2024-11-22DOCUMENT & INFORMATION CENT OF CHINESE ACAD OF SCI

Patent Information

Application Number
CN202411204950.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional methods for extracting entities and relationships from flowcharts in scientific and technological literature rely on manually labeled data, which is time-consuming and labor-intensive, making it difficult to efficiently process large-scale documents. Furthermore, the lack of automated and intelligent extraction methods results in low processing efficiency.

Method used

We employ a retrieval-enhanced approach, which involves collecting and detailing diverse scientific and technological literature flowcharts to construct positive and negative sample pairs. We introduce the LLAVA architecture for supervised training, use a visual encoder and a language model for image-text matching, establish a vector knowledge base, and combine a multimodal large model for entity and relation extraction.

Benefits of technology

It improves the accuracy and efficiency of entity and relation extraction from flowcharts, enabling the efficient and accurate extraction of key information from scientific and technological literature, and providing support for knowledge graph construction and intelligent information processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003788A_ABST
    Figure CN119003788A_ABST
Patent Text Reader

Abstract

The invention provides a scientific and technical literature flow chart entity and relation extraction method based on retrieval enhancement, and relates to the technical field of artificial intelligence. A visual encoder is supervised and trained, a vector knowledge base is constructed for the literature database, the target flow chart is received, vector conversion and similarity retrieval matching are carried out, and an image-text matching result is determined; and supervising and training the multi-modal large model, transmitting an image-text matching result to the multi-modal large model, executing identification interpretation and relation extraction, and performing structure conversion to generate a target knowledge graph. The technical problems that a traditional method depends on manual data labeling, time and labor are wasted, the processing requirements of a large number of scientific and technical literatures are difficult to meet, automatic and intelligent extraction means are lacked, and the efficiency is low when large-scale literatures are processed are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for extracting entities and relationships from a scientific and technological literature flowchart based on retrieval enhancement. Background Art

[0002] In scientific and technological literature, flowcharts are an important means of illustration and are widely used to display research methods, experimental processes, and results analysis. However, traditional scientific and technological literature information extraction methods mainly rely on manually annotated data and single-mode processing technology, which makes it difficult to efficiently and accurately extract key entities and relationships from complex documents, especially in multimodal data processing involving text and image information. Existing technologies are particularly inadequate.

[0003] Flowcharts in scientific literature contain rich visual and textual information, and the extraction of entities and relationships is of great significance for applications such as knowledge graph construction, intelligent retrieval, and knowledge discovery. However, when faced with multimodal data such as flowcharts, traditional methods often exhibit problems such as low processing efficiency, poor accuracy, and incomplete information extraction. This is mainly because existing methods lack a comprehensive understanding of the associated information between images and texts and the ability to integrate multimodal information, making it difficult to effectively identify and utilize key information in flowcharts. Summary of the invention

[0004] This application provides a method for extracting entities and relationships from scientific literature flowcharts based on retrieval enhancement, aiming to solve the technical problems that traditional methods rely on manual annotation of data, are time-consuming and labor-intensive, are difficult to cope with the processing needs of a large number of scientific literature, and lack automated and intelligent extraction methods, resulting in inefficiency in processing large-scale literature.

[0005] The present application discloses a method for extracting entities and relationships from a scientific literature flowchart based on retrieval enhancement, the method comprising: mining a sample data set based on scientific literature, wherein the sample data set is determined based on a flowchart data set and a text data set that have a mapping relationship, and the text data set is flowchart step identification data; introducing a contrast loss function, assisting an LLAVA architecture, taking vector conversion and image-text matching as processing requirements, and combining the sample data set to obtain a supervised trained visual encoder, wherein the LLAVA architecture includes three components: a language model, a visual encoder, and an intermediate connector; introducing a text encoder in the supervised trained language model, and obtaining a visual encoder based on the visual encoder. encoder, traverse the literature database, perform text-to-vector conversion, and build a vector knowledge base, wherein the vector knowledge base can perform bidirectional conversion from text to vector; receive the target flowchart, perform flowchart vector conversion in combination with the visual encoder, and traverse the vector knowledge base for similarity retrieval and matching, perform text conversion on the matched text vectors, and determine the image-text matching result; perform data-driven modeling, supervise the training of a multimodal large model, and the multimodal large model is guided by the extraction of relations and entities; transmit the image-text matching result to the multimodal large model, perform recognition interpretation and relation extraction, and perform structural conversion to generate a target knowledge graph.

[0006] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0007] Collect a variety of flowcharts from scientific and technological literature, and make a detailed natural language description of each flowchart, pair positive and negative samples, ensure the accuracy and comprehensiveness of the annotated text, and improve the quality of the data set through quality control; based on the supervised training of the visual encoder, it can accurately match the flowchart with its corresponding descriptive text, improve the accuracy and recall rate of image-text matching; convert the target flowchart into a vector representation, and traverse the vector knowledge base for matching, retrieve similar vectors in the vector knowledge base through the vectorized flowchart, determine the document block corresponding to the matching text vector, combine the multimodal large model to extract entities and relationships, and output structured output results. In this way, the comprehensiveness and accuracy of information extraction in scientific and technological literature can be significantly improved, providing strong support for knowledge graph construction and intelligent information processing.

[0008] In general, the entity and relationship extraction method for scientific literature flowcharts based on retrieval enhancement provided in this application realizes efficient and accurate entity and relationship extraction of flowcharts in scientific literature by combining image-text alignment training and vector library retrieval technology. After the user provides the flowchart, the retrieval enhancement technology is used to retrieve the paragraph vectors related to the flowchart from the vector knowledge base and align them with the flowchart. Figure 1The multimodal large model is input, and the model will combine this information to perform entity and relationship extraction tasks and output the corresponding entities and relationships. Through this method, the complementary information between images and texts can be fully utilized to improve the efficiency and accuracy of entity and relationship extraction of scientific and technological literature flow charts, providing strong support for knowledge graph construction and intelligent information processing, and has broad application prospects and practical value.

[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A flowchart of a method for extracting entities and relationships from scientific and technological literature based on retrieval enhancement is provided for the embodiment of the present application;

[0011] Figure 2 The embodiment of the present application provides a schematic flow chart of a sample data set based on scientific and technological literature mining in a method for extracting entities and relationships from a scientific and technological literature flowchart based on retrieval enhancement. DETAILED DESCRIPTION

[0012] The embodiment of the present application provides a method for extracting entities and relationships from a scientific literature flowchart based on retrieval enhancement, thereby solving the technical problems that traditional methods rely on manual annotation of data, are time-consuming and labor-intensive, are difficult to cope with the processing needs of a large number of scientific literature, and lack automated and intelligent extraction methods, resulting in low efficiency when processing large-scale literature.

[0013] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application will be specifically described below in conjunction with the drawings in the specification.

[0014] like Figure 1 As shown, the embodiment of the present application provides a method for extracting entities and relationships from a scientific literature flowchart based on retrieval enhancement, the method comprising:

[0015] A sample data set is mined based on scientific and technological literature, wherein the sample data set is determined based on a flowchart data set and a text data set that have a mapping relationship, and the text data set is flowchart step identification data.

[0016] Furthermore, if Figure 2 As shown, the sample data set based on scientific and technological literature mining includes:

[0017] Collect flowchart data sets from scientific and technological literature; based on natural language, traverse the flowchart data set to perform description text conversion to determine the text data set; based on a mapping relationship, determine positive sample pairs and negative sample pairs based on the flowchart data and the text data set, wherein the positive sample pairs are determined based on mapping relationship pairing, and the negative sample pairs are determined based on non-mapping relationship pairing; integrate the positive sample pairs and negative sample pairs, and perform data validity verification screening to determine the sample data set.

[0018] First of all, retrieval enhancement technology is a deep learning framework that combines content retrieval and sequence generation. It enhances the generation ability of the model by utilizing an external knowledge base. Compared with traditional generation models, its core advantage lies in its ability to access and integrate a large amount of non-parametric knowledge, thereby providing richer and more accurate output. Its technical architecture usually includes the following key components: retrieval component: responsible for retrieving the most relevant information fragments or documents from a large-scale knowledge base to the input query; encoder: converting the retrieved documents and input queries into high-dimensional vector representations for further processing; interaction component: interacting the retrieved information with the vector representation of the input query so that the model can better understand the context; generation component: generating the final response or output based on the output of the encoder and interaction component. In this application, retrieval enhancement can be applied to flowchart understanding and description generation tasks. For example, when the model needs to generate a detailed description of a specific flowchart, it can first retrieve document fragments related to the flowchart, and then combine these fragments with the visual information of the flowchart to generate a more accurate and detailed description. This method not only improves the accuracy of the description, but also provides richer details and background information, thereby enhancing the user experience.

[0019] Specifically, we collect flowcharts from various scientific and technological literature, including journal articles, conference papers, technical reports, patent documents, etc., to ensure that the flowcharts cover multiple aspects such as research methods, research problems, and experimental equipment. We integrate the collected flowcharts into a flowchart data set to ensure that the flowcharts in the data set are diverse, including different disciplines and different types of research content.

[0020] Traverse the flowchart dataset, use professionals to manually annotate each flowchart in detail to ensure the accuracy and comprehensiveness of the description text, use natural language to describe the content of the flowchart, and the description text should cover each step, key decision points, experimental equipment and results in the flowchart. For example, "the first step is to prepare the sample; the second step is to perform the treatment using the target method; the third step is to make a temperature control judgment", etc. Pair each flowchart with its description text to ensure that each flowchart has a corresponding description text, and determine the text dataset.

[0021] Based on the mapping relationship, each flowchart is paired with its corresponding description text as a positive sample pair. The positive sample pair is constructed in a way that ensures that the matching relationship between each flowchart and its description text is clear and correct; each flowchart is paired with the description text of other flowcharts as a negative sample pair. The negative sample pair is constructed in a way that ensures that the matching relationship between each flowchart and its non-corresponding description text is random and wrong.

[0022] Integrate all positive and negative sample pairs into a sample dataset, conduct data quality checks, use automated or semi-automated methods to verify data validity, remove incorrectly labeled and non-compliant samples based on the inspection results, ensure the accuracy and comprehensiveness of all description texts, and ensure the high quality of the dataset.

[0023] The contrast loss function is introduced to assist the LLAVA architecture, and vector conversion and image-text matching are used as processing requirements. Combined with the sample data set, a visual encoder after supervised training is obtained, wherein the LLAVA architecture includes three components: a language model, a visual encoder, and an intermediate connector.

[0024] The image-text matching model is constructed based on the LLAVA architecture, which includes three components: a language model, a visual encoder, and an intermediate connector. The visual encoder can use ResNet or Vision Transformer (ViT) to convert the input image (flowchart) into a vector representation; the language model has a built-in text encoder. For example, the text encoder uses the Vicuna model to convert the description text into a vector representation; the intermediate connector performs image-text vector mapping. During the training process, a contrast loss function, such as a cross-loss entropy function, is used for contrast learning. The loss function encourages the model to bring the vector representation of each image close to the vector representation of its corresponding text, and at the same time keep it away from the vector representation of the unmatched text. The sample is trained until convergence, and other components are removed to obtain the visual encoder after supervised training.

[0025] Furthermore, in combination with the sample data set, a visual encoder after supervised training is obtained, including:

[0026] Based on the LLAVA architecture, the sample data set is divided into components for mapping, and training samples are determined by integration; the training samples are traversed, and matching optimization based on text vectors and image vectors is performed based on the contrast loss function as a constraint and the contrast learning principle, and the samples are trained until convergence, and the visual encoder after supervised training is selected; wherein, supervised training includes text sample recognition and vectorization based on the language model, flowchart sample vectorization based on the visual encoder, and image-text vector mapping based on the intermediate connector.

[0027] Specifically, based on the LLAVA architecture, the sample data set is classified and mapped and associated and integrated using a language model, a visual encoder, and the intermediate connector to serve as the training sample.

[0028] Furthermore, each flowchart I i The visual encoder generates an image vector, and each description text T j A text vector is generated by the language model, wherein the language model has a built-in text encoder, so that N samples in a batch will generate two vector sets [I1, I2, ... I N ] and [T1,T2,...T N ].

[0029] The generated image vector and text vector are integrated. At the same time, the acquired image vector and the text vector are mapped and identified, and a specific training task is determined as sample training data. The two-stage training strategy of LLAVA is adopted to perform sample supervised training to obtain the visual encoder after supervised training.

[0030] Specifically, the training process includes a pre-training stage and a fine-tuning stage. The pre-training stage performs feature alignment on the visual encoder and the text encoder. In this embodiment, the sample training data, including mapping corresponding groups of IT sample pairs, is used to perform linear mapping training, that is, mapping the image vector output by the visual encoder to the word embedding space of the text vector of the text encoder, and generating the intermediate connector that represents the mapping relationship between the image and text vectors.

[0031] Furthermore, the visual encoder is frozen, and the intermediate connector and the text encoder are fine-tuned based on execution capability. Specifically, the required execution function is configured to obtain a multimodal instruction following data set, i.e., instruction execution data, such as context association, detailed description, and complex reasoning, etc. The intermediate connector and the text encoder are fine-tuned based on parameters so that they can understand and process images according to the execution instructions, and give corresponding answers or perform corresponding tasks. After the supervised training converges, only the visual encoder after the supervised training is retained.

[0032] Contrastive learning is a self-supervised learning method that learns representations by comparing different data points. In this application, contrastive learning is used to train the model to identify the matching relationship between the flowchart and its description text. The model is trained to maximize the similarity of correct matches (positive sample pairs) and minimize the similarity of incorrect matches (negative sample pairs), thereby learning to distinguish the vector representations of different flowcharts and their descriptions.

[0033] During the training process, through the contrastive learning method, the model learns how to bring the vector representations of correctly matched image-text pairs closer together, and move the vector representations of incorrectly matched image-text pairs farther apart. The samples are trained until convergence, for example, reaching a preset number of iterations, or reaching a preset contrastive learning goal.

[0034] Furthermore, the validation set is used to test the performance, evaluate the accuracy and recall of image-text matching, and adjust the training strategy and optimize the model performance based on the test results.

[0035] Furthermore, the contrastive learning goal is to maximize the cosine distance of the positive sample pair and minimize the cosine distance of the negative sample pair, wherein the mapped image and text are the positive sample pair and the non-mapped image and text are the negative sample pair.

[0036] N samples in a batch will generate an N×N similarity matrix, in which there are N positive sample pairs, among which the mapping image and text are positive sample pairs, and their cosine distance is expected to be close to 1; the remaining N 2 -N negative sample pairs, among which the non-mapped image and text are negative sample pairs, and their cosine distance is expected to be close to -1.

[0037] Introducing a text encoder in a supervised training language model, traversing a document database, performing text-to-vector conversion, and constructing a vector knowledge base, wherein the vector knowledge base can perform bidirectional conversion from text to vector;

[0038] Among them, the document database covers a massive amount of document data. Based on the text encoder in the language model after supervised training, each document in the document database is converted into a vector, and the text-vector mapping before and after the vector conversion is established to integrate and form the vector knowledge base.

[0039] Furthermore, the constructing of the vector knowledge base includes:

[0040] Determine a basic unit framework of a vector knowledge base; based on the basic unit framework, segment the document database to determine a plurality of document blocks; based on a text encoder in a language model after supervised training, perform text vector conversion on the plurality of document blocks, and integrate and determine the vector knowledge base.

[0041] Determine the basic unit framework of the knowledge base and pre-process the target documents uploaded by users, such as PDF, TXT, etc. The pre-processing steps include parsing the document content, dividing the document into multiple Chunks, and determining multiple document blocks. These document blocks serve as the basic units of the knowledge base to facilitate subsequent processing and retrieval.

[0042] In the embodiment of the present application, a preset number of words is used as a fast segmentation method for documents, and a sliding window technology is introduced to process long texts. For example, if each block of content is "2000 words", the sliding window is 5 words, etc.

[0043] The text encoder based on the supervised trained language model vectorizes all document blocks to generate document vector data. These vector representations can capture the semantic information of the document content.

[0044] The document vector data is stored in a dedicated vector knowledge base (such as Chroma DB) for subsequent fast retrieval and matching. Each vector data stored in the vector knowledge base is associated with the original document fragment, ensuring that relevant information can be quickly located and obtained during retrieval.

[0045] A vector knowledge base is a database system specially designed for storing and retrieving vector data. In this application, the vector knowledge base is used to store vectorized representations of document content. These vectors capture the semantic information of the document. Through the vector knowledge base, fast similarity retrieval can be achieved, thereby finding the text paragraphs most relevant to the query flowchart in a large number of documents.

[0046] The target flowchart is received, the flowchart vector is converted in combination with the visual encoder, the vector knowledge base is traversed for similarity search and matching, the matched text vector is converted into text, and the image-text matching result is determined.

[0047] Furthermore, the method further comprises:

[0048] Upload the target flow chart, perform vector conversion in combination with the visual encoder, and determine the image vector data; based on the image vector data, traverse the vector knowledge base for retrieval and matching to determine the image-text matching result.

[0049] When a user inputs a target flowchart, the flowchart is vectorized according to the visual encoder to determine the image vector data, ensuring that the input flowchart vector can accurately reflect the content and structural information of the image.

[0050] Based on the image vector data, retrieval is performed in the vector knowledge base, and the similarity between the query vector and all vectors in the knowledge base is calculated using metrics such as cosine distance. The retrieval process will match the top K paragraphs (Chunks) that are most similar to the query vector, and the most relevant paragraphs retrieved will be further processed with the query vector to ensure that these paragraphs are indeed relevant to the user's query content. The retrieval results are filtered and sorted to ensure that the returned results are highly relevant and accurate, and finally the valid document vector is determined.

[0051] Furthermore, after traversing the vector knowledge base for search and matching, the following steps are included:

[0052] Determine a similarity coefficient; perform positive serialization integration on the similarity coefficient to extract the first K document vector data in the vector knowledge base; perform verification screening on the first K document vector data to determine the valid document vector; traverse the valid document vector, convert and determine the document block corresponding to the valid document vector as the image-text matching result.

[0053] The similarity between the query vector of the target flowchart and the document vector in the knowledge base is calculated using a measurement method such as cosine distance to obtain a similarity coefficient. The results are sorted according to the size of the similarity coefficient, and the top K document vector data with the highest similarity are extracted.

[0054] Exemplarily, the vector subject of the flowchart can be a flowchart vector, a flowchart title vector, a subordinate vector, etc., and the vector knowledge base is combined for retrieval to match and query the first K paragraphs with the most similar vectors, that is, the K items with the largest similarity coefficients. Preferably, the accuracy of the match can be ensured by combining the vector itself or the flowchart context.

[0055] The first K extracted document vector data are further verified to ensure that these document vectors are indeed related to the target flow chart. Multiple verification rules, such as semantic matching, content relevance, etc., can be used to filter and sort the search results to ensure that the returned results are highly relevant and accurate. The valid document vector is determined after verification. Based on the mapping relationship between documents and vectors in the vector knowledge base, the document blocks corresponding to the valid document vectors are matched as the image-text matching results.

[0056] Data-driven modeling is performed to supervise the training of a large multimodal model, which is guided by the extraction of relations and entities.

[0057] Data-driven modeling is performed to supervise the training of a large multimodal model. The input data of the large multimodal model includes the image vector of the target flowchart, the text vector of the relevant document block, and the label data required for supervised learning, including the annotation of entities and relationships. The large multimodal model is supervised and trained to ensure that it can perform effective entity and relationship extraction tasks on the input data. The large multimodal model is guided by the extraction of relationships and entities, and is used to process and understand the input multimodal data (images and text), identify and extract entities in the data and the relationships between them, and finally convert entities and relationships into structured outputs (such as JSON format), where the output format is set independently.

[0058] Specifically, by identifying and integrating information such as flowchart images, flowchart context, and the specified prompt (output result form), entities and relationships are extracted based on text vectors and document blocks. Among them, the flowchart context can assist in extracting the type of entity and improve the effectiveness of the extraction results.

[0059] Among them, the extracted relationship type is determined preferentially, for example, a "before and after" relationship, a "for" relationship, and the like.

[0060] Then, the extracted entity type is determined based on the relationship type. For example, "before and after" (relationship type): [method 1, method 2, method 3, ...] (entity type); "used for" (relationship type): [(method 1, research question 1), (method 2, research question 1), ...] (entity type).

[0061] The image-text matching results are transmitted to the multimodal large model, recognition interpretation and relationship extraction are performed, and structural transformation is performed to generate the target knowledge graph.

[0062] The image-text matching results, including image vectors and text vectors, are packaged and submitted as input data to the multimodal large model. This input integration method ensures that the large model can simultaneously consider the semantic information and related description information of the flowchart, providing a richer semantic basis for subsequent models to perform understanding, induction and summary tasks.

[0063] The large model performs entity and relationship extraction tasks based on the designed prompt, combined with the input flowchart and related text paragraphs. The model accurately identifies the key entities in the flowchart (such as problems, methods, etc.) and the relationships between them (such as sequence, use relationship, etc.) by deeply analyzing and understanding the flowchart and its related text. The extraction results are output in a structured JSON format to ensure the clarity and ease of use of the results. Finally, the target knowledge graph is generated. In this way, the comprehensiveness and accuracy of information extraction can be significantly improved, providing strong support for knowledge graph construction and intelligent information processing.

[0064] Furthermore, generating a target knowledge graph includes:

[0065] The image-text matching results are subjected to data scanning, recognition and semantic analysis, and the extraction results are determined with key entities and relative relationships as extraction targets; based on the extraction results, with the generation of structural requirements as constraints, the structural transformation of the extraction results is performed to determine the target knowledge graph.

[0066] The image-text matching results are preliminarily processed, including data scanning, recognition and semantic analysis. Through data scanning, recognition and semantic analysis, potentially meaningful features can be extracted from the input data to facilitate subsequent relationship extraction. Key entities and their relative relationships are identified from the preliminary processing results, and the key entities and relative relationships in the image-text matching results are determined to determine the extraction results.

[0067] Convert the extraction results into a structured knowledge graph format, such as JSON format, to ensure that the format and structure of the output data meet the requirements of knowledge graph construction, and generate the final knowledge graph according to the structural requirements. This process effectively improves the accuracy and comprehensiveness of information extraction.

[0068] In summary, the method for extracting entities and relationships from scientific literature flowcharts based on retrieval enhancement provided in the embodiments of the present application has the following technical effects:

[0069] Collect a variety of flowcharts from scientific and technological literature, and give a detailed natural language description of each flowchart, pair positive and negative samples to ensure the accuracy and comprehensiveness of the annotated text, and improve the quality of the data set through quality control; use the text encoder and visual encoder of the image-text matching model to convert the descriptive text and flowchart into vector representations, and optimize the model through comparative learning so that it can accurately match the flowchart with its corresponding descriptive text, thereby improving the accuracy and recall rate of image-text matching; vectorize the documents uploaded by users and store them in a vector database. When users query, similar vectors are retrieved from the knowledge base through the vectorized flowchart, and the query vector and retrieval results are input into the multimodal large model for entity and relationship extraction, and structured JSON results are output. In this way, the comprehensiveness and accuracy of information extraction in scientific and technological literature can be significantly improved, providing strong support for knowledge graph construction and intelligent information processing.

[0070] The following is a specific embodiment provided in the embodiments of the present application, which is an example of input and output of a piece of data:

[0071] Flowchart image, flowchart title: The flowchart of the knowledge to be extracted in this task. The example is the flowchart title [Flowchart title: Figure 1 Scientific research method flow] and its corresponding flowchart image.

[0072] Knowledge pattern: A template containing the type of elements that need to be filled in, for example, to represent knowledge such as "a research method follows another research method" or "a research method is used for a research question". The knowledge pattern is defined as: research method [S] research method [S] before and after, research method [S] research question [S] is used for, where [S] is a special token defined.

[0073] Task description: Explain what kind of output results this task should give based on the input. The example is [Extract the relational tuple representing "a specific research method follows a specific research method" and the relational tuple representing "a specific research method is used for a specific research problem". Please output the tuple according to the template shown in the "Knowledge Model". If it does not exist, output "none"; if there are multiple groups, separate the groups with ";". ] Among them, the form of the tuple is not limited. Similarly, other task descriptions and knowledge models in the form of tuples can also be defined.

[0074] Flowchart related information text: The related information text that mentions the flowchart, extracts the most similar first K paragraphs from the vector library through the vector representation of the flowchart. The sample is [As can be seen from Figure a, method A is used after method B, and method B is used to solve problem X. ] Used to supplement knowledge: method A and method B are "research methods", and problem X is "research problem". Combined with the flowchart related information, flowchart title, and flowchart image, the overall knowledge extraction is performed.

[0075] The following is a feasible exemplary model structure:

[0076] Example model architecture and initialization: The visual encoder uses Vision Transformer (ViT) to convert the input image (flowchart) into a vector representation. The text encoder uses the Vicuna model to convert the description text into a vector representation. The knowledge extraction uses the LLAVA-v1.6-34b multimodal large model.

[0077] In general, the entity and relationship extraction method for scientific literature flowcharts based on retrieval enhancement provided in this application realizes efficient and accurate entity and relationship extraction of flowcharts in scientific literature by combining image-text alignment training and vector library retrieval technology. After the user provides the flowchart, the RAG technology is used to retrieve the paragraph vectors related to the flowchart from the vector library and align them with the flowchart. Figure 1 The LLAVA multimodal model will be inputted, and the model will combine this information to perform entity and relationship extraction tasks and output the corresponding entities and relationships. This method can make full use of the complementary information between images and texts, improve the efficiency and accuracy of entity and relationship extraction of scientific literature flow charts, provide strong support for knowledge graph construction and intelligent information processing, and has broad application prospects and practical value.

[0078] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting entities and relationships from scientific literature flow charts based on retrieval enhancement, characterized in that: The method comprises: Mining a sample data set based on scientific and technological literature, wherein the sample data set is determined based on a flowchart data set and a text data set that have a mapping relationship, and the text data set is flowchart step identification data; Introducing a contrast loss function to assist the LLAVA architecture, taking vector conversion and image-text matching as processing requirements, and combining the sample data set to obtain a visual encoder after supervised training, wherein the LLAVA architecture includes three components: a language model, a visual encoder, and an intermediate connector; Introducing a text encoder in a supervised training language model, traversing a document database, performing text-to-vector conversion, and constructing a vector knowledge base, wherein the vector knowledge base can perform bidirectional conversion from text to vector; Receive the target flowchart, perform flowchart vector conversion in combination with the visual encoder, traverse the vector knowledge base for similarity retrieval and matching, perform text conversion on the matched text vectors, and determine the image-text matching result; Conduct data-driven modeling and supervise the training of a large multimodal model, wherein the large multimodal model is guided by the extraction of relations and entities; The image-text matching results are transmitted to the multimodal large model, recognition interpretation and relationship extraction are performed, and structural transformation is performed to generate the target knowledge graph.

2. The method for extracting entities and relationships from a scientific literature flow chart based on retrieval enhancement as claimed in claim 1, characterized in that: The sample data set mined based on scientific and technological literature includes: Collect flowchart datasets from scientific literature; Based on natural language, traverse the flowchart data set to perform description text conversion to determine the text data set; Based on the mapping relationship, determine a positive sample pair and a negative sample pair based on the flowchart data and the text data set, wherein the positive sample pair is determined based on the mapping relationship pairing, and the negative sample pair is determined based on the non-mapping relationship pairing; The positive sample pairs and the negative sample pairs are integrated, and data validation screening is performed to determine the sample data set.

3. The method for extracting entities and relationships from a scientific literature flow chart based on retrieval enhancement as claimed in claim 2, characterized in that: Combined with the sample data set, a visual encoder after supervised training is obtained, including: Based on the LLAVA architecture, the sample data set is divided into components through component mapping, and training samples are determined through integration; Traversing the training samples, taking the contrast loss function as a constraint, based on the contrast learning principle, performing matching optimization based on the text vector and the image vector, training the samples until convergence, and selecting the visual encoder after supervised training; Among them, the supervised training includes text sample recognition and vectorization based on the language model, flowchart sample vectorization based on the visual encoder, and image-text vector mapping based on the intermediate connector.

4. The method for extracting entities and relationships from a scientific literature flow chart based on retrieval enhancement as claimed in claim 3, characterized in that: The contrastive learning goal is to maximize the cosine distance of the positive sample pair and minimize the cosine distance of the negative sample pair, wherein the mapped image and text are the positive sample pair and the non-mapped image and text are the negative sample pair.

5. The method for extracting entities and relationships from scientific and technological literature flow charts based on retrieval enhancement as claimed in claim 1, characterized in that: The constructing of the vector knowledge base includes: Determine the basic unit framework of the vector knowledge base; Based on the basic unit framework, the document database is segmented to determine a plurality of document blocks; Based on the text encoder in the language model after supervised training, the multiple document blocks are converted into text vectors, and the vector knowledge base is determined by integration.

6. The method for extracting entities and relationships from a scientific literature flow chart based on retrieval enhancement as claimed in claim 5, characterized in that: The method further comprises: Upload the target flow chart, perform vector conversion in combination with the visual encoder, and determine the image vector data; Based on the image vector data, the vector knowledge base is traversed to perform search and matching to determine the image-text matching result.

7. The method for extracting entities and relationships from a scientific literature flow chart based on retrieval enhancement as claimed in claim 6, characterized in that: After traversing the vector knowledge base for search and matching, it includes: Determine similarity coefficient; Performing positive serialization integration on the similarity coefficients to extract the first K document vector data in the vector knowledge base; Verify and screen the first K document vector data to determine the valid document vector; The valid document vectors are traversed, and the document blocks corresponding to the valid document vectors are converted and determined as the image-text matching results.

8. The method for extracting entities and relationships from a scientific literature flow chart based on retrieval enhancement as claimed in claim 1, characterized in that: The generating of the target knowledge graph comprises: Performing data scanning, recognition and semantic analysis on the image-text matching results, taking key entities and relative relationships as extraction targets, and determining extraction results; Based on the extraction results, with the generation structure requirement as a constraint, the structural transformation of the extraction results is performed to determine the target knowledge graph.

Citation Information

Patent Citations

  • Traditional Chinese medicine massage multi-modal knowledge graph construction method based on large model and comparative learning

    CN118136264A

  • Image recognition method, model training method of target recognition model and related equipment

    CN118298434A

  • Systems and methods for vision-and-language representation learning

    US20220391755A1

Cited By

  • Intelligent technology dynamic tracking system for science field

    CN119293342A

  • Archaeological cultural relic model training method and system

    CN119443284A

  • Method and device for constructing medical condition image-text knowledge base, and medium

    CN119782505A

  • A method, device and medium for constructing a scientific information graphic and text knowledge base

    CN119782505B

  • Scientific and technical literature knowledge graph construction method, device and equipment and medium

    CN120030172A