Method and device for constructing medical condition image-text knowledge base, and medium

Abstract

The invention discloses a medical condition image-text knowledge base construction method and device and a medium, and relates to the field of knowledge base construction, and the method comprises the steps: for each scientific research file, inputting a corresponding scientific research text and a scientific research picture into a preset scientific research picture-text matching model to obtain a scientific research picture-text pair; the preset scientific research picture-text matching model is obtained by adopting a training sample set to train a preset deep learning neural network; based on the scientific research picture-text pairs, extracting scientific research entities, corresponding attributes and a relationship between the two scientific research entities, and mapping the scientific research entities and the corresponding attributes and the relationship to a preset medical condition image-text knowledge base body to obtain a medical condition image-text knowledge base; the preset medical condition image-text knowledge base body comprises a plurality of triads, and each triad comprises two scientific research entities, the relationship between the two scientific research entities and the attribute of each scientific research entity. According to the method, the medical condition image-text knowledge base containing the image-text information is constructed, so that the value and the utilization efficiency of the data are improved.

Description

Technical Field

[0001] The present application relates to the field of knowledge base construction, and in particular to a method, device and medium for constructing a scientific information graphic knowledge base. Background Art

[0002] The big model technology is a deep learning model with a large parameter scale that uses a large amount of data and computing power for training. Compared with other artificial intelligence technologies, it has stronger generalization and learning capabilities. At present, the big models that support multimodality mainly include the Flamingo big model released by Google DeepMind, Chat GPT4 of OpenAI in the United States, and the Wudao 3.0Emu big model released by China Zhiyuan. The data sets used for multimodal big models are: 1) Flickr30k and its Chinese translation version Flickr30k-CNA, which mainly consists of pictures and a simple sentence description of the scene in the picture; 2) WUKONG data set of 100 million image pairs of China Zhiyuan Company, which mainly consists of pictures and texts describing them; 3) AIC-ICC (Chinese image description data set released in the first AI Challenger Global AI Challenge jointly sponsored by Sinovation Ventures, Sogou, and Toutiao) data set consists of pictures and multiple texts that describe the scenes in the pictures in detail. These data sets are all pictures oriented to tasks or life scenes, and the descriptions are only the information of the picture itself, without other background knowledge of the picture. The large model trained using the above dataset has good application effects in image retrieval, target detection, and image recognition, but it is not deep enough in mining scientific and technological images containing contextual knowledge, and is not suitable for training multimodal large models for scientific and technological intelligence tasks. Summary of the invention

[0003] The purpose of this application is to provide a method, device and medium for constructing a scientific information graphic knowledge base, thereby constructing a scientific information graphic knowledge base containing graphic information, thereby improving the value and utilization efficiency of data.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides a method for constructing a scientific information graphic knowledge base, comprising:

[0006] Acquire multiple scientific research documents containing images and perform preprocessing to obtain scientific research texts and scientific research images corresponding to each of the scientific research documents;

[0007] For each scientific research document, the corresponding scientific research text and the scientific research picture are input into a preset scientific research picture-text matching model to obtain a scientific research picture-text pair; the preset scientific research picture-text matching model is obtained by training a preset deep learning neural network using a training sample set;

[0008] Obtaining a preset scientific information graphic knowledge base ontology; the preset scientific information graphic knowledge base ontology includes a plurality of triples, each triple includes two scientific research entities, a relationship between the two scientific research entities, and an attribute of each scientific research entity;

[0009] Based on the scientific research image-text pairs, the scientific research entities and corresponding attributes, and the relationship between the two scientific research entities are extracted, and mapped to the preset scientific information image-text knowledge base ontology to obtain the scientific information image-text knowledge base.

[0010] In a second aspect, the present application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for constructing a scientific information graphic and text knowledge base.

[0011] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for constructing a scientific information graphic and text knowledge base.

[0012] According to the specific embodiments provided by the present application, the present application has the following technical effects: the present application provides a method, device and medium for constructing a scientific information picture and text knowledge base, which mines scientific research texts and scientific research pictures from scientific research documents, and then obtains a preset scientific research picture-text matching model that can capture the correlation correspondence between pictures and texts based on a preset deep learning neural network training. In the training process of the model, not only the language features of scientific research pictures are utilized, but also the language features of scientific research texts are utilized, so that the mining of scientific research pictures is more extensive. Finally, knowledge extraction is performed based on the obtained scientific research picture-text pairs and mapped to the preset scientific information picture and text knowledge base ontology to obtain a scientific information picture and text knowledge base. The scientific information picture and text knowledge base includes picture and text information, which realizes the deep mining of the information contained in the picture and improves the value of the data; subsequent picture and text retrieval based on the scientific information picture and text database can improve the utilization efficiency of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 A schematic diagram of a process for constructing a scientific information graphic knowledge base provided in one embodiment of the present application;

[0015] Figure 2A schematic diagram of a preset scientific research image-text matching model provided in one embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] Building a high-quality multimodal image and text dataset is a key factor in promoting the development of multimodal large model technology, and the scientific information image and text knowledge base in the field of scientific and technological intelligence is the key to fine-tuning the general multimodal large model technology into a multimodal large model in the field of scientific and technological intelligence. Based on this, this application explores the use of artificial intelligence technology to mine the correlation between images and texts from scientific papers, patent articles and scientific news, and establish a scientific information image and text knowledge base for multimodal large models. This application scheme not only provides image and text datasets for the improvement of general multimodal large models, but also provides support for the establishment of multimodal scientific and technological intelligence large models.

[0018] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0019] In an exemplary embodiment, Figure 1 As shown, a method for constructing a scientific information graphic knowledge base is provided. The method is executed by a computer device, and can be specifically executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In an embodiment of the present application, the method includes the following steps 101 to 104.

[0020] Step 101, obtain multiple scientific research documents containing pictures and perform preprocessing to obtain scientific research texts and scientific research pictures corresponding to each of the scientific research documents; wherein, the scientific research documents containing pictures include scientific research news documents, scientific research paper documents and scientific research patent documents. The pictures in scientific research news may be the appearance of equipment and products in new product releases, high-tech equipment and process flows of technological breakthroughs, actual scenes of scientific and technological applications, leaders in the science and technology industry or senior executives of science and technology companies, etc. From the pictures in scientific and technological articles, the structure and relationship of the research objects and the logic of the model algorithm can be mined. The pictures in scientific research papers and patents are combined with the content of the articles to concisely and clearly show the research ideas and research results, and can also enhance the breadth and depth of academic exchanges, such as architecture diagrams, flow charts, research roadmaps, experimental results diagrams, system operation diagrams, etc.

[0021] In a practical application, crawler technology can be used to collect and adopt multiple scientific research documents. That is, crawler technology is used to automatically collect all the page contents that it can access according to certain rules to obtain or update the contents of these websites, and then the obtained data is saved locally.

[0022] In another practical application, the step of preprocessing the scientific research file containing the image includes:

[0023] (11) For the scientific research news file, web page content recognition technology is used to obtain web page text and web page images. Specifically, web page content recognition technology can identify the structure, content, style and other information of the web page, and can also extract the required information therefrom to obtain web page text and web page images, wherein the web page text includes text content and content structure, the text content includes text information such as title, content, and picture description, and the content structure is a basic format, such as the order of title, abstract, and headline, and also includes the position information of the picture.

[0024] (12) The scientific research paper files and the scientific research patent files are generally in PDF format, and OCR (Optical Character Recognition, which can convert image information into text) technology is used to parse them to obtain document text and document images; the web page text and the document text both include text content and content structure. In addition, a third-party PDF file parsing library can also be used for parsing, and information such as document format, images and their locations can be retained.

[0025] (13) Mark the web page text and the document text as scientific research text, and mark the web page image and the document image as scientific research images.

[0026] Step 102, for each scientific research document, the corresponding scientific research text and the scientific research picture are input into a preset scientific research picture-text matching model to obtain a scientific research picture-text pair; the preset scientific research picture-text matching model is obtained by training a preset deep learning neural network using a training sample set.

[0027] In an application example, the preset scientific research image-text matching model includes a visual encoder, a text encoder, an external knowledge combination module, a cross-modal matching module, and a loss function module, such as Figure 2 As shown. The input end of the visual encoder is used to receive the scientific research image, and the output end of the visual encoder is connected to the first input end of the cross-modal matching module; the input end of the text encoder is used to receive the scientific research text, and the output end of the text encoder is connected to the second input end of the cross-modal matching module; the output end of the cross-modal matching module is connected to the loss function module.

[0028] The external knowledge combination module is arranged between the visual encoder and the text encoder, and the external knowledge combination module is used to: measure the distance between the scientific research picture and the corresponding text paragraph position in the scientific research text, and send it to the visual encoder and the text encoder respectively as a picture-text pair distance feature.

[0029] Specifically, the step of measuring the distance between the scientific research image and the corresponding text paragraph position in the scientific research text in the external knowledge combination module includes: if there is a caption below or above the scientific research image, the distance between the scientific research image and the corresponding text paragraph position in the scientific research text is 0; if there is no caption below or above the scientific research image, the distance is calculated using the following formula:

[0030]

[0031] The visual encoder is used to encode the visual semantic information contained in the scientific research images to obtain visual feature representation. Specifically, a convolutional neural network is used to model the scientific research images as a whole to obtain a global visual representation of the images; since global encoding will lose fine-grained semantic information, a self-attention mechanism or a graph convolutional network is used to strengthen the relationship between entities in the visual modality to further enhance visual semantic understanding; in addition, the image-text pair distance feature is combined to provide more accurate matching information for subsequent cross-modal matching.

[0032] The text encoder is used to capture the semantic information contained in the scientific research text to obtain the text modality feature representation. Specifically, the Transformer model is used to introduce the multi-head attention mechanism into the text encoder to model the text sentences to obtain the corresponding representation. In addition, the image-text pair distance feature is combined to provide more accurate matching information for subsequent cross-modal matching.

[0033] The cross-modal matching module is used to eliminate the semantic gap between different modalities so as to better align the semantic information of the two modalities and thus perform similarity estimation more accurately. Specifically, the cross-modal semantic similarity is measured by jointly embedding visual features and text features into a common space for cross-modal semantic similarity estimation.

[0034] The loss function module is used to constrain the learning of the image-matching model through different objective functions. Specifically, the same ranking function is used for learning, such as hinge ranking loss (also known as triple ranking loss) and bidirectional hinge ranking loss. In this module, the learning performance is improved by introducing a loss function. During the training process, when the value of the loss function reaches the preset loss condition, the training ends.

[0035] The image-text matching implemented in step 102 is to measure the semantic similarity between the image and the sentence, determine the degree of association between the image and the text, and finally output the scientific research image and the text segment in the scientific research text that matches the scientific research image. In scientific research papers and scientific research patents, the annotation of images and the position of text related to the images are more rigorous, and the scientific and technological image-text matching task based on this can have a higher matching accuracy.

[0036] Step 103, obtaining a preset scientific information graphic and text knowledge base ontology; the preset scientific information graphic and text knowledge base ontology includes multiple triples, each triple includes two scientific research entities, the relationship between the two scientific research entities and the attributes of each of the scientific research entities.

[0037] The goal of ontology is to capture the knowledge of related fields, provide a common understanding of the knowledge in the field, determine the commonly recognized vocabulary in the field, and give clear definitions of these words (terms) and the relationships between them from different levels of formal models to overcome differences in terminology and achieve semantic interoperability. Ontology expresses domain knowledge by capturing relevant concepts, instances and their attributes, semantic relationships and related semantic constraints in the field. It is the skeleton of the scientific and technological information graphic knowledge base, which is used to describe graphic data sets in the field of science and technology and establish corresponding knowledge specifications. Constructing ontology can achieve a certain degree of knowledge sharing and reuse as well as improve the ability of system communication interoperability and reliability.

[0038] In an application example, this application provides a preset scientific information graphic knowledge base ontology. Specifically, interviews are conducted with experts in the field of science and technology, and research is conducted on the main scientific research concepts and processes in different scientific research fields, so as to collect opinions and suggestions for the design of the meta-ontology in the field of scientific information graphic. Then, relevant technical personnel establish a meta-model that describes the basic entities, attributes, and relationships of scientific information graphic data. On this basis, the entities, attributes, and relationships can be automatically expanded through deep learning-based semantic analysis technology to complete the automatic expansion of the ontology.

[0039] In another application example, the scientific research entity includes: author, publishing institution, school department, paper ID and picture ID.

[0040] The attributes of the scientific research entity include: author attributes, institution attributes, department attributes, paper attributes and image attributes; the author attributes include name, date of birth, age, gender, professional title, research field and academic qualifications; the institution attributes include institution ID, institution name and institution introduction; the department attributes include college ID, college name, college introduction and college research field; the paper attributes include: paper title, paper abstract, paper keywords, paper field, paper publication journal and paper publication time; the image attributes include image name, image type, image keywords, paper to which the image belongs and the image field.

[0041] The relationship between the two scientific research entities includes: is-a relationship, subordinate relationship, cause-effect relationship, time relationship, homonymous relationship and hierarchical relationship.

[0042] In this application, the image types include flow charts, architecture diagrams, design diagrams, experimental result diagrams, system interface diagrams, character diagrams and product schematic diagrams. In addition, scientific research images have strong domain attributes, category attributes and association relationships. The scientific research field has a relatively complete system structure, but there are many interdisciplinary studies in existing scientific and technological articles, so the domain attributes of scientific research images belong to a one-to-one or one-to-many relationship. There are also different associations between different types of scientific research images. For example, a design diagram and an experimental result diagram may be a description of different stages of the same scientific research topic; the same design diagram may be a description of different aspects of the design of the same scientific research topic. Therefore, there are co-position relationships, hierarchical relationships, and affiliation relationships between different scientific research images.

[0043] The value contained in each scientific research image is also different. For example, the experimental result image can mine the performance indicators of the research direction, the character image can mine the experts in the field of the research direction, and the product diagram can obtain competitive intelligence information. The types of scientific research images are manually defined as above. In the later stage, through the analysis and mining of a large number of images, deep learning technology or machine learning cluster analysis is used to learn new entity types, attributes and relationship types from the data, and then the ontology is expanded and updated based on this new information.

[0044] Step 104, based on the scientific research image-text pair, extract the scientific research entity and the corresponding attributes, the relationship between the two scientific research entities, and map them to the preset scientific information image-text knowledge base ontology to obtain the scientific information image-text knowledge base. The scientific information image-text knowledge base supports the dual search function of scientific research images and scientific research texts to support scientific and technological intelligence analysis tasks; supports the batch export of scientific research image-text pairs to generate cross-modal data sets, and realizes the purpose of supporting the pre-training of cross-modal scientific and technological intelligence large models.

[0045] In an application example, for the images in the same scientific research document, after completing the matching of scientific research image-text pairs, knowledge extraction is required, mainly including the extraction of the field to which the image belongs, the image type, the relationship between multiple images, etc. In the specific extraction process, if the scientific research entity is an image ID, that is, the extraction of scientific research entities, relationships or attributes related to the image, then knowledge extraction is performed based on the content of the text paragraph corresponding to the scientific research image (that is, the text content in the scientific research image-text pair); in addition, relevant content assistance can be retrieved from the overall scientific research text as needed, such as titles, abstracts, etc.; if it involves the extraction of other scientific research entities, relationships or attributes related to authors and publishing institutions, then knowledge extraction is performed based on the overall scientific research text.

[0046] In an application example, the step of extracting the attributes corresponding to the scientific research entity includes:

[0047] When the attribute corresponding to the scientific research entity is the field to which the image belongs, the extraction process includes: using deep learning classification technology to classify the title, abstract or scientific research content of the scientific research document to which the scientific research image belongs, so as to obtain the field to which the image belongs. After completing the above classification, the field to which the image belongs can also be preliminarily proofread and verified according to the classification number of the public scientific research document as needed.

[0048] When the attribute corresponding to the scientific research entity is a picture type, the extraction process includes: classifying pictures based on the scientific research pictures and the corresponding texts in the scientific research picture-text pair to obtain a first picture type and a second picture type; when the first picture type is consistent with the second picture type, taking the first picture type as the final picture type and storing it; when the first picture type is inconsistent with the second picture type, storing the scientific research pictures in a waiting area; when the number of scientific research pictures in the waiting area reaches a preset number, clustering all the scientific research pictures to obtain a new picture type; adding the new picture type to the preset scientific information picture and text knowledge base ontology to realize ontology update.

[0049] Specifically, a deep learning convolutional network is used to classify scientific research images according to visual features to obtain a first image type; at the same time, a deep learning recurrent network is used to classify the scientific research images according to the text matched by the scientific research images to obtain a second image type. In addition, when clustering the area to be processed, a threshold is set. When the value of the new image category is greater than the set threshold, it is manually reviewed and added as a new image type after passing the review.

[0050] The step of extracting the relationship between the two scientific research entities includes: matching the image types of the scientific research images according to preset rules to obtain the relationship between the scientific research images. For those beyond the preset rules, a deep learning algorithm is used to extract the texts corresponding to the scientific research images to obtain the relationship between the images.

[0051] Finally, the text matched by the scientific research picture is used to extract the knowledge of the named entity, and the extracted entities and attributes are mapped to the ontology library. If the entity does not exist in the ontology library, the ontology library is updated as a new entity. After the knowledge is extracted, the scientific research pictures and texts as well as the extracted knowledge are mapped to an instance of the ontology to realize the construction of the scientific information picture and text knowledge base. In addition, the picture classification, picture clustering, picture-text matching, text knowledge extraction and other places in this application that require automatic processing and analysis are all realized by neural networks of different structures in deep learning technology. Graph database technology is also used to effectively store, manage and query complex relational data, thereby improving the value and utilization efficiency of the data, that is, the construction of the scientific information picture and text knowledge base in this application is based on the graph database technology.

[0052] In summary, the present application has the following advantages over the prior art:

[0053] (1) Construct a scientific information graphic text based on scientific research images in scientific and technological literature, deeply analyze the scientific research information contained in scientific and technological images, form a knowledge system based on scientific research images, and improve the application value of scientific and technological images.

[0054] (2) Match scientific research images with corresponding texts in scientific and technological literature and complete cross-modal data pre-annotation, thereby taking advantage of the rigorous writing style of scientific and technological literature to match images and texts in the literature, and then further mine relevant information of the images from the text describing the images. This application not only utilizes the visual features of scientific and technological images, but also utilizes the language features of related texts, making the mining of images more extensive.

[0055] (3) This application constructs a scientific information picture and text knowledge base, which has the function of exporting data sets for multimodal large model training, and also has the function of semantic analysis of scientific and technological pictures, making the use of scientific and technological pictures more extensive.

[0056] From the perspective of application prospects, this application can accelerate the development of multimodal large models in the vertical field of scientific and technological intelligence. The development of multimodal large models relies on the support of high-quality multimodal data sets. The multimodal scientific information picture and text knowledge base of this application can provide data set support for fine-tuning multimodal large models in the field of science and technology, and accelerate the development of multimodal large models.

[0057] This application can drive technological innovation. Scientific research images contain design information of scientific research plans, methods, process flows, etc., effect information of experimental results, information on the results of scientific research transformation, and information on experts, scholars and scientific research institutions in the field. The scientific research image and text knowledge base can deeply interpret this information, which will provide strong support for technological innovation.

[0058] This application can improve the research efficiency of think tanks. Semantic retrieval of scientific and technological images can enable think tank researchers to quickly and accurately use the knowledge brought by the images, which can maximize the research efficiency of think tanks, help think tank researchers better grasp scientific and technological trends, and provide timely and accurate intelligence support for decision-making.

[0059] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (I / O for short) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for constructing a scientific information graphic knowledge base is implemented.

[0060] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0061] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0062] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0063] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0064] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0065] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0066] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for constructing a scientific information graphic knowledge base, characterized in that: The method for constructing a scientific information graphic knowledge base comprises: Acquire multiple scientific research documents containing images and perform preprocessing to obtain scientific research texts and scientific research images corresponding to each of the scientific research documents; For each scientific research document, the corresponding scientific research text and the scientific research picture are input into a preset scientific research picture-text matching model to obtain a scientific research picture-text pair; the preset scientific research picture-text matching model is obtained by training a preset deep learning neural network using a training sample set; Obtaining a preset scientific information graphic knowledge base ontology; the preset scientific information graphic knowledge base ontology includes a plurality of triples, each triple includes two scientific research entities, a relationship between the two scientific research entities, and an attribute of each scientific research entity; Based on the scientific research image-text pairs, the scientific research entities and corresponding attributes, and the relationship between the two scientific research entities are extracted, and mapped to the preset scientific information image-text knowledge base ontology to obtain the scientific information image-text knowledge base.

2. The method for constructing a scientific information graphic knowledge base according to claim 1, characterized in that: The scientific research files containing images include scientific research news files, scientific research paper files and scientific research patent files; The steps of preprocessing the scientific research file containing the image include: For the scientific research news files, web page text and web page images are obtained using web page content recognition technology; For the scientific research paper file and the scientific research patent file, OCR technology is used to parse to obtain document text and document image; the web page text and the document text both include text content and content structure; The web page text and the document text are marked as scientific research texts, and the web page pictures and the document pictures are marked as scientific research pictures.

3. The method for constructing a scientific information graphic knowledge base according to claim 1, characterized in that: The scientific research entities include: author, publishing institution, school department, paper ID and image ID; The attributes of the scientific research entity include: author attributes, institution attributes, department attributes, paper attributes and image attributes; the author attributes include name, date of birth, age, gender, professional title, research field and academic qualifications; the institution attributes include institution ID, institution name and institution introduction; the department attributes include college ID, college name, college introduction and college research field; the paper attributes include: paper title, paper abstract, paper keywords, paper field, paper publication journal and paper publication time; the image attributes include image name, image type, image keywords, image paper and image field; The relationship between the two scientific research entities includes: is-a relationship, subordinate relationship, cause-effect relationship, time relationship, homonymous relationship and hierarchical relationship.

4. The method for constructing a scientific information graphic knowledge base according to claim 3, characterized in that: The image types include flow charts, architecture diagrams, design diagrams, experimental result diagrams, system interface diagrams, character diagrams and product schematics.

5. The method for constructing a scientific information graphic knowledge base according to claim 1, characterized in that: The preset scientific research image-text matching model includes a visual encoder, a text encoder, an external knowledge combination module, a cross-modal matching module and a loss function module; The input end of the visual encoder is used to receive the scientific research image, and the output end of the visual encoder is connected to the first input end of the cross-modal matching module; the input end of the text encoder is used to receive the scientific research text, and the output end of the text encoder is connected to the second input end of the cross-modal matching module; the output end of the cross-modal matching module is connected to the loss function module; The external knowledge combination module is arranged between the visual encoder and the text encoder, and the external knowledge combination module is used to: measure the distance between the scientific research picture and the corresponding text paragraph position in the scientific research text, and send it to the visual encoder and the text encoder respectively as a picture-text pair distance feature.

6. The method for constructing a scientific information graphic knowledge base according to claim 5, characterized in that: The step of measuring the distance between the scientific research image and the corresponding text paragraph position in the scientific research text in the external knowledge combination module includes: If there is a caption below or above the scientific research image, the distance between the scientific research image and the corresponding text paragraph position in the scientific research text is 0; If there is no legend below or above the scientific research image, the following formula is used to calculate the distance:

7. The method for constructing a scientific information graphic knowledge base according to claim 1, characterized in that: The step of extracting the attributes corresponding to the scientific research entity comprises: When the attribute corresponding to the scientific research entity is the field to which the image belongs, the extraction process includes: using deep learning classification technology to classify the title, abstract or scientific research content of the scientific research document to which the scientific research image belongs, so as to obtain the field to which the image belongs; When the attribute corresponding to the scientific research entity is a picture type, the extraction process includes: classifying the pictures based on the scientific research pictures and the corresponding texts in the scientific research picture-text pair to obtain a first picture type and a second picture type; when the first picture type is consistent with the second picture type, using the first picture type as the final picture type; when the first picture type is inconsistent with the second picture type, storing the scientific research pictures in a waiting area; when the number of scientific research pictures in the waiting area reaches a preset number, clustering all the scientific research pictures to obtain a new picture type; adding the new picture type to the preset scientific information picture and text knowledge base ontology to realize ontology update.

8. The method for constructing a scientific information graphic knowledge base according to claim 1, characterized in that: The step of extracting the relationship between the two scientific research entities includes: matching the image types of the scientific research images according to preset rules to obtain the relationship between the scientific research images.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing a scientific information graphic knowledge base as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing a scientific information graphic knowledge base described in any one of claims 1 to 8 is implemented.

Images