Multi-modal scientific data retrieval method and system based on abstract-assisted cognitive enhancement

Abstract

The invention belongs to the field of data retrieval, and provides a multi-modal scientific data retrieval method and system based on abstract-assisted cognitive enhancement. According to the technical scheme, a multi-modal scientific data semantic representation model based on a large model is constructed, professional domain knowledge and a structure guiding mechanism are combined, and a multi-modal scientific data semantic representation model is constructed; generating a structured abstract containing the key knowledge; and then, on the basis of knowledge such as keywords and description information provided by the structured abstract, a multi-modal scientific data retrieval model guided by directional thinking chain multipath is constructed, multi-dimensional correlation analysis of scientific data is realized through combination of cognitive prompt knowledge enhancement and multi-step dynamic reasoning, and the data resource retrieval performance in the field of scientific data is improved.

Description

Technical Field

[0001] The present invention belongs to the field of data retrieval, and in particular to a multimodal scientific data retrieval method and system based on summary-assisted cognitive enhancement. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the vigorous development of knowledge and technology in various scientific fields, scientific data analysis has become an important source and basis for scientific and technological innovation, and the ability to retrieve scientific big data will become the key to success in major scientific discoveries. At present, scientific data retrieval mainly relies on manual retrieval and analysis of relevant content from massive multimodal scientific data. However, scientific events themselves are low-probability events in the existing historical scientific data. The retrieval method using a general large model is not suitable for scientific data fields with strong terminology, complex knowledge associations, and diverse data structures. At the same time, scientific data retrieval requires high precision, strong logical associations, and traceable sources. Therefore, it is urgent to conduct research on effective retrieval methods based on scientific data to promote the open sharing of scientific data.

[0004] Existing retrieval methods are difficult to meet the requirements of "fast, accurate and complete" intelligent discovery of scientific data. Specifically, there is a lack of discovery of "small" probability scientific data in massive scientific data, and heterogeneous scientific data is difficult to efficiently integrate and retrieve. In addition, scientific data retrieval needs to meet real-time update requirements, so the current retrieval of massive historical scientific data is inefficient.

[0005] At present, some research institutions have begun to use large language models (LLM) for resource retrieval. At the same time, in order to avoid problems such as retrieval illusion, outdated knowledge and lack of knowledge in specialized fields, they are combined with retrieval-augmented generation (RAG) to expand it into an internal knowledge base that can access specific fields. RAG mainly includes three steps: indexing, retrieval and generation. By introducing professional knowledge prompts, the relevance, reliability and accuracy of LLM retrieval output in different fields are improved. The current mainstream frameworks of this form include Naive RAG, Advanced RAG and Modular RAG. Among them, Naive RAG follows the traditional RAG retrieval process and has problems such as inaccurate external knowledge retrieval and generation difficulties; Advanced RAG can effectively improve the retrieval quality. Currently, it mainly uses sliding window methods, fine-grained segmentation and metadata merging to improve its retrieval performance; Modular RAG mainly introduces modules, including iterative and adaptive retrieval methods, and has the advantages of retrieval flexibility and adaptability. The above methods mainly solve the complex knowledge association, but ignore the challenge of scientific data having multimodal data types. The difficulty in integrating heterogeneous data knowledge leads to inconsistent data representation, which in turn increases the difficulty of understanding the retrieved data. Summary of the invention

[0006] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a multimodal scientific data retrieval method and system based on summary-assisted cognitive enhancement, which combines professional domain knowledge with a structural guidance mechanism by constructing a multimodal scientific data semantic representation model based on a large model to generate a structured summary containing key knowledge; then, based on the keywords, descriptive information and other knowledge provided by the structured summary, a multimodal scientific data retrieval model with multi-path guidance of a directional thinking chain is constructed, and through cognitive prompt knowledge enhancement combined with multi-step dynamic reasoning, multi-dimensional correlation analysis of scientific data is realized, thereby improving the data resource retrieval performance in the scientific data field.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] The first aspect of the present invention provides a multimodal scientific data retrieval method based on summary-assisted cognitive enhancement, comprising the following steps:

[0009] Acquire multimodal scientific data and transform them into unified feature coding representation;

[0010] Extract keywords for the search questions;

[0011] Performing multimodal data retrieval based on keywords of the question to be searched to obtain related multimodal data, including: retrieving related text data from a text library based on feature coding representations of keywords of the question to be searched, and associating feature coding representations of other modalities based on feature coding representations of the related text data to obtain scientific data of other modalities;

[0012] Construct a RAG keyword entity tree for scientific data based on structured abstracts and their corresponding keywords and description information;

[0013] Based on the associated multimodal data, the multimodal retrieval process is optimized according to the cognitive hint knowledge output by the RAG keyword entity tree. After a multi-step dynamic reasoning retrieval process, the final retrieval content is obtained.

[0014] Furthermore, the unified representation of the multimodal data is converted into a unified feature coding representation, including:

[0015] The acquired multimodal data is semantically segmented and entity represented to obtain multimodal data after segmentation and representation;

[0016] Encode the multimodal data after block representation to obtain multimodal features;

[0017] Correlate multimodal features to obtain multimodal fusion features;

[0018] By stacking attention layers and fully connected layers, the fused features are reduced in dimension to obtain a multimodal unified encoding feature representation for scientific data classification.

[0019] Furthermore, after obtaining the multimodal scientific data associated with the keywords of the question to be searched, the multimodal scientific data is reordered, specifically including:

[0020] The question to be retrieved and the associated multimodal data are uniformly represented and connected with the separator "[SEP]", and the start symbol "[CLS]" is added as the input of the BERT model. The similarity score between the retrieved content and the question to be retrieved is output, and the recommendation list of the associated multimodal data is sorted according to the score and output.

[0021] Furthermore, a RAG keyword entity tree for scientific data is constructed based on the structured abstract and its corresponding keywords and description information, including:

[0022] According to the section titles corresponding to the text directory, each chapter is regarded as a text block;

[0023] Extracting keyword unified representations from unified representations of each text block and unified representations of directory titles;

[0024] The unified representation of keywords whose frequency meets the threshold is used as the unified representation of keyword entities, and the unified representation of keyword entities is used as the unified representation of the text block;

[0025] The keyword entity unified representation is associated with other entity unified representations associated with the catalog, and the keyword unified representations that are the same as other text entities are associated with it, thereby completing the construction of a primary RAG keyword entity tree for scientific data;

[0026] Based on the primary RAG keyword entity tree for scientific data, the unified representation of keyword entities, the unified representation of descriptions and the unified representation of context associations are clustered, and the unified representation of text summaries of the clustering clusters is generated, thereby completing the construction of the RAG keyword entity tree for scientific data from the bottom up.

[0027] Furthermore, in the RAG keyword entity tree for scientific data, nodes that are clustered together are sibling nodes, and the cluster features of clustered sibling nodes are represented by a parent node. The parent node contains a text summary of the cluster, and the clustered sibling nodes within the cluster represented by the parent node are child nodes of the parent node.

[0028] Furthermore, the step of retrieving relevant text data from a text library according to the feature coding representation of the keywords of the question to be searched includes:

[0029] The questions to be retrieved and massive texts are represented as a unified representation code and mapped to a unified vector space. The L2 distance of the FAISS method is used as the similarity function to obtain the correlation score between the representation code of the questions to be retrieved and the representation code of the text. The relevant text data is filtered out according to the correlation score threshold setting.

[0030] Furthermore, the multi-step dynamic reasoning retrieval process is expressed as follows:

[0031]

[0032] Among them, L θ (x) represents the pre-trained large model; θ represents the model parameters; x represents the query to be input into the pre-trained large model; y represents the expected output, which is determined by the expert; z1,...,z n Represents the acquired cognitive cue knowledge, iteratively outputting the retrieval data from the 1st to the nth time; represents the cognitive association process based on the RAG keyword entity tree, F represents the RAG keyword entity tree, F = {f1,f2,...,f n} contains scientific data keyword entities and the association structure between entities, κ = {k1, k2, ..., k n} is the retrieval fusion process of different modal data, A={a1,a2,...,a n ,ai ∈π} represents the set of keyword entity association actions, including association enhancement, association decomposition, association reduction, and association extinction; Indicates the positive and negative feedback from external experts on the output retrieval data.

[0033] A second aspect of the present invention provides a multimodal scientific data retrieval system based on summary-assisted cognitive enhancement, comprising:

[0034] A multimodal data unified representation module, which is used to obtain multimodal scientific data and convert the multimodal scientific data into a unified feature coding representation;

[0035] A multimodal data retrieval module is used to extract keywords for the search question; perform multimodal data retrieval based on the keywords of the search question to obtain related multimodal data, including: retrieve relevant text data from a text library based on the feature coding representation of the keywords of the search question, associate the feature coding representation of other modalities based on the feature coding representation of the related text data, and obtain scientific data of other modalities;

[0036] The dynamic reasoning retrieval module is used to construct a RAG keyword entity tree for scientific data based on structured abstracts and their corresponding keywords and description information; based on the associated multimodal data, the multimodal retrieval process is optimized according to the cognitive prompt knowledge output by the RAG keyword entity tree, and the final retrieval content is obtained through a multi-step dynamic reasoning retrieval process.

[0037] A third aspect of the present invention provides a computer-readable storage medium.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as described above.

[0039] A fourth aspect of the present invention provides a computer device.

[0040] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as described above are implemented.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. Aiming at the problem that scientific data retrieval lacks effective multimodal data knowledge fusion and associated retrieval methods, resulting in difficulty in data retrieval and poor semantic logic, the present invention constructs a unified representation module for multimodal data, converts multimodal scientific data such as text, tables, and structure diagrams into a unified feature coding representation, realizes cross-modal data fusion and strong associated representation, and ensures the rationality of semantic reasoning of output retrieval content;

[0043] 2. Since scientific data resources have problems such as strong professional terminology, complex data drill-down associations, and difficulty in tracing data sources, the present invention combines scientific data domain knowledge with a structural guidance mechanism, and uses a predefined structural template containing a unique ID, keywords, and semantic descriptions to guide the large model to generate retrieval knowledge containing key knowledge traceability, while improving the transparency and accountability audit tracking of multimodal scientific data;

[0044] 3. As scientific data involves many fields, has a large data base, and is updated quickly, traditional retrieval models are difficult to accurately and comprehensively extract the fusion knowledge and knowledge associations between data, resulting in the application trend of "big data, low value" for scientific data due to retrieval restrictions. In response to the above problems, the present invention constructs a multi-modal scientific data retrieval model with multi-path guidance of directional thinking chains, and guides the large model to realize multi-perspective semantic association retrieval through cognitive prompt knowledge enhancement combined with multi-step dynamic reasoning, thereby improving the data resource retrieval performance in the field of scientific data.

[0045] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 is a flow chart of a multimodal scientific data retrieval method based on summary-assisted cognitive enhancement provided by an embodiment of the present invention;

[0048] Figure 2 It is a multimodal data unified representation module provided by an embodiment of the present invention;

[0049] Figure 3 It is a multi-modal enhanced retrieval framework diagram for extracting keywords provided by an embodiment of the present invention;

[0050] Figure 4 It is a schematic diagram of multi-step dynamic reasoning retrieval based on cognitive prompt knowledge provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0052] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0054] In order to solve the problem mentioned in the background technology of this application, the existing technology mainly solves the complex knowledge association, but ignores the challenge of scientific data with multimodal data types. The difficulty in integrating heterogeneous data knowledge leads to inconsistent data representation, which in turn increases the difficulty of understanding the retrieved data.

[0055] First, the present invention constructs a multimodal scientific data semantic representation model based on a large model, converts multimodal scientific data such as text, tables, and images into a unified semantic representation, thereby realizing cross-modal data fusion and association representation; at the same time, it combines professional domain knowledge with a structural guidance mechanism, and uses a predefined structural template to guide the large model to generate a structured summary containing key knowledge, thereby improving the multi-perspective semantic association representation of multimodal scientific data. Subsequently, based on the knowledge such as keywords and descriptive information provided by the structured summary, a multimodal scientific data retrieval model with multi-path guidance of a directional thinking chain is constructed. Through cognitive prompt knowledge enhancement combined with multi-step dynamic reasoning, the large model is guided to achieve multi-perspective semantic association retrieval, thereby improving the data resource retrieval performance in the scientific data field. Finally, experiments were carried out based on a provincial scientific data management system. The results show that the performance of this method in multiple business evaluation indicators is better than the current mainstream method, which verifies the efficiency and feasibility of the proposed method, and can provide technical support for the hierarchical and classified opening of scientific data resources, and further strengthen the development and utilization of data resources of scientific and technological plan results by providing convenient and reliable public data acquisition and sharing services.

[0056] like Figure 1Based on multimodal scientific data such as text, tables, and images, the BERT model and ViT model are used to uniformly represent the multimodal data. After block division, the Transformer Encoder model is used to embed these blocks into vectors, put all vectors into the index and create an LLM prompt, so that the model can answer the user's query based on the context found in the search step, thereby constructing a multimodal scientific data retrieval method based on summary-assisted cognitive enhancement, thereby realizing cross-modal data fusion and associated representation. Its core framework consists of indexing, retrieval, and generation, and mainly includes the following three steps:

[0057] (1) Indexing: This process is usually performed offline. The original documents or data are cleaned and divided into blocks. The block knowledge is then used to generate semantic vectors through an embedding model and an index is created.

[0058] (2) Retrieval: For the question input by the user, the same embedding model is used to calculate the vector similarity between the question embedding and the semantic vector, and then the top N document blocks with the highest similarity are selected as the enhanced context information of the current question;

[0059] (3) Generation: The original question and N document blocks are merged into a new prompt, which is then fed into a large language model to obtain information to answer the question. Based on the built-in keyword entity tree, related information is merged into the prompt through feedback to achieve multi-round iterative scientific data retrieval.

[0060] Embodiment 1

[0061] This embodiment provides a multimodal scientific data retrieval method based on summary-assisted cognitive enhancement, comprising the following steps:

[0062] Step 1: Obtain multimodal data;

[0063] In this embodiment, the multimodal data comes from the data resources of the scientific data center, including text, table and image data, specifically including various data resources of business applications such as cloud platform intersection data, reported project data, experimental data sets, images, documents, etc.;

[0064] Step 2: Unify the acquired multimodal data;

[0065] Input multimodal data into a unified representation module, transform multimodal scientific data such as text, tables, and images into a unified feature coding representation, and realize cross-modal data fusion and correlation representation.

[0066] The unified representation module includes semantic segmentation and entity representation, a multimodal encoder, a text guidance unit and a reduction unit.

[0067] This embodiment uses image data and text data as examples, using the pre-trained BERT model as the text encoder and the ViT model as the image encoder. Figure 2 As shown:

[0068] The specific steps include:

[0069] Step 201: semantically segment and entity-represent the acquired multimodal data to obtain segmented and represented multimodal data;

[0070] Scientific data texts often have clear and standardized structural divisions, and their internal knowledge is systematic and logically related. In order to maintain the integrity of contextual semantics and the logic between text structure levels, this embodiment uses a structure-aware segmentation method to semantically segment sentences, paragraphs, sections or chapters of scientific data texts, so that the text blocks not only meet the LLM context window size limit, but also maintain the internal logic between the upper and lower blocks and the relevance of chapter titles.

[0071] The specific method is: define the text block as a "block entity type", including id, keywords and main text description. Similarly, define the "image entity type" based on the text extracted from the image data, including id, keywords and image-related description.

[0072] The "block entity type" id contains the article ID representing the globally unique text ID, the para Code representing the paragraph code in the text, and the sequential code id InPara representing each text block in the paragraph, thereby establishing a bidirectional relationship between the original text and its text blocks, ensuring the semantic structure association and content sequence logic of the text blocks, and at the same time supporting the association navigation and traceability between different knowledge granularities.

[0073] The "image entity type" id contains the image ID representing the globally unique image ID and the Struc Code representing the structural sequence code in the image, thereby establishing a bidirectional association between the internal entities of the image and the text blocks, ensuring the semantic correspondence and logical sequence association between the internal entities of the image and the text blocks.

[0074] Step 202: Encode the multimodal data after block representation to obtain multimodal features;

[0075] This example uses the pre-trained BERT base model to convert the block-represented text T into a token sequence X t ={t1,t2,...,t s}, the image I after block representation obtains the block sequence X p ={p1,p2,...,p r}, X pInput the ViT-B / 32 model with two multi-layer perceptrons pre-trained on ImageNet-1k to obtain image features X v ={v1,v2,...,v r}, text feature X t and image features X v As the input of the modal graph generation module GA, the text features obtained and image features

[0076] Step 203: Associating the multimodal features to obtain multimodal fusion features;

[0077] The text features are associated with the image features to realize the description of the image features by the text features, and at the same time, the image features enhance the logical correlation between the text features.

[0078] In this embodiment, the text keyword features are first associated with the image keyword features through sparse attention, and the feature vector correlation is calculated using a similarity function, that is, a 1d-CNN and pooling layer are connected to fuse all features, and similarity matching is performed, and the related features with high correlation are aligned and fused to obtain a fused feature with a unified representation and an associated metric representation, and then associated with the text features through a self-attention mechanism to obtain a fused feature with a unified representation and a sufficient text description.

[0079] Step 204: Reduce the dimension of the fused features by stacking attention layers and fully connected layers to obtain a multimodal unified coding feature representation for scientific data classification.

[0080] In addition, in order to implement the consistency constraint of feature representation, in this embodiment, the loss function loss uses relative L2 to construct an encoding distance evaluation function to measure the degree of fusion of graph features and text features. According to the feedback of the loss function, the parameters of the multimodal encoder, multimodal feature association and reduction unit are adjusted to bring the feature data with consistent representation closer to each other.

[0081] Step 3: Perform multimodal retrieval enhancement on the retrieval question, combine professional domain knowledge with the structure guidance mechanism, and generate a structured summary containing key knowledge;

[0082] Since scientific data resources have problems such as strong professional terminology, complex data drill-down associations, and difficulty in data fusion and utilization, this embodiment combines scientific data domain knowledge with a structure guidance mechanism, uses predefined structure templates, guides the large model to generate structured summaries containing key knowledge, and improves the multi-perspective semantic association representation of multimodal scientific data.

[0083] like Figure 3 As shown, the specific steps include:

[0084] Step 301: extract keywords from the search question;

[0085] In this implementation, a pre-trained large model is used to extract keywords for the search questions, including the name of the scientific and technological plan, project name, project undertaking unit, project execution period, time, location, scientific and technological terms, etc.

[0086] For example, the ChatGPT model is used to extract keywords from questions through prompt engineering. The prompt template is as follows:

[0087] Prompt=f"″"You are a keyword extractor. You will get a text containing keywords. Your task is to accurately identify and extract each keyword in the provided text. Your reply must be a keyword list. Each element in the list is an extracted keyword. You cannot reply with anything else not mentioned above.

[0088] [Response format]: Keyword 1, Keyword 2, ...

[0089] Here are two examples:

[0090] [Enter text]: What are the applicants and conditions for the 2024 Provincial Employee Innovation and Creativity Competition?

[0091] [Output keywords]: 2024, provincial employee innovation and efficiency competition, applicants, and conditions.

[0092] [Input text]: In the 2013-2023 science and technology plan projects, what are the names of the projects related to modal fusion?

[0093] [Output keywords]: 2013-2023, science and technology plan project, modal fusion, project name.

[0094] Please refer to the above example to extract keywords from the following input text:

[0095] [Input text]: {text}

[0096] [Output keywords]: ""

[0097] Step 302: Retrieve relevant text data from the text library based on the extracted keywords, associate the feature coding representations of other modalities with the feature coding representations of the relevant text data, and obtain scientific data of other modalities;

[0098] Retrieve relevant texts from the text library, that is, perform a unified multimodal data representation on keywords, obtain feature codes, and use enhanced retrieval to retrieve the feature codes of text data resources in scientific databases, including:

[0099] The multimodal data unified representation module is used to represent the questions to be retrieved and the massive texts as a unified representation code and map them to a unified vector space. Therefore, there will be overlapping areas of semantically related information in the vector space. This embodiment uses the L2 distance of the FAISS (Facebook AI Similarity Search) method as the similarity function to obtain a correlation score that measures the representation code of the questions to be retrieved and the representation code of the text, and selects relevant texts according to the correlation score threshold setting.

[0100] To solve the problem that the search results are poor due to the low frequency of keywords in the search questions, the keywords extracted in step 301 are added to the end of the search questions, thereby improving the recall rate of relevant content.

[0101] Subsequently, based on the obtained text representation code and the coding distance evaluation function of the unified representation module of multimodal data, the other modal data such as related graph data and video data are obtained.

[0102] Step 303: sorting the retrieved content by re-sorting to obtain re-sorted retrieved content;

[0103] In this embodiment, a re-ranking model is used to rank the retrieved recall content. The model is based on the pre-trained BERT model and is trained based on massive scientific data texts to enhance the applicability to documents in the field of scientific data. In the re-ranking model, the text mapping link is abandoned, and the multi-modal data of the questions to be retrieved and the recalled content are directly represented uniformly and connected with the separator "[SEP]", and the start symbol "[CLS]" is added as the input of the BERT model, and the similarity score between the retrieved recall content and the questions to be retrieved is output, and the recommendation list of multi-modal data is output according to the score.

[0104] The above scheme combines dense retrieval and sparse retrieval methods by adopting a multimodal enhanced retrieval strategy, thereby ensuring the accuracy of the retrieval while taking into account the comprehensive coverage of the recalled content.

[0105] Step 4: Construct a RAG keyword entity tree for scientific data based on the indicator entities, entity associations, logical rules and summaries of scientific data;

[0106] Currently, more and more scientific data are being aggregated, collected, and archived. Integrated data covers many fields, has a large data base, and is updated quickly. Traditional retrieval models are difficult to accurately and comprehensively extract the fused knowledge and knowledge associations between data, resulting in scientific data presenting an application trend of "big data, low value" due to retrieval restrictions.

[0107] In response to the above problems, this embodiment constructs a multimodal scientific data retrieval model with multi-path guidance of directional thinking chains based on structured abstracts and their corresponding keywords, descriptive information, etc., and guides the large model to realize multi-perspective semantic association retrieval through cognitive prompt knowledge enhancement combined with multi-step dynamic reasoning, thereby improving the data resource retrieval performance in the scientific data field.

[0108] Specifically, a pre-trained large model such as ChatGPT is used to generate summaries based on keywords and related recall content, combined with retrieval-augmented generation (RAG) to use a tree structure to represent the keyword association structure in scientific data, and through data iterative feedback recall between the pre-trained large model and RAG, multi-step dynamic reasoning retrieval is achieved. The specific architecture diagram is as follows Figure 4 As shown:

[0109] The multi-step dynamic reasoning retrieval consists of module 1 and module 2. Module 1 is composed of a parameterized pre-trained large model, which obtains reasoning retrieval content, keywords and generates summaries based on cognitive prompt knowledge and recall content association feedback; module 2 builds a cognitive prompt tree based on scientific data knowledge entities, entity associations, logical rules and recall retrieval content keywords of module 1, and gives new cognitive prompt knowledge feedback to system 1 based on its prompt pointing and associated keywords.

[0110] Through iterative interaction of data between the two channels, multi-step dynamic reasoning retrieval is realized, which can provide scientific data content retrieval such as decomposition, evaluation, analysis and trend exploration. Each step of summary generation is integrated into the output after scientific data-related knowledge, thus solving complex retrieval tasks and having the advantages of being explainable, perfectible and correctable.

[0111] The specific steps include:

[0112] Step 401: converting the original text, table, image and other data in the scientific database into a unified representation according to the process of step 2;

[0113] Step 402: According to the section titles corresponding to the text directory, each chapter is regarded as a text block, and ChatGPT is used to extract the keyword unified representation of each text block unified representation and the directory title unified representation, and the keyword unified representation whose word frequency meets the threshold is used as the keyword entity unified representation, and the keyword entity unified representation is used as the unified representation of the text block;

[0114] Step 403: Associate the keyword entity unified representation with other entity unified representations associated with the directory, and associate it with the same keyword unified representations as other text entities, thereby completing the construction of a primary scientific data-oriented RAG keyword entity tree.

[0115] Step 404: Based on the primary scientific data-oriented RAG keyword entity tree, clustering is performed according to unified representations (including keyword entity unified representations, description unified representations, and context association unified representations), and a text summary unified representation of the clustering clusters is generated, thereby completing the construction of the scientific data-oriented RAG keyword entity tree from the bottom up.

[0116] Among them, the nodes that are clustered together are sibling nodes; the cluster features of the clustered sibling nodes are represented by the parent node, which contains the text summary of the cluster, and the clustered sibling nodes in the cluster represented by the parent node are the child nodes of the parent node. This structure loads blocks representing different levels of context into the context of LLM, so as to simplify the number of nodes and improve the relevance of keyword retrieval, thereby efficiently answering questions at different levels.

[0117] Among them, clustering uses the Bayesian information criterion of the Gaussian mixture model to determine the optimal number of clusters, and a Gaussian mixture algorithm based on a probability threshold is used to perform recursive similarity clustering embedding until the similarity is less than the set minimum clustering threshold or until the number of unique clusters becomes 1.

[0118] The text summary extraction process of the parent node is as follows: starting from the root level of the tree, and retrieving k documents of its child nodes according to the cosine similarity of the unified representation embedding, combining the extracted keyword entity unified representation, description unified representation, and context association unified representation, and creating a text list containing the root document and summary as the text summary of the parent node.

[0119] This text is then stored into the vector storage.

[0120] In addition, various question types captured in the outcome question answering, such as true or false, summary, short answer, etc., are recorded as given questions and answers, thereby providing examples of questions created by human experts for module 1.

[0121] In the multi-step dynamic reasoning retrieval process, the RAG keyword entity tree is used to enhance the upper and lower level keywords of keyword association. The upper and lower level keywords are combined with their source scientific data text data to form associated cognitive prompt knowledge feedback to module 1.

[0122] Step 5: Optimize the multimodal retrieval process based on the cognitive hint knowledge output by the RAG keyword entity tree, and obtain the final retrieval content through a multi-step dynamic reasoning retrieval process.

[0123] The self-learning multi-step dynamic indexing reasoning process is essentially a process in which the large model integrates heuristic information based on the acquired associated cognitive prompt knowledge, and decomposes and eliminates invalid associated entities in the RAG keyword entity tree based on the number of retrieval knowledge updates and the positive and negative confirmation of the output results, thereby comprehensively improving the multimodal scientific data retrieval capabilities.

[0124] Multi-step dynamic index inference is defined as follows:

[0125]

[0126] Module 1: L θ (x) represents the pre-trained large model; θ represents the model parameters; x represents the query to be input into the pre-trained large model; y represents the expected output, which is determined by the expert; z1,...,z n It represents the retrieval data from the first to the nth time that is iteratively output by module 1 based on the cognitive hint knowledge obtained from module 2;

[0127] Module 2: It represents the cognitive association process based on the RAG keyword entity tree, which is specifically expressed as follows:

[0128] (1) F represents the RAG keyword entity tree, F = {f1,f2,...,f n} contains scientific data keyword entities and the association structure between entities, which can be used to guide the large model to output structured retrieval knowledge. f represents the keyword entity;

[0129] (2)κ={k1,k2,...,k n} is the retrieval fusion process of different modal data, specifically text encoding association graph encoding, obtaining association graph / video data;

[0130] (3) A = {a1, a2, ..., a n ,a i ∈π} represents the set of keyword entity association actions, including association enhancement, association decomposition, association reduction, and association extinction. The specific positive and negative feedback adjustment is confirmed by external experts on the output retrieval data;

[0131] (4) It indicates the positive and negative feedback from external experts on the output retrieval data;

[0132] The output of the multi-step dynamic index inference process is z1,...,z n , where z i ={f i ,k i ,a i ,r i}. Each step of the reasoning process They are all determined by the pre-trained large model, cognitive prompt knowledge, positive and negative feedback from external experts, and multimodal knowledge fusion. That is, the retrieval content organization response is determined by the pre-trained large model L in module 1. θTo achieve this, Module 2 constructs a self-learning and updated RAG keyword entity tree through cognitive prompt knowledge and positive and negative feedback from external experts, constructs an iterative prompt, retrieval, feedback, decision-making and updated structured dynamic index reasoning process, and continuously learns and optimizes the multimodal retrieval process of scientific data.

[0133] Experimental results and analysis

[0134] The experimental data depends on the opportunity of a provincial science and technology department to build a scientific data management system. The data sources include two parts. One part is based on the existing scientific and technological project data of the science and technology cloud platform, including material data such as project application, review, process management and final acceptance. The present invention extracts basic directory information related to scientific and technological plan projects and achievements, such as the name of the scientific and technological plan, the name of the project, the project undertaking unit, the project execution period, etc., and forms a scientific and technological project database, an expert database, etc. through extraction. Among them, a total of about 53,000 scientific and technological plan projects (including platforms and talents) were established from 2013 to 2020, with an average of about 7,600 projects per year; the other part of the data source is research data submitted in accordance with the national scientific data policy and the scientific data submission standards. This type of data belongs to the fields of natural sciences, engineering technology sciences, etc., based on data generated by basic research, applied research, experimental development, etc., as well as original data and its derivative data obtained and used for scientific research activities through observation, monitoring, investigation, inspection and testing, etc., and its types include but are not limited to audio, video, pictures, tables, text, etc. In addition, we collected questions and answers generated during various project defenses to create a dataset, manually checked the generated questions and answers, removed duplicate questions, and built a dataset containing 12,786 scientific data question-answer pairs. The dataset was divided into training and test sets in a ratio of 7:3. During the experiment, all prompts were evaluated using the OpenAI API's gpt-3.5-turbo-16k model.

[0135] Comparative analysis of methods

[0136] The present invention is based on the development history of RAG for large-scale language model retrieval output optimization. The selected comparative experimental framework types include basic LLM, Naive RAG combined with basic LLM, Advanced RAG combined with basic LLM, Modular RAG combined with basic LLM, and the method of the present invention, and the commonly used optimization methods are used for comparison based on different framework types.

[0137] The present invention uses manual evaluation accuracy and recall as the model retrieval performance evaluation indicators. Manual evaluation accuracy is to manually judge whether all the answers generated by the model retrieval are correct or not compared with the standard answers, that is, to calculate its accuracy (ACC):

[0138]

[0139] The present invention uses the Rouge-n indicator to evaluate the integrity of the answer generated by the model retrieval. The characters in the retrieval-generated answer and the standard answer are taken as basic elements, and the overlap rate of the basic elements of the two is calculated, namely, Rouge-n:

[0140]

[0141]

[0142] Among them, TP means that all the answers generated by the manual judgment model are correct, N means the number of samples in the test set, and recall i represents the percentage of overlapping characters between the i-th retrieval-generated answer and the standard answer i in the test set, and Rouge-n is the recall of all samples in the test set. i average value.

[0143] Table 1 shows the experimental comparison results of the accuracy of the above models on the test set and the Rouge-n indicator:

[0144] Table 1 Comparison of indicators in the test set

[0145]

[0146] Taking into account the domain expertise and semantic association complexity of scientific data, experiments show that the improvement of the large model combined with the NaiveRAG method is about 10% compared with the baseline without RAG, indicating that the scientific data domain as a retrieval prompt can effectively improve the retrieval performance of the large model; the large model combined with Adaptive RAG is significantly better than Naive RAG, indicating that the semantic association complexity between scientific data professional terms, including entity association, word meaning extension and textual implication, in the scientific data question-answering dataset, the diversity, ambiguity and semantic connection of professional terms require further alignment of data representation or retrieval association, so the sliding window optimization configuration, fine-grained segmentation and metadata merging can improve the quality of the indexed content, make the user's original question clearer, and clarify the retrieval target. Compared with Naive For the RAG method, the two indicators are improved by 11.37% and 11.31%, 12.23% and 13.80%, and 21.51% and 20.56% respectively. Among them, the metadata merging method has a significantly higher improvement effect than the other two methods, indicating that in the scientific data question-answering dataset, the consistency of data representation has a greater impact on the retrieval results; the test results of the large model combined with Modular RAG on the scientific data question-answering dataset are better than those of other RAG methods, the rewriting module is equivalent to the metadata merging effect, and the adaptive feedback mechanism improves the two indicators by 4.65% and 3.18% respectively compared with the metadata merging, indicating that the Modular RAG framework inserts an adaptive feedback mechanism, which can effectively perform index prompt reasoning on LLM based on the summarized scientific data sample set. This method is suitable for knowledge complex association reasoning retrieval tasks. The method of the present invention belongs to the Modular RAG framework inserted adaptive feedback mechanism. Since the scientific data types include but are not limited to multimodal formats such as audio, video, pictures, tables, text, etc., the processing method of unified data representation in Adaptive RAG is borrowed. By constructing a unified representation of multimodal data, multimodal entity association, ambiguity elimination, and association expansion are realized. Its ACC and Rouge-n are compared with the Modular RAG framework inserted adaptive feedback mechanism. Improved by 4.09% and 2.55% respectively, the experimental data illustrate the applicability of the model constructed by the present invention in multimodal scientific data retrieval.

[0147] Ablation comparative analysis

[0148] In the actual application environment of scientific data, the traceability of data sources is crucial, and it is necessary to ensure that each piece of shared data provided to the outside has clear traceability information of the source. In order to improve the reasoning logic and semantic relevance of scientific data, the present invention constructs data links and associations between entities in the basic semantic segmentation and entity representation links, and provides each entity with a unique ID code and text information description. Vectors are generated based on keywords and text information descriptions for similarity matching, thereby achieving knowledge alignment and source traceability. In order to illustrate the effectiveness of the method of the present invention, the RAGAS prompt evaluation framework is used to evaluate the content relevance and contextual logic of the retrieved data. Specifically, it includes three types of evaluation indicators, namely:

[0149] (1) Evaluate the fidelity of the answer: LLM is used to decompose the retrieved knowledge into individual sentence descriptions. LLM determines whether the answer can be described in each sentence based on its verification function. The ratio of the number of verifiable sentence descriptions to the total number of descriptions is used as the "fidelity score", which indicates whether the retrieved knowledge is consistent with the context.

[0150] (2) Evaluate answer relevance: Input the retrieval knowledge into the LLM to regenerate the question, and calculate the vector cosine similarity between the generated question and the original question to verify the relevance of the retrieval content and the retrieval target.

[0151] (3) Evaluate context relevance: Directly use LLM to extract sentences related to the original question, and the number of extracted sentences / the number of all sentences in the retrieval context is used as the context relevance score.

[0152] Table 2 Comparative analysis of ablation experiments

[0153]

[0154] The experiment shows that ID coding has the greatest impact on evaluating context relevance, which directly affects the logic of semantic reasoning. Text information describes direct relationship knowledge granularity alignment, so it has the greatest impact on evaluating answer relevance; RAG keyword entity tree is used in LLM to improve retrieval reasoning based on logical association, so it can effectively improve the reasoning consistency in semantic enhanced indexing and retrieval, and evaluate answer fidelity, answer relevance and context relevance. Ablation experiments prove the applicability of the method in the logical association of scientific data retrieval.

[0155] Embodiment 2

[0156] This embodiment provides a multimodal scientific data retrieval system based on summary-assisted cognitive enhancement, including:

[0157] A multimodal data unified representation module, which is used to obtain multimodal scientific data and convert the multimodal scientific data into a unified feature coding representation;

[0158] A multimodal data retrieval module is used to extract keywords for the search question; perform multimodal data retrieval based on the keywords of the search question to obtain related multimodal data, including: retrieve relevant text data from a text library based on the feature coding representation of the keywords of the search question, associate the feature coding representation of other modalities based on the feature coding representation of the related text data, and obtain scientific data of other modalities;

[0159] The dynamic reasoning retrieval module is used to construct a RAG keyword entity tree for scientific data based on structured abstracts and their corresponding keywords and description information; based on the associated multimodal data, the multimodal retrieval process is optimized according to the cognitive prompt knowledge output by the RAG keyword entity tree, and the final retrieval content is obtained through a multi-step dynamic reasoning retrieval process.

[0160] Embodiment 3

[0161] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps in the multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as described above are implemented.

[0162] Embodiment 4

[0163] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as described above are implemented.

[0164] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0165] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0168] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal scientific data retrieval method based on summary-assisted cognitive enhancement, characterized in that: The steps include: Acquire multimodal scientific data and transform them into unified feature coding representation; Extract keywords for the search questions; Performing multimodal data retrieval based on keywords of the question to be searched to obtain related multimodal data, including: retrieving related text data from a text library based on feature coding representations of keywords of the question to be searched, and associating feature coding representations of other modalities based on feature coding representations of the related text data to obtain scientific data of other modalities; Construct a RAG keyword entity tree for scientific data based on structured abstracts and their corresponding keywords and description information; Based on the associated multimodal data, the multimodal retrieval process is optimized according to the cognitive hint knowledge output by the RAG keyword entity tree. After a multi-step dynamic reasoning retrieval process, the final retrieval content is obtained.

2. The multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as claimed in claim 1, characterized in that: The unified representation of multimodal data is converted into a unified feature coding representation, including: The acquired multimodal data is semantically segmented and entity represented to obtain multimodal data after segmentation and representation; Encode the multimodal data after block representation to obtain multimodal features; Correlate multimodal features to obtain multimodal fusion features; By stacking attention layers and fully connected layers, the fused features are reduced in dimension to obtain a multimodal unified encoding feature representation for scientific data classification.

3. The multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as claimed in claim 1, characterized in that: After obtaining the multimodal scientific data associated with the keywords of the question to be searched, the multimodal scientific data is reordered, specifically including: The query to be retrieved and the associated multimodal data are uniformly represented and connected with the separator "[SEP]", and the start symbol "[CLS]" is added as the input of the BERT model. The similarity score between the retrieved content and the query to be retrieved is output, and the recommendation list of the associated multimodal data is sorted according to the score and output.

4. The multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as claimed in claim 1, characterized in that: Based on the structured abstract and its corresponding keywords and description information, a RAG keyword entity tree for scientific data is constructed, including: According to the section titles corresponding to the text directory, each chapter is regarded as a text block; Extracting keyword unified representations from unified representations of each text block and unified representations of directory titles; The unified representation of keywords whose frequency meets the threshold is used as the unified representation of keyword entities, and the unified representation of keyword entities is used as the unified representation of the text block; The keyword entity unified representation is associated with other entity unified representations associated with the catalog, and the keyword unified representations that are the same as other text entities are associated with it, thereby completing the construction of a primary RAG keyword entity tree for scientific data; Based on the primary RAG keyword entity tree for scientific data, the unified representation of keyword entities, the unified representation of descriptions and the unified representation of context associations are clustered, and the unified representation of text summaries of the clustering clusters is generated, thereby completing the construction of the RAG keyword entity tree for scientific data from the bottom up.

5. The multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as claimed in claim 4, characterized in that: In the RAG keyword entity tree for scientific data, nodes that are clustered together are sibling nodes. The cluster characteristics of clustered sibling nodes are represented by the parent node. The parent node contains the text summary of the cluster, and the clustered sibling nodes within the cluster represented by the parent node are the child nodes of the parent node.

6. The multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as claimed in claim 1, characterized in that: The method of retrieving relevant text data from a text library according to the feature coding representation of the keywords of the question to be searched includes: The questions to be retrieved and massive texts are represented as a unified representation code and mapped to a unified vector space. The L2 distance of the FAISS method is used as the similarity function to obtain the correlation score between the representation code of the questions to be retrieved and the representation code of the text. The relevant text data is filtered out according to the correlation score threshold setting.

7. The multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as claimed in claim 1, characterized in that: The multi-step dynamic reasoning retrieval process is expressed as follows: Among them, L θ (x) represents the pre-trained large model; θ represents the model parameters; x represents the query to be input into the pre-trained large model; y represents the expected output, which is determined by the expert; z1,...,z n Represents the acquired cognitive cue knowledge, and the iterative output of the retrieval data from the 1st to the nth time; z i ={f i ,k i ,a i ,r i }; represents the cognitive association process based on the RAG keyword entity tree, F represents the RAG keyword entity tree, F = {f1,f2,...,f n } contains scientific data keyword entities and the association structure between entities, κ = {k1, k2, ..., k n } is the retrieval fusion process of different modal data, A={a1,a2,...,a n ,a i ∈π} represents the set of keyword entity association actions, including association enhancement, association decomposition, association reduction, and association extinction; Indicates the positive and negative feedback from external experts on the output retrieval data.

8. A multimodal scientific data retrieval system based on summary-assisted cognitive enhancement, characterized in that: include: A multimodal data unified representation module, which is used to obtain multimodal scientific data and convert the multimodal scientific data into a unified feature coding representation; Multimodal data retrieval module, which is used to extract keywords for the search questions; Performing multimodal data retrieval based on keywords of the question to be searched to obtain related multimodal data, including: retrieving related text data from a text library based on feature coding representations of keywords of the question to be searched, and associating feature coding representations of other modalities based on feature coding representations of the related text data to obtain scientific data of other modalities; Dynamic reasoning retrieval module, which is used to build a RAG keyword entity tree for scientific data based on structured abstracts and their corresponding keywords and description information; Based on the associated multimodal data, the multimodal retrieval process is optimized according to the cognitive hint knowledge output by the RAG keyword entity tree. After a multi-step dynamic reasoning retrieval process, the final retrieval content is obtained.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the multimodal scientific data retrieval method based on summary-assisted cognitive enhancement as described in any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the multimodal scientific data retrieval method based on summary-assisted cognitive enhancement are implemented as described in any one of claims 1 to 7.

Images