Comparative learning-based chapter-level pre-training scientific literature representation and query method

Abstract

The invention discloses a chapter-level pre-training scientific literature representation and query method based on comparative learning. The literature representation method comprises the following steps: 1) constructing a scientific literature pre-training model which comprises a semantic information representation learning module, a citation network representation learning module, an auxiliary learning sub-module and a semantic-citation information comparison learning module; 2) constructing a semantic information data set and a citation network about the scientific literature; 3) utilizing a semantic information representation learning module to generate semantic information corresponding to the scientific literature according to the semantic information data set of each scientific literature; 4) acquiring deep citation network information of the scientific literature from the citation network by using a citation network representation learning module; 5) minimizing the difference between semantic information and deep quotation network information in the same literature, and optimizing the scientific literature pre-training model through global comparative learning; and 6) inputting the scientific literature a to be processed into the optimized scientific literature pre-training model to obtain the representation of the scientific literature a.

Description

Technical Field

[0001] The present invention relates to the fields of literature mining analysis and artificial intelligence technology, and particularly to the implementation of a passage-level pre-training scientific literature representation and query method based on contrastive learning, aiming to effectively mine and analyze the deep information contained in scientific and technological literature by constructing and utilizing a passage-level scientific literature contrastive learning pre-training model. Background Art

[0002] In today's era of information explosion, a vast amount of literature resources are crucial for academic research and knowledge discovery. When conducting academic research and literature reviews, researchers increasingly rely on scientific and technological literature databases. As an important knowledge carrier, scientific and technological literature carries rich information and research results, and is of great significance for scientific research, academic exchanges, and technological innovation. With the rapid development of information technology and artificial intelligence, the information age has brought a vast amount of scientific and technological literature. According to statistics, the number of scientific research documents generated globally each year increases exponentially. As a type of scientific data, scientific literature not only contains rich knowledge but also has characteristics such as being vast, multi-source, and having complex correlations. How to quickly and accurately retrieve the required information from it has become a significant challenge for scientific researchers.

[0003] For a long time, these databases have relied on traditional literature retrieval techniques such as keyword indexing, metadata tags, abstracts, and titles. Although these techniques can alleviate the demand for information retrieval to a certain extent, they have deficiencies: keyword indexing is difficult to cover the profound meaning and rich knowledge of the literature, metadata tags may have errors, and the information content of abstracts and titles is limited, making it impossible to deeply mine the deep knowledge in the literature, which affects the retrieval efficiency and accuracy. At the same time, the diversity of literature, such as the multimodality of different research fields and research methods, as well as the inherent characteristics of high-dimensionality and sparsity of text information, further increases the difficulty of parsing and application. In traditional retrieval systems such as PubMed and Web of Science, researchers usually use keyword searches, and the system provides relevant literature by matching keywords. However, in actual scientific research work, literature retrieval often requires a more in-depth semantic analysis ability to identify the main contributions, research methods, conclusions, etc. of the literature. Therefore, there is an urgent need for new scientific and technological literature representation methods to improve the quality and efficiency of information retrieval.

[0004] Traditional literature representation methods have some limitations when dealing with high-dimensional, sparse, and multimodal scientific and technological literature data. The high-dimensionality mentioned here means that literature data usually contains a large number of features or attributes, such as authors, keywords, abstracts, full texts, etc., which will lead to an increase in the complexity of data representation and computational burden. Sparsity means that the features in the literature data are often sparsely distributed, that is, each literature has only a few non-zero features, which brings difficulties to data modeling and analysis.

[0005] In recent years, passage-level document representation has become an important direction in the research of applied natural language processing. The purpose of passage-level document representation research is to capture high-level semantic information in documents and achieve effective information fusion, thereby improving the performance of tasks such as document retrieval, document recommendation, and knowledge discovery. Passage-level document representation usually involves the understanding and representation of the full text of a document or its main parts (such as abstract, introduction, method, result, and discussion). With the continuous progress of computers and algorithms, researchers can now improve document representation methods from multiple perspectives.

[0006] Improvements in scientific document representation are used to improve the accuracy and relevance of retrieval, and also contribute to document analysis and knowledge mining. For example, deep learning methods can mine potential trends and patterns based on external information such as the citation situation of articles and the research fields of supervisors. Contrastive learning can learn document vectors in an unsupervised manner, enabling effective document similarity analysis without explicit labels.

[0007] In addition, passage-level document representation is particularly important in interdisciplinary research. In fields such as computational biology, drug discovery, and medical diagnosis, researchers not only need to extract information from the literature in their own fields but also retrieve and integrate information in related fields. High-quality document representation methods can promote the transfer and integration of knowledge between different disciplines, driving disciplinary development and innovation.

[0008] With the globalization of academic research and the increase in interdisciplinary cooperation, effective scientific and technological document representation methods are increasingly valued for mining, understanding, and communicating scientific research results. Good document representation methods can improve the work efficiency of scientific researchers, accelerate scientific discovery, and provide a solid foundation for technological innovation and academic communication. Therefore, exploring scientific and technological document representation methods based on contrastive learning is not only a need to cope with the current massive growth of scientific and technological documents but also an important means to mine knowledge and accelerate innovation. Summary of the Invention

[0009] To solve the above problems, the present invention provides a passage-level pre-training scientific document representation method and query method based on contrastive learning. The present invention integrates data to construct semantic information and citation networks about scientific documents, and uses neural network algorithms and contrastive learning to deeply mine the potential information of scientific documents. Effectively learn the semantic information and correlation information of scientific documents, and systematically compare the differences between the semantic information of scientific documents and the information in the citation network field. The methods and steps of the present invention will be introduced in detail from multiple aspects such as overall architecture, data acquisition and processing, document mining algorithms, result analysis and application, etc.

[0010] 1) The overall architecture of the representation method of the present invention is as follows:

[0011] The core of the present invention is a pre-training architecture that integrates information extraction, contrastive learning, and literature information analysis. The overall architecture of the system includes the following main modules:

[0012] 11) Data acquisition and processing module: Acquire and process scientific literature data, and convert the original literature data into structured data that is easy to analyze and further process. Construct a semantic information dataset about scientific literature based on the ID, title, and abstract information of scientific literature; construct a citation network about scientific literature based on the citation relationship and entity inclusion relationship between scientific literatures.

[0013] 12) Semantic information representation learning module: Input the constructed semantic information data about scientific literature into the semantic information representation learning module, and capture the deep semantic information of scientific literature through a hierarchical Transformer model. The semantic information of scientific literature obtained by the semantic information representation learning module includes token-level semantic information and link-level citation relationship information of scientific literature;

[0014] 13) Citation network representation learning module: Input the constructed scientific literature citation network into the citation network representation learning module, and learn and extract the deep citation network information of scientific literature through a GAT model.

[0015] 14) Auxiliary learning subtask module: In addition, based on the constructed semantic information dataset and citation network of scientific literature, two auxiliary task hierarchical masked language models and citation relationship prediction models are respectively used to extract the token-level semantic information l i and link-level citation relationship information c i of the literature to improve the fine-grained performance of pre-training.

[0016] 15) The present invention designs a global contrastive learning to minimize the difference between the deep semantic information extracted by the semantic information representation learning module and the citation network information captured by the citation network representation learning module in the same literature. By discriminating whether the two learned information belongs to the same literature, the quality of literature representation is further improved.

[0017] In summary, the scientific literature set to be processed undergoes data preprocessing to obtain a semantic information dataset and a citation network; the semantic information dataset and the citation network are respectively input into the semantic information representation learning module and the citation network representation learning module to obtain the deep semantic representation and deep citation network representation of scientific literature; under the framework of global contrastive learning, the deep semantic representation and deep citation network representation are gradually aligned and optimized through contrastive learning, and at the same time, affected by two auxiliary learning tasks, continuous training finally forms a unified high-quality representation.

[0018] 16) Fine-tuning for downstream tasks:

[0019] In the application stage of this model, the general knowledge of the pre-trained model is transferred to specific tasks. Through the customization of downstream tasks and the fine-tuning of the model architecture, the pre-trained literature representation can be further optimized to adapt to specific tasks such as classification, citation prediction, and relation extraction. Here, a query method is specifically proposed, and taking this query method as an example, it is described how to fine-tune the literature representation generated by the model for application in downstream tasks.

[0020] Appendix Figure 1 Fig. shows the overall flowchart of the method for pre-training scientific literature representation based on contrastive learning according to the present invention.

[0021] The following is a detailed description of each module.

[0022] I. Data Acquisition and Processing Module

[0023] Data processing is a key link in the entire literature representation learning process, directly affecting the accuracy and efficiency of subsequent analysis. The main task of this module is to convert the original data into structured data that is easy to analyze and process and can be used for literature representation learning, including steps such as data acquisition, cleaning, standardization, and relationship modeling. The following is a detailed description of the data processing module.

[0024] 1) Data Acquisition

[0025] Data acquisition is the first step and also the most fundamental link of the present invention. The sources of data mainly include the following datasets:

[0026] 11) LitCovid dataset: The LitCovid dataset is a database maintained by the National Center for Biotechnology Information (NCBI) in the United States, specifically for COVID-19-related literature. This dataset aggregates scientific literature on coronavirus diseases and provides services for classifying, annotating, and organizing these literatures. LitCovid provides centralized access to 414,003 (and still increasing) relevant articles in PubMed.

[0027] 12) BioPMC dataset: The BioPMC dataset is a dataset related to the biomedical field, mainly involving biomedical literature in PubMed Central (PMC). It contains the text content of a large number of biomedical literatures from PMC, as well as related entity annotations (such as genes, proteins, compounds, etc.) and relationship annotations (such as gene-disease relationships, protein-compound relationships, etc.). All open-access articles in PubMed Central (PMC) in BioPMC are provided in BioC format, which provides a large number of full-text research articles for text mining and information retrieval research.

[0028] 13) Pubtator Dataset: The Pubtator dataset is a widely used dataset in the biomedical field for text mining and information extraction tasks. The Pubtator dataset contains a large number of biomedical literature abstracts and full texts, as well as related entity annotations (such as genes, proteins, compounds, etc.) and relationship annotations (such as gene-disease relationships, protein-compound relationships, etc.). These annotation information enables researchers to conduct text mining, information extraction, and knowledge discovery, thus accelerating the progress and discovery of scientific research.

[0029] 2) Data Construction

[0030] The original dataset mainly contains.xml files corresponding to each scientific literature. The.xml files mainly contain the title, ID, abstract, introduction, references, etc. of the current scientific literature. The data construction process extracts scientific literature information from the original XML files to construct a semantic information dataset and a citation network. It mainly includes the following two key steps:

[0031] 21) Constructing the Semantic Information Dataset:

[0032] Traverse the original dataset, scan the currently traversed.xml file, find the tags corresponding to PubMed Id, title, and abstract, and obtain the tag content, that is, extract the PubMed Id, title, and abstract of the currently traversed scientific literature as the original semantic information of a scientific and technological literature. Write all the original semantic information of the literature into a.csv file as the original semantic information dataset input to the model.

[0033] 22) Constructing the Citation Network:

[0034] Traverse the original dataset, scan the currently traversed.xml file, find the Referencelist tag of the current scientific literature, obtain the reference list of the current scientific literature, traverse the reference list to obtain all citation relationship pairs, record the relationship "1" between the two scientific literatures corresponding to each citation relationship pair, and establish an edge between the corresponding two nodes; at the same time, we also record all entity relationships included in the abstract of the current scientific literature. Traverse the abstract entity relationship records of all scientific literatures. If two scientific literatures contain the same entity relationship, obtain the entity inclusion relationship pair, record the entity inclusion relationship between the corresponding two scientific literatures as relationship "2", and establish an edge between the corresponding two nodes to obtain the cited network. We use the ID of each scientific literature as the "key" of the corresponding node of the current scientific literature, and use the set of all citation neighbor nodes and entity inclusion neighbor nodes with relationship "1" and relationship "2" with the current scientific literature as the "value", and store all the neighbor information of scientific literatures as a.pkl file, which is used as the original cited network for inputting the model.

[0035] 3) Data preprocessing

[0036] Since the original literature data is stored in formats such as XML and contains a lot of redundant information, directly using it for data mining may lead to problems such as reduced efficiency, affected accuracy, and increased noise. Unified preprocessing operations are required. Data cleaning and standardization are important steps in data preprocessing, aiming to remove or correct noise, redundancy, and incorrect data in the data. The main steps are as follows:

[0037] 31) Data cleaning: The cleaning operations mainly include removing duplicate data, handling missing values, correcting errors in the data, etc.

[0038] 311) Missing value handling: Check whether there are missing values in the extracted article ID, title, abstract, etc. If semantic information such as article id and title is missing, delete the corresponding labeled data.

[0039] 312) Removing redundant data: Clean all relationships, delete the scientific literature citation relationships / entity inclusion relationships that are not in the semantic information dataset, ensure that each scientific and technological literature has semantic information and its related cited network information; in addition, check whether there are self-citation relationships caused by the source data in the scientific literature citation relationships and delete them.

[0040] 32) Data standardization: Standardize situations such as incorrect data annotation and inconsistent data case in the data. The standardization process makes data from different sources have a unified format and structure for subsequent data mining.

[0041] The data after data preprocessing is more standardized and consistent, laying a foundation for subsequent representation learning.

[0042] Appendix Figure 2 Flowchart of the data acquisition and processing module.

[0043] II. Semantic Information Representation Learning Module

[0044] Scientific literature contains rich semantic information. In the semantic information representation learning module, the present invention mainly uses a hierarchical Transformer model to split scientific literature into tokens and encode them, thereby capturing the semantic information and context relationships of scientific literature. In the semantic information representation learning module, the multi-head attention mechanism is applied to amplify the signals of key tokens and reduce the influence of less important tokens.

[0045] III. Citation Network Representation Learning Module

[0046] Different from ordinary texts, scientific literature not only contains rich semantic information but also contains additional association information brought by the academic network, which affects its implicit spatial representation. In the citation network representation learning module, the present invention uses the GAT (Graph Attention Network) model to extract important relationships and information between documents from the citation network. The GAT model uses the attention mechanism to calculate the correlation between different documents and combines the representations of documents with weights according to the correlation.

[0047] IV. Auxiliary Learning Subtask Module

[0048] The present invention also proposes a pre-trained multi-task self-supervised learning objective to improve the overall performance and generalization ability of the model. This pre-training framework includes two auxiliary tasks: HMLM (Hierarchical Masked Language Model) and CRP (Citation Relationship Prediction). The hierarchical masked language model further introduces hierarchical context information to capture richer semantic levels in the text; the citation relationship prediction task is used to understand the citation relationship between documents, that is, to predict whether document A will cite document B based on the semantic content of document A, or the reason why document A cites document B. By combining global contrast learning with the hierarchical masked language model and the citation relationship prediction model to extract token-level semantic information and link-level citation relationship information, the pre-training framework of the present invention can share knowledge between different levels and tasks, enhance the generalization ability and representation ability of the model. This multi-task self-supervised learning method not only improves the understanding and generation ability of the model but also better addresses the challenges of various downstream tasks.

[0049] AppendixFigure 3 Shows the flowchart of the auxiliary learning sub-task module of the present invention.

[0050] V. Semantic-Citation Network Information Contrastive Learning Module

[0051] The present invention believes that based on the two most important factors of scientific literature: semantic information and citation network information, it is necessary to construct a pre-trained model that can simultaneously consider the semantic information and relevance information of scientific literature to support different types of downstream tasks. Using the contrastive learning method, the semantic information representation and citation network representation of the literature can be aligned with each other in the feature space to achieve a "contrastive" effect, that is, the semantic information and citation network information of the same literature should be closer in the representation space, while the semantic information and citation network information of different literatures should be as far away as possible. Through contrastive learning, the accuracy of literature understanding is improved, providing richer feature representations for subsequent literature analysis, recommendation and other tasks.

[0052] VI. Fine-tuning Applied to Downstream Tasks

[0053] Based on pre-trained representation learning, fine-tuning is a process of customizing the general literature representation output by the pre-trained model. By training on task-specific datasets and adjusting its model parameters, the representation of the literature can better serve specific downstream tasks such as classification, recommendation, retrieval, etc. For example, in a classification task, the output representation vector is further processed by a classification head (such as a fully connected layer) to output a class label. In a retrieval task, the output vector is used to calculate the similarity (such as cosine similarity) and compare it with other literature vectors. In addition, the present invention particularly proposes a query method: by calculating the Euclidean distance between the literature representations learned by the model, comparing the representation vector of the literature to be queried with the representation vectors of each literature in the literature library, calculating its similarity score, sorting the literatures in the literature library according to the similarity score, and returning several literatures with the highest similarity scores as the query results. This query method can efficiently identify the literatures most relevant to the literature to be queried in terms of content and citation network structure, providing accurate literature recommendations and auxiliary analysis for scientific researchers. In summary, fine-tuning can not only enhance the performance of the model on specific tasks, but also enable the model to show strong adaptability on new tasks.

[0054] The present invention also provides a query method, attached Figure 4 Shows the flowchart of the query method proposed by the present invention. Its steps include:

[0055] 1) Use the optimized pre-trained model of scientific literature to generate the representation of each literature in the literature library;

[0056] 2) Use the optimized pre-trained model for scientific literature to generate the representation of the literature to be queried and calculate the similarity with the representation of each literature in the literature library respectively;

[0057] 3) According to the similarity calculation results, select several literatures with the highest similarity as the query results of the literature to be queried and output them.

[0058] A server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the above method.

[0059] A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program realizes the above method when executed by a processor.

[0060] Compared with the prior art, the positive effects of the present invention are:

[0061] By introducing the combination of hierarchical Transformer and graph attention network (GAT), the present invention effectively integrates the semantic information of scientific literature and the correlation information of citation networks, thereby more comprehensively capturing the implicit spatial representation of the literature. Through the design based on contrastive learning, the model can simultaneously consider the relationship between the semantic encoding and citation encoding of the literature, which helps to more accurately judge the similarity and citation relationship between literatures, and improves the representation ability of scientific literature. In addition, the present invention also introduces the ideas of multi-task learning and self-supervised learning, and uses the hierarchical masked language model and citation relationship prediction task to further enhance the model's understanding of the literature content and citation network, and improve the quality of literature representation and the generalization ability of the model. Compared with the existing literature processing methods, the present invention can more effectively mine the multi-level information in scientific literature, provide richer literature representations, support a wider range of downstream tasks, such as research hotspot analysis, trend prediction, and literature recommendation, etc., thereby promoting the accuracy and practicality of scientific literature mining. Brief Description of the Drawings

[0062] Figure 1 It is the overall flowchart of the present invention.

[0063] Figure 2 It is the flowchart of the data acquisition and processing module.

[0064] Figure 3 It is the flowchart of the auxiliary learning sub-task module.

[0065] Figure 4 It is the flowchart of the query method proposed by the present invention.

[0066] Figure 5 It is the schematic diagram of the model framework structure of the present invention. Detailed implementation manners

[0067] To more clearly elaborate the objectives, technical solutions, and advantages of the embodiments of the present invention, the following will describe the technical solutions in detail with actual examples. Obviously, the described embodiments only represent some embodiments of the present invention, rather than all. According to this embodiment, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0068] An embodiment of the present invention provides a method for pre-training scientific literature representation based on contrastive learning, as Figure 1 shown, which mainly includes:

[0069] I. Scientific literature data processing

[0070] Data preprocessing mainly includes removing redundant data and constructing a semantic information dataset and a citation network for scientific literature

[0071] 1. The scientific literature.xml file contains a large amount of redundant information. Remove the redundant data in the scientific literature data to avoid affecting the effect of semantic information representation learning.

[0072] 2. Extract the PubMed Id, title, and abstract of the scientific literature as the original semantic information of a scientific and technological literature; construct a citation network in combination with the citation information and entity inclusion information of the scientific literature. In the citation network, each node represents a scientific literature, and each edge represents a citation relationship or an entity inclusion relationship between one scientific literature and another scientific literature.

[0073] Based on the above process, a semantic information dataset and a citation network for model training are obtained.

[0074] II. Constructing a pre-training model for text-level scientific literature representation learning

[0075] As Figure 5 shown, the scientific literature pre-training model constructed in the embodiment of the present invention mainly includes: a semantic information representation learning module, a citation network representation learning module, an auxiliary learning sub-task module, and a semantic-citation network information contrastive learning module.

[0076] In the pre-training model constructed in the present invention, we define two information sets of the semantic information and citation graph information of a constructed scientific literature L as: (1) Semantic information: S L ={title, abstract, …, paragraph N}, (2) Citation graph information: G L =(L, E), G LDenote the subgraph of the citation network constructed with the literature L as the center. We input the above two types of information into the corresponding feature extraction modules respectively and perform comparative learning by minimizing the difference between the output vectors. The goal of pre-training is:

[0077]

[0078] Then, in fine-tuning, various downstream literature tasks (such as citation prediction and domain classification) can be applied to optimize the pre-trained model.

[0079] In the semantic information representation learning module, the present invention uses hierarchical Transformers, including paragraph Transformers and literature Transformers, to extract the semantic information of a piece of literature. In the citation network representation module, the present invention uses a GAT encoder to capture citation network information; for generalization, the present invention designs multi-task and self-supervised learning objectives by automatically extracting supervision signals from unlabeled data. Next, each module will be described in detail.

[0080] 1. Semantic Information Representation Learning Module

[0081] Layout embedding uses the Transformer model to extract the semantic information of the literature, making it perform excellently in processing natural language text. Nowadays, variant models based on Transformer have been widely applied to training on large-scale datasets, especially in training large language models. The Transformer model first splits the input text into a series of n-grams and encodes these n-grams into tokens. By looking up the word embedding vocabulary, these tokens are converted into vector representations, and each vector contains the semantic information of a specific token. Each token interacts with other tokens within the context window to better capture the correlation between tokens.

[0082] It is worth noting that all tokens are visible to each other in each layer, that is to say, they can influence each other. To highlight the signals of key tokens and reduce the influence of less important tokens, the Transformer model adopts the multi-head attention mechanism. This mechanism allows the model to focus on key tokens while reducing the attention to irrelevant tokens. In this way, the model can better understand the important parts of the input text. A typical Transformer layer consists of alternating attention layers and feed-forward layers, and such a structure can perform repeated transformations on the vector representation. After multiple layers of transformations, the model gradually extracts richer and richer language information, including word meanings, syntactic structures, and contexts, etc. This process of layer-by-layer extraction enables the Transformer model to model more complex language relationships and better capture the deep meaning of the text, which can be written as:

[0083] H = NormLayer(X (l) ) + MultiHead(X (l) )

[0084] X (l+1) = NormLayer(H + F(H))

[0085] where is the input text of the l-th layer of the Transformer, s is the length of the input text, represents the dimension. NormLayer represents the normalization layer, and F represents a two-layer feed-forward layer with a ReLU activation function, and MultiHead represents multi-head attention:

[0086] MultiHead(X (l) = Concat(head1,…,head h ))W O

[0087]

[0088] where the matrices Q, K, and V are defined as the matrices whose i-th rows are the vectors q i , k i and v i respectively, that is:

[0089] q i = x i W Q

[0090] k i = x i W K

[0091] v i = x i W V

[0092] In summary, the output H of the attention calculation is concatenated and transformed using the output weight matrix W O ∈ R dh×d . Through the Transformer encoder, the final output embedding of a sequence can be obtained where each embedding contains the context information in the sequence. In the layout embedding, the present invention uses the paragraph Transformer to encode the text in the selected paragraph into a paragraph representation, and uses the literature Transformer to integrate multiple paragraphs into a literature representation. The paragraph Transformer and the literature Transformer are described in detail below:

[0093] 1.1 Paragraph Transformer

[0094] For each paragraph p j ∈ L i , assume its token sequence is {t1, t2,..., t |pi|}, and the corresponding token embeddings are {e1, e2,..., e |pi|}. In the proposed model, the present invention defines a new token [SOD] (start of document) and randomly initializes its embedding e [SOD] , which will be concatenated with the embedding sequence of p j into the paragraph Transformer. Then, the final output of [SOD] will be used as the representation of paragraph p j :

[0095]

[0096] where Transformer D () represents the forward function of the paragraph Transformer. The present invention uses the Word2vec method to initialize the input token embeddings. For each token, its input representation is constructed by adding its embedding to the embedding at its corresponding position in the text. Through the above process, multiple paragraph representations of the input document are obtained.

[0097] 1.2 Document Transformer

[0098] A document consists of multiple paragraphs, and each paragraph can be transformed into a corresponding representation vector by the paragraph Transformer. The document Transformer consists of multiple layers of bidirectional transformers and a mean pooling layer, which aggregates the paragraph representations into a document representation:

[0099]

[0100] where l i is the token-level semantic representation of document i. Pooling() represents the average of all paragraph embedding outputs. Transformer L () is the forward function of the document Transformer, which can collect information from other paragraphs through the self-attention mechanism and output the final document representation with different attention weights.

[0101] Through the above steps, we obtain the deep semantic information captured by the semantic information representation learning module mentioned above.

[0102] 2. Citation Network Representation Learning Module

[0103] Given a scientific document citation network First, the goal of the present invention is to extract the neighborhood information of scientific and technological literature from this network. GAT calculates the weights between nodes through the attention mechanism, and then performs a weighted sum on the node representations. By introducing the attention mechanism, the GAT model can adaptively model the relationships between nodes, thereby better capturing the information in the graph-structured data.

[0104] Due to memory limitations, traditional GAT takes the entire graph as the input of the model and is difficult to apply to large-scale graph data. In addition, the interconnected graph structure makes it impossible to perform parallel computing on the complete graph topology, making the propagation of GAT on large-scale graph data very time-consuming. Therefore, the present invention considers sampling subgraphs of G to construct the domain subgraph of scientific and technological literature a i denoted as G i which can retain the interactions and relationships between a i and other scientific and technological literatures in G. The citation embedding module uses the GAT model to encode G i into an embedding vector. The input of the GAT layer is the neighborhood nodes corresponding to the node a i corresponding to the i-th scientific literature, where N is the number of nodes and F is the number of features in each node. The GAT layer outputs a new set of node features (possibly with a different cardinality F′), parameterized by the weight matrix W ∈ R F×F′ through a linear transformation. Then GAT performs a self-attention mechanism on the nodes:

[0105] The attention coefficient represents the importance of the features of node a j to node a i . To make the coefficients comparable between different nodes, GAT normalizes them using the softmax function:

[0106]

[0107] The attention mechanism uses the LeakyReLU non-linear activation function with a negative input slope α = 0.4. The final output feature of each node is:

[0108]

[0109] Then, the present invention uses a readout function to aggregate all the obtained patch representations into a fixed-length graph-level representation as follows:

[0110]

[0111] where 0 ≤ l ≤ L C , represents the node a at the l-th layer of GATj the output hidden vector and is a neighbor in the l-th layer of G i Aggregation (l) (·) and Combine (l) (·) are composite functions of the l-th layer GAT, N(G i ) is the node set of G i . Finally, Readout(·) summarizes the output citation embedding, which represents the average function in the method proposed by the present invention.

[0112] Through the above steps, we obtain the citation network information extracted by the citation network representation learning module mentioned above.

[0113] 3. Auxiliary learning subtask module

[0114] 3.1 Hierarchical masked language model

[0115] According to the characteristics of the hierarchical Transformer proposed by the present invention, a hierarchical masked language model (HMLM) is proposed in model pre-training to further improve document representation. The traditional masked language model task randomly masks several tokens in the context and tries to predict them using the output embeddings of the encoder, while HMLM operates at a higher level and combines character-level and paragraph-level context information. Character-level context refers to masking certain characters or words in a paragraph or text, and then using the paragraph Transformer model to predict the labels at these masked positions. Paragraph-level context is to mask an entire paragraph in a paragraph or text, and the document Transformer model needs to fill in these missing paragraphs according to the context.

[0116] In pre-training, the hierarchical masked language model is used to train the hidden outputs of paragraphs and tokens of the hierarchical Transformer model. In this way, the hierarchical Transformer model can learn the bidirectional representation of the document by predicting character-level and paragraph-level masked positions. Such pre-training helps to extract the deep semantics and context relationships in the document, providing richer features for subsequent text understanding and downstream tasks. Therefore, given a scientific and technological document a iFor the semantic information dataset, the hierarchical masked language model realizes in-context learning through a dual masking mechanism. First, a part of the tokens in the paragraph are randomly masked to obtain the original masked tokens of the text representation sequence, and the masked tokens are replaced with [MASK]. The paragraph Transformer model predicts the masked characters based on the context within the paragraph; at the paragraph level, the entire paragraph content is masked, and the literature Transformer model reconstructs the overall representation of the masked paragraph through the context semantics of other paragraphs within the literature. Finally, through hierarchical context modeling, HMLM jointly utilizes local lexical associations and global paragraph dependencies to drive the hierarchical Transformer model to learn the bidirectional semantic representation of scientific literature from local vocabulary to global paragraph structure.

[0117] The loss of the HMLM task can be formulated as i the cross-entropy loss of all masked tokens in the paragraph of

[0118]

[0119] where t k represents the k-th masked token, and the true label is the one-hot encoding of t k , is the set of masked tokens in paragraph p j , c (ranging from 1 to V) is the traversal variable of the vocabulary index, and V represents the vocabulary size (i.e., the total number of all possible tokens that the model needs to predict). is the prediction probability of the model for token t k , usually generated by the softmax layer of the Transformer model, and is used to compare with the true label to calculate the cross-entropy loss.

[0120] 3.2 Citation Relationship Prediction Model

[0121] In addition to the token-level semantic representation, the present invention also uses citation relationship prediction as another auxiliary task. Citation prediction (CRP) is a key task in scientific literature analysis, aiming to predict which other literatures a literature will cite in the future, or predict which literatures will cite this literature. Citation prediction has important application values in fields such as academic research, scientific research trend analysis, and literature recommendation.

[0122] As mentioned above, in the citation representation learning module, a sampling strategy is used in each training step to extract the subgraph of a literature a i , so the link-level correlation between two literatures cannot be directly learned. Therefore, this model attempts to predict the number of hops between two literatures. Specifically:

[0123] Given node ai A sampled local reference graph G i , the model will randomly select 15% of the links among them as the input for the hop count prediction task, Denote the reference graph G from node a i by i the set of 15% of the links randomly sampled from r i,j refers to the direct link type from node a i to node a j . The output hidden vector of the L-th layer literature node a C of GAT is denoted as i The L-th layer is the last layer of GAT. The model inputs these representations into the linear transformation and softmax(σ(·)) functions: L C Therefore,

[0124]

[0125] represents the probability that the hop count of r ranges from 1 to H (this is the maximum hop count from a i,j to its farthest neighbor). The loss function of this task can be calculated as: i where

[0126]

[0127] is the one-hot encoding of the hop count of r r i,j , and H is the maximum hop count from a i to its neighbors in the local reference graph G i .

[0128] 4. Semantic-Citation Network Information Contrastive Learning Module

[0129] Contrastive learning is a method for formulating tasks for machine learning models to find similar and different things. Using this method, machine learning models can be trained to classify similar and dissimilar data samples. As mentioned above, we believe that the two most important factors for representing scientific literature are semantic information and citation network information, and there is a strong correlation between them. The present invention designs a global contrastive learning to maximize the similarity between the token-level semantic information l i (i.e., semantic embedding) captured by the semantic representation learning module and the link-level citation relationship information c i (i.e., citation embedding) extracted by the citation network representation learning module for the same literature, while minimizing the similarity between the semantic information l i in the same literature and the deep citation network information c iThe differences between them are used to distinguish whether they belong to the same document. The loss is calculated by the InfoNCE loss function to optimize the model that generates these embedding representations.

[0130] In summary, the core of the entire architecture is to optimize the total loss to improve the quality of document representations by combining global contrastive learning with two subtasks: hierarchical masked language model and citation relationship prediction. The overall architecture takes contrastive learning as the core goal, minimizing the differences between the semantic embeddings and citation embeddings of the same document, while maximizing the differences between the embeddings of different documents. Its loss is denoted as L GCL . In addition, the two auxiliary tasks are the hierarchical masked language model (HMLM) and citation relationship prediction (CRP). HMLM enhances the semantic representation ability of the model by randomly masking character-level and paragraph-level labels in the document and predicting the masked content. Its loss is denoted as L HMLM ; CRP learns the citation network features by predicting the number of hops between documents, and its loss is denoted as L HOP .

[0131] Combined with the above tasks for pre-training, the final loss of this model is expressed as:

[0132] L final (θ; D, G) = L GCL + λ1L HMLM + λ2L HOP

[0133] where λ1 and λ2 are hyperparameters used to balance the task weights. By continuously backpropagating to optimize the final loss, the model updates the embedding representations of semantics and citations, thereby achieving higher-quality document representations and enhancing its adaptability to downstream tasks. Based on the above process, the pre-training framework can share knowledge between different levels and tasks, enhancing the generalization ability and representation ability of the model. This multi-task self-supervised learning method not only improves the understanding and generation ability of the model but also better addresses the challenges of various downstream tasks.

[0134] Through all the above steps, we construct a passage-level pre-trained scientific document representation model based on contrastive learning, which can effectively generate high-quality representation vectors for documents. For any processed document a, after inputting it into the optimized document representation model, its representation vector v a can be obtained. The process of generating the representation vector is as follows:

[0135] ● The semantic information data of document a is input into the hierarchical Transformer for processing: the paragraph Transformer extracts multiple paragraph representations of document a, and the document Transformer aggregates the multiple paragraph representations into the semantic representation of document a, that is, the semantic embedding;

[0136] ● The citation network data of Document a is input into GAT for processing: First, subgraph sampling is performed on Document a, and the GAT model is used to encode the subgraph into an embedding vector. The embedding vectors of different subgraphs are aggregated to generate the embedding representation of the document in the citation network;

[0137] ● Next, the model fuses the semantic representation and citation embedding, and continuously updates and optimizes the final document representation vector v using global contrast learning a ; At the same time, two subtasks are used to assist contrast learning. The hierarchical masked language model (HMLM) combines character-level and paragraph-level context information, and uses the characteristics of the hierarchical Transformer to predict the labels at the masked positions, training the hidden outputs of the paragraphs and tokens of the hierarchical Transformer model. In this way, the hierarchical Transformer model can learn the bidirectional representation of the document by predicting the masked positions at the character level and paragraph level; The citation prediction task (CRP) attempts to predict the number of hops between two documents, learning the link-level correlation between the two documents, and ensuring that the generated citation embedding can accurately reflect the citation relationship and graph structure features between documents.

[0138] ● Finally, by combining global contrast learning with the two auxiliary learning subtasks, minimizing the sum of the losses of these three tasks, and continuously backpropagating to optimize the final loss, the model updates the embedding representations of semantics and citations to obtain the final high-quality document representation vector v a 。

[0139] III. Fine-tuning for Application to Downstream Tasks

[0140] After learning the semantic information and citation network information of scientific documents through the pre-training model for document-level scientific document representation learning constructed by the present invention, the model of the present invention can be fine-tuned to be applicable to different downstream tasks, mainly in the following two ways:

[0141] End-to-end mode: In this mode, we directly train the pre-training model and activation layer constructed by the present invention to fine-tune the downstream task The objective is:

[0142]

[0143] where θ ft and are the optimized parameters, and L ft is the loss function of the downstream task.

[0144] Feature-based mode: In this mode, we use the pre-training model constructed by the present invention as a feature extractor, and the downstream task will use the output features as input for training.

[0145] Each of the two modes has its own advantages and disadvantages. The end-to-end mode can further update the weights according to the downstream tasks, so as to extract more effective features. The feature-based mode is more efficient because it does not participate in the model training.

[0146] In the citation prediction task, the pre-trained model learns the semantics of the literature and the citation graph information. In the fine-tuning stage, it adapts to the task requirements by adding a task-specific output layer (e.g., a binary classifier), adjusting the input format (combining the literature representation obtained by the model and the graph embedding learned from the citation network to form an input vector), selecting an appropriate loss function (e.g., BCELoss binary cross-entropy loss), and optimizing the model.

[0147] In addition, the present invention particularly proposes a query method. The document to be queried and the documents in the document library are represented in vector form using the passage-level pre-trained scientific literature representation method based on contrastive learning proposed by the present invention, and efficient and accurate document retrieval is realized by using similarity calculation. As Figure 4 shown, first, the document representation model constructed by the present invention is used to generate the representation vectors of the document to be queried and all the documents in the document library. Then, the Euclidean distance between the representation vector of the document to be queried and the representation vector of each document in the document library is calculated, and the distance is converted into a similarity score. By comparing the similarity scores, the documents in the document library are sorted to obtain several documents most similar to the document to be queried. The specific steps are as follows:

[0148] 1. Calculate the Euclidean distance of the document representation vectors: First, the representation vectors of the document to be queried and each document in the document library are generated by the optimized document representation model. For the document to be queried a and each document d in the document library i , their representation vectors are respectively v a and Next, calculate the Euclidean distance between them:

[0149]

[0150] Among them, D i represents the Euclidean distance between the document to be queried and document d i , m is the dimension of the representation vector, and are the representations of document a and document d i in the j-th dimension respectively.

[0151] 2. Calculate the similarity score: According to the calculated Euclidean distance, the similarity score between documents can be expressed by converting the distance value into similarity. Generally, the similarity S i is inversely proportional to the Euclidean distance D i , that is:

[0152]

[0153] Thus, the smaller the distance, the higher the similarity score.

[0154] 3. Sort the documents in the literature library: According to the calculated similarity scores {S1, S2, …, S n}, sort the documents in the literature library in descending order to obtain the sorted document set D * :

[0155] D * = Sort(D, by S i )

[0156] Among them, the sorted document set D * is arranged in descending order according to the similarity scores.

[0157] 4. Return the query result: Select several documents with the highest similarity scores from the sorted document set D * (assuming the first k documents are selected) as the query result:

[0158]

[0159] Among them, is the final query result, including the k documents with the highest similarity to the document to be queried. Through this series of steps, the similarity score can be calculated based on the Euclidean distance between the document to be queried and the representation vectors of the documents in the literature library, and then the literature retrieval can be efficiently performed to return the most relevant documents as the query result.

[0160] This method can make full use of the semantic features and citation network features of the documents, and at the same time effectively combine the similarity calculation to efficiently screen out the most relevant documents to the document to be queried from the literature library, providing support for subsequent academic research or literature recommendation.

[0161] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those of ordinary skill in the art can modify or equivalently replace the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to what is described in the claims.

Claims

1. A method for representing scientific literature at the document level based on contrastive learning, the steps of which include: 1) Construct a scientific literature pre-training model, including a semantic information representation learning module, a citation network representation learning module, an auxiliary learning sub-module, and a semantic-citation information contrastive learning module; 2) Use the data acquisition and processing module to acquire scientific literature and convert it into structured data, and construct a semantic information dataset about scientific literature according to the ID, title, and abstract information of scientific literature, and construct a citation network about scientific literature according to the citation relationship and entity inclusion relationship between scientific literature; 3) Use the semantic information representation learning module to generate the semantic information of the corresponding scientific literature according to the semantic information dataset of each scientific literature; the semantic information of scientific literature includes the token-level semantic information and link-level citation relationship information of scientific literature; 4) Use the citation network representation learning module to obtain the deep citation network information of scientific literature from the citation network; 5) Minimize the difference between the semantic information and the deep citation network information in the same document, and optimize the scientific literature pre-training model through global contrastive learning; 6) For a scientific literature a to be processed, use the data acquisition and processing module to process the scientific literature a and input it into the optimized scientific literature pre-training model to obtain the representation of the scientific literature a; the representation of the scientific literature a includes the semantic information and citation network information of the scientific literature a.

2. The method according to claim 1, characterized in that The method for constructing the citation network: Take each scientific literature in the acquired scientific literature set as a node, traverse each scientific literature in the scientific literature set, obtain the reference list of the current scientific literature, traverse the reference list to obtain all citation relationship pairs, record the existence of relationship "1" between the two scientific literatures corresponding to each citation relationship pair and establish an edge between the corresponding two nodes; traverse the abstract entity relationship records of all scientific literatures, if two scientific literatures contain the same entity relationship, record the existence of relationship "2" between the corresponding two scientific literatures and establish an edge between the corresponding two nodes to obtain the citation network; Take the ID of the current scientific literature as the "key" of the node corresponding to the current scientific literature, and take the node set composed of the nodes having relationship "1" and relationship "2" with the current scientific literature as the "value" of the node corresponding to the current scientific literature.

3. The method according to claim 1, wherein Use the auxiliary learning sub-task module to extract the token-level semantic information of each scientific literature and predict the link-level citation relationship information of each scientific literature; the auxiliary learning sub-task module includes a hierarchical masked language model and a citation relationship prediction model; the hierarchical masked language model is used to extract the token-level semantic information of each scientific literature; the citation relationship prediction model is used to understand the citation relationship between documents and obtain the link-level citation relationship information of each scientific literature.

4. The method according to claim 3, wherein The method for the hierarchical masked language model to extract the token-level semantic information of each scientific literature is as follows: for the scientific and technological literature a i in the semantic information dataset, the hierarchical masked language model first masks a part of the tokens in the paragraphs of the scientific and technological literature a i to obtain the original masked tokens regarding the text representation sequence, replaces the masked tokens with [MASK], and the paragraph Transformer model predicts the masked characters based on the context within the paragraph; at the paragraph level, the entire paragraph content is masked, and the literature Transformer model reconstructs the overall representation of the masked paragraph through the context semantics of other paragraphs within the literature. Finally, through hierarchical context modeling, the HMLM jointly utilizes local lexical associations and global paragraph dependencies to drive the hierarchical Transformer model to learn the bidirectional semantic representation of scientific and technological literature from local vocabulary to global paragraph structure.

5. The method according to claim 1, characterized in that, Using the loss function L final (θ; D, G) = L GCL + λ1L HMLM + λ2L HOP Optimize the pre-trained model of the scientific literature; Among them, L GCL is the difference loss value between the semantic information and the citation network information in the same document, t j is the j-th original word or symbol randomly masked in paragraph p j where t k is the k-th masked token, is the set of masked tokens in paragraph p j of the scientific document, L i is the set of paragraphs of the scientific document, is the one-hot encoding of t k where is the predicted probability of the model for token t k where V is the vocabulary size, r i,j refers to the direct link type from node a i to node a j where is a set of 15% of the links randomly sampled from the citation graph G i of node a i i.e., where H is the maximum number of hops, is the one-hot encoding of the number of hops of r i,j where represents the probability that the number of hops of r i,j ranges from 1 to H, where λ1 and λ2 are hyperparameters.

6. The method according to claim 1, wherein The semantic information representation learning module generates the semantic information of the corresponding scientific literature through a hierarchical Transformer model; the citation network representation learning module uses the GAT model to obtain the domain information of scientific literature from the citation network.

7. A query method, the steps of which include: 1) Generate the representation of each document in the document library using the optimized scientific literature pre-training model in claim 1; 2) Generate the representation of the document to be queried using the optimized scientific literature pre-training model in claim 1 and calculate the similarity with the representation of each document in the document library respectively; 3) According to the similarity calculation results, select several documents with the highest similarity as the query results of the document to be queried and output them.

8. A server, characterized in that, It includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for executing any one of the methods in claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements any one of the methods in claims 1 to 7.

Images