A network and text content based document citation purpose classification system

Abstract

This invention utilizes methods from the field of cybersecurity to implement a document citation purpose classification system based on network and text content. The system achieves citation purpose classification through six steps: data preparation; training a weak classifier and using it to perform preliminary classification of unlabeled data; targeted manual annotation based on the coarse classification results; feature engineering on the obtained expanded labeled citation dataset; obtaining the optimal classifier based on the labeled data; and using the optimal classifier to classify documents. Ultimately, this system measures the importance of various citations appearing in the document text, identifies important citations, and thus assists researchers in exploring the knowledge transfer process between documents.

Description

Technical Field

[0001] This invention relates to the field of web search technology, and in particular to a classification system for the purpose of document citation based on web and text content. Background Technology

[0002] With the development of machine learning technologies, more and more traditional research fields have gained more powerful data-based analysis tools. Typically, for supervised learning tasks with labeled data, researchers need to collect a sufficient number of samples on the target dataset to train the machine learning model. Furthermore, the model's performance is highly dependent on the quality of the extracted features. Feature extraction captures key information from the real data distribution, and the machine learning model used can only see these features. Therefore, data acquisition and feature extraction are both crucial steps in machine learning tasks.

[0003] With the advent of the internet age, an increasing number of electronic document databases have flourished, such as Wikipedia, arXiv, and Google Scholar. Behind these platforms lies a vast amount of literature data, along with knowledge graphs formed by the attributes of the documents themselves and the relationships between them. Knowledge graphs have also become a popular research area in recent years, with researchers focusing on how to construct existing knowledge into network structures to better store, retrieve, and achieve higher-precision knowledge-driven research.

[0004] In the process of writing professional literature, new models and ideas are not always the result of sudden inspiration; most researchers draw inspiration from the work of predecessors. Therefore, most research results are not isolated; they are all more or less related to previous research. Most articles contain anywhere from a few to dozens of citations. If we consider articles as nodes and citations as edges, then these citations connect all articles in a directed graph, forming a citation network. Research on citation classification can better help researchers conduct academic impact analysis, moving beyond the problem of treating all citations as equally important in previous studies, and enabling a more fair and impartial evaluation of the achievements of articles.

[0005] However, data analysis in citation networks still faces the challenge of limited available labeled data. Taking academic paper citation networks as an example, when researchers analyze the citation behavior of one article to another, defining the purpose of the citation requires significant manual effort. Teufel et al. (2006), Valentina et al. (2015), and Jurgens et al. (2018) have all conducted research in this area and open-sourced their manually labeled datasets. However, compared to the tens of thousands of data points in classic machine learning applications in other fields, the available data in academic paper citations is currently only around 3,000. Furthermore, the datasets suffer from multiple classification criteria and a severe imbalance in the proportion of samples from different categories. In addition, traditional methods for feature extraction only consider some statistical features of the cited sentence or use LDA models to model the probability of text topics, resulting in a large number of repetitive and useless features.

[0006] Therefore, this invention aims to improve the data annotation problem in the field of academic paper citation research by using new methods, and at the same time, to obtain better model classification results by introducing data from the latest open source Microsoft Academic Knowledge Graph across domains. Summary of the Invention

[0007] To this end, this invention proposes a classification system for the purpose of document citation based on citation networks and citation texts.

[0008] The system classifies documents through six steps:

[0009] Step 1, Data Preparation: Obtain a labeled / unlabeled dataset containing the cited article, the target article, and the citation context. Use WPS document conversion tool to extract the text from the PDF into TXT text. Then use parsCit tool to perform structured extraction of any citations in the document, including the citation context. For the extracted citations, use title similarity search to confirm the identity of the cited article and record it in the cited text dataset.

[0010] Next, the labeled data is mapped to two categories: "important citations" and "non-important citations". This merges the existing labeled data into two unified categories. The two categories of labeled data are then matched with the cited text dataset. Successfully matched citation entries are collected as labeled citation data, while the remaining data entries in the cited text dataset are collected as unlabeled citation data.

[0011] Step 2: Train a weak classifier and use it to perform preliminary classification of the unlabeled data. The weak classifier is trained based on 358-dimensional features of the cited text and the cited article itself. The Random Forest classifier is trained in a supervised manner using the labeled citation data obtained in Step 1. The trained classifier is then used to classify the unlabeled citation data obtained in Step 1 to obtain coarse two-class classification results for the unlabeled citation dataset.

[0012] Step 3: Based on the coarse classification results, perform targeted manual annotation. Referring to the coarse classification results obtained in Step 2, use an annotation tool written in Python to manually review the citation context information classified as "important citations". If necessary, refer back to the PDF paper for review. Finally, determine whether the coarse classification results are correct. Collect the citation data entries correctly classified as "important citations" and inject them into the existing labeled citation dataset to obtain a labeled citation dataset with "important citations" significantly expanded. The annotation tool is an annotation program written in Python, which is used to facilitate the operation of annotators during the annotation process.

[0013] Step 4 involves feature selection operations. The expanded labeled citation dataset and the unlabeled citation dataset obtained in Step 3 are combined with a citation network based on the MAG academic knowledge graph for feature engineering. Features are selected, calculation methods are updated and replaced based on prior knowledge and experimental results. For the first time, citation network features are incorporated into classification features. Finally, each citation record in the labeled citation dataset corresponds to a 36-dimensional feature vector as the basis for classifier training.

[0014] Step 5: Based on the labeled data, obtain the optimal classifier. Use the ExtraTree classifier on the labeled reference dataset, combine the result of the feature selection part in Step 4 with the unlabeled reference data, and obtain the classifier with the best performance after debugging.

[0015] Step 6: Using the optimal classifier obtained in Step 5, classify the citation purpose of all articles for which citations, citation objects, and citation contexts can be obtained, and obtain accurate classification results for the citation purpose of the literature.

[0016] The data sources used in step 1 are: the ACL Anthology digital library of the Association for Computational Linguistics, and a small amount of annotated data from the ACL Anthology collection that were open-sourced online by Teufel et al. (2006), Valentina et al. (2015), and Jurgens et al. (2018). All 68,000 usable, undamaged PDF documents were downloaded from the ACL digital library.

[0017] The eight dimensions in step 4 are: extracting network features from labeled data based on Microsoft Academic Graph, including co-citation set similarity features, citation time-series correlation features, etc., totaling six dimensions:

[0018] (1) Total citation count of the citing and cited articles and the difference in publication year between the two papers: Since all researchers previously only counted mutual citations within the ACL electronic library, the total citation count in the original features was one-sided and needed to be updated. The difference in publication year had obvious errors in the previous dataset (such as 1000 and -4) and also needed to be updated.

[0019] (2) Cross-correlation coefficient between citations of the author and the cited article: Information from the MAG citation network can be used to obtain detailed information on the number of times each article is cited each year, thus creating a time series sequence of year-citation count for each article. For the citation time series of the author and the cited article, the correlation coefficient can be used to calculate whether there is any correlation between the time series, thereby observing whether the two articles have a relationship such as "increasing popularity at the same time" or "decreasing popularity at the same time".

[0020] (3) Cross-correlation coefficient of citation of the cited text and the cited text in the sliding window: the same as the feature in (2), but the sliding window is used to perform multiple calculations to capture the citation changes of the two articles within a finer granular time window.

[0021] (4) Similarity of other articles citing the citing text and the cited text: Due to the availability of the MAG academic citation network, it is possible to determine the number of articles that cite both the citing text and the cited text. By comparing this number with the number of articles that cite either the citing text or the cited text individually, it is possible to determine the strength of others' tendency to cite both the citing text and the cited text simultaneously, which can reflect, to some extent, others' judgments on the relevance of the two articles.

[0022] (5) Similarity between the cited text and the set of articles cited by the cited text: Similar to (4), but (5) compares the number of articles cited simultaneously in the citing text and the total number of articles cited by both articles. It describes the strength of the authors' tendency to cite the same articles from the authors' perspective. This feature represents whether the ideas and methods referenced by the two authors have a strong overlap when they are working, reflecting the relevance of the content of the two articles from the authors' writing perspective.

[0023] (6) Whether the author cites his own article: Thanks to the MAG academic knowledge graph which lists the authors of each article, we can now completely and accurately find out the author's self-citation situation.

[0024] Additionally, Doc2Vec is used to obtain the vectorized representation of the article summary, and cosine similarity is used to calculate the article content similarity features between the article containing the quoted sentence and the cited target.

[0025] The technical effects to be achieved by this invention are as follows:

[0026] This method measures the importance of various citations appearing in a document text, identifies important citations, and thus assists document researchers in exploring the process of knowledge transfer between documents. Attached Figure Description

[0027] Figure 1 A schematic diagram of the overall process of a classification system for the purpose of document citation based on network and text content;

[0028] Figure 2 Overall flowchart of the data annotation program Detailed Implementation

[0029] The following are preferred embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0031] This invention proposes a method for classifying the purpose of citations in literature based on citation networks and citation texts. This method measures the importance of various citations appearing in the literature text and identifies important citations, thereby assisting literature researchers in exploring the process of knowledge transfer between documents.

[0032] This invention includes the following steps:

[0033] Step 1, Data Preparation: Obtain labeled / unlabeled datasets containing the cited article, the target article, and the citation context. Data sources used: ACL Anthology (the digital library of the Association for Computational Linguistics), and a small amount of annotated data from the ACL Anthology collection released online by Teufel et al. (2006), Valentina et al. (2015), and Jurgens et al. (2018). Download all 68,000 usable, intact PDF documents from the ACL digital library. Use WPS Office to extract the text from the PDFs into TXT files. Then, use parsCit to extract the structure of any citations in the documents, including the citation context. For the extracted citations, use title similarity to confirm the identity of the cited article and record it in the cited text dataset.

[0034] Subsequently, label mapping was performed on the annotated data of Teufel, Valentina, and Jurgens, transforming the original classifications into two categories: "important citations" and "unimportant citations." This merged the existing annotated data into two unified categories. These two categories of annotated data were then matched against the cited text dataset. Successfully matched citation entries were collected as labeled citation data, while the remaining entries in the cited text dataset were collected as unlabeled citation data.

[0035] Step 2: Train a weak classifier and use it to perform preliminary classification of the unlabeled data. A Random Forest classifier is trained in a supervised manner using the labeled citation data obtained in Step 1. Features are trained using the 358-dimensional features based on the cited text and information from the cited article itself, as summarized by Jurgens et al. (2018). The trained classifier is then used to classify the unlabeled citation data obtained in Step 1, yielding coarse two-class classification results for the unlabeled citation dataset.

[0036] Step 3: Perform targeted manual annotation based on the coarse classification results. Referring to the coarse classification results obtained in Step 2, use a self-developed annotation tool to manually review the citation context information classified as "important citations." If necessary, refer back to the PDF paper for further review to ultimately determine whether the coarse classification results are correct. Collect the citation data entries correctly classified as "important citations" and inject them into the existing labeled citation dataset to obtain a significantly expanded labeled citation dataset for "important citations."

[0037] The self-developed annotation tool is an annotation program written in Python, designed to facilitate annotation operations for annotators. Its main program logic is illustrated below. Figure 2 content.

[0038] Step 4: Feature engineering is performed on the expanded labeled citation dataset obtained in Step 3. Features are selected, and calculation methods are updated and replaced based on prior knowledge and experimental results. The 358 dimensions of features extracted from the citation context and the article itself are reduced to 28 dimensions, and all calculation methods are updated. This ensures that multiple citations of the same target article appearing multiple times within an article can be considered as a single citation entry, increasing information reliability. Feature information gain is used in the experiments to analyze the importance of the 358-dimensional features, selecting features that are most helpful for citation classification. Eight new dimensions are added, and for the first time, citation network features are included in classification features. Network features are extracted from the labeled data based on Microsoft Academic Graph, including six dimensions such as co-citation set similarity features and citation time-series correlation features.

[0039] Specifically, it includes:

[0040] (1) Total citation count of the citing and cited articles and the difference in publication year between the two papers: Since all researchers previously only counted mutual citations within the ACL electronic library, the total citation count in the original features was one-sided and needed to be updated. The difference in publication year had obvious errors in the previous dataset (such as 1000 and -4) and also needed to be updated.

[0041] (2) Cross-correlation coefficient between citations of the author and the cited article: Information from the MAG citation network can be used to obtain detailed information on the number of times each article is cited each year, thus creating a time series sequence of year-citation count for each article. For the citation time series of the author and the cited article, the correlation coefficient can be used to calculate whether there is any correlation between the time series, thereby observing whether the two articles have a relationship such as "increasing popularity at the same time" or "decreasing popularity at the same time".

[0042] (3) Cross-correlation coefficient of citation of the cited text and the cited text in the sliding window: the same as the feature in (2), but the sliding window is used to perform multiple calculations to capture the citation changes of the two articles within a finer granular time window.

[0043] (4) Similarity of other articles citing the citing text and the cited text: Due to the availability of the MAG academic citation network, it is possible to determine the number of articles that cite both the citing text and the cited text. By comparing this number with the number of articles that cite either the citing text or the cited text individually, it is possible to determine the strength of others' tendency to cite both the citing text and the cited text simultaneously, which can reflect, to some extent, others' judgments on the relevance of the two articles.

[0044] (5) Similarity between the cited text and the set of articles cited by the cited text: Similar to (4), but (5) compares the number of articles cited simultaneously in the citing text and the total number of articles cited by both articles. It describes the strength of the authors' tendency to cite the same articles from the authors' perspective. This feature represents whether the ideas and methods referenced by the two authors have a strong overlap when they are working, reflecting the relevance of the content of the two articles from the authors' writing perspective.

[0045] (6) Whether the author cites his own article: Thanks to the MAG academic knowledge graph which lists the authors of each article, we can now completely and accurately find out the author's self-citation situation.

[0046] Additionally, Doc2Vec is used to obtain a vectorized representation of the article summary, and cosine similarity is used to calculate the article content similarity features between the article containing the cited sentence and the cited target. Finally, each citation record in the labeled citation dataset corresponds to a 36-dimensional feature vector as the basis for classifier training.

[0047] Step 5: Obtain the optimal classifier based on the labeled data. Using the ExtraTree classifier on the labeled citation dataset, the best-performing classifier was obtained after debugging. Its F1 score for important categories is 0.07 higher than the best value of all existing citation classifiers in academic research, which can verify the effectiveness of this method and the reliability of the final model.

[0048] Step 6: Using the citation target classifier obtained in Step 5, classify all articles for which citations, citation objects, and citation contexts can be obtained. By adjusting the classifier's output threshold, the accuracy and recall of the classifier's prediction results can be adjusted. Even with a precision of 1, there will still be a recall rate of about 20% for important citation categories. During the classification process using the trained model, a higher precision threshold can be used to minimize the number of "non-important citation" category samples classified as "important citations".

[0049] In step 1, the original articles were selected from a collection of over 70,000 academic papers in PDF format from the ACL Anthology digital library. These were converted to text files (.txt format) using WPS PDF to Word conversion tool. Then, the parsCit tool (a citation context extraction tool based on conditional random fields) was used to identify citations in the documents and extract all text of a certain length from the context of the citations as citation context text data. Currently, the data only contains citation entries and context data; there are no corresponding target articles. This invention uses a string similarity comparison strategy based on the difflib library in Python to compare the title and author information of the cited articles with the title and author information of other papers in the ACL paper library. If the similarity is higher than a threshold, it is considered a successful match. All successfully matched data entries are collected to obtain a complete unlabeled dataset of <citations, citation context, and target article information>.

[0050] Step 2 first obtains available citation label data from existing open-source datasets as labeled training data for the weak classifier. A random forest model based on information gain from machine learning and 360-dimensional features extracted from contextual information, summarized by previous researchers, are selected as the model and feature inputs for training the weak classifier, roughly dividing the unlabeled dataset obtained in Step 1. It is worth noting that this invention adopts the widely used six-category (background, motivation, comparison, inheritance, use, future prospects) citation purpose classification method. At this point, each citation in the unlabeled dataset has been roughly labeled into a certain citation category by the weak classifier.

[0051] Step 3 involves manual annotation based on the coarsely labeled data obtained in Step 2. The annotation program uses coarsely labeled tags for initial citation screening. This is because most articles contain a certain number of samples, while some citations that are significant in the study of knowledge dissemination context (mainly inheritance-type citations) have an extremely low proportion. Multiple articles may only contain one significant citation. Therefore, the initial screening process plays a crucial role in accelerating the manual annotation process and improving the efficiency of annotators. During the annotation process, annotators are shown information about the sentence and paragraph in which the citation appears, allowing them to manually annotate the purpose of the citation. This increased information helps reduce discrepancies in annotation results caused by cognitive biases among different annotators. It should also be noted that three domain experts were invited to perform the annotation work, and only citation entries with annotations of the same category were used. At this point, the project has obtained an unlabeled citation dataset and a labeled citation dataset for important categories, supplemented by extensive manual annotation.

[0052] Step 4, extracting relevant features from the dataset obtained in Step 3, can be referred to as the feature engineering part. The original 358-dimensional features originate from all text content-related features summarized in previous research. However, due to the weak expressive power of some features and the high repetition of some features, analysis revealed that very few features contribute significantly to the weak classifier overall. Excessive redundancy of features also negatively impacts classifier performance; therefore, it is necessary to screen the original features. The screening criteria are generally twofold: first, based on the information gain contribution of each feature according to the feature segmentation of the decision tree model; and second, adding new features relevant to the intended purpose or modifying the calculation and extraction process of some original features based on professional knowledge.

[0053] Regarding the first point, let's take the CART decision tree-based random forest model used in this project as an example. Random forests generate multiple trees by repeatedly taking subsets of the original data with replacement to create multiple smaller datasets, forming a forest. Each tree in the forest is a CART decision tree, i.e., a binary decision tree based on Gini coefficient gain. After training, for new samples, a final classification score is generated by voting based on the classification results of all decision trees in the forest. When calculating feature importance, for a non-leaf node 'a' in a single decision tree in the forest, let its left child be 'b' and its right child be 'c'. Let the corresponding uppercase letters A, B, and C represent the sample sets reaching the leaf nodes, respectively. Clearly, A = B ∪ C and... The importance of this split at node a can be calculated by the following formula:

[0054] Importance(a)=Gini(A)×Pr(A)-Gini(B)×Pr(B)-Gini(C)×Pr(C)

[0055] Where Pr(A) represents the ratio of the number of samples in set A to the number of samples in the original total dataset, the Gini coefficient Gini(A) is calculated as follows:

[0056]

[0057] Where k∈K represents a type of reference existing in set A, K is the set of all reference types, and p k Let A represent the proportion of referenced samples of type k in set A. Then, for the entire forest, we can use all nodes a that use a certain feature f as the splitting feature. i The average importance of f is used as a qualitative importance assessment indicator:

[0058]

[0059] Regarding the second point, since the latest full-text data from ACLAnthology was obtained, some older data could be recalculated with more accurate values. Newly added features extracted from the citation context typically include abstract similarity and keyword extraction. Abstract similarity is calculated using Doc2Vec text encoding to automatically encode the content of both the cited and target articles, and then the cosine similarity between the two encoded vectors is used for measurement. The Doc2Vec text encoding algorithm used in this invention is specifically the PV-DBOW model, which extracts continuous text of a fixed window size as training data. The window moves forward word by word until the entire document is traversed, generating a large number of continuous text segments as training data. The algorithm's idea is to extract all words in each continuous text segment as predicted words and use the segment representation to predict the predicted words as accurately as possible. This model assumes that the occurrence of a word is independent of its context words. If (w i-m ......w i+m Let m represent the sequence of words in a continuous text segment, where m represents the window size and i represents the current position of the window within the text segment. Then, the model training objective is:

[0060]

[0061] Where doc represents the text auto-encoding vector, the model is trained using a neural network model with randomly initialized fixed-length vectors for all words and randomly initialized doc vectors of the same length. Based on the training objective function, taking the logarithm and then the negative, the model loss function can be designed as follows:

[0062]

[0063] N is the total document length, and y represents the output of the penultimate layer in the neural network corresponding to the input. Since the dataset primarily consists of academic papers, encoding the article summaries using Doc2Vec is sufficient. Doc2Vec describes the distribution of words within a text segment; for similar topics, segments with similar word distributions will have similarly close encoding results. Experimental results show that the similarity measure of summaries is the most important classification rule for the classifier.

[0064] Meanwhile, this invention utilizes the paper citation network from Microsoft Academic Knowledge Graph as supplementary paper network data, incorporating network features previously unused in the field into the overall feature extraction results. Typical network features include co-citation network features and citation time-series correlation features. For example, in the co-citation network features, assuming the sets of articles citing papers P1 and P2 in the citation network are C(P1) and C(P2) respectively, the network co-citation Jaccard similarity feature can be characterized as follows:

[0065] Jaccard(P1,P2)=totalnum(C(P1)∩C(P2)) / totalnum(C(P1)∪C(P2))

[0066] The addition of citation network features based on the MAG academic knowledge graph allows the classifier to acquire citation network features beyond the context information during training. By increasing the information acquisition methods, the performance of the citation target classifier is effectively improved. In this step, through feature selection, feature update calculation, and feature addition, the previously summarized 354-dimensional content feature vector is reduced to a 28-dimensional vector, and 8 new features are added. The 258-dimensional features extracted from the citation context and the article itself were reduced to 28 dimensions. Since existing open-source datasets did not merge multiple citations of the same article, this project used statistical methods during dimensionality reduction. Maximum, minimum, and mean values ​​were used to statistically merge the features of multiple citations of an article, ensuring that citations to the same target article appearing multiple times within an article are considered as a single citation, increasing information reliability. In experiments, feature information gain was used to perform importance analysis on the 358-dimensional features, selecting features that were most helpful for citation classification. Furthermore, thanks to the citation network extracted from the Microsoft Academic Knowledge Graph, we were able to update inaccurate values ​​for features such as "number of citations" and "whether the citing and cited articles have common authors," further strengthening the information representation capabilities of the feature set. A total of 8 new dimensions were recalculated and added.

[0067] In step 5, the optimal classifier is constructed based on the labeled feature dataset obtained in step 4. The classifier is trained using the feature vector extracted for each reference and the corresponding label. Based on extensive experimental results, this invention designs ExtraTree, a variant of the random forest classifier, as the optimal classifier. Its general architecture is similar to that of random forest; however, when selecting the splitting feature for the decision tree, instead of using the splitting feature with the maximum information gain, it randomly selects one from the available features. This method reduces the prediction variance of the overall machine learning model but introduces a slight prediction bias. The process of acquiring unlabeled data in the ACL electronic library.

[0068] In general, there are three data sources: ACL digital library, open source datasets, and Microsoft MAG academic knowledge graph.

[0069] The open-source dataset includes citation purpose annotations and calculated features from previous researchers on citations in articles within the ACL digital library. Since they only provided the annotated citation purposes and the values ​​of all their calculated features, the project needed to obtain the complete full text from ACL to recalculate some feature values ​​during experiments.

[0070] The MAG academic knowledge graph is used to calculate newly added citation network features and update the values ​​of several features that were incorrectly calculated by previous researchers. Therefore, in Figure 1 The ACL (Accelerated Linking Array) is placed relatively early, while the MAG (Main Feature Group) is added only when new features are added or updated. According to experiments, this invention improves the F1 score (calculated as follows) for important category classification by 0.07 compared to the previous best classifier. Furthermore, based on ablation experiments, the manual annotation and feature engineering steps in this method play a crucial role in improving the results, strongly verifying the effectiveness and necessity of the method's steps.

[0071] In step 6, the classifier trained in step 5 is used to classify the citations extracted from all existing papers in ACL Anthology, and citations predicted to be of importance are extracted for research related to knowledge dissemination within the academic context. This method can be appropriately modified and extended to a wider range of literature based on usage, providing strong materials for research related to knowledge dissemination.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A classification system for the purpose of citations in literature based on citation networks and citation texts, characterized in that: The classification of documents is achieved through 6 steps: Step 1, Data Preparation: Obtain a labeled / unlabeled dataset containing the cited article, the target article, and the citation context. Use WPS document conversion tool to extract the text from the PDF into TXT text. Then use parsCit tool to perform structured extraction of any citations in the document, including the citation context. For the extracted citations, use title similarity search to confirm the identity of the cited article and record it in the cited text dataset. Next, the labeled data is mapped to two categories: "important citations" and "non-important citations". This merges the existing labeled data into two unified categories. The two categories of labeled data are then matched with the cited text dataset. Successfully matched citation entries are collected as labeled citation data, while the remaining data entries in the cited text dataset are collected as unlabeled citation data. Step 2: Train a weak classifier and use it to perform preliminary classification of the unlabeled data. The weak classifier is trained based on 358-dimensional features of the cited text and the cited article itself. The Random Forest classifier is trained in a supervised manner using the labeled citation data obtained in Step 1. The trained classifier is then used to classify the unlabeled citation data obtained in Step 1 to obtain coarse two-class classification results for the unlabeled citation dataset. Step 3: Based on the coarse classification results, perform targeted manual annotation. Referring to the coarse classification results obtained in Step 2, use an annotation tool written in Python to manually review the citation context information classified as "important citations". If necessary, refer back to the PDF paper for review. Finally, determine whether the coarse classification results are correct. Collect the citation data entries correctly classified as "important citations" and inject them into the existing labeled citation dataset to obtain a labeled citation dataset with "important citations" significantly expanded. The annotation tool is an annotation program written in Python, which is used to facilitate the operation of annotators during the annotation process. Step 4: Perform feature selection operations. The expanded labeled citation dataset and the unlabeled citation dataset obtained in Step 3 are combined with the citation network based on MAG academic knowledge graph for feature engineering. Based on prior knowledge and experimental results, features are selected, calculation methods are updated and replaced; for the first time, citation network features are included in classification features. Ultimately, each citation record in the labeled citation dataset will correspond to a 36-dimensional feature vector as the basis for classifier training; Step 5: Based on the labeled data, obtain the optimal classifier. Use the ExtraTree classifier on the labeled reference dataset, combine the result of the feature selection part in Step 4 with the unlabeled reference data, and obtain the classifier with the best performance after debugging. Step 6: Using the optimal classifier obtained in Step 5, classify the citation purpose of all articles for which citations, citation objects, and citation contexts can be obtained, and obtain accurate classification results for the citation purpose of the literature.

2. The document citation purpose classification system based on citation networks and citation texts as described in claim 1, characterized in that: The data sources used in step 1 are: the ACL Anthology digital library of the Association for Computational Linguistics, a small amount of annotated data from the ACL Anthology collection available online, and all 68,000 usable and undamaged PDF documents downloaded from the ACL digital library.

3. The document citation purpose classification system based on citation networks and citation texts as described in claim 1, characterized in that: Step 4 consists of eight dimensions: extracting network features from labeled data based on Microsoft Academic Graph, including six dimensions such as co-citation set similarity features and citation time-series relevance features. Specifically, these include: the total number of citations of the citing and cited articles, the difference in publication years between the two papers, the cross-correlation number of the citation time series of the citing and cited articles, the cross-correlation number of the citation time series of the citing and cited articles with a sliding window, the similarity of the set of other articles citing the citing and cited articles, the similarity of the set of articles cited by the citing and cited articles, and whether the author cites their own article. Additionally, Doc2Vec is used to obtain the vectorized representation of the article summary, and cosine similarity is used to calculate the article content similarity features between the article containing the cited sentence and the citation target.

Images