Scientific and technical literature type labeling method

Abstract

The invention relates to a scientific and technical literature type labeling method, which comprises the following steps of: firstly, determining type information of literature by utilizing a plurality of information sources based on technologies of character matching, semantic matching and the like, and then, formulating a priority rule according to information source characteristics and method characteristics so as to determine a final type when the type information of the plurality of information sources is inconsistent. According to the method, a multi-source information fusion strategy is introduced to determine the literature type, and the misjudgment risk possibly brought by a single information source is effectively reduced by integrating feature information from different channels. According to the scheme, only key features need to be efficiently coded and stored, and the problem that in a traditional method, resource consumption is too large due to comprehensive collection and processing of all possible information is solved. Under the fusion mechanism, the literature types can be accurately identified and labeled by utilizing limited computing resources, so that the problems of disordered literature management and low retrieval efficiency caused by insufficient information or misclassification are effectively prevented.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data classification, and particularly relates to a scientific and technological literature type labeling method, which realizes literature data classification dataset construction, and in particular to a Chinese scientific and technological literature type labeling method based on multiple information such as literature titles, abstracts and keywords. BACKGROUND

[0002] As an important carrier of knowledge inheritance and innovation, the number of scientific and technological literature is growing at an unprecedented rate. However, how to efficiently and accurately classify and organize it in order to facilitate information retrieval, analysis and application has become a key problem that needs to be solved in the field of scientific information management. High-quality labeled data, as the cornerstone supporting this process, is of great significance to promoting the natural language processing (NLP) research of scientific literature. However, on the one hand, the high-quality labeled corpus of current Chinese scientific literature is relatively scarce, and manual labeling has uneven quality and low efficiency, on the other hand, traditional classification methods are often based on fixed classification standards and frameworks, and the existing classification system may not fully meet the needs of modern management system and the differences between the unique classification systems of various units or institutions. These systems differ in classification standards, granularity, hierarchy, etc., resulting in inconsistent classification of literature when flowing across systems. SUMMARY

[0003] In view of the problems of lack of high-quality labeled corpus of current Chinese scientific literature and uneven quality and low efficiency of manual labeling, the application provides a scientific literature type labeling method, which considers the differences between different classification systems, formulates flexible mapping rules and conversion mechanisms, makes the classification results compatible with the classification systems of various units, and realizes literature data classification dataset construction.

[0004] In order to solve the above technical problems, the application provides a scientific literature type labeling method, characterized by comprising the following steps:

[0005] Step one, multiple information sources are used to construct a literature type feature set, and data preprocessing is performed, the literature type feature set includes the title, abstract, keyword, free word and classification number of the literature;

[0006] Step two, classify the preprocessed literature type and determine the literature type, the classification method includes character similarity, semantic similarity and type mapping strategy;

[0007] Step three, comprehensive integration, according to the priority and weight, comprehensive decision is made on the type information from different information sources to determine the final literature type.

[0008] In step one, the data preprocessing includes removing punctuation, stop words, numbers and special characters, and using a dictionary-based word segmentation method combined with new word discovery technology to segment Chinese text.

[0009] In step two, for key word and free word short text information, a character similarity calculation method is used to calculate the similarity between literature text and type label by constructing a Chinese key word dictionary and a synonym dictionary to expand the label set.

[0010] In step two, for title and abstract long text information, semantic similarity is used for judgment, and a pre-trained language model is used to calculate the semantic similarity between texts, and the weighted average of CLS character vector representation and sentence average vector representation is used as the final semantic similarity value.

[0011] In step two, for the classification system of different journals or databases, a type mapping table is established to map the classification labels under different systems to the pre-defined classification system.

[0012] The mapping relationship includes complete equivalence relationship, partial overlap relationship, upper inclusion relationship and lower refinement relationship.

[0013] In step three, the priority determination method is as follows:

[0014] First, the type obtained by mapping the classification number is the type with the highest priority, that is, if the type obtained by the type mapping method can be mapped to the pre-defined type, then the mapping result is used as the literature type;

[0015] If the classification number mapping cannot determine the type, the literature type obtained by the character similarity method is used as the secondary priority type;

[0016] If character matching still cannot determine the type, the similarity result calculated by the semantic similarity method is used as the secondary priority type.

[0017] In step three, the weight allocation method is to allocate weights to character similarity, semantic similarity and type mapping strategy respectively.

[0018] In step three, the comprehensive judgment method includes:

[0019] If the results of priority and weight method are consistent, then the result is directly used as the final literature type;

[0020] If the results are inconsistent, then the result with higher confidence is selected as the final determination result according to the confidence.

[0021] The method is suitable for type labeling of Chinese scientific literature.

[0022] Beneficial effects: The present application is based on the title, abstract, keywords and other information, using character matching, semantic matching and other advanced technologies, extracting the type characteristics of the literature from multiple dimensions, and formulating priority rules based on these characteristics to make reasonable decisions when the information of multiple information sources is inconsistent, realizing the type labeling of Chinese scientific literature.

[0023] Compared with the prior art, the present application introduces a multi-source information fusion strategy to determine the literature type, effectively reduces the misjudgment risk that may be caused by a single information source by integrating feature information from different channels. This scheme only needs to efficiently encode and store key features, avoiding the problem of excessive resource consumption caused by collecting and processing all possible information in traditional methods. Under this fusion mechanism, we can accurately identify and label the literature type using limited computing resources, thereby effectively preventing literature management confusion and low retrieval efficiency caused by insufficient or incorrect classification.

[0024] Through the above method, we can fully utilize the advantages of multi-source information sources to improve the accuracy and reliability of automatic literature type labeling. At the same time, this method also has certain flexibility and scalability, which can be adjusted and optimized according to actual needs. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The present application processing process method flow chart. DETAILED DESCRIPTION

[0026] In order to make the purpose, content and advantages of the present application clearer, the specific embodiments of the present application are described in further detail below.

[0027] The scientific literature type labeling method proposed by the present application specifically includes the following steps:

[0028] Step one, construct the literature type feature set and perform data preprocessing (information input)

[0029] In the present application, multiple information sources are used to construct the feature set of the literature type, mainly including the title, abstract, keywords, free words and classification number of the literature, and the above data sources are preprocessed, including removing punctuation marks, stop words, numbers and special characters. For Chinese text, a dictionary-based word segmentation method is used, and a new word discovery technology is combined to improve the accuracy of word segmentation.

[0030] Step two, classify the literature according to the literature features after preprocessing, and determine the literature type (type judgment)

[0031] According to the expert's suggestion, complete the literature type predefinition, determine the literature classification system and type label.

[0032] In the automatic labeling process, the type information obtained from each information source is fully utilized, and multiple strategies such as character similarity, semantic similarity and type mapping are combined to improve the accuracy of classification. The execution process of the three type judgment strategies is as follows:

[0033] Character similarity: For keyword, free word short text information, a character-based similarity calculation method is adopted, specifically the Jaccard similarity calculation method. In order to improve the effectiveness of evaluation, first, a Chinese keyword dictionary is constructed, and the phrase matching method is used to calculate the similarity between the literature text and the type label. In the literature type label, a synonym dictionary is constructed manually to expand the label set. The greater the similarity value, the higher the similarity. The calculation formula is as follows:

[0034]

[0035] Where A is the phrase set generated by literature short text segmentation; B is the phrase set under a certain classification label, generally 3-5 synonymous words are composed.

[0036] Semantic similarity: For title, abstract long text information, a pre-trained language model (such as BERT) is used to calculate the semantic similarity between texts. Specifically, the BERT model is used to represent characters, and the [CLS] character vector representation and sentence average vector representation are selected to calculate the similarity. Finally, the weighted average of the two is selected as the final semantic similarity value. The calculation formula is as follows:

[0037] sim(T1,T2)=α·CosineSimilarity(CLS1,CLS2)+(1-α)·CosineSimilarity(Avg1,Avg2)

[0038]

[0039] w i represents the character vector representation obtained by bert, Avg T represents the average vector representation, T1 represents the literature long text, T2 represents the literature classification label description information, which is written by artificial, and the character quantity is less than or equal to 50; n represents the character length of the sentence T, α represents the weight value, the range is between 0-1, CLS1 represents the [CLS] character vector representation of T1, CLS2 represents the [CLS] character vector representation of T2;

[0040] Type mapping: Considering that different journals or databases may have their own classification systems, a type mapping table is established to map the classification labels under different systems to the predefined classification system. Specifically, according to the document classification number and corresponding rules, the type of the literature is accurately found and determined, and the determined literature type is compared and analyzed with the predefined type system, considering the possible relationship between them, including but not limited to complete equivalence relationship (i.e. the connotation and extension are completely consistent), partial overlap relationship (part of the semantic content is consistent), upper inclusion relationship (the literature type is a specific branch of the predefined type) and lower refinement relationship (the predefined type is a broader category of the literature type), etc. Finally, according to the determination results of these relationships, a type mapping table is constructed.

[0041] Step three, comprehensive integration (result integration)

[0042] In the literature type determination stage, we propose two methods, namely priority-based and weight-based determination methods.

[0043] S3.1. Priority determination: According to the importance and reliability of information sources, different priorities are set for different information sources. In priority setting, considering that classification number information is self-contained information of literature, determined by the publisher, and has a certain authority, the type obtained by the type mapping method is taken as the highest priority type, that is, if the type obtained according to the classification number can be mapped to the predefined type, then the mapping result is taken as the literature type. Keywords and free words can reflect the core content and theme of the literature, and can further verify and supplement the determination of literature type, so if the mapping cannot determine the type, then the character similarity method is used to obtain the type information of the literature as the second priority type, that is, if the similarity value calculated is greater than the threshold value 0.3 (the threshold value is affected by the classification label set, the more comprehensive the word group representing the type label in the set, the higher the threshold value), it is considered that the classification label can be used as the literature type. If the type still cannot be determined, the similarity result calculated by the semantic similarity method is taken as the second priority, and the literature type is obtained according to the calculation result.

[0044] S3.2. Weight allocation: In addition to priority, we also consider allocating appropriate weights to different information sources for calculation, where the weight of the classification number mapping method is set to 0.4, the weight of the subject word and free word matching method is set to 0.2, and the weight of the title and abstract similarity calculation method is set to 0.1. These weights reflect the contribution of information sources in the classification process. Weight allocation can be adjusted and optimized based on experimental results to ensure the accuracy and stability of the classification results. The specific calculation process is as follows:

[0045] S = 0.4S1 + 0.2S2 + 0.2S3 + 0.1S4 + 0.1S5

[0046] S is the score of the literature belonging to one of the type labels, S i S represents the score obtained by the above method, which is standardized by the 0-1 standardization method.

[0047] Finally, the score of the literature belonging to each literature label can be calculated, and the maximum value is taken as the type label.

[0048] S3.3. Comprehensive judgment: In the comprehensive judgment stage, we make a comprehensive decision on the type information from different information sources according to the priority and weight. Specifically, if the results of the two methods are consistent, the result is directly taken as the final literature type; if the results are inconsistent, according to the actual situation, a certain confidence is given to each result, and the result with higher confidence is selected as the final determination result. For example, when the data source is relatively single and the authority is higher, the type is determined by the optimization level method, when the data is more complex and the source is diversified, the type is determined by the method based on weight, and the influence of multiple factors on the literature type is considered.

[0049] Combined with the above method, we designed a multi-source information fusion classification method for scientific and technological literature. This method uses character similarity, semantic similarity and type mapping table to realize classification and optimize the results, and finally obtains the final literature type.

[0050] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should be considered as the protection scope of the present application.

Claims

1. A method for marking the type of scientific and technological documents, characterized in that: The following steps are involved: Step 1: construct a document type feature set using multiple information sources and perform data preprocessing. The document type feature set includes the document title, abstract, keywords, free words and classification number; Step 2: classify the pre-processed document types to determine the document types, wherein the classification methods include character similarity, semantic similarity and type mapping strategy; Step 3: Comprehensive integration: make comprehensive decisions on the type information from different information sources based on priority and weight to determine the final document type.

2. The method according to claim 1, characterized in that In step 1, the data preprocessing includes removing punctuation marks, stop words, numbers and special characters, and segmenting the Chinese text using a dictionary-based word segmentation method combined with new word discovery technology.

3. The method according to claim 1, characterized in that In the second step, for the keywords and free word short text information, a character similarity calculation method is used to expand the label set by constructing a Chinese keyword dictionary and a synonym dictionary to calculate the similarity between the document text and the type label.

4. The method according to claim 1, wherein In the second step, semantic similarity is used to judge the long text information of the title and abstract, and the semantic similarity between the texts is calculated using a pre-trained language model. The weighted average of the CLS character vector representation and the sentence average vector representation is used as the final semantic similarity value.

5. The method according to claim 1, wherein In the second step, for the classification systems of different journals or databases, a type mapping table is established to map the classification labels under different systems to the predefined classification system.

6. The method according to claim 5, characterized in that The mapping relationships include complete equivalence relationships, partial overlap relationships, superordinate inclusion relationships, and subordinate refinement relationships.

7. The method according to claim 1, characterized in that In step 3, the priority determination method is as follows: First, the type obtained by mapping the classification number is taken as the type with the highest priority. That is, if the type obtained by the type mapping method can be mapped to the predefined type, the mapping result is used as the document type. If the classification number mapping cannot determine the type, the document type obtained by the character similarity method is used as the second priority type; If the character matching still cannot determine the type, the similarity result calculated by the semantic similarity method is used as the second priority type.

8. The method according to claim 1, characterized in that In the step three, the weight assignment method is: assigning weights to character similarity, semantic similarity and type mapping strategy respectively.

9. The method according to any one of claims 1 to 8, characterized in that In step 3, the comprehensive determination method includes: If the results of the priority and weight methods are consistent, the result is directly used as the final document type; If the results are inconsistent, the result with higher confidence level will be selected as the final judgment result according to the confidence level.

10. The method according to claim 1, characterized in that The method is applicable to the type annotation of Chinese scientific and technological literature.

Images