Method and device for classifying resources in discipline field of Ministry of Education

Abstract

The invention discloses a method and a device for classifying resources in the discipline field of the Ministry of Education, and belongs to the technical field of resource classification in the discipline field, and the method comprises the following steps: collecting article data of 120 discipline categories of the Ministry of Education; the method comprises the following steps: screening subject affiliations of collected article data by virtue of GPT, so as to realize AI labeling and obtain a sample set; dividing the sample set into a training set and a test set according to a set proportion; the method comprises the following steps of: converting a training set and a test set into a vector of 768 dimensions by using a vector model paraphrase-multilingual-mpnet-base-v2, and converting the training set and the test set into a vector of 768 dimensions by using the vector model paraphrase-multilingual-mpnet-base-v2; using 768-dimensional vectors of the training set and the test set to train the model; the defect of discipline classification in the Ministry of Resource Education in China is overcome, and high-quality resources are provided for discipline construction of the Ministry of Education.

Description

Technical Field

[0001] The present invention belongs to the technical field of subject field resource classification, and specifically relates to a method and device for subject field resource classification for the Ministry of Education. Background Art

[0002] The resource classification of the Ministry of Education in the subject area mainly includes philosophy, economics, law, education, literature, history, science, engineering, agriculture, medicine, management, art, etc. These subject categories cover a wide range of fields from basic theory to applied technology, providing a rich selection of majors for higher education.

[0003] To classify the subject field resources of the Ministry of Education, that is, to implement the subject field classification of the Ministry of Education on the literature resources, it is often necessary to use the existing technical solution with patent publication number "CN112966120B" to classify the subject field resources of the Ministry of Education, but this type of classification method is not very accurate.

[0004] The country is currently vigorously developing the disciplines of the Ministry of Education, but there is currently no technical method in the academic community that can accurately classify literature resources into the disciplines of the Ministry of Education. This is specifically reflected in the following aspects:

[0005] 1. The development and establishment of disciplines require literature resources, but the main method of the existing resource classification system is still the 5th edition of the Chinese Library Book Classification System published by Beijing Library Press in September 2010. This method has been difficult to adapt to the current academic community's requirements for resource classification.

[0006] 2. Currently, there is a lack of a complete subject classification system of the Ministry of Education in China. Summary of the invention

[0007] In order to solve the defects existing in the prior art, the present invention proposes a method and device for classifying resources in the subject field of the Ministry of Education, which collects article data of 120 subject categories of the Ministry of Education; screens the subject affiliation of the collected article data with the help of GPT, so as to realize AI labeling and obtain a sample set; divides the sample set into a training set and a test set according to a set ratio; uses the vector model paraphrase-multilingual-mpnet-base-v2 to convert both the training set and the test set into 768-dimensional vectors; uses the 768-dimensional vectors of the training set and the test set to train the model; improves the domestic deficiencies in the resource classification of the Ministry of Education, and provides high-quality resources for the discipline construction of the Ministry of Education.

[0008] The present invention utilizes the following technical solutions.

[0009] A method for classifying resources in subject areas of the Ministry of Education, including:

[0010] Step 1: Collect article data from 120 disciplines of the Ministry of Education;

[0011] Step 2: Screen the subject affiliation of the collected article data based on GPT to achieve AI annotation and obtain a sample set;

[0012] Step 3: Divide the sample set into training set and test set according to the set ratio;

[0013] Step 4: Use the vector model paraphrase-multilingual-mpnet-base-v2 to convert both the training set and the test set into 768-dimensional vectors;

[0014] Step 5: Train the model using the 768-dimensional vectors of the training and test sets.

[0015] Preferably, in step 1, the article data includes monographs, journals, books, newspapers, conference papers, scientific reports, standard documents, patent documents, dissertations, and government publications.

[0016] Preferably, in step 1, data of at least 20,000 articles are collected for each subject category of the Ministry of Education.

[0017] Preferably, in step 2, the method for screening the subject affiliation of the collected article data by means of GPT includes:

[0018] The collected article data is sent to GPT to obtain the subject affiliation of each article data.

[0019] Preferably, in step 2, a single article data and its subject affiliation constitute a sample data, and all the article data and its subject affiliation constitute a sample set.

[0020] Preferably, in step 3, the ratio is set to 4:1, so that the sample set is divided into a training set and a test set, that is, the sample set is divided into a training set and a test set with a ratio of 4:1.

[0021] Preferably, step 5 specifically includes the following stages:

[0022] Step 5-1: Custom F1-score function stage, which defines and implements a custom F1-score evaluation function for calculating the F1 score during model training and evaluation;

[0023] Step 5-2: Model building phase, its definition provides two neural network models with different architectures, namely _build_m1 and _build_m2;

[0024] Step 5-3: Model training phase, by defining training parameters, the 768-dimensional vectors of the training set and the test set are input into the two neural network models with different architectures, and the Adam optimizer and the category weight balancing method are used to optimize the model training process, and the best model is saved as the trained model;

[0025] Step 5-4: Model prediction stage, which loads the trained model, predicts the new article data, and returns the prediction results to achieve the classification of subject area resources for the Ministry of Education.

[0026] Preferably, step 5-1 specifically includes:

[0027] The custom F1-score function includes: the main function, the function for calculating precision, and the function for calculating recall;

[0028] The process of calculating the accuracy function includes:

[0029] Execute the following formula to calculate the accuracy:

[0030] Precision = number of true positive examples obtained by the neural network model / (number of predicted true positive examples + 0.0000001);

[0031] The process of calculating the recall function includes:

[0032] The following formula is used to calculate the recall rate:

[0033] Recall rate = number of true positive examples obtained by the neural network model / (number of possible true positive examples + 0.0000001);

[0034] The main function is used to call the function for calculating precision and the function for calculating recall in sequence and calculate the F1 score. The formula for calculating the F1 score is as follows:

[0035] F1 score = 2*(precision*recall) / (precision+recall).

[0036] Preferably, step 5-2 specifically includes:

[0037] Construct a neural network model of an architecture, which includes: the first layer of the neural network model contains 128 neurons, and the activation function is RELU; the second layer of the neural network model contains 64 neurons, and the activation function is RELU; the output layer of the neural network model contains a set number of neurons n, and the activation function is softmax;

[0038] A neural network model with an architecture is constructed, which includes: the first layer of the neural network model contains 256 neurons, and the activation function is RELU; the second layer of the neural network model contains 128 neurons, and the activation function is RELU; the output layer of the neural network model contains a set number of neurons n, and the activation function is softmax.

[0039] A device for classifying resources in subject areas of the Ministry of Education, comprising:

[0040] The collection module is used to collect article data from 120 disciplines of the Ministry of Education;

[0041] The screening module is used to screen the subject affiliation of the collected article data based on GPT, so as to achieve AI annotation and obtain a sample set;

[0042] A partitioning module is used to divide the sample set into a training set and a test set according to a set ratio;

[0043] The conversion module is used to convert both the training set and the test set into 768-dimensional vectors using the vector model paraphrase-multilingual-mpnet-base-v2;

[0044] The training module is used to train the model using 768-dimensional vectors of the training set and the test set.

[0045] The beneficial effects of the present invention are that, compared with the prior art, the technical effects of the present invention include:

[0046] 1. It improves the insufficiency of the subject classification of the Ministry of Education in China and provides high-quality resources for the subject construction of the Ministry of Education;

[0047] 2. Efficient model training: By customizing the F1-score evaluation function and balancing category weights, the performance of the model can be improved when processing unbalanced data; Multi-model architecture: Two different model architectures are provided, and users can choose the appropriate model according to specific tasks, which improves the flexibility and applicability of the device for the classification of subject resources of the Ministry of Education; Automatic model saving and loading: The device for the classification of subject resources of the Ministry of Education automatically saves the best model and automatically loads the model during prediction, which simplifies the model management process; High accuracy and multi-indicator evaluation: Through a variety of evaluation indicators (such as accuracy, precision, recall and F1 score), a more comprehensive model performance evaluation method is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of the method for classifying resources in subject areas of the Ministry of Education described in the present invention;

[0049] Figure 2 It is a partial structural diagram of the device for classifying resources in subject areas of the Ministry of Education described in the present invention;

[0050] Figure 3 It is a schematic diagram of part of the source code for dividing the sample set into a training set and a test set in the present invention;

[0051] Figure 4 It is a partial code schematic diagram of the custom F1-score function stage described in the present invention;

[0052] Figure 5 It is a partial code schematic diagram of the model building stage described in the present invention;

[0053] Figure 6 It is a schematic diagram of part of the code in the model prediction stage of the present invention;

[0054] Figure 7 It is a comparative schematic diagram of the number of subjects of the Ministry of Education calculated by INCITES for a single article and the insects described in the present invention;

[0055] Figure 8 It is a schematic diagram that 45% of the subjects in INCITES described in the present invention have scores below 0 in insects;

[0056] Fig. 9 It is a schematic diagram of the subject intersection and conflict between insects and INCITES described in the present invention;

[0057] Fig.10 It is a schematic diagram of the score distribution of the subjects in which INCITES are more numerous than insects in the present invention;

[0058] Fig.11 It is a schematic diagram of the score distribution of the subjects in which insects are more numerous than incites in the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention clearer, the following will be combined with the drawings in the embodiments of the present invention to clearly and completely express the technical solution of the present invention. The embodiments expressed in the present invention are only some embodiments of the present invention, not all embodiments. According to the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the protection scope of the present invention.

[0060] like Figure 1 As shown, the method for classifying resources in subject areas of the Ministry of Education described in the present invention includes:

[0061] Step 1: Collect article data from 120 disciplines of the Ministry of Education;

[0062] In a preferred but non-limiting embodiment of the present invention, in step 1, the article data includes monographs, journals, books, newspapers, conference papers, scientific reports, standard documents, patent documents, dissertations, and government publications.

[0063] In a preferred but non-limiting embodiment of the present invention, in step 1, at least 20,000 article data are collected for each discipline category of the Ministry of Education.

[0064] Step 2: Screen the subject attribution of the collected article data with the help of GPT to achieve AI annotation and obtain a sample set to ensure the accuracy of the subject attribution of the collected article data;

[0065] In a preferred but non-limiting embodiment of the present invention, in step 2, the method for screening the subject affiliation of the collected article data based on GPT includes:

[0066] The collected article data is sent to GPT to obtain the subject affiliation of each article data.

[0067] In a preferred but non-limiting embodiment of the present invention, in step 2, a single article data and its subject affiliation constitute a sample data, and all article data and its subject affiliation constitute a sample set.

[0068] Step 3: Divide the sample set into training set and test set according to the set ratio;

[0069] In a preferred but non-limiting embodiment of the present invention, in step 3, the ratio is set to 4:1, so that the sample set is divided into a training set and a test set, that is, the number ratio of the sample set to the training set and the test set is 4:1. The specific code for dividing the sample set into the training set and the test set is as follows: Figure 3 shown.

[0070] Step 4: Use the vector model paraphrase-multilingual-mpnet-base-v2 to convert both the training set and the test set into 768-dimensional vectors;

[0071] Step 5: Train the model using the 768-dimensional vectors of the training and test sets.

[0072] In a preferred but non-limiting embodiment of the present invention, step 5 specifically includes the following stages:

[0073] Step 5 consists of the custom F1-score function stage, model construction stage, model training stage, model prediction stage and the attached custom evaluation stage. The device for the classification of resources in the subject field of the Ministry of Education builds deep learning models of different architectures, conducts large-scale data training, and is finally used for the task of resource classification in the subject field of the Ministry of Education.

[0074] Key features:

[0075] Step 5-1: Custom F1-score function stage, which defines and implements a custom F1-score evaluation function to calculate the F1 score during model training and evaluation, and improve the model's ability to handle imbalanced data;

[0076] Step 5-2: Model building phase, its definition provides two neural network models with different architectures, namely _build_m1 and _build_m2, to meet the needs of different tasks;

[0077] Step 5-3: Model training phase, by defining training parameters (such as training cycle, batch size, early stopping mechanism, etc.), the 768-dimensional vectors of the training set and the test set are input into the two neural network models with different architectures, and the Adam optimizer and the category weight balancing method (the category is the subject category of the Ministry of Education, and the corresponding weight balance can be determined according to the actual requirements of the user) are used to optimize the model training process and save the best model as the trained model;

[0078] Step 5-4: Model prediction phase, which loads the trained model, predicts the new article data, and returns the prediction results to achieve the classification of subject field resources for the Ministry of Education. The specific code of the model prediction phase is as follows Figure 6 shown.

[0079] In a preferred but non-limiting embodiment of the present invention, step 5-1 specifically comprises:

[0080] The custom F1-score function includes: the main function, the function for calculating precision, and the function for calculating recall;

[0081] The process of calculating the accuracy function includes:

[0082] Execute the following formula to calculate the accuracy:

[0083] Precision = number of true positive examples obtained by the neural network model / (number of predicted true positive examples + 0.0000001), where 0.0000001 is set to prevent the denominator from being zero;

[0084] The process of calculating the recall function includes:

[0085] The following formula is used to calculate the recall rate:

[0086] Recall rate = number of true positive examples obtained by the neural network model / (number of possible true positive examples + 0.0000001), where 0.0000001 is set to prevent the denominator from being zero;

[0087] The main function is used to call the function for calculating precision and the function for calculating recall in sequence and calculate the F1 score. The formula for calculating the F1 score is as follows:

[0088] F1 score = 2*(precision*recall) / (precision+recall). The specific code of the custom F1-score function stage is as follows Figure 4 shown.

[0089] In a preferred but non-limiting embodiment of the present invention, step 5-2 specifically comprises:

[0090] Construct a neural network model of an architecture, which includes: the first layer of the neural network model contains 128 neurons, and the activation function is RELU; the second layer of the neural network model contains 64 neurons, and the activation function is RELU; the output layer of the neural network model contains a set number of neurons n, and the activation function is softmax;

[0091] Construct a neural network model with a certain architecture, which includes: the first layer of the neural network model contains 256 neurons, and the activation function is RELU; the second layer of the neural network model contains 128 neurons, and the activation function is RELU; the output layer of the neural network model contains a set number of neurons n, and the activation function is softmax. The specific code of the model construction stage is as follows Figure 5 shown.

[0092] The following is an evaluation of the Ministry of Education’s subject classification method at the article level through the custom evaluation stage:

[0093] Conclusion: The latest subject classification scheme of the Ministry of Education for insects (that is, the method of classifying resources in the subject field of the Ministry of Education of the present invention, referred to as insects here) has a great advantage in accuracy compared with the INCITES method of the prior art.

[0094] 500 articles were randomly selected from 97 MOE subjects in INCITES, and a total of 31,423 articles were obtained after deduplication. Then, each article was classified by subject using both INCITES and INCITES methods, and the number of subjects corresponding to each article was calculated.

[0095] Figure 7 A comparison of the number of MOE subjects calculated by INCITES for a single article is shown. Figure 7 It can be seen that insects are generally less than INCITES. The number of insects ranges from 1 to 7, while the number of INCITES ranges from 1 to 60. About 3% of the articles have more than 7 subject classifications.

[0096] Given that INCITES does not have detailed subject coefficients for MOE subjects calculated for individual articles, we used the scores calculated by INCITES for comparison. Figure 8 It shows that 45% of INCITES disciplines have scores below 0 in insects, which means that there is a strong conflict between insects and at least 45% of INCITES discipline classifications. Fig. 9 Schematic diagram of the interdisciplinary intersection and conflict between insects and INCITES.

[0097] Fig.10 for INCITES There are more subjects than insects Schematic diagram of the score distribution, from Fig.10 It can be seen that 60% of the subjects in incites have scores lower than 0.1 in insects, which means that there is a strong conflict between insects and incites in at least 60% of the subject classifications.

[0098] Randomly selecting the titles of 50 articles in a subject that INCITES has more subjects than insects (petroleum and natural gas engineering), it is obvious that incites is wrong to classify these articles as petroleum and natural gas engineering, and insects is correct not to classify these articles as petroleum and natural gas engineering.

[0099] Randomly selecting the titles of 50 articles in a subject (sociology) that has more INCITES subjects than insects, it is obvious that incites is wrong to classify these articles as sociology, and insects is right not to classify these articles as sociology.

[0100] Randomly selecting the titles of 50 articles in a subject (law) that has more INCITES subjects than insects, it is obvious that incites is wrong to classify these articles as law, and insects is right not to classify these articles as law.

[0101] Fig.11 for There are more insects than incites The score distribution of the subject is Fig.11 It can be seen that by randomly selecting the titles of some articles in the subject (petroleum and natural gas engineering) where INCITES is less common than insect subjects, it is obvious that insects are correct in classifying these articles as petroleum and natural gas engineering.

[0102] By randomly selecting the titles of some articles in a discipline that is less common than insects (sociology), it is clear that insects are correct in classifying these articles as oil and gas engineering.

[0103] By randomly selecting the titles of some articles in a subject (law) that is less common in INCITES than in insects, it is obvious that insects are correct in classifying these articles as law.

[0104] In summary, the test model accuracy of the method of the present invention is above 98%.

[0105] like Figure 2 As shown, the device for classifying resources in subject areas of the Ministry of Education described in the present invention includes:

[0106] The collection module is used to collect article data from 120 disciplines of the Ministry of Education (i.e., disciplines and majors of the Ministry of Education);

[0107] The screening module is used to screen the subject attribution of the collected article data based on GPT, so as to achieve AI annotation and obtain a sample set to ensure the accuracy of the subject attribution of the collected article data;

[0108] A partitioning module is used to divide the sample set into a training set and a test set according to a set ratio;

[0109] The conversion module is used to convert both the training set and the test set into 768-dimensional vectors using the vector model paraphrase-multilingual-mpnet-base-v2;

[0110] The training module is used to train the model using 768-dimensional vectors of the training set and the test set.

[0111] The beneficial effects of the present invention are that, compared with the prior art, the technical effects of the present invention include:

[0112] 1. It improves the insufficiency of the subject classification of the Ministry of Education in China and provides high-quality resources for the subject construction of the Ministry of Education;

[0113] 2. Efficient model training: By customizing the F1-score evaluation function and balancing category weights, the performance of the model can be improved when processing unbalanced data; Multi-model architecture: Two different model architectures are provided, and users can choose the appropriate model according to specific tasks, which improves the flexibility and applicability of the device for the classification of subject resources of the Ministry of Education; Automatic model saving and loading: The device for the classification of subject resources of the Ministry of Education automatically saves the best model and automatically loads the model during prediction, which simplifies the model management process; High accuracy and multi-indicator evaluation: Through a variety of evaluation indicators (such as accuracy, precision, recall and F1 score), a more comprehensive model performance evaluation method is provided.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not deviate from the spirit and scope of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for classifying resources in the subject area of ​​the Ministry of Education, characterized in that: include: Step 1: Collect article data from 120 disciplines of the Ministry of Education; Step 2: Screen the subject affiliation of the collected article data based on GPT to achieve AI annotation and obtain a sample set; Step 3: Divide the sample set into training set and test set according to the set ratio; Step 4: Use the vector model paraphrase-multilingual-mpnet-base-v2 to convert both the training set and the test set into 768-dimensional vectors; Step 5: Train the model using the 768-dimensional vectors of the training and test sets.

2. The method for classifying resources in subject areas of the Ministry of Education according to claim 1 is characterized in that: In step 1, the article data includes monographs, journals, books, newspapers, conference papers, scientific and technological reports, standard documents, patent documents, dissertations, and government publications.

3. The method for classifying resources in subject areas of the Ministry of Education according to claim 2 is characterized in that: In step 1, data on at least 20,000 articles were collected for each discipline category of the Ministry of Education.

4. The method for classifying resources in subject areas of the Ministry of Education according to claim 3 is characterized in that: In step 2, the method for screening the subject affiliation of the collected article data based on GPT includes: The collected article data is sent to GPT to obtain the subject affiliation of each article data.

5. The method for classifying resources in subject areas of the Ministry of Education according to claim 4 is characterized in that: In step 2, a single article data and its subject affiliation constitute a sample data, and all article data and its subject affiliation constitute a sample set.

6. The method for classifying resources in subject areas of the Ministry of Education according to claim 5 is characterized in that: In step 3, the ratio is set to 4:1, so that the sample set is divided into a training set and a test set, that is, the number ratio of the sample set to the training set and the test set is 4:

1.

7. The method for classifying resources in subject areas of the Ministry of Education according to claim 6 is characterized in that: Step 5 specifically includes the following stages: Step 5-1: Custom F1-score function stage, which defines and implements a custom F1-score evaluation function for calculating the F1 score during model training and evaluation; Step 5-2: Model building phase, its definition provides two neural network models with different architectures, namely _build_m1 and _build_m2; Step 5-3: Model training phase, by defining training parameters, the 768-dimensional vectors of the training set and the test set are input into the two neural network models with different architectures, and the Adam optimizer and the category weight balancing method are used to optimize the model training process, and the best model is saved as the trained model; Step 5-4: Model prediction stage, which loads the trained model, predicts the new article data, and returns the prediction results to achieve the classification of subject area resources for the Ministry of Education.

8. The method for classifying resources in subject areas of the Ministry of Education according to claim 7 is characterized in that: Step 5-1 specifically includes: The custom F1-score function includes: the main function, the function for calculating precision, and the function for calculating recall; The process of calculating the accuracy function includes: Execute the following formula to calculate the accuracy: Precision = number of true positive examples obtained by the neural network model / (number of predicted true positive examples + 0.0000001); The process of calculating the recall function includes: The following formula is used to calculate the recall rate: Recall rate = number of true positive examples obtained by the neural network model / (number of possible true positive examples + 0.0000001); The main function is used to call the function for calculating precision and the function for calculating recall in sequence and calculate the F1 score. The formula for calculating the F1 score is as follows: F1 score = 2*(precision*recall) / (precision+recall).

9. The method for classifying resources in subject areas of the Ministry of Education according to claim 8 is characterized in that: Step 5-2 specifically includes: Construct a neural network model of an architecture, which includes: the first layer of the neural network model contains 128 neurons, and the activation function is RELU; the second layer of the neural network model contains 64 neurons, and the activation function is RELU; the output layer of the neural network model contains a set number of neurons n, and the activation function is softmax; A neural network model with an architecture is constructed, which includes: the first layer of the neural network model contains 256 neurons, and the activation function is RELU; the second layer of the neural network model contains 128 neurons, and the activation function is RELU; the output layer of the neural network model contains a set number of neurons n, and the activation function is softmax.

10. A device for classifying subject-area resources of the Ministry of Education, characterized in that: include: The collection module is used to collect article data from 120 disciplines of the Ministry of Education; The screening module is used to screen the subject affiliation of the collected article data based on GPT, so as to achieve AI annotation and obtain a sample set; A partitioning module is used to divide the sample set into a training set and a test set according to a set ratio; The conversion module is used to convert both the training set and the test set into 768-dimensional vectors using the vector model paraphrase-multilingual-mpnet-base-v2; The training module is used to train the model using 768-dimensional vectors of the training set and the test set.

Images