Enterprise attachment industry method based on small sample classification

By combining few-shot learning and pre-trained models with prototype networks, the problem of insufficient sample size in the six major industry categories of enterprises affiliating with the Science and Technology Innovation Board was solved. This approach achieved high classification accuracy and low-cost model training, thereby improving the accuracy of evaluating the science and technology innovation attributes of enterprises.

CN118965102APending Publication Date: 2024-11-15SHANGHAI STOCK COMM CO LTD

Patent Information

Application Number
CN202410986619.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing technologies suffer from insufficient sample size and labeled data in the six major industry categories for companies listing on the Science and Technology Innovation Board, resulting in poor classification model performance. Furthermore, traditional models fail to fully utilize category label information.

Method used

We employ a few-shot learning approach, using the pre-trained Chinese model bert-base-chinese to convert text data into vector representations. We also expand the semantic information of the labels through web crawling, adjust the sample vectors using a prototype network, calculate Euclidean distance for classification, and optimize the model parameters using the cross-entropy loss function.

Benefits of technology

It improves classification accuracy under small sample conditions, reduces training costs and computational resource requirements, and enhances the model's performance in similar text classification tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118965102A_ABST
    Figure CN118965102A_ABST
Patent Text Reader

Abstract

The invention relates to a small sample classification-based enterprise attachment industry method, which comprises the following steps of: S1, acquiring original text data, and carrying out label labeling; s2, a Chinese pre-training model is used as an encoder, and text data is converted into vector representation; s3, expanding a label text vector; s4, generating a support set by using a prototype network algorithm in combination with the text vector and the expanded tag text vector, and adjusting a support set sample vector; s5, defining prototype vectors, and calculating Euclidean distances between the query set sample vectors and various prototype vectors; s6, selecting the industrial label corresponding to the prototype vector with the minimum Euclidean distance as a sample classification result of the query set; s7, calculating a cross entropy loss function and updating encoder parameters by adopting a back propagation algorithm so as to optimize the classification model; and S8, carrying out classification tasks of six major industries of the enterprise affiliation department wound board. The method has the advantages that the performance of a small sample task is remarkably improved, and the risk of overfitting on a small data set is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of enterprise affiliation industry methods, and specifically to an enterprise affiliation industry method based on small sample classification. Background Art

[0002] The scientific and technological innovation attributes can help enterprises implement their scientific and technological innovation positioning, support and encourage "hard technology" enterprises to be listed on the Science and Technology Innovation Board; accelerate the transformation of scientific and technological achievements into real productivity, and promote the transformation of economic development to innovation-driven. Based on the classification of "hard technology" enterprises, the Science and Technology Innovation Board focuses on supporting six high-tech industries and strategic emerging industries, namely, new generation information technology, high-end equipment, new materials, new energy, energy conservation and environmental protection, and biomedicine. However, the evaluation of the scientific and technological innovation attributes of enterprises cannot be separated from the comparison with enterprises in the same industry.

[0003] The industry that companies are affiliated with is an industry classification problem with a small sample size. Currently, there are two main types of industry classification technologies: one is to use artificially established word libraries to match business and product terms. This method focuses on building a complete and comprehensive industry word library to ensure fast, accurate and high-quality word matching; the other is to use machine learning methods. First, fine-tune the pre-trained model, use the fine-tuned model to convert the company's text information into text vector encoding, and then train the classification model with a large amount of labeled data to classify the corresponding industry of the company. This method focuses on how to obtain high-quality data to fine-tune the pre-trained model and how to optimize the classification model based on machine learning methods to improve the classification accuracy.

[0004] Among the existing technical solutions, the vocabulary-based method requires a lot of manual labor costs for vocabulary construction, calibration, and updating. Secondly, the matching model based on machine learning methods needs to be trained with a large amount of labeled data. However, for the classification task of enterprises affiliated with the six major industries of the Science and Technology Innovation Board, there is no large amount of labeled data for similar industries and products, resulting in poor performance of existing models in this scenario. Moreover, existing machine learning technologies are all trained based on labeled data. After the data is labeled, the labels often lose their actual connotations and are only codes to distinguish different categories. Traditional models do not take into account the information that can be fully utilized from category labels.

[0005] There is an urgent need to propose a method for enterprise affiliation based on small sample learning to train industry classification models and facilitate the horizontal comparison and evaluation of enterprise scientific and technological innovation attributes. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an enterprise affiliation industry method based on small sample classification, which is used to adjust the sample vector in small sample learning and improve the classification accuracy.

[0007] In order to achieve the above purpose, a method for enterprise affiliation industry based on small sample classification is provided, the method is as follows: S1. Obtain original text data from the prospectus of listed science and technology innovation board companies, and label them according to the six major industries of the science and technology innovation board; S2. Use the Chinese pre-trained model bert-base-chinese as an encoder to convert the text data into a vector representation; S3. For each industry label, retrieve its semantic information on Wikipedia through crawler technology, and combine the information with the industry label to obtain an expanded label text vector; S4. In the prototype network, combine the text vector with the expanded label text vector, and adjust the support set sample vector through vector operation; S5. Combine the new vector obtained by averaging the sample vectors of each class of support set as the prototype vector of the class, and calculate the Euclidean distance between the query set sample vector and each class of prototype vectors; S6. Select the industry label corresponding to the prototype vector with the smallest distance as the classification result of the query set sample according to the Euclidean distance; S7. Update the encoder parameters by calculating the cross entropy loss function and using the back propagation algorithm to optimize the classification model; S8. Use 6-way The classification model is trained with 1-shot data. After the training, the model parameters are fixed and the classification task of the six major industries of enterprises affiliated to the Science and Technology Innovation Board is carried out.

[0008] Preferably, step S1 of the present invention is specifically as follows: acquisition and segmentation of original data, the original enterprise data comes from the relevant text descriptions in the prospectuses of publicly disclosed and listed science and technology innovation board companies, and these original enterprise text data are used as texts for training industry classification models, and are labeled with the six major industries of new generation information technology, high-end equipment manufacturing, new materials, new energy, biomedicine, and energy conservation and environmental protection.

[0009] Preferably, step S2 of the present invention is as follows: processing the text and label of step S1, and converting the text and label into a vector input model through an encoder. In a 2-way 2-shot task, the original training sample is { label 1:[ text 11 , text 12 , text 13 , ...], label 2: [ text 21 , text 22 , text 23, ...], ...}, the samples used for training have multiple categories and samples, which are used to train a classifier that can complete the 2-way 2-shot task. The 2-way 2-shot classification task refers to a small sample classification task in which the samples to be classified have two categories and each has only two sample data.

[0010] Preferably, the original data annotated in step S1 of the present invention needs to be split into training text and label text and input into the encoder respectively.

[0011] Preferably, the method of the present invention uses a prototype network to train a classifier for a 2-way 2-shot task, and selects a 2-way 2-shot support set from the original training data for generating a prototype. Let the selected support set be { L A : [ A 1, A 2], L B :[ B 1, B 2]}; At the same time, it is necessary to randomly select another sample from the original data of this 2-way as the query set, which will be used to verify the prediction results. Let the query set be { L A : [ A 3]}, where [ A 1, A 2, B 1, B 2, A 3] is the training text; [ L A , L B ] is the label text, representing category A and category B. Before entering the encoder, the label text needs to be further processed; the crawler method is used to obtain the semantics and interpretation corresponding to the label text on Wikipedia, and the label and semantics are spliced ​​and combined into the text form of "label: semantics", still in [ L A , L B ] represents the expanded label text; use the pre-trained model bert-base-chinese as the initial encoder to map the input Chinese text to a vector. Let the vectors mapped from the input training text and label text be: A 1:[ a 11 , a 12 , a 13 , a 14 , a15 ]; A 2:[ a 21 , a 22 , a 23 , a 24 , a 25 ]; A 3:[ a 31 , a 32 , a 33 , a 34 , a 35 ]; B 1:[ b 11 , b 12 , b 13 , b 14 , b 15 ]; B 2:[ b 21 , b 22 , b 23 , b 24 , b 25 ]; L A :[ l 11 , l 12 , l 13 , l 14 , l 15 ]; L B :[ l 21 , l 22 , l 23 , l 24 , l 25 ]; The new vector obtained by summing and averaging the sample vectors of each class in the support set is the prototype vector of that class: A =(A 1+ A 2) / 2, B =( B 1+ B 2) / 2, and then calculate the Euclidean distance from the query set sample vector to the prototype vector of each category, and take the category corresponding to the minimum distance as the classification result of the query set; query set sample A 3 Off-prototype vector A The distance is the smallest and is predicted to be class A, while the sample label is L A , it is actually Class A.

[0012] Preferably, step S7 of the present invention is as follows: after prediction using the query set, according to the correctness of the classification result, the loss function is calculated and back-propagated to update the parameters of the encoder, and the model is trained to minimize the loss function so that it fits the data to improve the classification accuracy. The loss function used is the cross entropy loss, and the formula is expressed as: ;in C represents the number of categories, p i is the true value, q i is the predicted value.

[0013] Preferably, step S8 of the present invention is specifically as follows: using the prototype network improved by steps S2-S7, using 6-way1-shot data to train a model suitable for six classification tasks; after the model is trained, the model parameters are fixed to complete the classification task; when the prototype network is classified, the same operations of data input, encoding, prototype generation and prediction are performed, but the parameters are not updated, and only the classification results are output; the actual operation is to use the corporate texts of the existing six major industries as the support set and the corporate texts to be classified as the query set, generate prototypes corresponding to the six major industries through the support set, calculate the Euclidean distance from the query set to the prototypes of the six major industries, and take the industry category corresponding to the prototype with the smallest distance as the classification result output.

[0014] Compared with the prior art, the present invention has the following advantages: 1. This patent involves a small sample classification task, using a pre-trained BERT model. The pre-trained model has been trained on a large amount of Chinese corpus and can capture rich text feature representations. These features can be transferred to new tasks to improve the performance of the model. Therefore, the use of a pre-trained model can bring significant performance improvements on small sample tasks and reduce the risk of overfitting on small data sets.

[0015] 2. The pre-trained model has been trained on a large-scale dataset, so the high computational cost of training the model from scratch can be avoided. Therefore, only fine-tuning on the dataset of the corresponding task is required, which usually requires less time and computing resources. Moreover, during fine-tuning, the model can focus on learning features related to the target task.

[0016] 3. This method mainly generates vectors by encoding with a pre-trained model, and then the prototype network realizes classification. The prototype network is very simple and only requires the Euclidean distance of the vector, so the main complexity is in the encoder, and the pre-trained model + fine-tuning method used by the encoder greatly reduces the training cost. Therefore, the method proposed in this patent is relatively simple and effective to implement, and can be quickly and effectively reused on similar text classification tasks.

[0017] 4. In addition to the convenience of the model, this patent also introduces label information into text vector generation through semantic expansion. This move allows the model to capture the similarity and correlation between words while capturing the similarity between different category labels and between different words and labels. Especially in multi-label classification tasks, label embedding can help the model better handle the dependency between labels and improve classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 , is a flow chart of the present invention; Figure 2 , is a schematic diagram of the improved prototype network of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, principle and structure of the present invention more clear, it is further described below in conjunction with the drawings and specific embodiments.

[0020] The present invention provides an enterprise affiliation industry method based on small sample classification, which aims to expand the classification label information by using the knowledge of Wikipedia to adjust the sample vector in small sample learning and improve the classification accuracy. The overall framework is as follows Figure 1 As shown. Now, taking the small sample classification task of 2 categories with 2 samples each (2-way 2-shot) as an example, the detailed description of the technical operation using the prototype network is as follows: See also Figure 1 , Figure 2 .

[0021] Step 1: Acquisition and division of original data. The original enterprise data comes from the relevant text descriptions in the prospectus of publicly disclosed and listed companies on the Science and Technology Innovation Board. The relevant texts in the prospectus mainly include: "Operation of the Issuer's Main Business" in "Section 2 Overview"; "Issuer's Main Business and Main Products and Services", "Company's Core Technology", "Issuer's Sales and Major Customers", "Issuer's Raw Material Procurement and Major Suppliers" in "Section 6 Business and Technology". Based on the six major industry divisions of the Science and Technology Innovation Board, these original enterprise text data are used as texts for training industry classification models, and are labeled with the six major industries of new generation information technology, high-end equipment manufacturing, new materials, new energy, biomedicine, and energy conservation and environmental protection.

[0022] Step 2: Model training. Figure 1 As shown in Figure 1, the text and labels are processed and converted into vectors for input into the model through the encoder. Figure 2 The prototype network shown in the figure is used as an example to illustrate the specific operation process. In the 2-way 2-shot task, let's assume that the original training samples are {label1: [text11, text12, text13, ...], label2: [text21, text22, text23, ...], ...}. The samples used for training have multiple categories and samples, mainly used to train a classifier that can complete the 2-way 2-shot task. The 2-way 2-shot classification task refers to a small sample classification task in which the samples to be classified have two categories and each has only two sample data.

[0023] Step 2.1: Data input. Figure 1 As shown, the original training data needs to be split into training text and label text and input into the encoder separately. Figure 2 (1) in the figure details how to split: we use the prototype network to train the classifier for the 2-way 2-shot task. We need to select the 2-way 2-shot support set from the original training data to generate the prototype. Let the selected support set be {LA: [A1,A2],LB:[B1,B2]}; at the same time, we need to randomly select another sample from the 2-way original data as the query set. The query set will be used to verify the prediction result. Let the query set be {LA: [A3]}. Among them, [A1,A2,B1,B2, A3] is the training text; [LA,LB] is the label text, representing class A and class B. Before entering the encoder, the label text must be further processed.

[0024] Step 2.1.1: Label text expansion. Label texts are nouns with practical meanings. We can use crawlers to obtain the semantics and interpretations of label texts on Wikipedia, and combine the labels and semantics into a text form of "label: semantics", and still use [LA, LB] to represent the expanded label text.

[0025] Step 2.2: Vector generation. Figure 2 As shown in (2), we use the pre-trained model bert-base-chinese as the initial encoder to map the input Chinese text into a vector. The actual length of the vector is set to 768. For ease of understanding, let's assume that the vector mapped from the input training text and the label text is: A1:[a11,a12,a13,a14,a15]; A2:[a21,a22,a23,a24,a25]; A3:[a31,a32,a33,a34,a35]; B1:[b11,b12,b13,b14,b15]; B2:[b21,b22,b23,b24,b25]; LA:[l11,l12,l13,l14,l15]; LB:[l21,l22,l23,l24,l25].

[0026] Step 2.3: Adjust the vector. Figure 2 As shown in (3), the support vector of the training text is adjusted using the label text vector. [A1, A2, B1, B2] is used as the support set sample vector, and the corresponding expanded label text vector is added and averaged to obtain: A1=(A1+LA) / 2, A2=(A2+LA) / 2, B1=(B1+LB) / 2, B2=(B2+LB) / 2.

[0027] So the adjusted vector is: A1=[(a11+l11) / 2, (a12+l12) / 2, (a13+l13) / 2, (a14+l14) / 2, (a15+l15) / 2], A2=[(a21+l11) / 2, (a22+l12) / 2, (a23+l13) / 2, (a24+l14) / 2, (a25+l15) / 2], B1=[(b11+l21) / 2, (b12+l22) / 2, (b13+l23) / 2, (b14+l24) / 2, (b15+l25) / 2], B2=[(b21+l21) / 2, (b22+l22) / 2, (b23+l23) / 2, (b24+l24) / 2, (b25+l25) / 2].

[0028] Step 2.4: Generate prototypes and predictions. Figure 2 As shown in (4), we sum and average the sample vectors of each class in the support set to obtain a new vector, which is the prototype vector of that class: A=(A1+A2) / 2, B=(B1+B2) / 2. Then calculate the Euclidean distance between the query set sample vector and the prototype vector of each class, and take the class corresponding to the minimum distance as the classification result of the query set. In the figure, the query set sample A3 has the shortest distance from the prototype vector A and is predicted to be class A, while the sample label is LA, which is actually class A.

[0029] Step 2.5: Back propagation and parameter update. Figure 2 As shown in (5), after using the query set for prediction, the loss function is calculated and back-propagated according to the correctness of the classification result to update the encoder parameters and train the model to minimize the loss function so that it fits the data to improve the classification accuracy. The loss function we use is the cross entropy loss, which is expressed as: ; Where C represents the number of categories, pi is the true value, and qi is the predicted value.

[0030] Step 3: Classification. According to the needs of enterprises to be affiliated with the six industries of the Science and Technology Innovation Board, we need to perform six classification tasks. Using the improved prototype network, a model suitable for six classification tasks can be trained using 6-way 1-shot data. After the model is trained, the model parameters are fixed to complete the classification task. When classifying the prototype network, similar to model training, the same operations of data input, encoding, prototype generation and prediction are performed, but the parameters are not updated, and only the classification results are output. The actual operation is to use the corporate texts of the six major industries as the support set and the corporate texts to be classified as the query set. The prototypes corresponding to the six major industries are generated through the support set, and the Euclidean distance from the query set to the prototypes of the six major industries is calculated. The industry category corresponding to the prototype with the smallest distance is taken as the classification result output.

[0031] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent substitutions or changes according to the technical solutions and novel concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for enterprise affiliation based on small sample classification, characterized by The method is as follows: S1. Obtain original text data from the prospectuses of listed Sci-Tech Innovation Board companies and label them according to the six major industries of the Sci-Tech Innovation Board; S2. Use the Chinese pre-trained model bert-base-chinese as an encoder to convert the text data into a vector representation; S3. For each industry tag, retrieve its semantic information on Wikipedia using crawler technology, and combine the information with the industry tag to obtain an expanded tag text vector; S4. In the prototype network, the text vector and the expanded label text vector are combined and adjusted through vector operation to obtain the support set sample vector; S5. The new vector obtained by averaging the sample vectors of each class of support set is used as the prototype vector of the class, and the Euclidean distance between the sample vectors of the query set and the prototype vectors of each class is calculated; S6. Select the industry label corresponding to the prototype vector with the smallest distance according to the Euclidean distance as the classification result of the query set sample; S7. Optimize the classification model by calculating the cross entropy loss function and updating the encoder parameters using the back propagation algorithm; S8. Use 6-way 1-shot data to train the classification model. After training, fix the model parameters and carry out the classification task of the six industries of enterprises affiliated to the Science and Technology Innovation Board.

2. The method for enterprise affiliation based on small sample classification as claimed in claim 1, characterized in that The step S1 is specifically as follows: acquisition and division of original data. The original enterprise data comes from the relevant text descriptions in the prospectuses of publicly disclosed and listed science and technology innovation board companies. These original enterprise text data are used as texts for training industry classification models, and are labeled with the six major industries of new generation information technology, high-end equipment manufacturing, new materials, new energy, biomedicine, and energy conservation and environmental protection.

3. The method for enterprise affiliation based on small sample classification as claimed in claim 1, characterized in that The step S2 is as follows: process the text and labels of step S1, and convert the text and labels into vectors for input into the model through the encoder. In the 2-way 2-shot task, the original training sample is { label 1: [ text 11 , text 12 , text 13 , ...], label 2: [ text 21 , text 22 , text 23 , ...], ...}, the samples used for training have multiple categories and samples, which are used to train a classifier that can complete the 2-way 2-shot task. The 2-way 2-shot classification task refers to a small sample classification task in which the samples to be classified have two categories and each has only two sample data.

4. The method for enterprise affiliation based on small sample classification as claimed in claim 1, characterized in that The original data annotated in step S1 needs to be split into training text and label text and input into the encoder respectively.

5. The method for enterprise affiliation based on small sample classification as claimed in claim 4, characterized in that The method uses a prototype network to train a classifier for a 2-way 2-shot task, and selects a 2-way 2-shot support set from the original training data to generate a prototype. Let the selected support set be { L A : [ A 1, A 2], L B :[ B 1, B 2]}; At the same time, it is necessary to randomly select another sample from the original data of this 2-way as the query set, which will be used to verify the prediction results. Let the query set be { L A : [ A 3]}, where [ A 1, A 2, B 1, B 2, A 3] is the training text; [ L A , L B ] is the label text, representing class A and class B. Before entering the encoder, the label text needs to be further processed; Use crawler methods to obtain the semantics and interpretation of the tag text on Wikipedia, and combine the tag and semantics into a text form of "tag: semantics", still in [ L A , L B ] represents the expanded label text; Use the pre-trained model bert-base-chinese as the initial encoder to map the input Chinese text to a vector. Suppose the vectors mapped from the input training text and the label text are: A 1:[ a 11 , a 12 , a 13 , a 14 , a 15 ]; A 2:[ a 21 , a 22 , a 23 , a 24 , a 25 ]; A 3:[ a 31 , a 32 , a 33 , a 34 , a 35 ]; B 1:[ b 11 , b 12 , b 13 , b 14 , b 15 ]; B 2:[ b 21 , b 22 , b 23 , b 24 , b 25 ]; L A :[ l 11 , l 12 , l 13 , l 14 , l 15 ]; L B :[ l 21 , l 22 , l 23 , l 24 , l 25 ]; The new vector obtained by summing and averaging the sample vectors of each class in the support set is the prototype vector of that class: A =( A 1+ A 2) / 2, B =( B 1+ B 2) / 2, and then calculate the Euclidean distance from the query set sample vector to the prototype vector of each category, and take the category corresponding to the minimum distance as the classification result of the query set; query set sample A 3 Off-prototype vector A The distance is the smallest and is predicted to be class A, while the sample label is L A , it is actually Class A.

6. The method for enterprise affiliation based on small sample classification as claimed in claim 1, characterized in that The step S7 is specifically as follows: after prediction is performed using the query set, the loss function is calculated and back-propagated according to the correctness of the classification result to update the parameters of the encoder, and the model is trained to minimize the loss function so that it fits the data to improve the classification accuracy. The loss function used is the cross entropy loss, which is expressed as follows: ; in C represents the number of categories, p i is the true value, q i is the predicted value.

7. The method for enterprise affiliation based on small sample classification as claimed in claim 1, characterized in that The step S8 is specifically as follows: using the prototype network improved by steps S2-S7, a model suitable for six classification tasks is trained using 6-way 1-shot data. After the model is trained, the model parameters are fixed to complete the classification task. When the prototype network is classified, the data input, encoding, prototype generation and prediction operations are also performed, but the parameters are not updated, and only the classification results are output. The actual operation is to use the corporate texts of the six major industries as the support set and the corporate texts to be classified as the query set, generate prototypes corresponding to the six major industries through the support set, calculate the Euclidean distance from the query set to the prototypes of the six major industries, and take the industry category corresponding to the prototype with the smallest distance as the classification result output.

Citation Information

Patent Citations

  • Small sample text automatic classification method and system

    CN117520551A

  • Semantic attention element learning method based on Bayesian estimation

    CN117523278A

  • Event classification model training method and device, event classification model using method and device and medium

    CN117851601A

Cited By

  • Multi-modal classification model training method, multi-modal classification method and storage medium

    CN120565114A