Document multi-classification method based on cooperation of large language model and deep learning algorithm

Abstract

The invention discloses a document multi-classification method based on cooperation of a large language model and a deep learning algorithm, relates to the technical field of document classification, and solves the technical problem of low final classification efficiency caused by the fact that different document types are not considered in the prior art. According to the method, user documents and historical data are collected, the user documents are input into an LLM model, document key information and important keywords are extracted, prediction tags are obtained by combining document theme prediction, and the prediction tags and the important keywords are bound to obtain classification tags; setting an information coefficient based on historical data, calculating according to the classification label and a set correct prediction label to obtain a prediction correct rate, and combining the prediction correct rate with the information coefficient to obtain a document classification confidence coefficient; setting a confidence coefficient threshold according to the historical data, comparing the document classification confidence coefficient with the confidence coefficient threshold, and obtaining a classification state according to a comparison result; and adjusting and optimizing the large language model based on the classification state to obtain a final classification confidence coefficient.

Description

Technical Field

[0001] The present invention belongs to the field of document classification, and specifically is a document multi-classification method that collaborates with a large language model and a deep learning algorithm. Background Art

[0002] In the current document management field, document classification is not only the basis of information retrieval and knowledge management, but also a key link to promote business process automation and improve decision-making efficiency. With the advent of the big data era, the number and complexity of documents have increased dramatically. Traditional document classification methods such as manual classification or automatic classification technology based on simple rules can no longer meet the current high standards for classification accuracy and efficiency.

[0003] The invention patent with application number CN2020102156429 discloses a training method, document classification method, device and equipment for a document classification model. The invention collects a large number of training sample documents, reconstructs the training sample documents through a preset reconstruction algorithm to obtain reconstructed sample documents, and trains a neural network model through the reconstructed sample documents to obtain a document classification model; when this method reconstructs the sample documents, the accuracy of the reconstructed sample documents for the final document classification result is unpredictable. For different types of documents, the required classification requirements will be different due to their different types, which may cause the final classification results to not meet the requirements, and the resource cost of retraining is high, resulting in low classification efficiency.

[0004] The present invention provides a document multi-classification method that collaborates with a large language model and a deep learning algorithm to solve the above technical problems. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a document multi-classification method that collaborates with a large language model and a deep learning algorithm, which is used to solve the technical problem that the prior art does not take into account the differences in document types, resulting in low final classification efficiency.

[0006] To achieve the above objectives, the first aspect of the present invention provides a document multi-classification method that cooperates with a large language model and a deep learning algorithm, comprising:

[0007] Step 1: Collect user documents, input user documents into the LLM model to obtain key information of the documents, and extract several sets of historical data from the database;

[0008] Step 2: Classify user documents based on key information of the documents to obtain classification labels;

[0009] Step 3: Obtain document classification confidence based on classification labels; set a confidence threshold based on historical data, and compare the document classification confidence with the confidence threshold to obtain the classification status;

[0010] Step 4: Optimize the document classification confidence based on the classification status to obtain the final classification confidence; based on the final classification.

[0011] Preferably, classifying the user document based on the key information of the document to obtain a classification label includes:

[0012] Extract key information of the document; wherein the key information of the document includes the document subject, important paragraphs of the document and the subject of the paragraph of the document;

[0013] Important keywords in important paragraphs of the document are extracted through natural language processing technology, and the document topic and important keywords are input into the large language model as input data to predict the document label and obtain the predicted label of the document. Each important keyword is bound to the corresponding predicted label to obtain the classification label.

[0014] Preferably, obtaining the document classification confidence according to the classification label includes:

[0015] The number of all different prediction labels in the classification label is counted, and the correct prediction label is manually selected, and the ratio of the number of correct prediction labels to the number of all prediction labels is calculated to obtain the prediction accuracy; the information coefficient is set according to the key information of historical documents in the historical data, and the prediction accuracy is multiplied by the information coefficient to obtain the document classification confidence.

[0016] Preferably, the information coefficient is set according to the key information of the historical document, including:

[0017] Extracting key information of historical documents; wherein the key information of historical documents includes the subject of historical documents, important historical paragraphs and the subject of historical paragraphs;

[0018] Mark the historical data with the same historical document theme as the document theme as historical matching data, and determine whether the historical paragraph theme in the historical matching data is the same as the historical document theme; if yes, set the corresponding paragraph coefficient to α; if no, set the corresponding paragraph coefficient to 1-α; where 0.5<α<1;

[0019] The paragraph coefficients of all important historical paragraphs are counted and the average is calculated, and the average of the paragraph coefficients is used as the information coefficient.

[0020] Preferably, setting the confidence threshold according to historical data includes:

[0021] Extract historical data; the historical data includes historical classification labels and historical correct labels obtained by the model for document recognition;

[0022] The historical prediction rate is obtained based on the historical classification labels and historical correct labels of the historical matching data; the confidence threshold is obtained by multiplying the information coefficient by the historical prediction rate.

[0023] Preferably, the step of obtaining the historical prediction rate according to the historical classification labels and the historical correct labels of the historical matching data includes:

[0024] Input the historical classification label into the demand setting model to obtain the demand coefficient; determine whether the historical classification label is the same as the historical correct label; if yes, mark the corresponding historical classification label as the historical prediction correct label; if no, mark the corresponding historical classification label as the historical prediction wrong label; wherein the demand setting model is constructed based on the neural network model;

[0025] The historical prediction rate is obtained by calculating the ratio of the number of historical prediction correct labels to the number of historical classification labels and multiplying it by the demand coefficient.

[0026] Preferably, the demand setting model is constructed based on a neural network model, including:

[0027] Historical classification labels and corresponding historical demand coefficients are extracted from the database; the historical classification labels and corresponding historical demand coefficients are used as training data and test data to train the neural network model. After the training, the training effect is tested through the test data, and the neural network model is adjusted according to the test results to obtain a bone demand setting model with the input data as the classification label and the output data as the demand coefficient.

[0028] Preferably, the step of comparing the document classification confidence with a confidence threshold to obtain the classification status includes:

[0029] Extract document classification confidence and confidence threshold;

[0030] Determine whether the document classification confidence is less than the confidence threshold; if yes, mark the classification status as unsatisfactory; if no, mark the classification status as satisfactory.

[0031] Preferably, the optimizing the document classification confidence based on the classification state to obtain the final classification confidence includes:

[0032] When the classification status is up to standard, the document classification confidence is used as the final classification confidence;

[0033] When the classification status is not up to standard, the LLM model is adjusted based on the document classification confidence and the confidence threshold, and the document classification confidence obtained after adjustment is marked as the optimized classification confidence; it is judged whether the optimized classification confidence is greater than the document classification confidence; if so, the optimized classification confidence is used as the final classification confidence; if not, the document classification confidence is used as the final classification confidence.

[0034] Preferably, the adjusting the LLM model based on the document classification confidence and the confidence threshold includes:

[0035] Calculate the confidence difference between the document classification confidence and the confidence threshold; normalize the confidence difference to obtain a standard confidence difference;

[0036] The parameters initialized by the LLM model are marked as initial parameters, the initial parameters are subtracted from the standard confidence difference to obtain the optimized parameters, and the optimized parameters are input into the LLM model for updating to obtain the adjusted model.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. The present invention collects user documents and historical data, inputs user documents into the LLM model to extract key document information from user documents, extracts important keywords from important paragraphs of documents, obtains prediction labels based on document topics and important keywords, and binds prediction labels with important keywords to obtain classification labels, which helps to more accurately reflect the core content of documents, thereby improving the accuracy of classification; sets information coefficients based on historical data, calculates prediction accuracy based on classification labels and set correct prediction labels, and obtains document classification confidence by combining prediction accuracy and information coefficient, so that the system can more accurately evaluate the importance of each feature, thereby improving the accuracy of classification; matches historical document topics in historical data with document topics to obtain historical matching data, obtains historical prediction rates based on historical matching data, and obtains confidence thresholds by combining historical prediction rates and information coefficients; obtains classification status based on the comparison results of document classification confidence and confidence thresholds, and adjusts and optimizes the large language model based on the classification status to obtain final classification confidence, which can improve the quality of classification decisions and reduce the possibility of misclassification, and can flexibly adjust classification standards under different circumstances, thereby ensuring the stability and consistency of classification results.

[0039] 2. The present invention calculates the information coefficient based on historical document information, obtains the document classification confidence in combination with the prediction accuracy, sets the confidence threshold based on historical data, compares the document classification confidence with the confidence threshold, and obtains the classification status based on the comparison result; when the classification status is not up to standard, the LLM model is adjusted based on the document classification confidence and the confidence threshold to obtain the final classification confidence, so that the system can more accurately evaluate the importance of each feature, thereby improving the accuracy of classification, and the model adjustment and optimization enables the model to continuously learn and improve from the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 The following is a flowchart of an embodiment of the present invention.

[0042] Figure 2 The following is a complete flowchart for obtaining document classification confidence in one embodiment of the present invention.

[0043] Figure 3 The complete flowchart of obtaining the final classification confidence in one embodiment of the present invention. DETAILED DESCRIPTION

[0044] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1-Figure 3 The first aspect of the present invention provides a document multi-classification method that combines a large language model with a deep learning algorithm, including:

[0046] Step 1: Collect user documents, input the user documents into the LLM model to obtain the key information of the documents, and extract several sets of historical data from the database.

[0047] Exemplarily, a user document uploaded by a user is received, and the user document is input into an LLM model (large language model) to extract document topics, important paragraphs of the document, and document paragraph topics of important paragraphs of the document; and several groups of classified historical data are extracted from a database.

[0048] In some other preferred embodiments, if the user document exceeds the maximum input length of the LLM model, the user document can be segmented into document blocks, and important paragraphs in the user document can be extracted and reassembled to obtain the overall key content of the document.

[0049] Step 2: Classify user documents based on document key information to obtain classification labels.

[0050] Exemplarily, important keywords in important paragraphs of a document are extracted through natural language processing technology, and the document topic and several extracted important keywords are respectively input into a large language model as several groups of input data, and the category label of the document is predicted to obtain several predicted labels predicted by the model based on the important keywords, and each important keyword is bound to the corresponding predicted label to obtain several groups of classification labels.

[0051] In some other preferred embodiments, important paragraphs of the document may be directly input as input data into a large language model for classification prediction. Compared with using important keywords for classification prediction, the amount of data using important paragraphs of the document is less, but the prediction accuracy is higher.

[0052] Step 3: Get the document classification confidence based on the classification label; set the confidence threshold based on historical data, and compare the document classification confidence with the confidence threshold to obtain the classification status.

[0053] Exemplarily, the number of all different prediction labels in the classification labels is counted. In this embodiment, a total of 5 prediction labels are predicted, namely, Class A prediction labels, Class B prediction labels, Class C prediction labels, Class D prediction labels and Class E prediction labels; the number of prediction labels for Class A prediction labels is 35, the number of prediction labels for Class B prediction labels is 12, the number of prediction labels for Class C prediction labels is 3, the number of prediction labels for Class D prediction labels is 2, and the number of prediction labels for Class E prediction labels is 3; in this embodiment, the correct prediction label selected should be the Class A prediction label, and the ratio of the correct prediction label to the total number of prediction labels is calculated to obtain a prediction accuracy of 64%.

[0054] Match the historical document topics in the historical data with the document topic of the current document, and mark the historical data with the same document topic as the current document as the historical matching data; determine whether the historical paragraph topic in the historical matching data is the same as the historical document topic, if the same, mark the paragraph coefficient of the paragraph as α, if different, mark the paragraph coefficient of the paragraph as 1-α; in this embodiment, set α to 0.8, and the paragraph coefficient is set by the staff, indicating the degree of correlation between the paragraph and the document topic, and 0.5<α<1; calculate the average value of the paragraph coefficients of all historical important paragraphs in the historical document to obtain the information coefficient; multiply the prediction accuracy by the information coefficient to obtain the document classification confidence, so that the system can more accurately evaluate the importance of each feature, thereby improving the accuracy of classification.

[0055] Extract historical classification labels and corresponding historical demand coefficients from the database; use the historical classification labels and corresponding historical demand coefficients as training data and test data to train the neural network model. After the training, test the training effect through the test data, and adjust the neural network model according to the test results to obtain a demand setting model with the input data as the classification label and the output data as the demand coefficient; input the historical classification labels in the historical matching data into the demand setting model to obtain the demand coefficient. In this embodiment, the demand coefficient obtained is 3.

[0056] It should be noted that the demand coefficient represents the degree of demand for classification accuracy of different types of documents. For example, office documents have a higher demand for classification accuracy, while similar material demand documents have a slightly lower demand for classification accuracy.

[0057] Determine whether the historical classification label is the same as the historical correct label; if they are the same, mark the corresponding historical classification label as the historical prediction correct label; if they are different, mark the corresponding historical classification label as the historical prediction error label; calculate the ratio of the number of historical prediction correct labels to the total number of historical classification labels, and multiply it by the demand coefficient to obtain the historical prediction rate; analyze the prediction accuracy of the LLM model through historical data, and a large amount of historical data can obtain the average prediction accuracy of the LLM model; multiply the historical prediction rate by the information coefficient to obtain the confidence threshold.

[0058] The document classification confidence of the current document is compared with the confidence threshold. In this embodiment, if the document classification confidence of the current document is less than the confidence threshold, the prediction accuracy of the LLM model for the current document fails to reach the average level, and the corresponding classification status is marked as a non-compliant state.

[0059] Step 4: Optimize the document classification confidence based on the classification status to obtain the final classification confidence; based on the final classification.

[0060] Exemplarily, the classification state in this embodiment is a non-standard state, and the confidence difference between the document classification confidence and the confidence threshold is calculated; the confidence difference is normalized to obtain a standard confidence difference; the parameters initialized by the LLM model are marked as initial parameters, and the initial parameters are obtained by subtracting the standard confidence difference to obtain optimized parameters, and the optimized parameters are input into the LLM model for updating to obtain an adjusted model, and the document classification confidence is recalculated through the adjusted model, and the recalculated document classification confidence is marked as the optimized classification confidence; the optimized classification confidence is compared with the document classification confidence, if the optimized classification confidence is greater than the document classification confidence, the optimized classification confidence is selected as the final classification confidence; if the document classification confidence is greater than the optimized classification confidence, the document classification confidence is selected as the final classification confidence; the classification result with the highest confidence in the final classification confidence is selected as the final result, which can improve the quality of classification decisions and reduce the possibility of misclassification, and can flexibly adjust the classification standards under different circumstances, thereby ensuring the stability and consistency of the classification results.

[0061] In some other preferred embodiments, if the classification status of the current document is up to standard, the document classification confidence is used as the final classification confidence, indicating that the prediction of the current document reaches an average level.

[0062] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0063] Working principle of the present invention:

[0064] The present invention collects user documents and historical data, inputs user documents into the LLM model to extract key information of the document, extracts important keywords, and obtains prediction labels in combination with document topic prediction, and binds the prediction labels with important keywords to obtain classification labels; sets information coefficients based on historical data, calculates the prediction accuracy according to the classification labels and the set correct prediction labels, and obtains the document classification confidence by combining the prediction accuracy and the information coefficient; sets a confidence threshold according to historical data, compares the document classification confidence with the confidence threshold, and obtains the classification status according to the comparison result; and adjusts and optimizes the large language model based on the classification status to obtain the final classification confidence.

[0065] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A document multi-classification method using a large language model and a deep learning algorithm, characterized in that: include: Step 1: Collect user documents, input user documents into the LLM model to obtain key information of the documents, and extract several sets of historical data from the database; Step 2: Classify user documents based on key information of the documents to obtain classification labels; Step 3: Obtain document classification confidence based on classification labels; set a confidence threshold based on historical data, and compare the document classification confidence with the confidence threshold to obtain the classification status; Step 4: Optimize the document classification confidence based on the classification status to obtain the final classification confidence; Based on the final classification.

2. According to claim 1, a document multi-classification method using a large language model and a deep learning algorithm is characterized in that: The step of classifying the user document based on the key information of the document to obtain a classification label includes: Extract key information of the document; wherein the key information of the document includes the document subject, important paragraphs of the document and the subject of the paragraph of the document; Important keywords in important paragraphs of the document are extracted through natural language processing technology, and the document topic and important keywords are input into the large language model as input data to predict the document label and obtain the predicted label of the document. Each important keyword is bound to the corresponding predicted label to obtain the classification label.

3. According to claim 1, the method for multi-classification of documents using a large language model and a deep learning algorithm is characterized in that: The step of obtaining the document classification confidence according to the classification label includes: The number of all different prediction labels in the classification label is counted, and the correct prediction label is manually selected, and the ratio of the number of correct prediction labels to the number of all prediction labels is calculated to obtain the prediction accuracy; the information coefficient is set according to the key information of historical documents in the historical data, and the prediction accuracy is multiplied by the information coefficient to obtain the document classification confidence.

4. According to claim 3, the method for multi-classification of documents using a large language model and a deep learning algorithm is characterized in that: The step of setting the information coefficient according to the key information of the historical document includes: Extracting key information of historical documents; wherein the key information of historical documents includes the subject of historical documents, important historical paragraphs and the subject of historical paragraphs; Mark the historical data with the same historical document theme as the document theme as historical matching data, and determine whether the historical paragraph theme in the historical matching data is the same as the historical document theme; if yes, set the corresponding paragraph coefficient to α; if no, set the corresponding paragraph coefficient to 1-α; where 0.5<α<1; The paragraph coefficients of all important historical paragraphs are counted and the average is calculated, and the average of the paragraph coefficients is used as the information coefficient.

5. According to claim 1, the method for multi-classification of documents using a large language model and a deep learning algorithm is characterized in that: The step of setting the confidence threshold according to the historical data includes: Extract historical data; the historical data includes historical classification labels and historical correct labels obtained by the model for document recognition; The historical prediction rate is obtained based on the historical classification labels and historical correct labels of the historical matching data; the confidence threshold is obtained by multiplying the information coefficient by the historical prediction rate.

6. The method for multi-classification of documents using a large language model and a deep learning algorithm according to claim 5, characterized in that: The obtaining of the historical prediction rate according to the historical classification labels and the historical correct labels of the historical matching data includes: Input the historical classification label into the demand setting model to obtain the demand coefficient; determine whether the historical classification label is the same as the historical correct label; if yes, mark the corresponding historical classification label as the historical prediction correct label; if no, mark the corresponding historical classification label as the historical prediction wrong label; wherein the demand setting model is constructed based on the neural network model; The historical prediction rate is obtained by calculating the ratio of the number of historical prediction correct labels to the number of historical classification labels and multiplying it by the demand coefficient.

7. The method for multi-classification of documents using a large language model and a deep learning algorithm according to claim 6, characterized in that: The demand setting model is constructed based on a neural network model, including: Historical classification labels and corresponding historical demand coefficients are extracted from the database; the historical classification labels and corresponding historical demand coefficients are used as training data and test data to train the neural network model. After the training, the training effect is tested through the test data, and the neural network model is adjusted according to the test results to obtain a bone demand setting model with the input data as the classification label and the output data as the demand coefficient.

8. The document multi-classification method using a large language model and a deep learning algorithm in collaboration according to claim 1, characterized in that: The step of comparing the document classification confidence with the confidence threshold to obtain the classification status includes: Extract document classification confidence and confidence threshold; Determine whether the document classification confidence is less than the confidence threshold; if yes, mark the classification status as unsatisfactory; if no, mark the classification status as satisfactory.

9. The method for multi-classification of documents using a large language model and a deep learning algorithm according to claim 1, characterized in that: The step of optimizing the document classification confidence based on the classification state to obtain the final classification confidence includes: When the classification status is up to standard, the document classification confidence is used as the final classification confidence; When the classification status is not up to standard, the LLM model is adjusted based on the document classification confidence and the confidence threshold, and the document classification confidence obtained after adjustment is marked as the optimized classification confidence; it is judged whether the optimized classification confidence is greater than the document classification confidence; if so, the optimized classification confidence is used as the final classification confidence; if not, the document classification confidence is used as the final classification confidence.

10. The method for multi-classification of documents using a large language model and a deep learning algorithm according to claim 9, characterized in that: The step of adjusting the LLM model based on the document classification confidence and the confidence threshold includes: Calculate the confidence difference between the document classification confidence and the confidence threshold; normalize the confidence difference to obtain a standard confidence difference; The parameters initialized by the LLM model are marked as initial parameters, the initial parameters are subtracted from the standard confidence difference to obtain the optimized parameters, and the optimized parameters are input into the LLM model for updating to obtain the adjusted model.

Images