A large language model-based cascading commonality question mining method
By employing a cascaded common problem mining method, combined with small and large language models, and optimizing computational resources, the accuracy and efficiency issues of text problem clustering were resolved, enabling efficient and rapid discovery and processing of common problems.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHENGJIANG PUBLIC INFORMATION
- Filing Date
- 2023-11-10
- Publication Date
- 2026-06-05
AI Technical Summary
Existing text clustering techniques suffer from insufficient accuracy, low processing efficiency, and susceptibility to noise. Furthermore, large language models require excessive computational resources in practical applications, limiting their universality.
A cascaded common problem mining method is adopted, which combines small language embedding models and large language embedding models, uses clustering algorithms to extract coarse and fine-grained common problems, and combines them with a general large language model for summarization and named entity recognition to build a common problem database and optimize the utilization of computing resources.
It achieves higher accuracy and faster speed in mining common text problems, reduces dependence on computing resources, improves the system's adaptability and availability, and adapts to text problem processing in different scenarios.
Smart Images
Figure CN117493447B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method for mining cascaded common problems based on large language models. Background Technology
[0002] In the current era of big data, the explosive growth of text data and information necessitates the rapid and accurate discovery and mining of common issues for tasks such as decision support, problem-solving, and information retrieval. These common issues include, but are not limited to, language habits, text themes, emotional attitudes, and intent understanding. Existing text question clustering techniques mainly rely on keyword matching and topic modeling, which often have limitations, such as insufficient accuracy, low processing efficiency, and susceptibility to noise.
[0003] With the emergence of large language models, such as the GPT series and BERT pre-trained models, these models utilize deep learning techniques for deep text understanding, enabling them to discover and handle deeper common problems. However, the inference of large language models requires substantial computational resources, which limits their widespread deployment in practical applications. Therefore, designing a text clustering and analysis system that achieves a good balance between efficiency and accuracy and is adaptable to various scenarios is a pressing technical problem that needs to be solved. Summary of the Invention
[0004] In order to solve at least one of the technical problems mentioned in the background art, the present invention aims to provide a cascaded common problem mining method based on a large language model, which can mine and discover common problems in text with higher accuracy and faster speed, and can also effectively utilize the deep learning characteristics of the large language model to reduce the dependence on a large amount of computing resources.
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A method for mining cascading commonality issues based on a large language model includes the following steps:
[0007] S1 uses a small language embedding model to represent text problems as feature vectors, and extracts the first number of coarse-grained common problems of all text problems through a clustering algorithm; retains the second-highest number of coarse-grained common problems with the highest number of sub-problems;
[0008] S2, obtain the original text questions corresponding to the first second number of coarse-grained common questions, and use a large language embedding model to express the original text questions as feature vectors. Extract the third number of fine-grained common questions of the original text questions through a clustering algorithm; retain the fourth number of fine-grained common questions with the highest number of sub-questions.
[0009] S3, for the first four types of fine-grained common problems, the general model of large language is used to summarize and generalize to obtain the fifth type of common problems;
[0010] S4. For the fifth number of common problems obtained, name entity recognition is performed using a general language model, and the common problems are ranked by importance.
[0011] In some embodiments of the present invention, step S4 is followed by:
[0012] S5. Construct a database of common problems based on historical data; use the feature vector of the text problem as the key K and the common problem as the value V, and save a corresponding dictionary to store the meta-information of the text problem and the common problem.
[0013] In some embodiments of the present invention, when adding a new text question to the common question database, the similarity between the feature vector of the new text question and the feature vector of the existing questions in the common question database is calculated; if the similarity is greater than a set first similarity threshold, the new text question is added.
[0014] In some embodiments of the present invention, step S5 is followed by:
[0015] S6. For newly added text questions, obtain their feature vector representation, search in the common question database, and add the newly added text question to the common question to which the original text question with the highest similarity belongs as its sub-question.
[0016] In some embodiments of the present invention, when the similarity between the newly added text question and the existing questions is less than the second similarity threshold, the following method is used for classification: constructing prompt words, taking the extracted common questions as a candidate set, and using a general language model to predict the common question to which the newly added text question belongs.
[0017] In some embodiments of the present invention, when the similarity between the newly added text question and the existing questions is less than the second similarity threshold, the following method two is used for classification: the mapping pair of <text question, common question> in steps S1 to S3 is used as the training set to train the student model; the trained student model is used to predict the common question to which the newly added text question belongs.
[0018] In some embodiments of the present invention, the student model employs a focal-loss loss function.
[0019] In some embodiments of the present invention, when two classification methods are used, a generalized large language model is used to vote on the two classification results and select the classification result with the higher number of votes.
[0020] In some embodiments of the present invention, step S6 is followed by:
[0021] S7. After adding new text questions to the common question database, the database is iteratively updated and reordered according to importance.
[0022] In some embodiments of the present invention, the clustering algorithm employs the K-means clustering algorithm.
[0023] A cascaded commonality problem mining system based on a large language model includes:
[0024] The coarse-grained common problem filtering module uses a small language embedding model to represent text problems as feature vectors, and extracts the first number of coarse-grained common problems from all text problems through a clustering algorithm; it retains the second-highest number of coarse-grained common problems with the highest number of sub-problems.
[0025] The fine-grained common problem filtering module obtains the original text problems corresponding to the second-to-last number of coarse-grained common problems, and uses a large language embedding model to express the feature vectors of the original text problems. It then extracts the third-to-last number of fine-grained common problems from the original text problems through a clustering algorithm. Finally, it retains the fourth-to-last number of fine-grained common problems with the highest number of sub-problems.
[0026] The general model of large language uses the general model of large language to summarize and generalize the first four number of fine-grained common problems to obtain the fifth number of common problems.
[0027] The entity naming recognition module uses a general language model to recognize named entities for the fifth number of common problems obtained, and ranks the common problems by importance.
[0028] In some embodiments of the present invention, the cascaded common problem mining system based on a large language model further includes: a common problem database construction module, which constructs a common problem database based on historical data; and uses the feature vector of the text problem as the key K and the common problem as the value V to store a corresponding dictionary for storing the meta-information of the text problem and the common problem.
[0029] In some embodiments of the present invention, the cascaded common problem mining system based on a large language model further includes: a new problem expansion and classification module, which obtains the feature vector expression of a newly added text problem, searches in the common problem database, and adds the newly added text problem to the common problem to which the original text problem with the highest similarity belongs as its sub-problem.
[0030] In some embodiments of the present invention, the cascaded common problem mining system based on a large language model further includes a database iterative update module, which iteratively updates the database and re-sorts the importance of new text problems after they are added to the common problem database.
[0031] Compared with the prior art, the beneficial effects of the present invention are:
[0032] 1. High Efficiency and Precision: This invention's cascaded common problem mining system based on a large language model can more effectively mine and discover common problems in text. The deep understanding of text by the large language model enables our system to achieve higher accuracy when mining common problems. Simultaneously, through optimized computational strategies and efficient algorithm design, our system can significantly improve processing speed while maintaining high accuracy.
[0033] 2. Resource Optimization: Although large language models require significant computational resources when processing text problems, this invention, through special model design and computational optimization, greatly reduces the demand for computational resources. This allows our system to operate in resource-constrained environments, improving the system's adaptability and availability.
[0034] 3. Strong adaptability: The system design of this invention has strong adaptability and scalability. Whether dealing with different types of text problems or adapting to different usage scenarios, it can be quickly adjusted and optimized through modular system design and parameter configuration.
[0035] 4. Continuous learning: Compared with traditional rule-based text question mining methods, the deep learning-based model design of this invention can continuously learn and optimize as data grows, thereby achieving better results.
[0036] 5. Beneficial for decision support: Because it can quickly and accurately uncover common problems in text, this system helps improve the accuracy of decision-making, especially in scenarios that require processing large amounts of text information, such as public opinion analysis, user feedback processing, and market research. Attached Figure Description
[0037] Figure 1 This is a flowchart of the overall method of the present invention.
[0038] Figure 2 This is a system block diagram of the present invention. Detailed Implementation
[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0040] Language embedding models are pre-trained language models that convert natural language text input into corresponding embedding vector representations of the text output. Their size is typically determined by the number of parameters; models with fewer than 1 billion parameters are generally considered small, and vice versa. For example, BERT (Bidirectional Encoder Representation from Transformers) is a small language embedding model, while the Embedding interface provided by OpenAI can be considered a large language embedding model.
[0041] Large-scale general language models refer to large-scale pre-trained language models that can generate output natural language text based on natural language text input, possessing certain reasoning, question-answering, and summarizing abilities. Currently successful examples include ChatGPT and GPT4.
[0042] KNN (K Nearest Neighbors) is a machine learning classification algorithm in which each sample can be represented by its K nearest neighbors.
[0043] Example 1:
[0044] Please see Figure 1 This embodiment provides a method for mining cascaded common problems based on a large language model, including the following steps:
[0045] S1 uses a small language embedding model to represent text problems as feature vectors, and extracts the first number of coarse-grained common problems of all text problems through a clustering algorithm; retains the second-highest number of coarse-grained common problems with the highest number of sub-problems;
[0046] Specifically, using small language embedding models such as BERT as word vector embedders, we can obtain word vector embeddings for each text problem. The feature vector is represented in 3D. Then, the K-means clustering algorithm is used to cluster all... The feature vectors of the text problem are coarsely screened to initially extract... There are clusters, and the original text corresponding to these clusters is... A common problem. The reason for using small language models is that they have fewer model parameters (less than 1 billion) and lower computational cost.
[0047] The process of the K-means clustering algorithm is as follows:
[0048] First, you need to choose a suitable one. Numerical values, that is, to initially obtain from The number of questions extracted from a set of questions. The ultimate goal is to obtain... A cluster of common problems: The optimization goal is:
[0049]
[0050] in, It is a cluster The mean vector can be considered as the feature representation vector of common problems.
[0051]
[0052] The algorithm flow is as follows:
[0053] The input is a sample set Cluster tree of clustering Maximum number of iterations
[0054] The output is a cluster partition.
[0055] 1) From the dataset Random selection One sample as the initial One centroid vector:
[0056] 2) For
[0057] a) Initialize the cluster partition C to... .
[0058] b) For Calculate samples and each centroid vector Distance: , Will The smallest one is Corresponding category Update now .
[0059] c) For , right Recalculate the new centroid for all sample points. .
[0060] d) If all If none of the centroid vectors have changed, proceed to step 3.
[0061] 3) Output cluster partitioning .
[0062] For the extracted There are several common problems. Further filtering is then performed based on the number of sub-problems within each common problem. Sub-problems are sorted from highest to lowest number, and only the top few are retained. A common problem It can be pre-set manually. That is, those common problems with a small number of subproblems are discarded, because the fewer the number of subproblems, the less common the problem can be approximated.
[0063] S2, obtain the original text questions corresponding to the first second number of coarse-grained common questions, and use a large language embedding model to express the original text questions as feature vectors. Extract the third number of fine-grained common questions of the original text questions through a clustering algorithm; retain the fourth number of fine-grained common questions with the highest number of sub-questions.
[0064] Specifically, the data retained in step S1 The original text questions (sub-questions) corresponding to the common questions are sent to the initial screening question candidate set, which is used to obtain the results at this time. One original text question ( Next, a large language embedding model with more powerful semantic representation capabilities (such as the Embedding interface provided by OpenAI) is used to... The original text questions obtain their respective... 3D feature vector representation. The K-means clustering algorithm is used again to extract the feature vectors with higher accuracy. A common problem at the fine-grained level.
[0065] After clustering, fine-grained common problems are further filtered, and the obtained... For the common problem, repeat the screening process in step S1 and retain the top [cases]. A common problem It can be preset manually.
[0066] S3, for the first four types of fine-grained common problems, the general model of large language is used to summarize and generalize to obtain the fifth type of common problems;
[0067] Specifically, the data obtained in step S2 These common problems are summarized and generalized using more powerful general language models (such as ChatGPT or GPT4). That is, the input... Given a common problem to a general language model, the task is to analyze and output a concise and refined list of these common problems, specifying the number of common problems to be output. Whether determined by the general language model or user settings, but maintaining... .this There may still be repetitions or overlaps in these questions.
[0068] S4. For the fifth number of common problems obtained, name entity recognition is performed using a general language model, and the common problems are ranked by importance.
[0069] Specifically, the extract from step S3 A common problem is the use of the Large Language Model (LLM) for named entity recognition. First, the important entities (time, location, organization, etc.) are extracted. Then, K-means clustering is performed to obtain coarse-grained entity clusters. The frequency information of these entities is collected, and their importance is initially determined based on the frequency. Finally, based on entity clusters with different importance... The general language model generates an impact factor by comprehensively considering factors such as relevant laws and regulations, historical cases, recent occurrence time, frequency of occurrence (number of sub-problems), money involved, and number of people affected. ,right We rank the common problems by weight, that is, by importance. Specifically, we will... A common problem and entity clusters with different levels of importance The input is fed into the general language model, which is required to analyze the risks that these problems may involve, generate impact shadows, and generate corresponding weight coefficients (such as a score of 0-9) based on the understanding of the general language model.
[0070] S5. Construct a common problem database based on historical data; using the feature vector of the text problem as the key K and the common problem as the value V, store a corresponding dictionary to store the meta-information of the text problem and the common problem. When adding a new text problem to the common problem database, calculate the similarity between the feature vector of the new text problem and the feature vector of the existing problems in the common problem database; if the similarity is greater than a set first similarity threshold, then add the new text problem.
[0071] Specifically, the extract from step S3 A common problem is identified, and a structured database of common problems is constructed based on historical data. The textual problems obtained in step S3 will be used to... 3D feature vector As a database key Common problem text as value It also saves a corresponding dictionary to store the metadata of the original and common problems reported in this round (such as the importance, weight, update time, etc.) for subsequent retrieval.
[0072] Finally, it is necessary to maintain a database of common problems, that is, to gradually add new problems to it, and use the corresponding feature vector for each new problem. The feature vector of each question that has been entered into the question bank Calculate the cosine similarity: And design a similar threshold If and only if If a question is deemed not to overlap with any questions already added to the question bank, it is added to the question bank; otherwise, the question is discarded.
[0073] S6. For newly added text questions, obtain their feature vector representation, search in the common question database, and add the newly added text question to the common question to which the original text question with the highest similarity belongs as its sub-question.
[0074] Specifically, for newly added text problems, the simplest solution is to first obtain their embedding vector representation, then search the database built in step six, and add the new problem to the most similar common problems as a new sub-problem.
[0075] However, when the similarity between the newly added question and the existing questions is low, that is, when the similarity between the newly added text question and the existing questions is less than the second similarity threshold, in order to further improve the accuracy of the judgment, we designed the following two schemes:
[0076] The following classification method is used:
[0077] Based on the KNN concept: construct prompt words, take the extracted common questions as a candidate set, and use a general language model to predict the common questions to which newly added text questions belong.
[0078] Specifically, let's say the new problem to be added is... The existing problem is retrieval results The question is Therefore, by constructing reasonable prompts, the extracted set of common questions can be used as a candidate set. This allows the general language model to predict the common problem to which newly added problems belong. .
[0079] The following method two is used for classification:
[0080] Based on the idea of model distillation: the mapping pairs of <text problem, common problem> in steps S1 to S3 are used as the training set to train the student model; the trained student model is used to predict the common problem to which a newly added text problem belongs.
[0081] Specifically, steps S1 to S3 above contain many mapping pairs of <textual problems, common problems>, which can be used as a training set for training the student model (the BERT-large-Chinese model can be considered for weight initialization). Considering the potential long-tail problem, i.e., the number of problems corresponding to different common problems is not the same, which would lead to insufficient training for common problems corresponding to fewer problems, to alleviate this problem, the traditional cross-entropy loss is not used. ,in To predict the probability of a sample being positive, focal loss is used, i.e. By setting appropriate hyperparameters (such as...) and (This allows for dynamic adjustment of the weights of different common problems to achieve the best training results.)
[0082] When two classification methods are used, a general language model is used to vote on the two classification results and select the classification result with the higher number of votes.
[0083] Specifically, a self-voting mechanism could be considered, allowing the large model to vote on the answers generated by different methods, selecting the one with the most votes as the final common problem for judgment. That is: ,in These are the common problems output by the two schemes mentioned above, one using the KNN idea and the other using the model distillation idea.
[0084] Of course, if the above two approaches still cannot adequately address the common problem classification of new problems, a large language model combined with existing general knowledge can be used to establish a new classification for the problem and create new common problems.
[0085] S7. After adding new text questions to the common problem database, the database is iteratively updated and reordered according to importance.
[0086] Specifically, to ensure the database maintains high accuracy and universality even after new problems are added over time, iterative updates are necessary. Specifically, considering both the total number of newly added sub-problems and the passage of time, the weight ranking and clustering accuracy of the problems in the database are updated and re-ranked. That is, after adding a new sub-problem, the number of sub-problems covered by the common problem, the most recent occurrence time, the common problem description, and other influencing factor parameters mentioned in step S4 are updated. These changes are then used to recalculate the new weight of the common problem according to the influencing factor weight coefficients in step S4, and the problems in the common problem database are reordered in descending order of weight.
[0087] Example 2:
[0088] Please refer to Figure 2 This embodiment provides a cascaded common problem mining system based on a large language model, including:
[0089] The coarse-grained common problem filtering module uses a small language embedding model to represent text problems as feature vectors, and extracts the first number of coarse-grained common problems from all text problems through a clustering algorithm; it retains the second-highest number of coarse-grained common problems with the highest number of sub-problems.
[0090] The fine-grained common problem filtering module obtains the original text problems corresponding to the second-to-last number of coarse-grained common problems, and uses a large language embedding model to express the feature vectors of the original text problems. It then extracts the third-to-last number of fine-grained common problems from the original text problems through a clustering algorithm. Finally, it retains the fourth-to-last number of fine-grained common problems with the highest number of sub-problems.
[0091] The general language model summary module summarizes and generalizes the first four fine-grained issues using the general language model to obtain the fifth common problem.
[0092] The entity naming recognition module uses a general language model to recognize named entities for the fifth number of common problems obtained, and ranks the common problems by importance.
[0093] The common problem database construction module builds a common problem database based on historical data; it uses the feature vector of the text problem as the key K and the common problem as the value V, and saves a corresponding dictionary to store the meta-information of the text problem and the common problem;
[0094] The new problem expansion and classification module obtains the feature vector representation of newly added text problems, searches the common problem database, and adds the newly added text problem to the common problem to which the original text problem with the highest similarity belongs as its sub-problem.
[0095] The database iteration update module updates the database iteratively after adding new text questions to the common problem database and re-sorts the database by importance.
[0096] In this embodiment, the specific working principles and processes of each module are the same as in Embodiment 1, and will not be repeated here.
[0097] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.
Claims
1. A method for mining cascaded commonalities based on a large language model, characterized in that, Includes the following steps: S1 uses a small language embedding model to represent text problems as feature vectors, and extracts the first number of coarse-grained common problems of all text problems through a clustering algorithm; retains the second-highest number of coarse-grained common problems with the highest number of sub-problems; S2, obtain the original text questions corresponding to the second-to-last number of coarse-grained common questions, the original text questions being the text questions expressed by feature vectors using a small language embedding model in S1; and express the original text questions by feature vectors using a large language embedding model, and extract the third-to-last number of fine-grained common questions of the original text questions through a clustering algorithm; retain the fourth-to-last number of fine-grained common questions with the highest number of sub-questions; S3, for the first four types of fine-grained common problems, the general model of large language is used to summarize and generalize to obtain the fifth type of common problems; S4. For the fifth number of common problems obtained, name entity recognition is performed using a general language model, and the common problems are ranked by importance.
2. The method for mining cascaded commonalities based on a large language model according to claim 1, characterized in that, Following step S4, the following is also included: S5. Construct a database of common problems based on historical data; use the feature vector of the text problem as the key K and the common problem as the value V, and save a corresponding dictionary to store the meta-information of the text problem and the common problem.
3. The method for mining cascaded commonalities based on a large language model according to claim 2, characterized in that, When adding a new text question to the common question database, the similarity between the feature vector of the new text question and the feature vector of the existing questions in the common question database is calculated; if the similarity is greater than a set first similarity threshold, the new text question is added.
4. The method for mining cascaded commonalities based on a large language model according to claim 3, characterized in that, Following step S5, the following is also included: S6. For newly added text questions, obtain their feature vector representation, search in the common question database, and add the newly added text question to the common question to which the original text question with the highest similarity belongs as its sub-question.
5. The method for mining cascaded commonalities based on a large language model according to claim 4, characterized in that, When the similarity between the newly added text question and the existing questions is less than the second similarity threshold; The classification is carried out using the following method: constructing prompt words, taking the extracted common questions as a candidate set, and using a general language model to predict the common questions to which newly added text questions belong.
6. A method for mining cascaded commonalities based on a large language model according to claim 4 or 5, characterized in that, When the similarity between the newly added text question and the existing questions is less than the second similarity threshold; The following method two is used for classification: the mapping pairs of <text problem, common problem> in steps S1 to S3 are used as the training set to train the student model; the trained student model is used to predict the common problem to which a newly added text problem belongs.
7. The method for mining cascaded commonalities based on a large language model according to claim 6, characterized in that, The student model uses the focal-loss loss function.
8. The method for mining cascaded commonalities based on a large language model according to claim 6, characterized in that, When two classification methods are used, a general language model is used to vote on the two classification results and select the classification result with the higher number of votes.
9. The method for mining cascaded commonalities based on a large language model according to claim 4, characterized in that, Following step S6, the following is also included: S7. After adding new text questions to the common question database, the database is iteratively updated and reordered according to importance.
10. The method for mining cascaded commonalities based on a large language model according to claim 1, characterized in that, The clustering algorithm used is the K-means clustering algorithm.