An unsupervised multi-keyword text clustering method based on word frequency sorting and pruning
This unsupervised multi-keyword text clustering method, which employs word frequency ranking and pruning, addresses the challenges of inefficient and unproductive text classification analysis. It provides meaningful preliminary classification results, assisting analysts in further analysis and improving the relevance and efficiency of text classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGDONG ESHORE TECH
- Filing Date
- 2022-12-27
- Publication Date
- 2026-05-29
AI Technical Summary
Existing technologies in text classification suffer from problems such as lack of analytical direction, lack of focus on specific problems, and low efficiency, especially lacking analytical schemes for word frequency ranking and pruning.
An unsupervised multi-keyword text clustering method based on word frequency ranking and pruning is adopted. By segmenting words, counting word frequencies, defining pruning and keyword thresholds, the word frequency of keyword combinations is calculated, unnecessary calculations are pruned, and meaningful preliminary classification results are provided.
It implements text content importance ranking, provides meaningful preliminary classification results, assists analysts in further analysis, and balances result quality and execution efficiency.
Smart Images
Figure CN115840822B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of text analysis, and in particular relates to an unsupervised multi-keyword text clustering method based on word frequency ranking and pruning. Background Technology
[0002] In many industry work order systems or other text-based business systems, data mining problems frequently arise involving the classification of different text records. Some text classifications can be derived manually, but many others cannot be determined through human experience. In such cases, we need to devise algorithms to discover possible classifications, followed by manual verification. For cases where classification is manually verified, the used model algorithm is considered a supervised method. Conversely, models that do not require manual verification are considered unsupervised methods. Unsupervised methods can largely overcome the limitations of manual analysis, such as high labor costs, lack of analytical direction, lack of focus, and low efficiency.
[0003] For example, Chinese invention patent CN201811508368.3 discloses a text clustering method, a text clustering device, and a terminal device, including: acquiring training text and performing word segmentation preprocessing on the training text to obtain multiple words to be trained; using the words to be trained to train a preset conversion model to obtain a trained conversion model; acquiring text to be clustered and performing word segmentation preprocessing on the text to be clustered to obtain multiple text feature words; using the trained conversion model to convert the text feature words into word vectors respectively, and superimposing all the word vectors in the text to be clustered to obtain the text vector of the text to be clustered; and clustering the text vectors to obtain a clustering result.
[0004] However, the aforementioned Chinese invention patent CN201811508368.3 does not propose an analysis scheme for word frequency ranking and pruning, including ranking, quality, and efficiency. Summary of the Invention
[0005] Therefore, the purpose of this invention is to provide an unsupervised multi-keyword text clustering method based on word frequency ranking and pruning. This method overcomes the limitations of manual text analysis, provides meaningful preliminary classification results for analysts' reference, and assists analysts in further analysis. First, word frequency ranking can, to a certain extent, represent the importance ranking of text content. Second, it proposes multiple keywords to describe text classification. Finally, it provides an analysis method that balances result quality and execution efficiency through pruning.
[0006] The technical solution of this invention is: an unsupervised multi-keyword text clustering method based on word frequency ranking and pruning, comprising the following steps:
[0007] Step S1. Collect text sample data;
[0008] Step S2. Segment words and count word frequencies;
[0009] Step S3. Define the pruning threshold;
[0010] Step S4. Define keyword thresholds;
[0011] Step S5. Calculate the word groups and word frequencies of the nth text;
[0012] Step S6. Calculate the word frequency of the m-th word in the n-th text record and determine whether pruning is necessary;
[0013] Step S7. Calculate the (m+1)th word word_m+1 of the nth text record, and calculate the word frequency of the two keywords word_m and word_m+1 appearing at the same time;
[0014] Step S8. Take the (m+2)th word word_m+2 from the nth text record, and calculate the word frequency of the three keywords word_m, word_m+1, and word_m+2 appearing simultaneously;
[0015] Step S9. Continue to calculate and obtain a number of keywords. Determine whether the number of keywords is greater than the keyword threshold. Only those that are greater than the threshold can be classified as a category.
[0016] Step S10. Return to step S5 and continue calculating the (n+1)th text record.
[0017] This invention addresses the problems of indecisiveness, lack of focus, and low efficiency in text classification analysis through the aforementioned technical solutions. Word frequency ranking can, to a certain extent, effectively represent the importance ranking of text content. A multi-keyword approach is proposed to describe text classification, aiding business personnel in summarizing their findings. A pruning method is employed to provide an analysis approach that balances result quality and execution efficiency.
[0018] In one specific implementation, in step S1, the text sample data is text data from a certain period of time in the production business system.
[0019] In one specific implementation, step S2 involves performing word segmentation on all collected text data, then counting how many work orders each segmented word appears in, and using the result as the word frequency.
[0020] In one specific implementation, step S3 specifically involves defining a word frequency threshold as the criterion for pruning.
[0021] In one specific implementation, step S4 involves defining a keyword threshold as a classification criterion.
[0022] In one specific implementation, step S5 specifically involves calculating the word groups and word frequencies contained in the nth text record, where n is the nth record of all training sample data, denoted as word_1, word_2, word_3,... and freq_1, freq_2, freq_3,... for the nth text record.
[0023] In one specific implementation, step S6 involves taking the m-th word_m of the n-th text record, determining whether the corresponding word frequency freq_m is greater than the pruning threshold T. If it is greater than or equal to T, the process continues; otherwise, the calculation is skipped.
[0024] In one specific implementation, step S7 involves taking the (m+1)th word (word_m+1) of the nth text record, calculating the word frequency of the two keywords (word_m and word_m+1) appearing simultaneously, and continuing if the frequency is greater than or equal to the pruning threshold T; otherwise, skipping this calculation.
[0025] In one specific implementation, in step S8, the (m+2)th word of the nth text record is taken, and the word frequency of the three keywords word_m, word_m+1, and word_m+2 appearing at the same time is calculated. Pruning is then performed based on the pruning threshold.
[0026] The beneficial effects of this invention are as follows:
[0027] This invention overcomes the limitations of manual text analysis, providing meaningful preliminary classification results for analysts' reference and assisting them in further analysis. Firstly, word frequency ranking can, to a certain extent, represent the importance ranking of text content. Secondly, it proposes using multiple keywords to describe text classification. Finally, it provides an analysis method that balances result quality and execution efficiency through pruning.
[0028] This invention addresses the problems of being at a loss when to start text classification analysis, lack of focus in analysis, and low efficiency. Word frequency ranking can, to a certain extent, represent the importance ranking of text content. It proposes using multiple keywords to describe text classification, helping business personnel to summarize information. Through pruning, it provides an analysis method that balances result quality and execution efficiency.
[0029] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description
[0030] Figure 1This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0031] The directional terms such as up, down, left, right, front, back, front, back, top, and bottom mentioned or possibly used in this specification are defined relative to their structure and are relative concepts. Therefore, they may vary depending on their location and usage. Consequently, these or other directional terms should not be interpreted as restrictive.
[0032] The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of embodiments consistent with some aspects of this disclosure.
[0033] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” used in this disclosure are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure refers to and includes any or all possible combinations of one or more associated listed items. Unless stated to the contrary, ordinal numbers such as “first,” “second,” etc., used in this disclosure are used to distinguish multiple objects and are not used to define the order, sequence, priority, or importance of the multiple objects.
[0034] Furthermore, it should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0035] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0036] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0037] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."
[0038] The technical solution provided by the present invention will now be described in further detail with reference to the accompanying drawings.
[0039] Please refer to Figure 1 As shown, an unsupervised multi-keyword text clustering method based on word frequency ranking and pruning includes the following steps:
[0040] Step S1. Collect text sample data;
[0041] Step S2. Segment words and count word frequencies;
[0042] Step S3. Define the pruning threshold;
[0043] Step S4. Define keyword thresholds;
[0044] Step S5. Calculate the word groups and word frequencies of the nth text;
[0045] Step S6. Calculate the word frequency of the m-th word in the n-th text record and determine whether pruning is necessary;
[0046] Step S7. Calculate the (m+1)th word word_m+1 of the nth text record, and calculate the word frequency of the two keywords word_m and word_m+1 appearing at the same time;
[0047] Step S8. Take the (m+2)th word word_m+2 from the nth text record, and calculate the word frequency of the three keywords word_m, word_m+1, and word_m+2 appearing simultaneously;
[0048] Step S9. Continue to calculate and obtain a number of keywords. Determine whether the number of keywords is greater than the keyword threshold. Only those that are greater than the threshold can be classified as a category.
[0049] Step S10. Return to step S5 and continue calculating the (n+1)th text record.
[0050] In this embodiment, in step S1, the text sample data is text data from a certain period of time in the production business system. For example, if it is a work order system, then work order text data from the past year can be collected.
[0051] Specifically, the aforementioned production business system can be a specialized production business management system. This system is applicable to specialized production industries and includes BOM entry and modification, ERP and supplier collaboration, and programming work order reporting. It can meet information needs such as delivery tracking, material tracking, quality tracking, cost tracking, and efficiency tracking.
[0052] Furthermore, in step S2, specifically, word segmentation is performed on all collected text data, and then the number of work orders in which each word appears is counted, and the result is used as the word frequency.
[0053] Specifically, in step S3, a word frequency threshold is defined as the criterion for pruning. For example, if the pruning threshold T = 1000 is set, then words with a frequency less than 1000 will be pruned.
[0054] Furthermore, in step S4, a keyword threshold is defined as the classification criterion. For example, setting the keyword threshold Y = 5 means that a category is formed only when the number of keywords is greater than 5.
[0055] Furthermore, in step S5, specifically, for the nth text record, where n is the nth record of all training sample data, calculate the word groups and word frequencies contained in it, denoted as word_1, word_2, word_3,... and freq_1, freq_2, freq_3,...
[0056] Specifically, in step S6, the m-th word word_m of the n-th text record is taken, and the corresponding word frequency freq_m is determined to be greater than the pruning threshold T. If it is greater than or equal to T, the calculation continues; otherwise, the calculation is skipped.
[0057] Furthermore, in step S7, specifically, the (m+1)th word word_m+1 of the nth text record is taken, and the word frequency of the two keywords word_m and word_m+1 appearing at the same time is calculated. If it is greater than or equal to the pruning threshold T, the calculation is continued; otherwise, the calculation is skipped.
[0058] Furthermore, in step S8, the (m+2)th word of the nth text record is taken, and the word frequency of the three keywords word_m, word_m+1, and word_m+2 appearing simultaneously is calculated. Pruning is then performed based on the pruning threshold.
[0059] This embodiment uses a method to acquire hundreds of thousands of text records from a work order system, perform word segmentation on them, define pruning thresholds and keyword thresholds, and then iteratively calculate word frequencies for all work order records to obtain several keywords. Since the time complexity of ordinary multi-keyword clustering algorithms reaches exponential levels, the computational load is so large that it is almost impossible to execute. Therefore, the pruning mechanism provided in this invention greatly reduces the computation time. The analysis results obtained through the text clustering method of this invention are then delivered to business personnel for manual screening to determine which keyword combinations can form a category, and then useless keywords are collected. The analysis continues iteratively, and this closed loop improves the clustering effect. This invention is a unique text clustering analysis method that includes word frequency ranking, pruning, and multi-keyword characteristic clustering methods.
[0060] Alternatively, the present invention may also include a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method of the embodiments of the present application.
[0061] Furthermore, a computer-readable storage medium may be provided, which stores a computer program that, when executed by one or more processors, implements the steps of the method provided in the embodiments of this application.
[0062] The present invention has the following technical effects:
[0063] This invention overcomes the limitations of manual text analysis, providing meaningful preliminary classification results for analysts' reference and assisting them in further analysis. Firstly, word frequency ranking can, to a certain extent, represent the importance ranking of text content. Secondly, it proposes using multiple keywords to describe text classification. Finally, it provides an analysis method that balances result quality and execution efficiency through pruning.
[0064] This invention addresses the problems of being at a loss when to start text classification analysis, lack of focus in analysis, and low efficiency. Word frequency ranking can, to a certain extent, represent the importance ranking of text content. It proposes using multiple keywords to describe text classification, helping business personnel to summarize information. Through pruning, it provides an analysis method that balances result quality and execution efficiency.
[0065] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.
Claims
1. An unsupervised multi-keyword text clustering method based on word frequency ranking and pruning, characterized in that, Includes the following steps: Step S1. Collect text sample data, wherein the text sample data is text data from the production business system; Step S2. Segment and count word frequency. Specifically, perform word segmentation on all collected text data, then count how many work orders each segmented word has appeared in, and use the statistical results as the word frequency. Step S3. Define the pruning threshold; Step S4. Define keyword thresholds; Step S5. Calculate the word groups and word frequencies of the nth text; Step S6. Calculate the word frequency of the m-th word in the n-th text record and determine whether pruning is needed. Specifically: take the m-th word word_m in the n-th text record and determine whether the corresponding word frequency freq_m is greater than the pruning threshold T. If it is greater than or equal to, continue to the next step; otherwise, skip this calculation. Step S7. Calculate the (m+1)th word word_m+1 of the nth text record, and calculate the word frequency of the two keywords word_m and word_m+1 appearing at the same time. Specifically: take the (m+1)th word word_m+1 of the nth text record again, and calculate the word frequency of the two keywords word_m and word_m+1 appearing at the same time. If it is greater than or equal to the pruning threshold T, continue to the next step; otherwise, skip this calculation. Step S8. Take the (m+2)th word word_m+2 of the nth text record, calculate the word frequency of the three keywords word_m, word_m+1, and word_m+2 appearing at the same time, and prune according to the pruning threshold; Step S9. Continue to calculate and obtain a number of keywords. Determine whether the number of keywords is greater than the keyword threshold. Only those that are greater than the threshold can be classified as a category. Step S10. Return to step S5 and continue calculating the (n+1)th text record.
2. The unsupervised multi-keyword text clustering method based on word frequency ranking and pruning according to claim 1, characterized in that, In step S1, the text sample data is text data from the production business system within a certain period of time.