A clustering-based text supervision checking method, device and computer equipment

Abstract

The application relates to a clustering-based text supervision checking method and device and a computer device. Feature word sets corresponding to a plurality of question description texts in a question list are obtained by respectively extracting feature words from the question description texts. The word frequency of each feature word in the question description text, the first inverse document frequency in all question description texts and the second inverse document frequency in all checked units are respectively calculated to obtain a feature word weight set of the question description text of each checked unit. A plurality of question description text clusters and a plurality of initial high-correlation-degree feature word sets corresponding to the question description text clusters are obtained through clustering. Finally, the initial high-correlation-degree feature word set is optimized according to a chi-square statistic to obtain an accurate high-correlation-degree feature word set. The method saves the calculation resources while ensuring the accuracy of the results by optimizing the feature word weight calculation, adopting a clustering algorithm for preliminary clustering and optimizing the clustering results through chi-square statistics.

Description

Technical Field

[0001] This application relates to the field of text analysis technology, and in particular to a cluster-based text supervision and inspection method, apparatus, and computer device. Background Technology

[0002] Currently, the analysis of supervisory and inspection archives still relies on manual input, summarization, and organization, which is inefficient and highly subjective. With the explosive growth in the number of archives, how to rationally mine and analyze textual data to improve the management efficiency of newly acquired archives has become an urgent problem to be solved. To reduce the probability of similar problems occurring in similar inspected units, it is crucial to fully utilize existing historical supervisory and inspection data, analyze the characteristics of problems, and explore the potential patterns among various problems. This is of great significance for supervisory and inspection managers to implement targeted management measures.

[0003] To improve the efficiency and accuracy of historical archive data analysis, efforts have been made in areas such as business process informatization, electronic data collection, and standardized text data formatting. However, in text analysis, currently popular machine learning methods such as natural language processing are not suitable for certain fields with insufficient historical labeled data, and they also have high demands on computing hardware. Therefore, there is an urgent need for a lightweight method that meets the intelligent classification needs of text problems. Summary of the Invention

[0004] Therefore, it is necessary to provide a cluster-based text supervision and inspection method, apparatus, and computer equipment to address the aforementioned technical problems.

[0005] A clustering-based text supervision and inspection method, the method comprising:

[0006] Obtain a list of issues for supervision and inspection; the list includes multiple inspected units and multiple problem description texts corresponding to each inspected unit;

[0007] Feature words are extracted from multiple problem description texts of each of the inspected units to obtain feature word sets for each problem description text. The word frequency of each feature word in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspected units are calculated. Based on the word frequency, the first reverse file frequency, and the second reverse file frequency, the feature word weight set of each problem description text is obtained.

[0008] Clustering calculations are performed on multiple problem description texts based on the feature word weight set to obtain multiple problem description text clusters and corresponding multiple initial high-relevance feature word sets;

[0009] Calculate the chi-square statistic between each feature word in the initial high-relevance feature word set and the problem description text cluster, and obtain the precise high-relevance feature word set for each problem description text cluster based on the chi-square statistic.

[0010] In one embodiment, the step of extracting feature words from the plurality of problem description texts to obtain feature word sets for each of the problem description texts includes:

[0011] Based on the pre-set supervision and inspection word segmentation table, the multiple problem description texts are segmented to obtain the initial feature word set of each problem description text;

[0012] Stop words and incorrect word segments in each of the first feature word sets are removed based on a pre-set filter word list, resulting in a processed feature word set.

[0013] In one embodiment, calculating the word frequency of each feature word in the feature word set in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all examined units includes:

[0014] For each unit being inspected, calculate the word frequency of each feature word in the feature word set within the current problem description text:

[0015]

[0016] Where TF(m,n) is the word frequency of the nth feature word in the mth problem description text, f(m,n) represents the number of times the nth feature word appears in the mth text, and K m This represents the total number of words in the m-th text.

[0017] Calculate the first inverse file frequency of each feature word in the feature word set across all problem description texts:

[0018]

[0019] Among them, IDF M (n) represents the first inverse document frequency of the nth feature word, M N M represents the total number of problem description texts. n This represents the number of question description texts containing the nth feature word. To avoid the denominator of the log function being zero, we use M... n Then add a correction term of 1;

[0020] Calculate the second reverse file frequency of each feature word in the feature word set across all inspected units:

[0021]

[0022] Among them, IDFP (n) represents the second inverse file frequency of the nth feature word, P N P represents the total number of units inspected. n This represents the number of inspected units containing the nth feature word, also in P n Then add a correction term of 1.

[0023] In one embodiment, the feature word weight set of each problem description text is obtained based on the word frequency, the first reverse file frequency, and the second reverse file frequency, including:

[0024] The initial feature word weight set for each inspected unit's problem description text is obtained based on the term frequency, first inverse document frequency, and second inverse frequency of each feature word in the feature word set.

[0025]

[0026] Among them, W pn Let n be the weight of the nth feature term in the pth inspected unit, and let n be the initial feature word weight set of the pth inspected unit text.

[0027] Normalizing the weights of each feature word in the initial feature word weight set yields the feature word weight set for each inspected unit's problem description text:

[0028]

[0029] In one embodiment, the feature word weight set performs clustering calculations on multiple problem description texts to obtain multiple problem description text clusters and corresponding multiple initial high-relevance feature word sets, including:

[0030] Randomly select k problem description texts as initial cluster centroids, use text similarity calculation methods to calculate the distance between each problem description text and the k initial cluster centroids, and select the centroid with the shortest distance as the centroid of each problem description text;

[0031] Calculate the distances from the problem description text with the same centroid to other centroids in turn, and select the centroid with the shortest distance as the new centroid of the problem description text;

[0032] Repeat the last two steps above until the centroid no longer changes, thus determining the final multiple problem description text clusters and their corresponding initial high-relevance feature word sets. In one embodiment, calculating the chi-square statistic between each feature word in the initial high-relevance feature word set and the problem description text cluster, and obtaining the precise high-relevance feature word set for each problem description text cluster based on the chi-square statistic, includes:

[0033]

[0034] Where, χ 2 (t n h k ) is the characteristic word t n With the problem description text cluster h k Chi-square statistic between, h k Let A be a text cluster describing the k-th type of problem, containing the feature word t. n And it belongs to the problem description text cluster h k The number of texts, B is the number of texts containing the feature word t. n However, it does not belong to the problem description text cluster h. k The number of texts, C, is the number of texts that do not contain the feature word t. n However, it belongs to the problem description text cluster h k The number of texts, D, that does not contain the feature word t n And it does not belong to the problem description text cluster h k The number of texts, M = A + B + C + D, is the total number of problem description texts for supervision and inspection.

[0035] The chi-square statistics of the feature words in the initial high-relevance feature word set of each problem description text cluster are ranked, and the chi-square statistics of each category and feature word are calculated. 2 The values ​​are sorted in descending order, and the top 10 feature words for each category are selected to describe that category. Based on the business logic, the categories are defined according to the highly relevant feature words for each category, resulting in the final set of highly relevant feature words.

[0036] A clustering-based text supervision and inspection device, the device comprising:

[0037] The acquisition module is used to acquire a list of issues for supervision and inspection; the list of issues includes multiple inspected units and corresponding multiple issue description texts.

[0038] The weight calculation module is used to extract feature words from the multiple problem description texts respectively, to obtain the feature word set of each problem description text, to calculate the word frequency of each feature word in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspected units, and to obtain the feature word weight set of each problem description text based on the word frequency, the first reverse file frequency, and the second reverse file frequency.

[0039] The clustering module is used to perform clustering calculations on multiple problem description texts based on the feature word weight set, to obtain multiple problem description text clusters and corresponding multiple initial high-relevance feature word sets;

[0040] The selection module is used to calculate the chi-square statistic between each feature word in the initial high-relevance feature word set and the problem description text cluster, and to obtain the precise high-relevance feature word set for each problem description text cluster based on the chi-square statistic.

[0041] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0042] Obtain a list of issues for supervision and inspection; the list includes multiple inspected entities and corresponding problem description texts;

[0043] Feature words are extracted from the multiple problem description texts to obtain the feature word set of each problem description text. The word frequency of each feature word in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspected units are calculated. The feature word weight set of each problem description text is obtained based on the word frequency, the first reverse file frequency, and the second reverse file frequency.

[0044] Clustering calculations are performed on multiple problem description texts based on the feature word weight set to obtain multiple problem description text clusters and corresponding multiple initial high-relevance feature word sets;

[0045] Calculate the chi-square statistic between each feature word in the initial high-relevance feature word set and the problem description text cluster, and obtain the precise high-relevance feature word set for each problem description text cluster based on the chi-square statistic.

[0046] The aforementioned clustering-based text supervision and inspection method, apparatus, and computer equipment extract feature words from multiple problem description texts in the problem list to obtain corresponding feature word sets. The frequency of each feature word in each feature word set within the specific problem description text, its first inverse file frequency across all problem description texts, and its second inverse file frequency across all inspected units are calculated to obtain the feature word weight set for each problem description text. Since the inspected units and problem description texts in the problem list have a one-to-many relationship, the first inverse file frequency and the second inverse file frequency of the feature words across all problem description texts are comprehensively considered, making the feature word weights more accurate. This provides accurate input data for the clustering calculation of the problem description texts, resulting in multiple problem description text clusters and corresponding initial high-relevance feature word sets. Finally, the initial high-relevance feature word sets are optimized using chi-square statistics to obtain the precise high-relevance feature word set. This method, through a two-step calculation approach—modifying the feature word weight calculation formula, using a clustering algorithm for initial clustering, and optimizing the clustering results using chi-square statistics—saves computational resources while ensuring the accuracy of the results.

[0047] In response to the current situation where the information technology transformation and organization of existing supervision and inspection archive text analysis is insufficient in historical data accumulation, lack of intelligent computing hardware and software conditions, and text analysis highly relies on manual identification, interpretation, and summarization, this method takes the intelligent classification of feature words describing problems in the text as its core, and designs and implements a full-process solution of structured document framework, cluster analysis, and optimized classification. It provides supervision and inspection managers with an intelligent assistant for problem text analysis and organization based on past experience, and gradually guides the transformation and upgrading from non-electronic to electronic and intelligent management. Compared with existing technologies, this method has the following significant advantages: (1) It can directly process data from offline electronic forms or even paper archives and can operate independently of information systems. (2) It has low requirements for computer hardware and software environment configuration, and the algorithm has universality of application. (3) It can calculate interpretable results regardless of whether the amount of accumulated text data is small or large. (4) In response to the situation of one-to-many relationship between the inspected unit and the problem text in supervision and inspection problems, the conventional TF-IDF weight calculation equation is modified to comprehensively consider the word frequency of feature words in a single text and in the text cluster of the inspected unit, so that the feature word weights are more accurate. (5) The problem is first classified by clustering algorithm, and then the classification is optimized by chi-square statistics. The two-step process saves computing resources and ensures the accuracy of the results. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating a clustering-based text supervision and inspection method in one embodiment;

[0049] Figure 2 This is a schematic diagram illustrating the calculation process of the feature word weight set in one embodiment;

[0050] Figure 3 This is a flowchart illustrating the clustering calculation process in one embodiment;

[0051] Figure 4 This is a flowchart illustrating the selection of calculations in one embodiment;

[0052] Figure 5 This is a structural block diagram of a cluster-based text supervision and inspection device in one embodiment;

[0053] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] In one embodiment, such as Figure 1As shown, a clustering-based text supervision and inspection method is provided, including the following steps:

[0056] Step 102: Obtain the list of issues for supervision and inspection.

[0057] The problem list includes multiple inspected units and multiple problem description texts for each inspected unit.

[0058] The problem list can be obtained by importing an Excel spreadsheet. This spreadsheet includes the inspection time, inspected unit, inspection item, nature of the problem, and specific manifestations (including cause analysis) for the problems discovered during supervision and inspection. In this embodiment, the problem list can be presented in tabular form, with the nature and specific manifestations of the problem grouped into one column named "Problem Description Text," and the inspected unit in another column. Each row includes different inspected units and their corresponding problem description texts, with a one-to-many relationship between inspected units and problem description texts.

[0059] Step 104: Extract feature words from multiple problem description texts of each inspected unit to obtain the feature word set of each problem description text. Calculate the word frequency of each feature word in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspection items. Obtain the feature word weight set of each problem description text based on the word frequency, the first reverse file frequency, and the second reverse file frequency.

[0060] Based on a pre-set supervision and inspection word segmentation table, multiple problem description texts are segmented to obtain an initial feature word set for each problem description text. Stop words and incorrect word segments in each initial feature word set are removed based on a pre-set filtering word list to obtain a processed feature word set.

[0061] The feature word weight set for each inspected entity's problem description text is obtained based on the term frequency, the first inverse document frequency, and the second inverse document frequency. The first inverse document frequency of a feature word in all problem description texts reflects the importance of that feature word in the problem description text; similarly, the second inverse document frequency of a feature word in all inspected entities reflects the importance of that feature word in the inspected entity.

[0062] Since the checked units and problem description texts in the problem list have a one-to-many relationship, this study comprehensively considers the first inverse file frequency of feature words in all problem description texts and the second inverse file frequency in all checked units, making the feature word weights more accurate and providing accurate input data for subsequent clustering calculations of problem description texts.

[0063] Step 106: Cluster the problem description texts of multiple inspected units according to the feature word weight set to obtain multiple problem description text clusters and corresponding multiple initial high-relevance feature word sets.

[0064] Based on the characteristics of the problem description text itself, the appropriate number of target clusters is 3 to 10. The number of feature words in the initial high-relevance feature word set of each cluster should not exceed 20. The basis for determining whether clustering results should be iterated is whether it meets the needs of specific supervision and inspection business.

[0065] Step 108: Calculate the chi-square statistic of each feature word in the initial high-relevance feature word set with the problem description text cluster, and obtain the precise high-relevance feature word set of each problem description text cluster based on the chi-square statistic.

[0066] The chi-square statistics of each cluster and feature word are ranked, and a preset number of feature words are selected to describe the corresponding cluster based on the ranking. Combined with business logic, the problem description text cluster is divided and defined according to the high relevance feature word set of each cluster, so as to obtain the accurate high relevance feature word set of the inspection problem description text cluster.

[0067] The aforementioned clustering-based text supervision and inspection method extracts feature words from multiple problem description texts in the problem list to obtain corresponding feature word sets. It then calculates the frequency of each feature word in its respective problem description text, its first inverse file frequency across all problem description texts, and its second inverse file frequency across all inspected units. This yields the feature word weight set for each problem description text. Since the inspected units and problem description texts in the problem list have a one-to-many relationship, the method comprehensively considers both the first inverse file frequency and the second inverse file frequency across all problem description texts, making the feature word weights more accurate. This provides accurate input data for the clustering calculation of the problem description texts, resulting in multiple problem description text clusters and corresponding initial high-relevance feature word sets. Finally, the initial high-relevance feature word sets are optimized using chi-square statistics to obtain the precise high-relevance feature word set. This method, through a two-step calculation approach—modifying the feature word weight calculation formula, using a clustering algorithm for initial clustering, and optimizing the clustering results using chi-square statistics—saves computational resources while ensuring the accuracy of the results.

[0068] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0069] In one embodiment, such as Figure 2 As shown, the calculation process for the feature word weight set is provided, including the following steps:

[0070] Step 202: Calculate the word frequency of each feature word in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspected units.

[0071] For each unit being inspected, calculate the word frequency of each feature word in the feature word set within the current problem description text:

[0072]

[0073] Where TF(m,n) is the word frequency of the nth feature word in the mth problem description text, f(m,n) represents the number of times the nth feature word appears in the mth text, and K m This represents the total number of words in the m-th question description text.

[0074] Considering that the longer the text, the greater the frequency of a certain feature word in the feature word set of its corresponding problem description text, in order to reduce the imbalance in word frequency calculation between long and short texts, that is, to reduce the bias towards long texts, this method uses the total number of words in the problem description text as the denominator.

[0075] Calculate the first inverse file frequency of each feature word in the feature word set across all problem description texts:

[0076]

[0077] Among them, IDF M (n) represents the first inverse document frequency of the nth feature word, M N M represents the total number of problem description texts. n This represents the number of question description texts containing the nth feature word. To avoid the denominator of the log function being zero, we use M... n Then add a correction term of 1;

[0078] Calculate the second reverse file frequency of each feature word in the feature word set across all inspected units:

[0079]

[0080] Among them, IDF P (n) represents the second inverse file frequency of the nth feature word, P N P represents the total number of units inspected. n This represents the number of inspected units containing the nth feature word, also in P n Then add a correction term of 1.

[0081] Step 204: Obtain the initial feature word weight set for each descriptive text based on the word frequency, the first inverse document frequency, and the second inverse frequency of each feature word in the feature word set.

[0082]

[0083] Among them, W pn Let n be the weight of the nth feature term in the pth inspected unit, and let n be the initial feature word weight set of the pth inspected unit text.

[0084] Step 206: Normalize the weights of each feature word in the initial feature word weight set to obtain the feature word weight set of each inspected unit's problem description text:

[0085]

[0086] In one embodiment, such as Figure 3 The diagram illustrates the clustering calculation process. The feature word weight set is used to cluster multiple problem description texts, resulting in multiple problem description text clusters and corresponding initial high-relevance feature word sets, including:

[0087] Step 302: Use the Skip-gram word vector model to train the corpus of highly related feature words obtained after word segmentation, and obtain the word vector of each feature word.

[0088] Step 304: Calculate the frequency, position, and word distance of feature words for each text.

[0089] Step 306: Randomly select k texts as initial cluster centroids. The appropriate number of target clusters is 3 to 10. The number of feature words in the initial high-relevance feature word set of each cluster does not exceed 20. Use the text similarity calculation method to calculate the distance between each text and these k cluster centroids, and select the centroid with the shortest distance as its own centroid.

[0090] Step 308: Calculate the distances from texts with the same centroid to other centroids in turn, and select the centroid with the shortest distance as the new centroid of the text.

[0091] Step 310: Repeat the last two steps above until the centroid no longer changes, and finally determine the final multiple problem description text clusters and the corresponding initial high-relevance feature word set.

[0092] In one embodiment, such as Figure 4 The diagram illustrates the selection calculation process. It calculates the chi-square statistic between each feature word in the initial high-relevance feature word set and the question description text cluster. Based on the chi-square statistic, the precise high-relevance feature word set for each question description text cluster is obtained, including:

[0093] Step 402: Calculate the chi-square statistic of each feature word in the initial high-relevance feature word set and the problem description text cluster.

[0094]

[0095] Where, χ 2 (t n h k ) is the characteristic word t n With the problem description text cluster h k Chi-square statistic between, h k Let A be a text cluster describing the k-th type of problem, containing the feature word t. n And it belongs to the problem description text cluster h k The number of texts, B is the number of texts containing the feature word t. n However, it does not belong to the problem description text cluster h. k The number of texts, C, is the number of texts that do not contain the feature word t. n However, it belongs to the problem description text cluster h k The number of texts, D, that does not contain the feature word t n And it does not belong to the problem description text cluster h k The number of texts, M = A + B + C + D, is the total number of problem description texts for supervision and inspection.

[0096] Step 404: Rank the chi-square statistics of the feature words in the initial high-relevance feature word set for each problem description text cluster, and calculate the chi-square statistics for each category and feature word. 2 The values ​​are sorted in descending order, and the top 10 feature words for each category are selected to describe that category.

[0097] Step 406: Based on the business logic, define the categories according to each type of highly relevant feature words to obtain the final set of highly relevant feature words.

[0098] In one embodiment, such as Figure 5 As shown, a clustering-based text supervision and inspection device is provided, including an acquisition module, a weight calculation module, a clustering module, and a selection module, wherein:

[0099] The acquisition module is used to obtain a list of issues for supervision and inspection; the list of issues includes multiple inspected units and corresponding problem description texts.

[0100] The weight calculation module is used to extract feature words from multiple problem description texts to obtain the feature word set of each problem description text. It calculates the word frequency of each feature word in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspected units. Based on the word frequency, the first reverse file frequency, and the second reverse file frequency, the feature word weight set of each problem description text is obtained.

[0101] The clustering module is used to perform clustering calculations on multiple problem description texts based on the feature word weight set, to obtain multiple problem description text clusters and corresponding multiple initial high-relevance feature word sets;

[0102] The selection module is used to calculate the chi-square statistics of each feature word in the initial high-relevance feature word set and the problem description text cluster, and obtain the accurate high-relevance feature word set of each problem description text cluster based on the chi-square statistics.

[0103] Specific limitations regarding the clustering-based text supervision and inspection device can be found in the limitations of the clustering-based text supervision and inspection method above, and will not be repeated here. Each module in the aforementioned clustering-based text supervision and inspection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0104] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores a list of problems. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a clustering-based text supervision and inspection method.

[0105] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0106] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A text supervision and inspection method based on clustering, characterized in that, The method includes: Obtain a list of issues for supervision and inspection; the list includes multiple inspected units and multiple problem description texts corresponding to each inspected unit; Feature words are extracted from multiple problem description texts of each of the inspected units to obtain feature word sets for each problem description text. The word frequency of each feature word in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspected units are calculated. Based on the word frequency, the first reverse file frequency, and the second reverse file frequency, the feature word weight set of each problem description text is obtained. Clustering calculations are performed on multiple problem description texts based on the feature word weight set to obtain multiple problem description text clusters and corresponding multiple initial high-relevance feature word sets; Calculate the chi-square statistic between each feature word in the initial high-relevance feature word set and the problem description text cluster, and obtain the precise high-relevance feature word set for checking the problem description text cluster based on the chi-square statistic; The calculation of the frequency of each feature word in the feature word set in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspected units includes: For each unit being inspected, calculate the word frequency of each feature word in the feature word set within the current problem description text: in, For the first n The feature word in the first m Word frequency in the problem description text Indicates the first m The first in the text n The number of times each feature word appears This represents the total number of words in the m-th question description text; Calculate the first inverse file frequency of each feature word in the feature word set across all problem description texts: in, Indicates the first n The first inverse file frequency of each feature word This indicates the total amount of problem description text. Indicates containing the first n The number of problem description texts for each feature word, to avoid the denominator of the log function being zero, in Then add a correction term of 1; Calculate the second reverse file frequency of each feature word in the feature word set across all inspected units: in, Indicates the first n The second inverse file frequency of each feature word This indicates the total number of units inspected. Indicates containing the first n The number of units examined for each feature word, in Then add a correction term of 1.

2. The method according to claim 1, characterized in that, Feature words are extracted from multiple problem description texts to obtain feature word sets for each description text, including: Based on the pre-set supervision and inspection word segmentation table, the multiple problem description texts are segmented to obtain the initial feature word set of each problem description text; Stop words and incorrect word segments in each of the first feature word sets are removed based on a pre-set filter word list, resulting in a processed feature word set.

3. The method according to claim 2, characterized in that, The feature word weight set for each problem description text is obtained based on the word frequency, the first reverse file frequency, and the second reverse file frequency, including: The initial feature word weight set for each inspected unit's problem description text is obtained based on the term frequency, first inverse document frequency, and second inverse frequency of each feature word in the feature word set. in, For the first p The first of the inspected units n The weight of the i-th feature term, the i-th p The initial feature word weight set of each examined unit text is ; Normalize the weights of each feature word in the initial feature word weight set to obtain the feature word weight set of each inspected unit's problem description text: 。 4. The method according to claim 1, characterized in that, Clustering calculations are performed on multiple problem description texts based on the feature word weight set to obtain multiple problem description text clusters and corresponding initial high-relevance feature word sets, including: Random selection k Using the problem description texts as the initial cluster centroids, text similarity calculation methods are used to calculate the similarity between each problem description text and the target cluster centroids. k The distance between the initial cluster centroids is used to select the centroid with the shortest distance as the centroid of each question description text. Calculate the distances from the problem description text with the same centroid to other centroids in turn, and select the centroid with the shortest distance as the new centroid of the problem description text; Repeat the last two steps above until the centroid no longer changes, thus determining the final clusters of multiple problem description texts and the corresponding initial set of highly relevant feature words.

5. The method according to claim 4, characterized in that, The calculation of the chi-square statistics of each feature word in the initial high-relevance feature word set and the problem description text cluster, and the obtaining of the precise high-relevance feature word set for each problem description text cluster based on the chi-square statistics, includes: in, Characteristic words With problem description text cluster Chi-square statistic between For the first k A class of problem description text clusters, where A is the cluster containing feature words. And it belongs to the problem description text cluster. The number of texts, B is the number of texts containing feature words. However, it does not belong to the problem description text cluster. The number of texts, C, is the number of texts that do not contain feature words. However, it belongs to the problem description text cluster. The number of texts, D, does not contain feature words. And it does not belong to the problem description text cluster. The number of texts, This refers to the total number of problem description texts from the supervision and inspection process; The chi-square statistics of the feature words in the initial high-relevance feature word set of each problem description text cluster are ranked, and the chi-square statistics of each category and feature word are ranked. The values ​​are sorted in descending order, and the first p feature words of each category are selected to describe that category. Based on the business logic, the categories are divided and defined according to the highly relevant feature words of each category, resulting in the final set of highly relevant feature words.

6. A cluster-based text supervision and inspection device, used to implement the cluster-based text supervision and inspection method according to any one of claims 1 to 5, characterized in that, The device includes: The acquisition module is used to acquire a list of issues for supervision and inspection; the list of issues includes multiple inspected units and corresponding multiple issue description texts. The weight calculation module is used to extract feature words from multiple problem description texts of each of the inspected units to obtain feature word sets of each problem description text, calculate the word frequency of each feature word in the current problem description text, the first reverse file frequency in all problem description texts, and the second reverse file frequency in all inspected units, and obtain the feature word weight set of each problem description text based on the word frequency, the first reverse file frequency, and the second reverse file frequency; The clustering module is used to perform clustering calculations on multiple problem description texts based on the feature word weight set, to obtain multiple problem description text clusters and corresponding multiple initial high-relevance feature word sets; The selection module is used to calculate the chi-square statistic between each feature word in the initial high-relevance feature word set and the problem description text cluster, and to obtain the precise high-relevance feature word set for each problem description text cluster based on the chi-square statistic.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

Images