Defect classification method, system and equipment based on text mining and medium

Abstract

The invention discloses a defect classification method, system and device based on text mining and a medium, and the method comprises the steps: obtaining a to-be-recognized defect report text in a power system, inputting the to-be-recognized defect report text into a trained classifier, and obtaining a defect type of the defect report text, according to the method, the system, the equipment and the medium, the efficiency of defect classification and root cause analysis can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of defect identification, and relates to a defect classification method, system, equipment and medium based on text mining. Background Art

[0002] Software defect analysis is an important part of software quality management. By analyzing historical defects, we can find the nature and source of defects, the causes of defects, obtain some useful experience and lessons, and form management measures to apply to the current or subsequent software development process, such as software requirements analysis, software design, software implementation and software testing. In this way, measures can be taken in the early stages of the software development life cycle to prevent the occurrence of these defects, reduce future maintenance costs, and improve software quality. However, existing defect reports usually contain a large amount of unstructured data (title, description, comments). Software defect descriptions may have a lot of redundant information and irregular syntax. In addition, the classification and root cause analysis of defects rely on human judgment, making it difficult to quickly extract effective information from them. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a defect classification method, system, device and medium based on text mining, which can effectively improve the efficiency of defect classification and root cause analysis.

[0004] To achieve the above object, the present invention discloses a defect classification method based on text mining, comprising:

[0005] Obtain the report text of the defects to be identified in the power system;

[0006] The defect report text to be identified is input into the trained classifier to obtain the defect category of the defect report text.

[0007] The further improvement of the defect classification method based on text mining described in the present invention is:

[0008] Furthermore, before inputting the defect report text to be identified into the trained classifier, the process further includes:

[0009] Build a dataset;

[0010] Build a classifier;

[0011] The classifier is trained and optimized using the data set by cross-validation and hyperparameter tuning, and the best classifier parameters are selected to obtain a trained classifier.

[0012] Furthermore, the process of constructing the data set is:

[0013] Collect files related to the power system and export defect report texts in the files, wherein the defect report texts include title, description and comment fields;

[0014] The defect report text is cleaned and segmented;

[0015] Generate a feature vector of the defect report text according to high-frequency keywords in the word segmentation result;

[0016] Clustering and classifying each defect report text according to its feature vector, and labeling each defect report text with a different defect category;

[0017] A dataset is constructed based on each defect report text and its corresponding defect category.

[0018] Furthermore, TF-IDF is used to identify high-frequency keywords in the word segmentation results.

[0019] Furthermore, the defect report text is segmented using the Chinese word segmentation tool Jieba.

[0020] The present invention discloses a defect classification system based on text mining, comprising:

[0021] An acquisition module, used for acquiring the defect report text to be identified in the power system;

[0022] The recognition module is used to input the defect report text to be recognized into the trained classifier to obtain the defect category of the defect report text.

[0023] The further improvement of the defect classification system based on text mining of the present invention is:

[0024] Furthermore, it also includes:

[0025] The first building module is used to build a data set;

[0026] The second building module is used to build a classifier;

[0027] The training module is used to train and optimize the classifier using the data set by cross-validation and hyperparameter tuning, select the best classifier parameters, and obtain the trained classifier.

[0028] Furthermore, the first building block includes:

[0029] A collection module, used to collect files related to the power system and export defect report texts in the files, wherein the defect report texts include title, description and comment fields;

[0030] A preprocessing module, used for cleaning and segmenting the defect report text and generating a feature vector of the defect report text according to high-frequency keywords in the segmentation result;

[0031] A generation module, used for clustering each defect report text according to the feature vector of each defect report text and identifying the defect topic after clustering using a topic model;

[0032] The annotation module is used to annotate different defect categories for each defect report text according to the clustered defect themes;

[0033] The third construction module is used to construct a data set according to each defect report text and its corresponding defect category.

[0034] The present invention discloses a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the defect classification method based on text mining are implemented.

[0035] The present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the defect classification method based on text mining are implemented.

[0036] The present invention has the following beneficial effects:

[0037] The text mining-based defect classification method, system, device and medium described in the present invention, when in specific operation, applies automated text analysis technology to software defect management, can effectively process large-scale unstructured text, and provide possibilities for defect classification and root cause analysis, thereby reducing manual intervention and significantly improving the efficiency and accuracy of defect classification and root cause analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0039] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] In the description of the present invention, it should be understood that the terms “include” and “comprises” indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0042] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0043] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects are in an "or" relationship.

[0044] It should be understood that, although the terms first, second, third, etc. may be used to describe preset ranges, etc. in the embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0045] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. Usually, the components described and shown in the accompanying drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0047] Various structural schematic diagrams according to the disclosed embodiments of the present invention are shown in the accompanying drawings. These figures are not drawn to scale, where for the purpose of clear expression, some details are enlarged and some details may be omitted. The shapes of various regions and layers shown in the figures and their relative sizes and positional relationships are merely exemplary. In practice, there may be deviations due to manufacturing tolerances or technical limitations, and those skilled in the art can design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0048] Embodiment 1

[0049] Reference Figure 1 , the defect classification method based on text mining according to the present invention includes the following steps:

[0050] 1) Data collection;

[0051] Collect files related to the power system, and export the defect reports in the files. The defect reports include title, description, and comment fields.

[0052] 2) Text cleaning;

[0053] Remove irrelevant characters in the defect reports, such as punctuation marks, HTML tags, special characters, etc., and then standardize the defect reports. Specifically, convert all characters in the defect reports to lowercase to reduce unnecessary feature redundancy, and remove stop words in the defect reports, such as "de", "shi", "he", to improve the efficiency of text analysis.

[0054] Word segmentation: The defect report text is segmented using the Chinese word segmentation tool Jieba, and the sentence is split into individual words or phrases. The words are converted to their original forms through stemming or lemmatization.

[0055] 3) Feature extraction;

[0056] TF-IDF is used to identify high-frequency keywords in the word segmentation results and measure the importance of high-frequency keywords in the defect report set to help identify specific high-frequency keywords in certain modules or error types. Feature vectors are generated for the defect reports to which they belong based on the high-frequency keywords. The word vector model Word2Vec is used to convert the words in the defect report into vectors to capture the semantic relationship between words and find connections between similar words.

[0057] 4) Clustering and classification;

[0058] K-means clustering is applied to group defect reports based on feature similarity, where each defect is assigned to a cluster based on the feature vector, and defects with similar error types or modules are found. DBSCAN is used for density-based clustering to identify abnormal data points or edge cases in defect reports, where DBSCAN is suitable for discovering noise points and abnormal problems in defect reports.

[0059] Use topic modeling (LDA) to automatically identify topics in bug report text and map them to possible categories.

[0060] Using the Support Vector Machine supervised learning algorithm, a classifier is trained to classify defect reports into predefined categories, such as module, error type, and root cause.

[0061] 5) Verification and testing;

[0062] Dataset division: Divide the dataset into training set, validation set and test set. The training set is used for model learning, the validation set is used for model tuning, and the test set is used to evaluate the final performance of the model.

[0063] Evaluation indicators: Use standard indicators such as accuracy, precision, recall, and F1-score to measure the performance of the classification model in defect classification and root cause identification tasks. Focus on evaluating the classification accuracy of the model in different categories to ensure good coverage of all modules and error types.

[0064] 6) Model optimization;

[0065] Through cross-validation and hyperparameter tuning, the best model parameters are selected to improve the performance of the classifier. For categories with poor performance, problems in feature selection and model design are analyzed and optimized accordingly.

[0066] Embodiment 2

[0067] The present invention discloses a defect classification system based on text mining, comprising:

[0068] An acquisition module, used for acquiring the defect report text to be identified in the power system;

[0069] The recognition module is used to input the defect report text to be recognized into the trained classifier to obtain the defect category of the defect report text.

[0070] As an embodiment of the present invention, it also includes:

[0071] The first building module is used to build a data set;

[0072] The second building module is used to build a classifier;

[0073] The training module is used to train and optimize the classifier using the data set by cross-validation and hyperparameter tuning, select the best classifier parameters, and obtain the trained classifier.

[0074] As an embodiment of the present invention, the first building block includes:

[0075] A collection module, used to collect files related to the power system and export defect report texts in the files, wherein the defect report texts include title, description and comment fields;

[0076] A preprocessing module, used for cleaning and segmenting the defect report text and generating a feature vector of the defect report text according to high-frequency keywords in the segmentation result;

[0077] A generation module, used for clustering each defect report text according to the feature vector of each defect report text and identifying the defect topic after clustering using a topic model;

[0078] The annotation module is used to annotate different defect categories for each defect report text according to the clustered defect themes;

[0079] The third construction module is used to construct a data set according to each defect report text and its corresponding defect category.

[0080] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0081] Embodiment 3

[0082] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the defect classification method based on text mining when executing the computer program, for example, comprising: collecting files related to an electric power system, exporting defect report texts in the files, wherein the defect report texts comprise title, description and comment fields; cleaning and segmenting the defect report texts; generating feature vectors of the defect report texts according to high-frequency keywords in the segmentation results; clustering and classifying the defect report texts according to the feature vectors of the defect report texts, and labeling different defect categories for the defect report texts; constructing a data set according to the defect report texts and their corresponding defect categories; constructing a classifier; using the data set to train and optimize the classifier by means of cross-validation and hyperparameter tuning, selecting the best classifier parameters, and obtaining a trained classifier; obtaining defect report texts to be identified in the electric power system; inputting the defect report texts to be identified into the trained classifier to obtain the defect category of the defect report text. The memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc. The processor, network interface, and memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs. Specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0083] Embodiment 4

[0084] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of implementing the defect classification method based on text mining are implemented, for example, including: collecting files related to the power system, exporting the defect report text in the file, the defect report text including the title, description and comment fields; cleaning and segmenting the defect report text; generating a feature vector of the defect report text according to the high-frequency keywords in the segmentation result; clustering and classifying each defect report text according to the feature vector of each defect report text, and marking different defect categories for each defect report text; constructing a data set according to each defect report text and its corresponding defect category; constructing a classifier; using the data set to train and optimize the classifier by cross-validation and hyperparameter tuning, selecting the best classifier parameters, and obtaining the trained classifier; obtaining the defect report text to be identified in the power system; inputting the defect report text to be identified into the trained classifier to obtain the defect category of the defect report text. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.

[0085] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and disclosure of the invention. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0090] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0091] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A defect classification method based on text mining, characterized in that: include: Obtain the report text of the defects to be identified in the power system; The defect report text to be identified is input into the trained classifier to obtain the defect category of the defect report text.

2. The defect classification method based on text mining according to claim 1 is characterized in that: Before inputting the defect report text to be identified into the trained classifier, the method further includes: Build a dataset; Build a classifier; The classifier is trained and optimized using the data set by cross-validation and hyperparameter tuning, and the best classifier parameters are selected to obtain a trained classifier.

3. The defect classification method based on text mining according to claim 1, characterized in that: The process of constructing the data set is as follows: Collect files related to the power system and export defect report texts in the files, wherein the defect report texts include title, description and comment fields; The defect report text is cleaned and segmented; Generate a feature vector of the defect report text according to high-frequency keywords in the word segmentation result; Clustering and classifying each defect report text according to its feature vector, and labeling each defect report text with a different defect category; A dataset is constructed based on each defect report text and its corresponding defect category.

4. The defect classification method based on text mining according to claim 3 is characterized in that: Use TF-IDF to identify high-frequency keywords in the word segmentation results.

5. The defect classification method based on text mining according to claim 3 is characterized in that: The defect report text is segmented using the Chinese word segmentation tool Jieba.

6. A defect classification system based on text mining, characterized in that: include: An acquisition module, used for acquiring the defect report text to be identified in the power system; The recognition module is used to input the defect report text to be recognized into the trained classifier to obtain the defect category of the defect report text.

7. The text mining-based defect classification system according to claim 6, characterized in that: Also includes: The first building module is used to build a data set; The second building module is used to build a classifier; The training module is used to train and optimize the classifier using the data set by cross-validation and hyperparameter tuning, select the best classifier parameters, and obtain the trained classifier.

8. The text mining-based defect classification system according to claim 7, characterized in that: The first building block includes: A collection module, used to collect files related to the power system and export defect report texts in the files, wherein the defect report texts include title, description and comment fields; A preprocessing module, used for cleaning and segmenting the defect report text and generating a feature vector of the defect report text according to high-frequency keywords in the segmentation result; A generation module, used for clustering each defect report text according to the feature vector of each defect report text and identifying the defect topic after clustering using a topic model; The annotation module is used to annotate different defect categories for each defect report text according to the clustered defect themes; The third construction module is used to construct a data set according to each defect report text and its corresponding defect category.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the defect classification method based on text mining as described in any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the defect classification method based on text mining as described in any one of claims 1 to 5 are implemented.

Images