Sample imbalance data processing diagnosis system and method

Abstract

The invention discloses a sample imbalance data processing diagnosis system and method, the system comprises a data import module, a sample classification module, a sample number averaging module, a sample balance module and a data analysis module, the data import module is connected with the sample classification module, the sample classification module is connected with the sample number averaging module, and the sample number averaging module is connected with the sample balance module. The sample number averaging module is connected with a sample balance module, and the sample balance module is connected with a data analysis module; according to the invention, through the use of the sample classification module, automatic classification of data is realized, and errors during group-by-group import are prevented; meanwhile, through the cooperation between the sample number averaging module and the sample balancing module, better data balancing is realized, and offset during model establishment is avoided; meanwhile, through cooperation between the data analysis module and the data prediction module, a data model can be generated on the basis of data analysis, and the application range of the system is widened.

Description

Technical Field

[0001] The present invention relates to the technical field of data science analysis, and in particular to a sample imbalance data processing and diagnosis system and method. Background Art

[0002] Sample imbalanced data refers to the situation where the number of samples of different categories in a data set varies greatly. For example, in a binary classification problem, if most samples belong to one category and only a few samples belong to another category, then the data set is imbalanced. The use of sample imbalanced data will lead to deviations in model training, and the model will tend to predict the majority class. The existing sample imbalanced data processing and diagnosis system can basically meet the needs of production, but it still has certain shortcomings; First, the existing sample imbalance data processing and diagnosis system needs to import data category by category; second, the existing system has poor ability to balance data; in addition, the existing system can only make result analysis and cannot make further predictions; first, the current system needs to import data category by category, which makes the operation cumbersome and inefficient. Each time a sample is processed, the data of each category must be imported separately, which not only increases time and labor costs, but also easily leads to errors and inconsistencies in data processing; second, when faced with the problem of category imbalance, the system cannot effectively resample or adjust the samples, which may lead to bias in the model training results; in addition, the existing system can only perform result analysis and cannot make further predictions, which limits the scope of application and practical value of the system, and cannot generate a prediction model based on data analysis, nor can it provide dynamic prediction support for decision-making, affecting the overall effectiveness and practicality of the system; Therefore, it is necessary to design a sample imbalance data processing diagnosis system and method. Summary of the invention

[0003] The object of the present invention is to provide a sample imbalance data processing diagnosis system and method to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a sample imbalance data processing and diagnosis system, comprising a data import module, a sample classification module, a sample number averaging module, a sample balancing module and a data analysis module, the data import module is connected to the sample classification module, the sample classification module is connected to the sample number averaging module, the sample number averaging module is connected to the sample balancing module, the sample balancing module is connected to the data analysis module, and the data analysis module is connected to the data prediction module.

[0005] As a further technical solution of the present invention, the sample classification module is composed of a label establishment module, a data search module, a data classification module, a classification data storage module, a label number storage module and a data quantity storage module. The label establishment module is connected to the data search module, the data search module is connected to the data classification module, and the data classification module is connected to the classification data storage module.

[0006] As a further technical solution of the present invention, the label establishment module is connected to a label number storage module.

[0007] As a further technical solution of the present invention, the data classification module is connected to a data volume storage module.

[0008] As a further technical solution of the present invention, the sample number averaging module is composed of a data volume acquisition module, a data volume summing module, a label number acquisition module, a data volume averaging module and an average value storage module. The data volume acquisition module is connected to the data volume summing module, and the data volume summing module is connected to the data volume averaging module.

[0009] As a further technical solution of the present invention, the data volume averaging module is connected to a label number acquisition module, and the data volume averaging module is connected to an average value storage module.

[0010] As a further technical solution of the present invention, the sample balancing module is composed of a classification data acquisition module, an average value acquisition module, a data volume comparison module, a sample generation module, a sample reduction module and a balanced data storage module. The classification data acquisition module is connected to the data volume comparison module, and the data volume comparison module is connected to the average value acquisition module.

[0011] As a further technical solution of the present invention, the data volume comparison module is connected to a sample generation module, the data volume comparison module is connected to a sample reduction module, the data volume comparison module is connected to a balanced data storage module, the balanced data storage module is connected to a sample generation module, and the balanced data storage module is connected to a sample reduction module.

[0012] As a further technical solution of the present invention, the data prediction module is composed of an analysis data acquisition module, a prediction model establishment module, a prediction model prediction module and a prediction data storage module. The analysis data acquisition module is connected to the prediction model establishment module, the prediction model establishment module is connected to the prediction model prediction module, and the prediction model prediction module is connected to the prediction data storage module.

[0013] A sample imbalance data processing and diagnosis method comprises the following steps: step one, data import; step two, data classification; step three, data analysis; step four, data prediction; In the above step 1, the data import module imports the data to be analyzed into the system; In the above step 2, the label establishment module establishes labels according to the requirements, and stores the label number through the label number storage module, and then the data search module searches in the imported data in turn according to the established labels so as to facilitate the data classification module to classify, and the amount of data of each category during classification is stored by the data amount storage module, and the classified data is stored by the classification data storage module; Among them, in the above step three, the data volume acquisition module acquires the data stored in the data volume storage module and then the data volume summing module sums them up, the label number acquisition module acquires the label number data stored in the label number storage module, and then the data volume averaging module uses the summed data and the label number data to calculate the average value, and the calculated average value is stored in the average value storage module, the classified data acquisition module acquires the classified data stored in the classified data storage module, and the average value acquisition module acquires the average value data in the average value storage module, and then the two types of data are respectively imported into the data volume comparison module, if the data volume of a certain type of data is lower than the average value, then the data of this type is sent to the sample generation module, the sample generation module randomly generates samples and adds them to the data of this type until the data volume reaches the average value, if the data volume of a certain type of data is higher than the average value, then the data of this type is sent to the sample reduction module, the sample reduction module randomly reduces the samples in the data of this type until the data volume reaches the average value, and the modified sample data is sent to the balanced data storage module for storage, if the data volume of a certain type of data is equal to the average value, then the data of this type is directly sent to the balanced data storage module for storage, and then the data analysis module analyzes the balanced data; In the above step four, the analysis data acquisition module obtains the analysis results in the data analysis module, the prediction model establishment module establishes a prediction model based on the results and system imported data, the prediction model prediction module can predict the results of new data based on the prediction model, and the predicted results are stored in the prediction data storage module.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: in this sample imbalance data processing and diagnosis system and method, the label establishment module establishes labels according to needs, and stores the number of labels through the label number storage module, and then the data search module searches in the imported data in turn according to the established labels to facilitate the data classification module to classify, and the amount of data of each category during classification is stored by the data amount storage module, and the classified data is stored by the classified data storage module, thereby realizing automatic classification of data and preventing errors when importing group by group; at the same time, the data amount acquisition module acquires the data stored in the data amount storage module and then sums the data by the data amount summing module, the label number acquisition module acquires the label number data stored in the label number storage module, and then the data amount averaging module uses the summed data and the label number data to calculate the average value, and the average value after calculation is stored in the average value storage module, the classified data acquisition module acquires the classified data stored in the classified data storage module, and the average value acquisition module acquires the average value data in the average value storage module, and then the two types of data are respectively imported into the data storage module. The data volume comparison module, if the data volume of a certain type of data is lower than the average value, then the data of this type is sent to the sample generation module, the sample generation module randomly generates samples and adds them to the data of this type until the data volume reaches the average value, if the data volume of a certain type of data is higher than the average value, then the data of this type is sent to the sample reduction module, the sample reduction module randomly reduces the samples in this type of data until the data volume reaches the average value, and the modified sample data is sent to the balanced data storage module for storage, if the data volume of a certain type of data is equal to the average value, then the data of this type is directly sent to the balanced data storage module for storage, and then the data analysis module analyzes the balanced data to achieve better data balancing and avoid offset when establishing the model; at the same time, the analysis data acquisition module obtains the analysis results in the data analysis module, the prediction model establishment module establishes a prediction model according to the results and the system import data, the prediction model prediction module can predict the results of the new data according to the prediction model, and the predicted results are stored in the prediction data storage module, and the data model can be generated based on the analysis data, thereby improving the application scope of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a system flow chart of the present invention; Figure 2 It is a system framework diagram of the present invention; Figure 3 is a schematic diagram of the architecture of the sample classification module of the present invention; Figure 4 Schematic diagram of the architecture of the sample number averaging module of the present invention; Figure 5 is a schematic diagram of the architecture of the sample balancing module of the present invention; Figure 6 It is a flowchart of the operation of the sample balancing module of the present invention; Figure 7 is a schematic diagram of the architecture of the data prediction module of the present invention; Figure 8 The figure is a flow chart of the method of the present invention.

[0016] In the figure: 10, data import module; 20, sample classification module; 21, label establishment module; 22, data search module; 23, data classification module; 24, classification data storage module; 25, label number storage module; 26, data volume storage module; 30, sample number averaging module; 31, data volume acquisition module; 32, data volume summing module; 33, label number acquisition module; 34, data volume averaging module; 35, average value storage module; 40, sample balancing module; 41, classification data acquisition module; 42, average value acquisition module; 43, data volume comparison module; 44, sample generation module; 45, sample reduction module; 46, balanced data storage module; 50, data analysis module; 60, data prediction module; 61, analysis data acquisition module; 62, prediction model establishment module; 63, prediction model prediction module; 64, prediction data storage module. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Please refer to the attached Figure 1 -Attached Figure 7An embodiment of the present invention provides: a sample imbalance data processing and diagnosis system, including a data import module 10, a sample classification module 20, a sample number averaging module 30, a sample balancing module 40 and a data analysis module 50, the data import module 10 is connected to the sample classification module 20, the sample classification module 20 is connected to the sample number averaging module 30, the sample number averaging module 30, the sample number averaging module 30 is connected to the sample balancing module 40, the sample balancing module 40 is connected to the data analysis module 50, and the data analysis module 50 is connected to the data prediction module 60; the sample classification module 20 is composed of a label establishment module 21, a data search module 22, The data classification module 23, the classification data storage module 24, the label number storage module 25 and the data volume storage module 26 are composed of the label establishment module 21 connected to the data search module 22, the data search module 22 is connected to the data classification module 23, and the data classification module 23 is connected to the classification data storage module 24; the label establishment module 21 is connected to the label number storage module 25; the data classification module 23 is connected to the data volume storage module 26; the sample number averaging module 30 is composed of the data volume acquisition module 31, the data volume summing module 32, the label number acquisition module 33, the data volume averaging module 34 and the average value storage module 35, the data volume acquisition module 3 1 is connected to a data volume summing module 32, and a data volume averaging module 34 is connected to the data volume summing module 32; the data volume averaging module 34 is connected to a label number acquisition module 33, and the data volume averaging module 34 is connected to an average value storage module 35; the sample balancing module 40 is composed of a classification data acquisition module 41, an average value acquisition module 42, a data volume comparison module 43, a sample generation module 44, a sample reduction module 45 and a balanced data storage module 46, the classification data acquisition module 41 is connected to a data volume comparison module 43, and the data volume comparison module 43 is connected to the average value acquisition module 42; the data volume comparison module 43 is connected to the sample generation module 44, the data volume comparison module 43 is connected to a sample reduction module 45, the data volume comparison module 43 is connected to a balanced data storage module 46, the balanced data storage module 46 is connected to a sample generation module 44, and the balanced data storage module 46 is connected to a sample reduction module 45; the data prediction module 60 is composed of an analysis data acquisition module 61, a prediction model establishment module 62, a prediction model prediction module 63 and a prediction data storage module 64, the analysis data acquisition module 61 is connected to a prediction model establishment module 62, the prediction model establishment module 62 is connected to a prediction model prediction module 63, and the prediction model prediction module 63 is connected to a prediction data storage module 64.

[0019] See also Figure 8 , an embodiment provided by the present invention: a sample imbalance data processing and diagnosis method, comprising the following steps: step one, data import; step two, data classification; step three, data analysis; step four, data prediction; In the above step 1, the data import module 10 imports the data to be analyzed into the system; In the above step 2, the label establishment module 21 establishes labels according to the requirements, and stores the label number through the label number storage module 25. Then, the data search module 22 searches the imported data in turn according to the established labels so as to facilitate the classification by the data classification module 23. During the classification, the amount of data in each category is stored by the data amount storage module 26, and the classified data is stored by the classification data storage module 24. In the above step 3, the data volume acquisition module 31 acquires the data stored in the data volume storage module 26 and then the data volume summing module 32 sums them up, the label number acquisition module 33 acquires the label number data stored in the label number storage module 25, and then the data volume averaging module 34 uses the summed data and the label number data to calculate the average value, and the calculated average value is stored in the average value storage module 35, the classified data acquisition module 41 acquires the classified data stored in the classified data storage module 24, and the average value acquisition module 42 acquires the average value data in the average value storage module 35, and then the two types of data are respectively imported into the data volume comparison module 41. Block 43, if the data volume of a certain type of data is lower than the average value, the data of this type is sent to the sample generation module 44, the sample generation module 44 randomly generates samples and adds them to the data of this type until the data volume reaches the average value, if the data volume of a certain type of data is higher than the average value, the data of this type is sent to the sample reduction module 45, the sample reduction module 45 randomly reduces the samples in the data of this type until the data volume reaches the average value, and the modified sample data is sent to the balanced data storage module 46 for storage, if the data volume of a certain type of data is equal to the average value, the data of this type is directly sent to the balanced data storage module 46 for storage, and then the data analysis module 50 analyzes the balanced data; In the above step four, the analysis data acquisition module 61 obtains the analysis results in the data analysis module 50, the prediction model establishment module 62 establishes a prediction model based on the results and system imported data, and the prediction model prediction module 63 can predict the results of new data based on the prediction model, and the predicted results are stored in the prediction data storage module 64.

[0020] Based on the above, the advantage of the present invention is that when the present invention is used, the label establishment module 21 establishes labels according to needs, and stores the number of labels through the label number storage module 25. Then, the data search module 22 searches the imported data in turn according to the established labels to facilitate the data classification module 23 to classify. During classification, the amount of data of each category is stored by the data amount storage module 26, and the classified data is stored by the classified data storage module 24, thereby realizing automatic classification of data and preventing errors when importing group by group; at the same time, the data amount acquisition module 31 acquires the data stored in the data amount storage module 26 and then sums it with the data amount summing module 32, the label number acquisition module 33 acquires the label number data stored in the label number storage module 25, and then the data amount averaging module 34 uses the summed data and the label number data to calculate the average value, and the average value after calculation is stored in the average value storage module 35, the classified data acquisition module 41 acquires the classified data stored in the classified data storage module 24, and the average value acquisition module 42 acquires the average value data in the average value storage module 35, and then the two types of data are respectively imported into the data amount comparison Module 43, if the amount of a certain type of data is lower than the average value, the data of this type is sent to the sample generation module 44, the sample generation module 44 randomly generates samples and adds them to the data of this type until the data amount reaches the average value, if the amount of a certain type of data is higher than the average value, the data of this type is sent to the sample reduction module 45, the sample reduction module 45 randomly reduces the samples in the data of this type until the data amount reaches the average value, and the modified sample data is sent to the balanced data storage module 46 for storage, if the amount of a certain type of data is equal to the average value, the data of this type is directly sent to the balanced data storage module 46 for storage, and then the data analysis module 50 analyzes the balanced data to achieve better data balancing and avoid offset when establishing the model; at the same time, the analysis data acquisition module 61 obtains the analysis results in the data analysis module 50, the prediction model establishment module 62 establishes a prediction model according to the results and the system import data, the prediction model prediction module 63 can predict the results of the new data according to the prediction model, and the predicted results are stored in the prediction data storage module 64, and a data model can be generated based on the analysis data, thereby improving the application scope of the system.

[0021] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A sample imbalance data processing and diagnosis system, comprising a data import module (10), a sample classification module (20), a sample number averaging module (30), a sample balancing module (40) and a data analysis module (50), characterized in that: The data import module (10) is connected to a sample classification module (20), the sample classification module (20) is connected to a sample number averaging module (30), the sample number averaging module (30), the sample number averaging module (30) is connected to a sample balancing module (40), the sample balancing module (40) is connected to a data analysis module (50), and the data analysis module (50) is connected to a data prediction module (60).

2. The sample imbalance data processing and diagnosis system according to claim 1, characterized in that: The sample classification module (20) is composed of a label establishment module (21), a data search module (22), a data classification module (23), a classification data storage module (24), a label number storage module (25) and a data quantity storage module (26); the label establishment module (21) is connected to the data search module (22), the data search module (22) is connected to the data classification module (23), and the data classification module (23) is connected to the classification data storage module (24).

3. The sample imbalance data processing and diagnosis system according to claim 2, characterized in that: The label creation module (21) is connected to a label number storage module (25).

4. The sample imbalance data processing and diagnosis system according to claim 2, characterized in that: The data classification module (23) is connected to a data volume storage module (26).

5. The sample imbalance data processing and diagnosis system according to claim 1, characterized in that: The sample number averaging module (30) is composed of a data volume acquisition module (31), a data volume summing module (32), a tag number acquisition module (33), a data volume averaging module (34) and an average value storage module (35); the data volume acquisition module (31) is connected to the data volume summing module (32), and the data volume summing module (32) is connected to the data volume averaging module (34).

6. The sample imbalance data processing and diagnosis system according to claim 5, characterized in that: The data volume averaging module (34) is connected to a tag number acquisition module (33), and the data volume averaging module (34) is connected to an average value storage module (35).

7. The sample imbalance data processing and diagnosis system according to claim 1, characterized in that: The sample balancing module (40) is composed of a classification data acquisition module (41), an average value acquisition module (42), a data volume comparison module (43), a sample generation module (44), a sample reduction module (45) and a balanced data storage module (46); the classification data acquisition module (41) is connected to the data volume comparison module (43), and the data volume comparison module (43) is connected to the average value acquisition module (42).

8. The sample imbalance data processing and diagnosis system according to claim 7, characterized in that: The data volume comparison module (43) is connected to a sample generation module (44), the data volume comparison module (43) is connected to a sample reduction module (45), the data volume comparison module (43) is connected to a balanced data storage module (46), the balanced data storage module (46) is connected to a sample generation module (44), and the balanced data storage module (46) is connected to a sample reduction module (45).

9. The sample imbalance data processing and diagnosis system according to claim 1, characterized in that: The data prediction module (60) is composed of an analysis data acquisition module (61), a prediction model establishment module (62), a prediction model prediction module (63) and a prediction data storage module (64); the analysis data acquisition module (61) is connected to the prediction model establishment module (62), the prediction model establishment module (62) is connected to the prediction model prediction module (63), and the prediction model prediction module (63) is connected to the prediction data storage module (64).

10. A method for diagnosing sample imbalance data processing, comprising the following steps: Step 1: data import; Step 2: data classification; Step 3: data analysis; Step 4: data prediction; It is characterized by: In the above step 1, the data import module (10) imports the data to be analyzed into the system; In the above step 2, the label establishment module (21) establishes labels according to requirements, and stores the number of labels through the label number storage module (25), and then the data search module (22) searches the imported data in sequence according to the established labels so as to facilitate the data classification module (23) to classify, and the amount of data in each category during classification is stored by the data amount storage module (26), and the classified data is stored by the classification data storage module (24); In the above step 3, the data volume acquisition module (31) acquires the data stored in the data volume storage module (26), and then the data volume summing module (32) sums the data. The label number acquisition module (33) acquires the label number data stored in the label number storage module (25). Then, the data volume averaging module (34) uses the summed data and the label number data to calculate the average value, and the calculated average value is stored in the average value storage module (35). The classified data acquisition module (41) acquires the classified data stored in the classified data storage module (24). The average value acquisition module (42) acquires the average value data in the average value storage module (35), and then the two types of data are respectively imported into the data volume. The comparison module (43) sends the data of a certain type to the sample generation module (44) if the data volume of the data of a certain type is lower than the average value. The sample generation module (44) randomly generates samples and adds them to the data of the certain type until the data volume reaches the average value. If the data volume of a certain type is higher than the average value, the data of the certain type is sent to the sample reduction module (45). The sample reduction module (45) randomly reduces the samples in the data of the certain type until the data volume reaches the average value. The modified sample data is sent to the balanced data storage module (46) for storage. If the data volume of a certain type of data is equal to the average value, the data of the certain type is directly sent to the balanced data storage module (46) for storage. Then, the data analysis module (50) analyzes the balanced data. In the above step 4, the analysis data acquisition module (61) obtains the analysis results in the data analysis module (50), the prediction model establishment module (62) establishes a prediction model based on the results and system imported data, and the prediction model prediction module (63) can predict the results of new data based on the prediction model, and the predicted results are stored in the prediction data storage module (64).

Images