A data analysis method, device and medium based on improved random forest

Abstract

The application discloses a data analysis method and device based on an improved random forest and a medium. The method comprises the following steps: integrating basic multi-source data; obtaining data samples based on the basic multi-source data; constructing a basic model based on a random forest algorithm; dividing the basic model based on a space division strategy to obtain an improved model; training a subspace based on a training set and a pre-test set to obtain a weight value corresponding to each subspace, and analyzing the subspace by using a corresponding weighted voting mechanism through the weight value; and training the improved model based on the training set and a test set, and performing data analysis through the trained improved model. The method can effectively analyze the data characteristics of basic information and application information data, and can effectively realize dimension reduction of high-dimensional data and obtain main control indexes of relevant analysis data, so that production guidance can be better performed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computers, in particular to a data analysis method based on improved random forest, device and medium. BACKGROUND

[0002] In basic data management, a large amount of data with various types and complex structures are involved in the basic data system, such as analyzing the fault reasons of equipment or transportation pipelines in the field of safety production. Sometimes more reliable and accurate data information is needed, so it is necessary to perform correlation analysis and sensitivity analysis, eliminate irrelevant and redundant features, generate new feature data, realize dimensionality reduction of high-dimensional data, and obtain the master control index of related data, so as to better guide production.

[0003] For a large amount of high-dimensional multi-source heterogeneous data, the traditional data mining method based on structured database has been difficult to meet the business needs. In actual production process, the operation and maintenance data in production have high nonlinear characteristics, and the data volume is huge. The traditional data statistics and feature analysis method has been not suitable for current business needs. SUMMARY

[0004] In order to solve the above problems, the present application provides a data analysis method based on improved random forest, comprising:

[0005] Integrating basic multi-source data and preprocessing the basic multi-source data;

[0006] Obtaining data samples based on the basic multi-source data, and dividing the data samples into a training set, a test set and a pre-test set;

[0007] Constructing a basic model based on a random forest algorithm, wherein the basic model comprises a plurality of decision trees;

[0008] Dividing the basic model based on a space division strategy to obtain an improved model, wherein the improved model comprises a plurality of subspaces, and each subspace comprises a plurality of decision trees;

[0009] Training the subspaces based on the training set and the pre-test set to obtain a weight value corresponding to each subspace, and analyzing the subspaces by using a corresponding weighted voting mechanism through the weight value;

[0010] Training the improved model based on the training set and the test set, and performing data analysis through the trained improved model.

[0011] In another aspect, the present application also provides a data analysis device based on improved random forest, comprising:

[0012] at least one processor; and

[0013] a memory in communication with the at least one processor; wherein

[0014] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the improved random forest-based data analysis method described in the above examples.

[0015] In another aspect, the present application also provides a non-volatile computer storage medium storing computer executable instructions configured to perform the improved random forest-based data analysis method described in the above examples.

[0016] The improved random forest-based data analysis method provided by the present application can bring the following beneficial effects:

[0017] The improved model in the present application can be used to realize feature extraction, data analysis and data classification of multi-source data at the grassroots level, and can effectively analyze the data characteristics of grassroots basic information and application information data. On the basis of the traditional random forest algorithm, the spatial division strategy and the weighted voting mechanism are added, which can effectively realize the dimension reduction of high-dimensional data and obtain the main control indicators of the relevant analysis data, so as to better guide the production and lay the foundation for the grassroots data work. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of this application and illustrate embodiments of the present application and its description, which serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 FIG. 1 is a flowchart of the improved random forest-based data analysis method in the embodiments of the present application;

[0020] Figure 2 FIG. 2 is a schematic diagram of the improved random forest-based data analysis method in one case in the embodiments of the present application;

[0021] Figure 3 FIG. 3 is a subspace division schematic diagram in one case in the embodiments of the present application;

[0022] Figure 4 FIG. 4 is an algorithm flowchart of the improved model in one case in the embodiments of the present application;

[0023] Figure 5 FIG. 5 is a schematic diagram of the hyperparameter experiment result in one case in the embodiments of the present application;

[0024] Figure 6For one case in the embodiments of the present application, a comparison diagram of the improved model and the traditional model is shown.

[0025] Figure 7 For one case in the embodiments of the present application, a result diagram of data characteristics is shown.

[0026] Figure 8 For the data analysis device based on the improved random forest in the embodiments of the present application, a diagram is shown. DETAILED DESCRIPTION

[0027] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with the embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0028] The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.

[0029] With the development of artificial intelligence, machine learning has shown advantages in data analysis and data governance. Among them, the ensemble learning algorithm is widely used in data classification and regression tasks in various fields due to its high accuracy, easy understanding and strong practicability, and has become a popular direction in machine learning.

[0030] Random forest is based on Bagging algorithm, which uses a simple random sampling method with replacement. In classification problems, it can effectively evaluate the importance of each feature in classification problems, avoid the performance improvement bottleneck and overfitting problem caused by single classifiers such as decision tree and support vector machine algorithm, and effectively improve the prediction performance of the algorithm, which has obvious advantages in extracting and processing high-dimensional feature data.

[0031] However, from the bottom implementation principle, the traditional random forest algorithm uses average majority voting method when making classification decision, without distinguishing the difference between strong classifiers and weak classifiers, which will affect the final classification effect of the whole random forest.

[0032] As shown in Figure 1 The embodiments of the present application provide a data analysis method based on an improved random forest, which comprises:

[0033] S101: integrating and preprocessing the basic multi-source data.

[0034] As shown in Figure 1 and Figure 2As shown, with the continuous promotion and reform of basic-level governance mode, basic-level data covers different types and different characteristics of data, and in the aspect of safety production, there are many types of sensors and a large amount of production control data (as one of basic-level multi-source data) collected, which are seriously affected by noise. Due to different development time or management departments, there are multiple heterogeneous information systems (as data sources to provide basic-level multi-source data), which run on different platforms, resulting in independent systems or account data, and difficulties in data communication, sharing and fusion.

[0035] At this time, the data in the EXCEL account and the database can be distributed and centrally stored and processed, the interface form data needs to be transmitted and preprocessed (including format conversion, etc.), and the same data is regularized and processed according to the rules or requirements.

[0036] S102: Obtain data samples based on the basic-level multi-source data, and divide the data samples into a training set, a test set and a pre-test set.

[0037] After obtaining the data samples, the data samples can be further divided by type, condition and content according to the requirements, so as to select the most needed data samples to divide the training set, the test set and the pre-test set. In the data samples, a part of the data samples is selected as the pre-test set, and the remaining data samples are divided into the training set and the test set according to the ratio of 7:3, which are used to train the model and test the model.

[0038] S103: Construct a basic model based on a random forest algorithm, wherein the basic model comprises a plurality of decision trees.

[0039] The random forest algorithm is an ensemble learning method based on Bagging idea and taking decision tree as basic structure. The algorithm uses a simple random sampling method with replacement, which can effectively evaluate the importance of each feature in the classification problem.

[0040] The random forest algorithm uses multiple decision trees to jointly predict. The random forest trains each decision tree in turn during training. The training sample of each tree is obtained by random sampling from the original training set. The following is the calculation process of the basic model constructed based on the random forest algorithm:

[0041] Suppose there is a data set (for the basic model and the improved model in the embodiments of the present application, the training sample can be referred to during training, and the basic-level multi-source data actually collected can be referred to during actual use) , which has n features, and is sampled with replacement until a sample space is generated . Wherein, represents the nth feature of the ith sample, y i represents a target variable or label of the i-th sample, and m and n represent the size of the space, which is set based on actual needs.

[0042] The base model learner is constructed, sampling is performed in the sampling space, and for each sample , a corresponding decision tree is generated, and the calculation result of each decision tree is recorded.

[0043] The base model is trained T times (a preset number of times) so that , wherein is the final output result of the base model, is a voting algorithm. For example, it can be an average majority voting method, each decision tree has the same voting weight, and in a random forest, the class with the largest number of votes among the prediction results of all weak learners on the sample x is the final classification class. If multiple classes obtain the highest votes, one is randomly selected as the final class.

[0044] S104: The base model is divided based on a space division strategy to obtain an improved model, the improved model including a plurality of subspaces, and each subspace including a plurality of decision trees.

[0045] As shown in Figure 3 , a random forest is obtained by integrating a plurality of decision trees, and if direct decision tree weight assignment is performed, a particularly large weight value is likely to occur, so that the final voting is concentrated on a few trees. The space division strategy divides a large random forest into subspaces (subForest) according to a small scale, and the relative majority voting result in the subspace is taken as the weighted object, so that the classification ability of an excellent subspace can be fully considered, and the situation that a few trees determine the entire forest can be avoided. The random forest is divided into n subspaces subForest1, subForest2, …, subForestN, and the relative majority voting principle is used in each subspace to fully consider all decision trees in the subspace and integrate the final voting result of each subspace. The number of subspaces can be determined based on the number of all decision trees included in the base model, so that the number of decision trees in each subspace is higher than a preset number, and the numbers are relatively close.

[0046] S105: The subspaces are trained based on the training set and the pre-test set to obtain a weight value corresponding to each subspace, so that the subspaces are analyzed by using a corresponding weighted voting mechanism based on the weight value.

[0047] For the assignment principle of random forest subspace weight, both the accuracy of classification result and the time complexity of algorithm should be considered. Considering the above two factors, a weighted voting mechanism is introduced. In addition to the training set and the test set, a part of the pre-test set is also divided. The pre-test set is used to calculate the accuracy on the trained subForest, and then the weight value of each subspace is obtained. Compared with the traditional random forest model (such as the base model mentioned above), it usually uses average majority voting method for voting, and each decision tree has the same voting weight, which is difficult to distinguish the difference between strong classifier and weak classifier. If the decision tree weight is directly assigned, it is easy to have a particularly large weight, so that the final voting is concentrated on a few trees. Therefore, by using the weighted voting mechanism for subspace, it can distinguish between strong and weak classifiers while avoiding the final voting concentrated on a few trees.

[0048] As shown in Figure 4 , the improved random forest algorithm uses space division strategy and weighted voting mechanism. First, the training set (Train set) is divided to train the subspace (subForest). The trained subspace calculates the test accuracy on the pre-test set (Pretest set), and then according to the accuracy, the corresponding weight value (weight) is given to the subspace. The test accuracy of each subspace is positively correlated with its corresponding weight value.

[0049] The improved random forest (WRF) based on space division and weighted voting mechanism is constructed. The divided test set data (Test set) is added to obtain the classification result (Ptype) of each subspace. Based on the weighted voting mechanism, the final classification result (Type) of each test data is determined.

[0050] Specifically, , where is the number of sample sets, is the number of samples classified correctly by the i-th subspace on the sample set x, is the weight of the i-th subspace.

[0051] Suppose a n-class problem | j }, according to the weighted classification result of the subForest, that is, the output probability of the improved model for each classification category is as formula: . Where, , is the probability of the j-th classification category, is a parameter variable, when the classification result of the i-th subspace is , the value of is 0, when the classification result of the i-th subspace is not time value is 1, is the weight value of the i-th subspace, and n is the number of subspaces.

[0052] S106: Based on the training set and the test set, the improved model is trained, and data analysis is performed through the trained improved model.

[0053] After training each subspace and assigning a weight value, the improved model can be trained, and the final classification result of the improved random forest algorithm is obtained by combining the weighted classification results of all random forest subspaces, that is, the output result of the improved model for all classification categories is as follows: ; wherein type is the final output classification category, is the probability of classification category j, is the corresponding function. The result with the maximum probability in all classifications is the final classification result of the improved random forest algorithm, thereby realizing data analysis.

[0054] The present application provides a data feature extraction method based on an improved random forest algorithm combining a space division strategy and a weighted voting mechanism, including data preprocessing, data set division, dividing the size of the random forest using the space division strategy, calculating the weight of the random forest subspace using the weighted voting mechanism, and constructing an improved random forest algorithm data feature extraction model. The accuracy of the feature extraction of the basic multi-source data is compared to achieve the purpose of applying the model to extract the basic data features.

[0055] Through the improved model in the present application, the feature extraction, data analysis and data classification of the basic multi-source data can be realized, and the basic information and application information data can be effectively analyzed. Based on the traditional random forest algorithm, the space division strategy and the weighted voting mechanism are added, which can effectively realize the dimension reduction of high-dimensional data and obtain the main control indicators of the related analysis data, thereby better guiding production and laying a foundation for the basic data work.

[0056] In one or more embodiments of the present application, in order to verify the performance and effectiveness of the proposed feature extraction method, under the same fault data set, the improved random forest algorithm based on the space division strategy and the weighted voting mechanism and the traditional random forest algorithm are used to extract features, and then the neural network is used for classification comparison experiment, the random forest scale is selected from 100, 200, 300, 400, 500, 600 six scales, and 10 rounds of independent experiments are carried out respectively, and the average value of the classification accuracy of the 10 rounds is taken as the final accuracy of each random forest scale as the algorithm performance comparison basis, wherein the optimal subspace division level (that is, the number of subspace division) of the improved random forest algorithm of the present application is determined through the hyperparameter optimization experiment.

[0057] Taking the data of a regional basic pipeline fault section as an example, the characteristic factors causing the fault need to be analyzed, because there are many parameters and large data involved, so the data is input into the improved model in the present application for training, and the hyperparameter experiment result diagram as shown in Figure 5 and the comparison diagram of the improved model and the traditional model as shown in Figure 6 are obtained, through the hyperparameter optimization experiment, the optimal subspace division level of the random forest algorithm based on the space division strategy and the weighted voting mechanism is determined to be [10, 15], and 12 is the optimal.

[0058] As shown in Figure 6 , under the same conditions, the classification accuracy of the improved random forest algorithm based on the space division strategy and the weighted voting mechanism is higher than that of the traditional random forest algorithm, and with the increase of the forest scale, the accuracy will be improved to a certain extent.

[0059] Based on this, it can be determined that the data analysis method proposed in the present application has more excellent performance than the traditional random forest method, can effectively extract relevant data features, and can improve the fault classification accuracy. For example, the data features with the top 20 weight proportions are shown in Figure 7 .

[0060] As shown in Figure 8 , the present application also provides a data analysis device based on an improved random forest, comprising:

[0061] at least one processor; and

[0062] a memory in communication connection with the at least one processor; wherein

[0063] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data analysis method based on the improved random forest of any of the above embodiments.

[0064] The embodiment of the present application further provides a non-volatile computer storage medium, which stores computer executable instructions, and the computer executable instructions are configured to implement the data analysis method based on the improved random forest.

[0065] The embodiments in the present application are described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other. Each embodiment mainly describes the difference from other embodiments. Especially, the device and medium embodiments are basically similar to the method embodiments, and thus the description is relatively simple, and the related parts can be referred to the description of the method embodiments.

[0066] The device and medium provided by the embodiments of the present application are one-to-one corresponding to the method, and thus the device and medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be described here.

[0067] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0068] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The function of one flow or multiple flows and / or blocks Figure 1 The function of one block or multiple blocks.

[0069] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable storage medium produce a manufactured product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The function of one flow or multiple flows and / or blocks Figure 1 The function of one block or multiple blocks.

[0070] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1 One or more flowchart or block diagrams can be used to describe embodiments of the present application as can be set forth in the accompanying drawings. However, it is to be understood that the functions noted in the blocks can be carried out in various other sequences than those described or shown in the figures. For example, described embodiments can be carried out in the manner in which functions are performed in other orders or overlapping, as will be readily understood and appreciated by those skilled in the art. One or more blocks can be combined or integrated with other blocks contrary to the flow direction.

[0071] In one typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0072] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, which can store information such as data, metadata, and / or instructions that can be used by one or more of the components of the computing device. The memory is an example of computer readable media.

[0073] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0074] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that includes the recited element.

[0075] ​The above merely provides an example of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A data analysis method based on improved random forest, characterized in that: It is used to solve the technical problems of large production control data volume and serious noise impact, which makes it difficult to select main control indicators and improve production guidance capabilities, including: Integrating grassroots multi-source data and preprocessing the grassroots multi-source data; Obtaining data samples based on the base multi-source data, and dividing the data samples into a training set, a test set, and a pre-test set; Building a basic model based on a random forest algorithm, wherein the basic model includes multiple decision trees; Based on a spatial partitioning strategy, the basic model is partitioned to obtain an improved model, wherein the improved model includes multiple subspaces, and each subspace includes multiple decision trees; the spatial partitioning strategy is to divide the random forest into subspaces according to scale, and use the relative majority voting results within the subspace as the weighted action object; Based on the space partitioning strategy, the basic model is divided to obtain an improved model, which specifically includes: Dividing the number of all decision trees included in the basic model into multiple subspaces so that each subspace contains at least a predetermined number of decision trees; Based on the training set and the pre-test set, the subspaces are trained to obtain weight values ​​corresponding to each subspace, and the subspaces are analyzed using a corresponding weighted voting mechanism according to the weight values; Based on the training set and the test set, the improved model is trained, and data analysis is performed using the trained improved model; The grassroots multi-source data includes at least production control data; the data sources of the grassroots multi-source data are multiple heterogeneous information systems, and the multiple heterogeneous information systems run on different platforms; the preprocessing includes at least format conversion; the production control data includes sensor data, EXCEL ledgers and database data.

2. The method according to claim 1, characterized in that Build a basic model based on the random forest algorithm, including: Construct a sampling space for an existing dataset; Constructing a base learner, sampling in the sampling space, generating a corresponding decision tree for each sampling, and recording the calculation results of the decision tree; For the calculation results of all decision trees, the average majority voting method is used to select the classification category with the most votes to determine the final prediction result.

3. The method according to claim 1, characterized in that Based on the training set and the pre-test set, the subspace is trained to obtain a weight value corresponding to each subspace, specifically including: Training the subspace based on the training set, and testing the trained subspace using the pre-test set to obtain a test accuracy corresponding to the subspace; According to the accuracy, a corresponding weight value is assigned to the subspace; wherein the test accuracy of each subspace is positively correlated with its corresponding weight value.

4. The method according to claim 3, characterized in that The output probability of each classification category of the improved model is as follows: ; in, , is the probability of the jth classification category, is a parameter variable. When the classification result of the i-th subspace is hour The value is 0, when the classification result of the i-th subspace is not hour The value is 1, is the weight value of the i-th subspace, and n is the number of subspaces.

5. The method according to claim 4, characterized in that The output results of the improved model for all classification categories are as follows: ; Among them, type is the classification category of the final output, is the probability of classification category j, is the corresponding function.

6. The method according to claim 1, characterized in that The method further comprises: The subspace division level is determined through a hyperparameter optimization experiment, and the range of the division level is [10, 15].

7. A data analysis device based on improved random forest, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: the data analysis method based on improved random forest as described in any one of claims 1 to 6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are set to: the data analysis method based on improved random forest according to any one of claims 1 to 6.

Images