Millimeter wave cloud radar non-cloud echo filtering method based on random forest algorithm

Abstract

The invention belongs to the technical field of non-cloud echo filtering, and provides a millimeter wave cloud radar non-cloud echo filtering method based on a random forest algorithm, and the method comprises the steps: to-be-classified radar base data collection, space-time texture and vertical gradient calculation, to-be-classified feature vector construction, and random forest classification model classification. The training process of the random forest classification model comprises the steps of data integration and grouping, data marking, label number statistics, echo type determination, training data set extraction, model training and model verification. According to the method, data such as the vertical gradient and the space-time texture are extracted, so that the model training data volume is enriched, and the model accuracy is improved; the reflectivity factors of four adjacent distance libraries are fused with normal cloud echo data, so that the distinguishing capability of the model on regional characteristic differences is improved; through a normal cloud echo training data set and a non-cloud echo training data set, the defect that a traditional method depends on radar power spectrum data is overcome, and the expansibility and replicability of the model are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of non-cloud echo filtering, and in particular to a millimeter wave cloud radar non-cloud echo filtering method based on a random forest algorithm. Background Art

[0002] Faced with the complex and changeable clouds in nature, various data quality problems are inevitable during millimeter-wave cloud radar detection, mainly non-cloud echoes, range sidelobe echoes, and velocity ambiguity. Among them, non-cloud echoes are one of the important factors affecting the accuracy of low cloud recognition by millimeter-wave cloud radars. Their echo characteristics are weak intensity, low speed, multi-peak power spectrum, and strong linear depolarization ratio. For the identification and filtering of non-cloud echoes, many observation data show that non-cloud echoes have the characteristics of very weak reflectivity factor Z and very strong depolarization ratio LDR as a whole, and are filtered out under the conditions of Z<-10dBZ and LDR>-20dB. Statistics show that non-cloud echoes can be effectively filtered out with Z<-5dBZ and LDR>-22dB, combined with K-domain frequency filtering. Based on radar power spectrum data, an improved neural network algorithm is used to identify and filter non-cloud echoes, with an accuracy rate of more than 80%. Using the three optimally weighted characteristic parameters of height, reflectivity factor intensity and spectrum mean, a neural network algorithm is used to identify non-cloud echoes, with an identification rate of more than 88.38%.

[0003] Although the above-mentioned related technologies have a certain effect on filtering out non-cloud echoes, there are still problems such as differences in radar performance parameters and regional characteristics, which lead to poor filtering effects. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a millimeter-wave cloud radar non-cloud echo filtering method based on a random forest algorithm, which enriches the model training data volume and improves the model accuracy by extracting data such as vertical gradients and spatiotemporal textures; by fusing the reflectivity factors of four adjacent distance libraries with normal cloud echo data, the model's ability to distinguish regional characteristic differences is improved; through normal cloud echo training data sets and non-cloud echo training data sets, the defect of traditional methods relying on radar power spectrum data is solved, and the scalability and replicability of the model are improved.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A millimeter wave cloud radar non-cloud echo filtering method based on random forest algorithm, comprising:

[0007] Collect radar-based data to be classified;

[0008] Calculating the spatiotemporal texture and the vertical gradient using the parameters of the radar-based data to be classified to obtain the data to be classified;

[0009] Constructing a feature vector to be classified according to the data to be classified;

[0010] Inputting the feature vector to be classified into a pre-trained random forest classification model to obtain quality-controlled data;

[0011] The training process of the random forest classification model includes:

[0012] Read the pre-collected millimeter-wave cloud radar base data;

[0013] Extracting the height data, reflectivity factor, radial velocity and spectral width of the millimeter wave cloud radar base data;

[0014] The reflectivity factor is used to calculate the spatiotemporal texture and vertical gradient to obtain texture data and gradient data;

[0015] Integrating the height data, the reflectivity factor, the radial velocity, the spectral width, the texture data, and the gradient data to obtain a data set, and randomly dividing the data set into a training set and a test set according to a preset ratio;

[0016] Using a reflectivity threshold and a polarization ratio threshold to label the distance library in the training set to obtain a label set;

[0017] Setting a detection window, counting the number of target tags of the tag set in the detection window, and obtaining window tag data;

[0018] Determining the echo type of the distance library according to the window tag data; the echo type includes: normal cloud echo and non-cloud echo;

[0019] Extracting data corresponding to the distance library in which the echo type in the training set is non-cloud echo, to obtain a non-cloud echo training data set;

[0020] Extracting the data corresponding to the distance library in which the echo type is a normal cloud echo in the training set and the reflectivity factors corresponding to four adjacent distance libraries to obtain a normal cloud echo training data set;

[0021] Inputting the non-cloud echo training data set and the normal cloud echo training data set into the random forest classification original model for training to obtain a trained temporary classification model;

[0022] The test set is used to obtain the accuracy data of the temporary classification model, and the temporary model whose accuracy meets the classification standard is output to obtain the random forest classification model.

[0023] Preferably, determining the echo type of the distance library according to the window tag data includes:

[0024] Determine the echo type of the distance library whose window label data is greater than or equal to a preset label threshold as a normal cloud echo;

[0025] The echo type of the distance library whose window tag data is less than a preset tag threshold is determined as a non-cloud echo type.

[0026] Preferably, the calculation formula of the texture data is:

[0027]

[0028] Among them, Zr i,j =Z i,j -Z i-1,j ; TdBZ is the texture data; Zr i,j is the gradient data; M is the number of the distance library; N is the number of radar radials; Z i,j is the reflectivity factor of the ith distance bin in the jth radar radial direction.

[0029] The present invention discloses the following technical effects:

[0030] The present invention provides a millimeter-wave cloud radar non-cloud echo filtering method based on a random forest algorithm. By extracting data such as vertical gradients and spatiotemporal textures, the problems of insufficient model training data volume and data dimension are solved, and the model training data volume is enriched; by fusing the reflectivity factors of four adjacent distance libraries with normal cloud echo data, the defect of poor regional characteristics of the traditional model is solved, and the ability to distinguish regional characteristic differences is improved; through normal cloud echo training data sets and non-cloud echo training data sets, the defect of traditional methods relying on radar power spectrum data is solved, and the scalability and replicability of the classification model are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0032] Figure 1 A schematic diagram of a non-cloud echo filtering process for a millimeter-wave cloud radar based on a random forest algorithm provided in an embodiment of the present invention;

[0033] Figure 2 A flowchart of non-cloud echo filtering of millimeter-wave cloud radar based on a random forest algorithm provided in an embodiment of the present invention;

[0034] Figure 3A comparison chart of filtering effects provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The purpose of the present invention is to provide a millimeter-wave cloud radar non-cloud echo filtering method based on a random forest algorithm, which enriches the model training data volume and improves the model accuracy by extracting data such as vertical gradients and spatiotemporal textures; improves the model's ability to distinguish regional characteristic differences by fusing the reflectivity factors of four adjacent distance libraries with normal cloud echo data; and solves the defect of traditional methods that rely on radar power spectrum data through normal cloud echo training data sets and non-cloud echo training data sets, thereby improving the scalability and replicability of the model.

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0038] Figure 1 A schematic diagram of a non-cloud echo filtering process for a millimeter-wave cloud radar based on a random forest algorithm provided in an embodiment of the present invention, Figure 2 A flowchart of filtering non-cloud echoes of a millimeter-wave cloud radar based on a random forest algorithm provided in an embodiment of the present invention is shown in FIG. Figure 1 and Figure 2 As shown, the present invention provides a method for filtering non-cloud echoes of a millimeter-wave cloud radar based on a random forest algorithm, comprising:

[0039] Step 100: Collect radar-based data to be classified;

[0040] Step 200: Calculate the spatiotemporal texture and vertical gradient using the parameters of the radar-based data to be classified to obtain the data to be classified;

[0041] Step 300: construct a feature vector to be classified according to the data to be classified;

[0042] Step 400: input the feature vector to be classified into the pre-trained random forest classification model to obtain quality-controlled data;

[0043] The training process of the random forest classification model includes:

[0044] Read the pre-collected millimeter-wave cloud radar base data;

[0045] Extract the height data, reflectivity factor, radial velocity and spectral width of millimeter wave cloud radar base data;

[0046] The reflectivity factor is used to calculate the spatiotemporal texture and vertical gradient to obtain texture data and gradient data;

[0047] Integrate the height data, reflectivity factor, radial velocity, spectral width, texture data and gradient data to obtain a data set, and randomly divide the data set into a training set and a test set according to a preset ratio;

[0048] The distance library in the training set is labeled using the reflectivity threshold and the polarization ratio threshold to obtain a label set;

[0049] Set the detection window, count the number of target labels in the detection window, and obtain the window label data;

[0050] Determine the echo type of the distance library according to the window label data; the echo types include: normal cloud echo and non-cloud echo;

[0051] Extract the data corresponding to the distance library of the echo type of non-cloud echo in the training set to obtain the non-cloud echo training data set;

[0052] Extract the data corresponding to the distance library with normal cloud echo type in the training set and the reflectivity factors corresponding to the four adjacent distance libraries to obtain the normal cloud echo training data set;

[0053] The non-cloud echo training data set and the normal cloud echo training data set are input into the original random forest classification model for training to obtain a temporary classification model that has been trained.

[0054] The test set is used to obtain the accuracy data of the temporary classification model, and the temporary model whose accuracy meets the classification standard is output to obtain the random forest classification model.

[0055] Further, the echo type of the distance library is determined according to the window label data, including:

[0056] Determine the echo type of the distance library whose window label data is greater than or equal to a preset label threshold as a normal cloud echo;

[0057] The echo type of the distance library whose window label data is less than a preset label threshold is determined as a non-cloud echo type.

[0058] Specifically, the calculation formula for texture data is:

[0059]

[0060] Among them, Zr i,j =Z i,j -Z i-1,j; TdBZ is the texture data; Zr i,j is the gradient data; M is the number of the distance library; N is the number of radar radials; Z i,j is the reflectivity factor of the ith distance bin in the jth radar radial direction.

[0061] Preferably, data preprocessing part. Radar parameter acquisition and spatiotemporal texture and vertical gradient calculation: read the millimeter wave cloud radar base data file to obtain parameters such as reflectivity factor Z, radial velocity V, and spectrum width W. The reflectivity spatiotemporal texture TdBZ reflects the short-term radial change of Z in time and radial direction. The reflectivity vertical gradient Zr reflects the change size and direction of Z in the vertical direction within the local range. Its calculation formula is as follows:

[0062]

[0063] Zr i,j =Z i,j -Z i-1,j

[0064] Where: M is the number of distance bins, N is the number of radar radials, i and j are from 0 to M and 0 to N respectively. In this algorithm, M = N = 3 is set. When any of the adjacent distance bins is the default value, the spatiotemporal texture is not calculated, and the denominator is reduced by 1.

[0065] Furthermore, a data set is created: a data set is created based on six parameters, namely, height H, reflectivity factor Z, radial velocity V, spectral width W, reflectivity vertical gradient Zr, and reflectivity spatiotemporal texture TdBZ.

[0066] Preferably, the model training part: Establishing a training data set: extracting 70% of the data from the complete data set as the training data set.

[0067] Further, labeling: non-cloud echoes and cloud echoes are marked according to their significant features such as weak echo intensity, poor time continuity, shallow echo, low echo height, and strong depolarization ratio. The specific labeling steps are as follows:

[0068] S1: For each distance library, determine whether there is a distance library that satisfies the reflectivity factor Z lower than the threshold (Z thresh ) and the linear depolarization ratio LDR is higher than the threshold (LDR thresh ), traverse all distance libraries and find the distance library category C that meets the condition i,j If yes, it is recorded as 1, otherwise, it is recorded as 0. The calculation formula is as follows:

[0069]

[0070] S2: Based on the echo thickness H and time continuity T, assume a window of H×T and count all the distance libraries C in the window. i,j = 1. If the number is less than NUM, all the distance library labels in the window are marked as 1, that is, non-cloud echo. Otherwise, all the distance library labels in the window are marked as 0, that is, cloud echo.

[0071] Specifically, the training feature variables are extracted: based on the screening results of non-cloud echoes and cloud echoes, the height H, reflectivity factor Z, radial velocity V, spectral width W, reflectivity vertical gradient Zr, and reflectivity spatiotemporal texture TdBZ of the distance library where the non-cloud echo is located are extracted as non-cloud echo training feature variables. At the same time, according to the characteristic differences between cloud echoes and non-cloud echoes, the reflectivity factor Z of the four adjacent distance libraries where the non-cloud echo is located, that is, the reflectivity factors before, after, above, and below the distance library are recorded as Zleft, Zright, Zup, and Zdown respectively, as a supplement to the non-cloud echo training feature variables, forming a non-cloud echo training data set of 10 feature parameters. Similarly, a cloud echo training data set of 10 feature parameters is also formed.

[0072] Further, a random forest classification model is constructed: The random forest classification algorithm is an integrated learning method that improves the accuracy and stability of classification by constructing multiple decision trees. Each decision tree is generated independently during the training process, and they are based on randomly selected samples and feature subsets. When a new sample enters the random forest, each tree will classify it, and the final result is determined by majority voting or other ensemble methods (such as average probability). Therefore, a random forest classification model for cloud echo and non-cloud echo classification is constructed based on the random forest method. The specific steps are:

[0073] S1: Preliminary construction of random forest classification model. First, the training data set is randomly sampled with replacement for n consecutive times to obtain n training subsets; secondly, m training feature variables are randomly selected from one of the training subsets to generate the feature subset of the root node of the decision tree for classifying cloud echoes and non-cloud echoes; then each node is split until the splitting stop condition is met (m feature parameters are split and the maximum depth of the decision tree is reached), and a decision tree is generated. Among them, the node splitting process completes the splitting of the training subset according to the optimal splitting feature value, so that the discrimination between non-cloud echo and cloud echo data after the split is higher than that before the split, and the classification result of cloud echo and non-cloud echo at this node is obtained; finally, the cloud echo and non-cloud echo classification results of all decision trees are counted, and the cloud echo and non-cloud echo classification results of the random forest classification model are obtained according to the majority voting principle, that is, the training result of the model is obtained.

[0074] S2: Use the accuracy rate as an evaluation indicator to evaluate the model training results. The formula is as follows:

[0075] accuracy=(TP+TN) / (P+N)

[0076] Where accuracy is the correct rate, P is the number of non-cloud echo distance libraries, N is the number of cloud echo distance libraries, TP is the number of non-cloud echo distance libraries correctly classified by the random forest classification model, and TN is the number of cloud echo distance libraries correctly classified by the random forest classification model.

[0077] S3: Based on the model evaluation results, the grid search method is used to traverse the hyperparameter combination scheme and 3-fold cross validation to optimize the hyperparameters and find the best hyperparameter combination to improve the performance of the random forest classification model.

[0078] The specific steps are as follows:

[0079] Determine the set of possible hyperparameter values, namely the number of decision trees in the random forest classification model (n_estimators), the depth of the decision tree (max_depth), the minimum number of samples required for internal node splits (min_samples_split), and the minimum number of samples required for leaf nodes (min_samples_leaf);

[0080] Build a hyperparameter grid: combine each hyperparameter and its possible values ​​into a "grid", that is, all possible hyperparameter combinations;

[0081] Model evaluation and 3-fold cross validation: 3-fold cross validation is used on the training set to evaluate the model performance for each hyperparameter combination. First, the training data set is divided into three parts; second, two parts are used for training each time and the remaining part is used for testing, and the model evaluation of this hyperparameter combination is repeated three times; finally, the average of the three evaluation results is used as the evaluation result of this hyperparameter combination. According to the evaluation results, the hyperparameter combination with the highest accuracy in identifying non-cloud echoes and cloud echoes is selected as the optimal hyperparameter combination.

[0082] S4: Construct the optimal random forest classification model for non-cloud echo and cloud echo according to the above optimal hyperparameter combination scheme.

[0083] Preferably, the model prediction verification part. Call the random forest classification model: input the predicted feature variables as input variables into the optimal random forest classification model of non-cloud echo and cloud echo. The output of the above model is the classification result of cloud echo or non-cloud echo in the prediction data set. The prediction label label is marked as 1, which means that the data in the distance library is predicted as non-cloud echo by the optimal random forest classification model. The prediction label label is marked as 0, which means that the data in the distance library is predicted as cloud echo by the optimal random forest classification model.

[0084] Furthermore, based on the above prediction results, the distance library where the non-cloud echo is located is screened (the prediction label is 1), and the data information such as reflectivity factor Z, radial velocity V, and spectral width W in the distance library where the non-cloud echo is located is recorded as invalid values. Finally, the data information after filtering out the non-cloud echo is formed, which is the data product after quality control. According to the above method, a case of weak precipitation clouds, cumulus clouds and high-altitude cirrocumulus clouds observed over a certain radar from 20:00 on September 27, 2023 to 08:00 on September 28, 2023 was selected to verify the filtering effect. It can be seen from the figure that the random forest algorithm can better filter out non-cloud echoes (reference Figure 3 ).

[0085] The beneficial effects of the present invention are as follows:

[0086] The present invention enriches the model training data volume and improves the model accuracy by extracting data such as vertical gradient and spatiotemporal texture; improves the model's ability to distinguish regional characteristic differences by fusing the reflectivity factors of four adjacent distance libraries with normal cloud echo data; and solves the defect of traditional methods that rely on radar power spectrum data through normal cloud echo training data sets and non-cloud echo training data sets, thereby improving the scalability and replicability of the model.

[0087] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0088] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A millimeter wave cloud radar non-cloud echo filtering method based on random forest algorithm, characterized in that: include: Collect radar-based data to be classified; Calculating the spatiotemporal texture and the vertical gradient using the parameters of the radar-based data to be classified to obtain the data to be classified; Constructing a feature vector to be classified according to the data to be classified; Inputting the feature vector to be classified into a pre-trained random forest classification model to obtain quality-controlled data; The training process of the random forest classification model includes: Read pre-collected millimeter-wave cloud radar base data; Extracting the height data, reflectivity factor, radial velocity and spectral width of the millimeter wave cloud radar base data; The reflectivity factor is used to calculate the spatiotemporal texture and vertical gradient to obtain texture data and gradient data; Integrating the height data, the reflectivity factor, the radial velocity, the spectral width, the texture data, and the gradient data to obtain a data set, and randomly dividing the data set into a training set and a test set according to a preset ratio; Using a reflectivity threshold and a polarization ratio threshold to label the distance library in the training set to obtain a label set; Setting a detection window, counting the number of target tags of the tag set in the detection window, and obtaining window tag data; Determining the echo type of the distance library according to the window tag data; the echo type includes: normal cloud echo and non-cloud echo; Extracting data corresponding to the distance library in which the echo type in the training set is non-cloud echo, to obtain a non-cloud echo training data set; Extracting the data corresponding to the distance library in which the echo type is a normal cloud echo in the training set and the reflectivity factors corresponding to four adjacent distance libraries to obtain a normal cloud echo training data set; Inputting the non-cloud echo training data set and the normal cloud echo training data set into the random forest classification original model for training to obtain a trained temporary classification model; The test set is used to obtain the accuracy data of the temporary classification model, and the temporary model whose accuracy meets the classification standard is output to obtain the random forest classification model.

2. The method for filtering non-cloud echoes of millimeter-wave cloud radar based on random forest algorithm according to claim 1, characterized in that: Determining the echo type of the distance library according to the window label data includes: Determine the echo type of the distance library whose window label data is greater than or equal to a preset label threshold as a normal cloud echo; The echo type of the distance library whose window tag data is less than a preset tag threshold is determined as a non-cloud echo type.

3. The method for filtering non-cloud echoes of millimeter-wave cloud radar based on random forest algorithm according to claim 1, characterized in that: The calculation formula of the texture data is: Among them, Zr i,j =Z i,j -Z i-1,j ; TdBZ is the texture data; Zr i,j is the gradient data; M is the number of the distance library; N is the number of radar radials; Z i,j is the reflectivity factor of the ith distance bin in the jth radar radial direction.

Images