ROA-RF-based clear sky echo recognition method
By adopting the ROA-RF-based clear sky echo recognition method in millimeter wave cloud radar, using improved random forest and fish optimization algorithm, combined with threshold method and refined labeling method, the problem of clear sky echo filtering is solved, high-accuracy recognition effect is achieved, and the analysis quality of meteorological radar data is improved.
Patent Information
- Application Number
- CN202411382938.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-06-20
AI Technical Summary
The lack of technology for clear sky echo filtering of millimeter-wave cloud radars in the existing technology, resulting in errors in radar map analysis and affecting meteorological observation and research.
The clear sky echo recognition method based on ROA-RF is adopted, and the training set is analyzed and feature extraction is performed, and the clear sky echo recognition model is constructed using improved random forest and fish optimization algorithms, and feature screening and noise removal are performed through threshold method and refined labeling.
It realizes accurate recognition of clear sky echoes, improves recognition accuracy and overall accuracy, reduces errors in the model learning process, and provides new tools for the processing and analysis of meteorological radar data.
Smart Images
Figure CN120180254A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of atmospheric science and technology, and specifically develops a clear-sky echo recognition method based on ROA-RF. Background Art
[0002] Millimeter-wave cloud radar is a new type of meteorological radar that uses high-frequency electromagnetic waves to detect meteorological targets with high precision. It mainly utilizes the backscattering effect of meteorological particle groups on electromagnetic waves to obtain the echo intensity, radial velocity, velocity spectrum width, and polarization information of meteorological targets, and further generates secondary data products.
[0003] It can play a decisive role in severe convective weather warning, research on the causes of extreme weather, and formulation of disaster weather emergency plans.
[0004] However, during detection, it is inevitably contaminated by a large number of clutter signals from non-meteorological targets. Common non-meteorological echoes include ground clutter, clear-sky echoes, and super-refraction echoes, etc. Clear-sky echoes are generated by moving low-altitude insects, birds, and atmospheric turbulence, etc.; clear-sky echoes are mostly manifested in the weak echo area near the radar. In each pixel point detected by the radar, as long as there are a small number of insects, a certain value of reflectivity factor can appear, which is easily confused with low-cloud echoes. The superposition of these clear-sky echoes and meteorological echoes is extremely likely to cause errors in radar map analysis, bringing difficulties to meteorological researchers in meteorological observation and research. Summary of the Invention
[0005] The purpose of the present invention is to provide a clear-sky echo recognition method based on ROA-RF, and develop a feature extraction algorithm for echo data based on the threshold method, so as to solve the technical problem of the lack of clear-sky echo filtering for millimeter-wave cloud radar in the prior art;
[0006] The present invention proposes a clear-sky echo recognition method based on ROA-RF, which is characterized in that it includes the following steps:
[0007] S1. Conduct attribute analysis and statistics according to the given training set to obtain the differences between clear-sky echoes and meteorological echoes in basic features, and perform secondary calculation of features based on the basic features to expand the differences between the two features;
[0008] S2. Mark clear-sky echoes, ground clutter echoes, and meteorological echoes pixel by pixel according to the refined labeling method to reduce the error in the model learning process;
[0009] S3. Analyze the distribution of echo features in the data set through experiments, and select features as input features for model training;
[0010] S4. Use the improved random forest as the basic learner, and use the remora optimization algorithm to calculate the hyperparameters of the number and depth of decision trees in the model, so as to obtain the clear sky echo recognition model.
[0011] Furthermore, according to the attribute analysis in step S1, draw THI images for each basic feature parameter in the clear sky echo data and meteorological echo data, and statistically analyze the differences in basic features; calculate secondary feature parameters based on the basic feature parameters to obtain a line graph of the relationship between each feature parameter and the data volume; determine the differences in the distribution of each feature parameter according to the statistical results of the line graph.
[0012] Furthermore, the basic feature parameters in step S1 include: reflectivity factor Z1, radial velocity V1, velocity spectrum width W1, and signal-to-noise ratio SNR1; secondary product feature parameters: echo intensity texture TDBZ, texture, mean reflectivity factor Average-Z1, variance of reflectivity factor SDVE-Z1, mean velocity Average-V1, and velocity variance SDVE-V1; the calculations of these quantities are defined as follows:
[0013]
[0014] Where NA and NR represent the calculation ranges defined in the azimuth and distance directions, Zi,j is the echo intensity at any point, TDBZ mainly reflects the local change size of the echo intensity; MDVE represents the radial velocity value of a point after median filtering, MDVE represents the average value of this radial velocity in this range, and SDVE is the variance of the radial velocity. For physical quantities related to echo intensity, it is stipulated that NA = 3 and NR = 3; for radial velocity and velocity spectrum width, it is stipulated that NA = 3 and NR = 9. Parametric calculations are only performed when the valid data within these ranges exceeds a certain range.
[0015] Furthermore, set the threshold according to the refined labeling method in step S2, and set the threshold according to the meteorological echoes at different heights and the surrounding particle distribution; use the pixel-level refined manual labeling method for the data to label each pixel point of the clear sky echo and meteorological echo.
[0016] Furthermore, according to the statistical results of the THI map and the parameter line graph of secondary features, determine the discrimination ability of each feature parameter for clear sky echo data and meteorological echo data; according to the discrimination ability, screen the feature parameter data in the clear sky echo data and meteorological echo data.
[0017] Further, in step S4, the bootstrap sampling method is used to randomly sample the training set, and different subsets are used to train the individual learners to expand the model diversity; the optimized data is imported into the ROA optimization algorithm, and the adaptation effect is calculated using eq. 21; then the ROA position is updated and the boundary is checked to determine whether the stopping condition is satisfied. If the stopping condition is not satisfied, parameter iteration continues, otherwise the iteration is terminated, and the RF parameters are obtained; the RF parameters are loaded into the RF model for training to obtain the ROA-RF prediction model; the data is divided into a training set and a test set, and the original echo data and the extracted feature data are selected to train the ROA-RF model, and finally the classification effect of the model is verified on the test set.
[0018] Further, the present invention provides feature extraction of echo data based on the threshold method, including: setting appropriate thresholds according to meteorological echoes at different heights and the surrounding particle distribution, and dividing the threshold setting into a global threshold and a local threshold. The global threshold compares the pixel values of the entire image with a set single threshold, thereby classifying the pixels into two categories: target and background. The local threshold adaptively sets the threshold according to the local area around the pixel and is applicable to the situation of uneven illumination or complex background.
[0019] Further, the present invention provides a clear-sky echo noise reduction method, which obtains the noise image of the clear-sky echo; classifies and discusses the distance and height of the noise points, and the ratio of the number of clear-sky echo points to meteorological echo points within the same time to determine the clear-sky echo points and meteorological echo points.
[0020] Further, a method for identifying the characteristics of marked data proposes a method for extracting echo data characteristics based on an improved threshold method; appropriate thresholds are set according to meteorological echoes at different heights and the surrounding particle distribution, so as to effectively screen out data points with information and extract them as features.
[0021] In summary, the present invention has the following beneficial effects:
[0022] 1. By analyzing the attribute importance of the training set, the differences between the two in basic features are determined, and quadratic features are calculated to expand the differences, enabling the model to more accurately identify clear-sky echoes. The recognition accuracy of the ROA-RF algorithm for clear-sky echoes is as high as 99%, showing the excellent performance of this method in clear-sky echo recognition. The pixel-level refined manual marking method is adopted to mark clear-sky echoes and meteorological echoes pixel by pixel, further reducing the error in the model learning process.
[0023] 2. By experimentally analyzing the distribution of echo features on the dataset, the optimal features are selected as the input features for model training, improving the training efficiency and recognition performance of the model. The improved random forest is used as the basic learner, and the remora optimization algorithm is utilized to optimize the hyperparameters of the number and maximum depth of decision trees in the model, obtaining better model parameters. An echo data feature extraction algorithm based on the threshold method is proposed. Appropriate thresholds are set according to the meteorological echoes at different heights and the distribution of surrounding particles, effectively screening out the data points with important information.
[0024] 3. By classifying and discussing the distance, height of the noise points, and the ratio of clear sky echo points to meteorological echo points within the same time, the clear sky echo points and meteorological echo points are determined, effectively removing the clear sky echo noise. It is not only applicable to the recognition of clear sky echoes but also can be used to distinguish ground echoes from meteorological echoes, providing a new tool for the processing and analysis of meteorological radar data. The overall accuracy of clear sky echoes is 94%, showing good performance of this method in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is the THI diagram of clear sky echoes and meteorological echoes;
[0026] Figure 2 is the distribution diagram of SNR1 and Z1 after data annotation;
[0027] Figure 3 is the relationship diagram between the secondary characteristic parameters of clear sky echoes and the dataset;
[0028] Figure 4 is the flow chart of the training process of the improved random forest algorithm;
[0029] Figure 5 is the three-dimensional iteration diagram of the clear sky echo recognition algorithm of ROA-RF;
[0030] Figure 6 is the clear sky echo recognition result diagram before and after model optimization;
[0031] Figure 7 is the clear sky echo recognition result diagram after filtering. DETAILED DESCRIPTION OF THE INVENTION
[0032] Refer to the appended Figure 1-7 to further illustrate the present invention. The technical solutions of the present invention will be further described in detail through specific embodiments as follows:
[0033] A method for recognizing clear sky echoes based on ROA-RF according to an embodiment of the present invention includes the following steps:
[0034] Step S1: Perform attribute analysis and statistics on the input training set to obtain the differences in basic features between clear-sky echoes and meteorological echoes. Calculate secondary features based on the basic features to expand the differences in their features.
[0035] Step S2: Mark clear-sky echoes, ground clutter echoes, and meteorological echoes pixel by pixel according to the refined labeling method to reduce the error in the model learning process.
[0036] Step S3: Analyze the distribution of echo features on the data set through experiments and select the features for use as input features for model training.
[0037] Step S4: Use the improved random forest as the basic learner and use the remora optimization algorithm to calculate the hyperparameters of the number and depth of decision trees in the model, so as to obtain a clear-sky echo recognition model.
[0038] According to the attribute analysis in Step S1, first draw THI images for each basic feature parameter in the clear-sky echo data and meteorological echo data, and statistically analyze the differences in basic features. Then, calculate secondary feature parameters on the basis of the basic feature parameters to obtain a line chart of the relationship between each feature parameter and the data volume. Finally, determine the differences in the distribution of each feature parameter according to the statistical results of the line chart.
[0039] Implement the attribute importance analysis of the given training set in Step S1. First, determine the differences in basic features between clear-sky echoes and meteorological echoes. The basic feature parameters include reflectivity factor Z1, radial velocity V1, velocity spectrum width W1, and signal-to-noise ratio SNR1. Then, calculate secondary features on the basis of the basic features, such as echo intensity texture TDBZ, texture Texture, mean reflectivity factor Average-Z1, variance of reflectivity factor SDVE-Z1, mean velocity Average-V1, and velocity variance SDVE-V1, to expand the differences in their features and provide more powerful feature support for subsequent recognition. Specifically, analyze the distribution characteristics of clear-sky echoes and meteorological echoes on the THI image to determine the differences in basic features. Add 6 more feature parameters on the basis of the basic features, analyze the number distribution chart of all basic feature parameters, and analyze the differences in their features again. The calculations of these quantities are defined as follows:
[0040]
[0041] Among them, NA and NR represent the calculation ranges defined in the azimuth and range directions. Zi,j is the echo intensity at any point. TDBZ mainly reflects the local change magnitude of the echo intensity. MDVE represents the radial velocity value of a certain point after median filtering. MDVE represents the average value of this radial velocity in this range, and SDVE is the variance of the radial velocity. For physical quantities related to the echo intensity, it is stipulated that NA = 3 and NR = 3; for the radial velocity and velocity spectrum width, it is stipulated that NA = 3 and NR = 9. Parametric calculations are only performed when the valid data within these ranges exceed a certain range.
[0042] In step S2, the refined labeling method is adopted to mark clear-sky echoes and meteorological echoes pixel by pixel: Specifically, first, appropriate thresholds are set according to the meteorological echoes at different heights and the surrounding particle distribution conditions to accurately distinguish the echo types at different heights. Then, the refined manual labeling method at the pixel level is used to mark clear-sky echoes and meteorological echoes pixel by pixel. It can ensure that each pixel is accurately classified, thereby reducing the error in the model learning process and improving the recognition accuracy. As shown in Table 1:
[0043] ROA-RF Statistical Index for Clear-Sky Echo Recognition
[0044]
[0045] In step S3, first, the distribution of echo features on the dataset is analyzed through experiments, including the statistics and visualization of various echo features in the training set (such as basic features like reflectivity factor Z1, radial velocity V1, velocity spectrum width W1, signal-to-noise ratio SNR1, etc., and secondary features like echo intensity texture TDBZ, texture, average reflectivity factor Average-Z1, variance of reflectivity factor SDVE-Z1, average velocity Average-V1, variance of velocity SDVE-V1, etc.) to understand the distribution laws and differences of these features in clear-sky echoes and meteorological echoes. Then, according to the statistical results such as the THI diagram and the broken line diagram of secondary feature parameters, the discrimination ability of each feature parameter for clear-sky echo data and meteorological echo data is determined. The feature parameters with strong discrimination ability for the two types of echoes are selected as the input features for subsequent model training. Finally, according to the discrimination ability screening results, the optimal feature parameter data is selected from all feature parameters as the input for model training to construct a prediction model that can accurately identify clear-sky echoes and meteorological echoes. It can ensure that the model can make full use of the most discriminative feature information and improve the recognition accuracy and efficiency.
[0046] In step S4, an improved random forest is adopted as the base learner, and the remora optimization algorithm is used to perform hyperparameter optimization on the number of decision trees and the maximum depth in the model, so as to obtain an optimized clear sky echo recognition model, that is, the ROA-RF model. First, with RF as the base learner, the bootstrap sampling method is used to randomly sample the training set. By randomly drawing multiple subsets from the original training set with replacement, each subset is used to train an individual learner (i.e., a decision tree). This can expand the diversity of the model and improve the generalization ability of the model. Then, the optimized data (including the data after attribute importance analysis, refined label processing, and feature selection) is imported into the ROA optimization algorithm. The ROA algorithm is an algorithm for global optimization that can find the optimal solution in the search space. The ROA algorithm is used to optimize the number of decision trees and the maximum depth in the RF model to find the best parameter combination. During the ROA optimization process, a specific calculation formula (such as eq.21) is used to evaluate the model performance under the current parameter combination. Then, according to the evaluation results, the position of ROA is updated, and the boundary conditions are checked to determine whether the stopping conditions are met. If the stopping conditions are not met, the parameter iteration continues; otherwise, the iterative process is exited, and the current optimal RF parameter combination is obtained. Finally, the best parameters are loaded into the RF model for training to obtain the ROA-RF prediction model. During the training process, the data is divided into a training set and a test set, and the original echo data and the extracted feature data are used to train the ROA-RF model, and the classification effect of the model is verified on the test set. Through this process, an ROA-RF model that can accurately identify clear sky echoes and meteorological echoes can be obtained.
[0047] The method for removing the noise points after the clear sky echo recognition is as follows: For the near-ground noise points, when the height is greater than 100, if the ratio of the number of clear sky echo points to the number of meteorological echo points within the same time is greater than 0.8, then all of them are set as clear sky echo points; For the noise points around 300 meters, when the height is greater than 80 and less than 120, if the ratio of the number of clear sky echo points to the number of meteorological echo points within the same time is greater than 1 and the background points are greater than 15, then all of them are set as clear sky echo points; For the clear sky echo noise points in the high altitude, when the height is greater than 150, if the number of clear sky echo points within the same time is less than 10, then all of them are set as meteorological echo points; For the continuous clear sky echo noise points in the high altitude, when the height is greater than 100, if there are 10 consecutive points recognized as meteorological echoes within the same time, then fill downwards starting from the first non-background point until the first point with a value of 2 is filled.
[0048] In an embodiment, the method for extracting the characteristics of echo data according to the threshold method is as follows: The threshold setting is divided into a global threshold and a local threshold. The global threshold compares the pixel values of the entire image with a set single threshold, thereby classifying the pixels into two categories: target and background. The local threshold adaptively sets the threshold according to the local area around the pixel and is applicable to the situation of uneven illumination or complex background.
[0049] The above embodiments are only for illustrating the technical concept and features of the present invention, and the purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and shall not be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit and essence of the present invention shall be covered within the protection scope of the present invention.
[0050] Of course, the above are only typical examples of the present invention. In addition, the present invention can also have many other specific implementation manners. Any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the scope required to be protected by the present invention.
Claims
1. A clear sky echo recognition method based on ROA-RF, characterized by: The following steps are involved: Step S1, performing attribute analysis and statistics based on the input training set to obtain the difference in basic features between the clear sky echo and the weather echo, and performing secondary feature calculation based on the basic features to obtain the difference in features between the two; Step S2: label the clear sky echo, ground object echo and weather echo pixel by pixel according to the refined labeling method to reduce the error in the model learning process; Step S3, analyzing the distribution of echo features on the data set through experiments, and selecting features for input features of model training; Step S4, using the improved random forest as the basic learner, using the remora optimization algorithm to calculate the hyperparameters of the number and depth of decision trees in the model, and obtaining the clear sky echo recognition model.
2. The clear sky echo identification method based on ROA-RF according to claim 1, characterized in that: According to the attribute analysis in step S1, firstly, a THI image is drawn for each basic characteristic parameter in the clear sky echo data and the meteorological echo data, and the difference of the basic characteristics is statistically analyzed; Then, the secondary characteristic parameters are calculated based on the basic characteristic parameters to obtain a line graph of the relationship between each characteristic parameter and the data volume; finally, the difference in the distribution of each characteristic parameter is determined based on the statistical results of the line graph.
3. The clear sky echo identification method based on ROA-RF according to claim 1, characterized in that: The basic characteristic parameters in step S1 include: reflectivity factor Z1, radial velocity V1, velocity spectrum width W1 and signal-to-noise ratio SNR1; the secondary product characteristic parameters include: echo intensity texture TDBZ, texture Texture, reflectivity factor average Average-Z1, reflectivity factor variance SDVE-Z1, velocity average Average-V1 and velocity variance SDVE-V1; the calculation definitions of these quantities are as follows: Among them, NA and NR represent the calculation range defined in the azimuth and distance directions, and j represents the echo intensity of any point. TDBZ mainly reflects the local change of echo intensity; MDVE represents the radial velocity value of a point after median filtering. MDVE represents the average value of this radial velocity in this range, and SDVE is the variance of the radial velocity. For physical quantities related to echo intensity, NA=3, NR3 are specified; for radial velocity and velocity spectrum width, NA=3, NR9 are specified. Parameter calculation is performed only when the valid data exceeds a certain range within these ranges.
4. A clear sky echo recognition method based on ROA-RF according to claim 1, characterized in that: The threshold is set according to the refined labeling method in step S2, and the threshold is set according to the meteorological echoes at different heights and the distribution of surrounding particles; the data is subjected to pixel-by-pixel refined manual labeling using a pixel-level refined manual labeling method to label the clear sky echo and the meteorological echo.
5. A clear sky echo identification method based on ROA-RF according to claim 2, characterized in that: According to the statistical results of the THI graph and the parameter line graph of the secondary features, the ability of each characteristic parameter to distinguish between clear sky echo data and meteorological echo data is determined; based on the distinguishing ability, the characteristic parameter data in the clear sky echo data and meteorological echo data are screened.
6. A clear sky echo recognition method based on ROA-RF according to claim 1, characterized in that: In step S4, the training set is randomly sampled by the self-service sampling method, and individual learners are trained with different subsets to expand the model diversity; the optimized data is imported into the ROA optimization algorithm, and the adaptation effect is calculated using eq.21; then the ROA position is updated and the boundary is checked to determine whether the stopping condition is met. If the stopping condition is not met, the parameter iteration continues, otherwise the iteration is jumped out to obtain the RF parameters; the RF parameters are loaded into the RF model for training to obtain the ROA-RF prediction model; the data is divided into a training set and a test set, and the original echo data and the extracted feature data are selected to train the ROA-RF model, and finally the classification effect of the model is verified on the test set.
7. A clear sky echo noise reduction method, characterized by: Obtain the noise image of the clear sky echo; classify and discuss the distance and height of the noise points, and the ratio of the number of clear sky echo points to the meteorological echo points at the same time, and determine the clear sky echo points and meteorological echo points.
8. A method for identifying features of labeled data, characterized in that: A feature extraction method for echo data based on the improved threshold method is proposed. According to the meteorological echoes at different heights and the distribution of surrounding particles, appropriate thresholds are set to effectively screen out data points with information and extract them as features.