A wind profile data intelligent processing method

Abstract

This invention relates to an intelligent processing method for wind profiler data, which solves the problems of existing wind profiler data processing methods, such as reliance on manual intervention for identifying missing height layers and the lack of structural consistency guarantees in interpolation methods. This method achieves automatic structural analysis and intelligent completion of missing height layers in wind profiler data through multi-step collaborative processes. The steps sequentially execute data access and format standardization, height layer structure parsing and index construction, intelligent identification of missing height layers, interpolation condition judgment, adaptive interpolation calculation, data structure reconstruction, and database storage. It automatically identifies missing height layers and adaptively selects interpolation algorithms to generate interpolation results based on data change trends, ensuring the integrity and continuity of the data structure and providing a reliable data foundation for wind field analysis models. This invention can be widely applied to wind profiler radar data platforms, meteorological data operational processing systems, and other scenarios.

Description

Technical Field

[0001] This invention relates to the field of meteorological data processing technology, and in particular to an intelligent processing method for wind profiler observation data. This invention belongs to the interdisciplinary technical field of environmental measurement data processing and intelligent meteorological data analysis, and is applicable to applications in wind profiler radar data platforms, environmental measurement data sharing platforms, and meteorological data operational processing systems for the automatic analysis and structural reconstruction of discontinuous altitude layer data. Background Technology

[0002] With the widespread application of wind profiler radar systems in meteorological observation, wind profiler data has gradually become an important data source for describing the structure of the atmospheric boundary layer. Wind profiler radar acquires wind speed, direction, and vertical velocity information at different altitudes through layered observation, forming a dataset with a height sequence structure. In practical applications, due to radar signal attenuation, changes in equipment operating status, and complex meteorological conditions, wind profiler observation data often suffers from missing altitude layers, resulting in gaps in what should be a continuous altitude layer sequence. Missing altitude layers not only affect data integrity but also introduce errors into subsequent wind field analysis models, leading to distortions in wind field structure analysis and decreased accuracy in numerical forecasts. Existing data processing methods typically use simple interpolation to fill in missing altitude layers, but these methods mostly lack a systematic data structure analysis mechanism, failing to guarantee the consistency of the data structure before and after interpolation. Furthermore, existing methods often perform interpolation updates directly at the database level, lacking a unified data structure management mechanism, which can easily lead to data duplication or disordered order, increasing the complexity of data maintenance. Furthermore, existing methods typically require manual specification of the interpolation height range, lacking automatic identification capabilities, resulting in low data processing efficiency and failing to meet the real-time and automation requirements of meteorological operations. Therefore, it is necessary to develop a method capable of automatically identifying missing height layers and performing intelligent interpolation processing to achieve automatic reconstruction of wind profile data structures, thereby improving data processing efficiency and reliability. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing wind profiler data processing methods, such as reliance on manual intervention for identifying missing height layers, lack of structural consistency guarantees in interpolation methods, and imperfect data management mechanisms. This invention proposes an intelligent wind profiler data processing method. By constructing a multi-step collaborative workflow, this method enables automatic structural analysis of wind profiler observation data before or after it enters the database. It also automatically performs interpolation to complete missing height layers, thereby forming a structurally complete and highly continuous data set, providing a reliable data foundation for subsequent wind field analysis models and operational applications.

[0004] The technical problem solved by this invention is achieved through the following technical solution: A method for intelligent processing of wind profile data includes the following steps: Step 1: Wind profile data access and format standardization processing; Step 2: Height layer structure parsing and index construction; Step 3: Intelligent identification of missing height layers and generation of interpolation set; Step 4: Interpolation condition judgment and boundary validity verification; Step 5: Adaptive interpolation calculation and consistency verification; Step 6: Data structure reconstruction and database update.

[0005] The standardized data generated in step 1 is used as the input for step 2. The height sequence and index constructed in step 2 are used as the input for step 3. The set of heights to be interpolated generated in step 3 is used as the input for step 4. The verification result of step 4 determines whether to execute step 5. The valid interpolated data generated in step 5 and the original height data in step 2 are used as the input for step 6. After step 6 completes the data reconstruction, the complete data is written to the database, and the data availability is fed back to step 1, forming a closed-loop processing flow.

[0006] Furthermore, the specific method of step 1 includes the following steps: Step 1.1: Build a multi-source data access interface to support reading data from various data sources such as real-time data streams from wind profiler radar, historical databases, and local files; Step 1.2: Preprocess the raw data to remove invalid data caused by equipment failure or signal interference, such as negative wind speed or wind direction outside the 0-360° range; Step 1.3: Convert the preprocessed multi-source data into a unified structured format and define standard data fields including: height, wind speed, wind direction, vertical velocity, and acquisition time; Step 1.4: Perform deduplication on the standardized dataset. If data with the same height and the same acquisition time exist, only the data with the highest quality is retained. Data quality is determined by the signal-to-noise ratio (SNR). The higher the SNR value, the better the data quality. Output the standardized dataset.

[0007] Furthermore, the specific method of step 2 includes the following steps: Step 2.1: Perform structural parsing on the standardized data set output in Step 1, traverse all data records, extract the height field, and form an initial height set; Step 2.2: Sort the initial height set in ascending order, remove duplicate height values, and obtain an ordered and unique height sequence; Step 2.3: Construct a height-data mapping index using a hash table structure, associating each height value with the corresponding complete data record to achieve fast query from height value to data record, optimizing the query time complexity to O(1); Step 2.4: Calculate the average step size of the height sequence, which is used for the subsequent generation of the standard height model.

[0008] Furthermore, the specific method of step 3 includes the following steps: Step 3.1: Filter outliers from the ordered height sequence obtained in Step 2, removing height values ​​that are outside the observation range of the wind profiler radar or that deviate from the average step size by ±50%, to obtain the effective height sequence. Step 3.2: Calculate the percentage of valid data in the valid height sequence. If the percentage of valid data is less than 80%, a data anomaly alarm is triggered, the current identification process is terminated, and the process returns to Step 1 to re-access data; if the percentage of valid data is greater than or equal to 80%, the subsequent steps are executed. Step 3.3: Generate a standard height layer sequence model, starting with the minimum height of the effective height sequence and ending with the maximum height, using the average step size as the interval to generate a continuous standard height sequence. Step 3.4: Compare the effective height sequence with the standard height sequence item by item to identify the height values ​​that do not appear in the effective height sequence, which are the missing height layers, forming a missing height set; Step 3.5: Calculate the percentage of missing heights. If the percentage of missing heights is >30%, the data is deemed insufficient, and the process returns to Step 1 to retrieve data again. If the percentage of missing heights is ≤30%, each missing height is matched with two adjacent valid heights, the mapping relationship is recorded, a set of heights to be interpolated is generated, and the set of heights to be interpolated and the mapping relationship are output.

[0009] Furthermore, the specific method of step 4 includes the following steps: Step 4.1: Take out each missing height sequentially from the set of heights to be interpolated output in Step 3, and obtain its corresponding adjacent valid heights according to the mapping relationship; Step 4.2: Using the height-data mapping index built in Step 2, query the corresponding complete data record; Step 4.3: Verify the consistency of timestamps in the boundary data and calculate the time difference between the acquisition of two adjacent valid height data. If the time difference is > 5 minutes, the boundary data is determined to be invalid, the missing height is marked as non-interpolable, the height layer is skipped, and the next missing height is processed; if the time difference is ≤ 5 minutes, the boundary data is determined to be valid, the interpolation condition is met, and step 5 is executed.

[0010] Furthermore, the specific method of step 5 includes the following steps: Step 5.1: For missing heights that meet the interpolation conditions, calculate the height difference between them and adjacent valid heights; Step 5.2: Calculate the rate of change of wind speed at the upper and lower boundaries for adaptive selection of the interpolation algorithm; Step 5.3 Adaptive selection of interpolation algorithm: If the wind speed change rate is <0.5m / s / 100m, a linear interpolation algorithm is used to generate the missing height data; if the wind speed change rate is ≥0.5m / s / 100m, a cubic spline interpolation algorithm is used to generate the missing height data. The cubic spline interpolation formula is constructed based on the continuity condition of the second derivative of adjacent effective heights to ensure the smoothness of the interpolated data. Step 5.4: Perform multi-dimensional consistency verification on the generated interpolated data. The verification criteria include: wind speed ∈ [0,60] m / s, wind direction ∈ [0,360]°, and the gradient of vertical velocity is consistent with the adjacent effective height. Step 5.5: If the verification passes, mark the interpolated data as valid and proceed to step 5.7; if the verification fails, start the retry mechanism, readjust the interpolation parameters and recalculate. The number of retries is recorded as count, with an initial value of count=0. Step 5.6: If count < 3, then count = count + 1, return to step 5.3 and re-execute the interpolation calculation; if count ≥ 3, then the interpolation is determined to have failed, the missing height is marked as uninterpolable, and the height layer is skipped. Step 5.7: Perform precision normalization processing on the effective interpolation data, retain two decimal places for wind speed and vertical speed, retain integer places for wind direction, unify the acquisition time to the average time, record metadata such as interpolation algorithm type, verification results, number of retries, etc., and output the effective interpolation data set.

[0011] Furthermore, the specific method of step 6 includes the following steps: Step 6.1: Merge the standardized dataset output from Step 1 with the valid interpolated dataset output from Step 5 to obtain the merged dataset. Step 6.2: Sort the merged data set in ascending order according to the height field to obtain an ordered data set; Step 6.3: Perform duplicate height detection on the ordered dataset. If multiple data records with the same height exist, only retain the valid interpolated data or the original data to ensure that each height value corresponds to only one data record. Step 6.4: Write the deduplicated ordered data set into the business database, overwriting the original data, to ensure that the database stores wind profile data that is structurally complete and highly continuous. Step 6.5: Generate a data processing log to record information such as the data source, data volume, number of missing heights, number of valid interpolations, abnormal situations, and processing results of this processing. At the same time, it will provide feedback on data availability to Step 1 and trigger the processing flow of the next batch of data to form a closed-loop control.

[0012] The beneficial effects of this invention are: (1) This invention achieves unified processing of multi-source wind profile data through standardized data access process and efficient height index construction. Combined with outlier filtering and proportion judgment mechanism, it improves the reliability of data preprocessing and provides a high-quality data foundation for subsequent identification and interpolation. At the same time, the height query efficiency is optimized to O(1) through hash index, which significantly improves the overall processing speed.

[0013] (2) This invention proposes a standard height model generation method based on average step size. By combining the dual proportion judgment of effective data ratio and missing height ratio, it realizes intelligent identification of missing height layers without manual intervention. At the same time, by mapping adjacent effective heights, it provides accurate boundary support for interpolation processing, solving the problems of low identification efficiency and poor accuracy of existing methods.

[0014] (3) This invention introduces the wind speed change rate as the basis for selecting the interpolation algorithm, proposes a method for adaptive switching between linear interpolation and cubic spline interpolation, and combines multi-dimensional consistency verification and retry mechanism to ensure the accuracy and rationality of the interpolation data. At the same time, through innovative formula construction, the quantitative calculation of the interpolation process is realized, which improves the scientificity and repeatability of the method.

[0015] (4) This invention ensures the consistency and continuity of the data structure after interpolation through a complete process of data merging, sorting and deduplication, avoiding the problems of data duplication and disordered order. At the same time, through a closed-loop processing mechanism, it realizes continuous iterative optimization of data, which can flexibly adapt to different types of wind profiler radar data sources and meteorological business needs, and has strong scalability. Attached Figure Description

[0016] Figure 1 This is a flowchart of the overall method of the present invention; Figure 2 This is a flowchart of the missing height layer identification process of the present invention; Figure 3 This is a flowchart of the adaptive interpolation process of the present invention. Detailed Implementation

[0017] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0018] A method for intelligent processing of wind profile data, such as Figure 1 As shown, it includes the following steps: Step 1: Wind profile data access and format standardization processing.

[0019] The core of this step is to achieve unified access and standardization of multi-source wind profile data, providing a consistent data foundation for subsequent processing. The specific implementation method is as follows: (1) Build a multi-source data access interface and adopt a modular design to adapt to the real-time data stream of wind profile radar, historical database and local files respectively, so as to ensure that data from different sources can be successfully accessed.

[0020] (2) Preprocess the raw data and set data validity judgment rules: records with wind speed less than 0 m / s or greater than 60 m / s, wind direction less than 0° or greater than 360°, and absolute value of vertical speed greater than 10 m / s are all judged as invalid data and directly removed; at the same time, data records with signal-to-noise ratio less than 10 dB are removed to ensure data quality.

[0021] (3) Convert the preprocessed multi-source data into a unified structured format and define 5 standard data fields: height h, wind speed v, wind direction d, vertical velocity w, and acquisition time t.

[0022] (4) The standardized dataset is deduplicated, and "height + collection time" is used as the unique identifier. If there are multiple data records with the same identifier, their SNR values ​​are compared and the record with the highest SNR value is retained. Finally, the standardized dataset is output. , where n is the total number of data records.

[0023] Step 2: Height layer structure parsing and index construction.

[0024] This step is used to parse the height structure of the standardized data and build a fast query index to support subsequent identification of missing height layers and interpolation processing. The specific implementation method is as follows: (1) Perform structural parsing on the standardized data set output in step 1. By traversing all data records, extract the height field to form an initial height set.

[0025] (2) Sort the initial height set in ascending order, remove duplicate height values, and obtain an ordered and unique height sequence.

[0026] (3) Construct a height-data mapping index using a hash table structure. The key of the hash table is the height value, and the value is the corresponding complete data record. In this way, when it is necessary to query the data corresponding to a certain height, it can be obtained quickly through the hash table. The query time complexity is optimized from O(m) to O(1), which greatly improves the efficiency of subsequent processing.

[0027] (4) Calculate the average step size of the height sequence for use in the generation of the subsequent standard height model to ensure that the standard height sequence matches the actual observation step size.

[0028] Step 3: Intelligent identification of missing height layers and generation of interpolation set.

[0029] See Figure 2 This step is used to automatically identify missing height layers, generate a set of heights to be interpolated, and ensure data quality through dual proportion judgment. The specific implementation method is as follows: (1) The ordered height sequence obtained in step 2 is filtered for outliers to remove height values ​​that are outside the observation range and abnormal, thus obtaining an effective height sequence.

[0030] (2) Calculate the effective data percentage of the effective height sequence. If the effective data percentage is ≥80%, no data anomaly alarm will be triggered, and the subsequent steps will continue. If the effective data percentage is <80%, an alarm will be triggered, the current identification process will be terminated, and the process will return to step 1 to reconnect the data.

[0031] (3) Generate a standard height layer sequence model, starting with the minimum height of the effective height sequence and ending with the maximum height, and using the average step size as the interval to generate a continuous standard height sequence. ,in This represents the average step size.

[0032] (4) Compare the effective height sequence with the standard height sequence item by item, traverse each height value in the standard height sequence, determine whether it exists in the effective height sequence, identify the height values ​​that do not appear in the effective height sequence, which are the missing height layers, and form a missing height set.

[0033] (5) Calculate the percentage of missing heights. If the percentage of missing heights is ≤30%, the data integrity is deemed to meet the requirements. Match each missing height with two adjacent valid heights, record the mapping relationship, generate a set of heights to be interpolated, and output the set of heights to be interpolated and the mapping relationship. If the percentage of missing heights is >30%, the data integrity is deemed insufficient, and return to step 1 to obtain data again.

[0034] Step 4: Interpolation condition judgment and boundary validity verification.

[0035] This step is used to determine whether each missing height layer meets the interpolation conditions, ensuring the validity of the interpolation boundary data. The specific implementation method is as follows: (1) Take out each missing height from the set of heights to be interpolated output in step 3 in sequence, and obtain its corresponding adjacent valid height according to the mapping relationship.

[0036] (2) Quickly query the complete data record corresponding to the height by using the height-data mapping index built in step 2.

[0037] (3) Verify the consistency of the timestamps of the boundary data, calculate the time difference between the collection of two adjacent valid height data, if the time difference is ≤ 5 minutes, the boundary data is determined to be valid and meets the interpolation conditions, and step 5 is executed; if the time difference is > 5 minutes, the boundary data is determined to be invalid, the missing height is marked as non-interpolable, the height layer is skipped, and the next missing height is processed.

[0038] Step 5: Adaptive interpolation calculation and consistency verification.

[0039] See Figure 3 This step is used to adaptively select the interpolation algorithm based on the data change trend, generate interpolated data, and perform verification to ensure interpolation accuracy. The specific implementation method is as follows: (1) For missing heights that meet the interpolation conditions, calculate the height difference between them and the adjacent valid heights.

[0040] (2) Calculate the rate of change of wind speed at the upper and lower boundaries. The calculation formula is as follows: in, The minimum effective height that is greater than the missing height to be interpolated. The maximum effective height is less than the missing height to be interpolated. , They are respectively , The corresponding wind speed value.

[0041] (3) Adaptive selection of interpolation algorithm: The threshold is set to 0.5 m / s / 100 m. If the wind speed change rate is ≥ 0.5 m / s / 100 m, the wind speed changes drastically, and cubic spline interpolation algorithm is used to generate the missing height data; if the wind speed change rate is < 0.5 m / s / 100 m, the wind speed changes gently, and linear interpolation algorithm is used. The linear interpolation formula is as follows. Taking wind speed as an example, the interpolation formulas for wind direction and vertical speed are similar: in, , These are the missing height and the adjacent valid height, respectively. , The height difference.

[0042] The formula for cubic spline interpolation is as follows: in, , , , The coefficients are cubic spline interpolation coefficients, obtained by solving for the boundary conditions.

[0043] (4) Perform multi-dimensional consistency verification on the generated interpolated data.

[0044] (5) If a certain interpolated data fails the verification, the retry mechanism is started, the interpolation parameters are readjusted and the calculation is performed. The initial value of the number of retry count is 0. If count < 3, the retry continues. If count ≥ 3, the interpolation is determined to be failed and the missing height is marked as uninterpolable.

[0045] (6) Perform precision standardization processing on the effective interpolation data, retain two decimal places for wind speed and vertical speed, retain integer places for wind direction, unify the collection time to the average time, record metadata such as interpolation algorithm type, verification result, number of retries, etc., and finally output the effective interpolation data set.

[0046] Step 6: Data structure reconstruction and database update.

[0047] This step is used to merge the original data and the interpolated data, reconstruct the complete height sequence, and write it into the database, forming a closed-loop process. The specific implementation method is as follows: (1) Merge the standardized data set output in step 1 with the effective interpolated data set output in step 5 to obtain the merged data set.

[0048] (2) Sort the merged data set in ascending order according to the height field to obtain an ordered data set.

[0049] (3) Perform duplicate height detection on the ordered data set, using height as the unique identifier. If there are multiple data records with the same height, only the interpolated data is retained.

[0050] (4) Write the deduplicated ordered data set into the business database and use batch insertion to overwrite the original data to ensure that the database stores wind profile data with complete structure and high continuity, which is convenient for subsequent wind field analysis model calls.

[0051] (5) Generate a data processing log to record information such as the data source, data volume, number of missing heights, number of valid interpolations, abnormal situations and processing results of this processing; at the same time, report the data availability to step 1 to trigger the processing flow of the next batch of data and form a closed-loop control.

Claims

1. A method for intelligent processing of wind profile data, characterized in that, Includes the following steps: (1) Wind profile data access and format standardization. A multi-source data access interface was built to be compatible with multiple data sources and multiple data formats; the raw data was preprocessed to remove invalid data and low signal-to-noise ratio data that did not meet the specifications; the preprocessed data was converted into a unified structured format, and standard fields such as height, wind speed, wind direction, vertical velocity, and acquisition time were defined; the standardized dataset was deduplicated, and the data with the highest signal-to-noise ratio at the same height and time was retained, and a standardized dataset was output. (2) Height layer structure parsing and index construction. Parse the standardized data and extract all height values ​​to form an initial height set; sort and remove duplicates from the initial height set to obtain an ordered and unique height sequence; use a hash table structure to build a mapping index between height and data records to achieve fast height lookup; calculate the average step size of the height sequence for subsequent standard height model construction; (3) Intelligent identification of missing height layers and generation of interpolation set. Filter out abnormal height values ​​in the ordered height sequence to obtain the effective height sequence; calculate the proportion of effective data. If the proportion does not meet the standard, trigger an alarm to terminate the process; if it meets the standard, continue; generate a standard height sequence with the extreme value of the effective height as the starting and ending point and the average step size as the interval; compare the effective height with the standard height to identify the missing height layer and form a missing set; calculate the proportion of missing height. If the proportion exceeds the standard, reacquire data; if it meets the standard, match each missing height with an adjacent effective height to generate the interpolation set and mapping relationship. (4) Interpolation condition judgment and boundary validity verification. Traverse the height to be interpolated and obtain the corresponding adjacent valid heights according to the mapping relationship; query the complete data of adjacent valid heights through the height index; verify the collection time difference of adjacent data. If the time difference exceeds the threshold, it is marked as not interpolable. If the time difference meets the threshold, it is determined that the interpolation condition is met and enter the subsequent interpolation stage. (5) Adaptive interpolation calculation and consistency verification. Calculate the height difference between the missing height and the adjacent valid height; calculate the wind speed change rate based on the wind speed difference and height difference between adjacent heights; adaptively select linear interpolation or cubic spline interpolation algorithm to generate missing height data according to the wind speed change rate threshold; perform multi-dimensional rationality verification on the interpolated data, and if the verification fails, start the retry mechanism; if the retry meets the standard, the interpolated data is retained; if the retry fails, it is marked as uninterpolable; perform precision normalization processing on the valid interpolated data, supplement the metadata, and output the valid interpolation set; (6) Data structure reconstruction and database update. Merge the original standardized data and effective interpolated data to form a complete dataset; sort the merged dataset in ascending order of height, check and remove duplicate height data to ensure that each height corresponds to a single data record; write the normalized complete data into the database in batches to overwrite the original data; generate a data processing log to record the entire process information, and at the same time, provide feedback on the data status to trigger the next batch of data processing to maintain closed-loop operation.

Images