Near space environment data quality improvement method

Abstract

The invention relates to a near space environment data quality improvement method, belongs to the technical field of near space environment detection, and solves the problems of low availability and accuracy of near space environment data in the prior art. Comprising the following steps: acquiring to-be-processed near space environment data files; performing analysis processing on each near space environment data file to obtain first near space environment data; performing abnormal data processing on the first near space environment data to obtain second near space environment data; performing resolution processing on the second near space environment data to obtain processed near space environment data; wherein the first near space environment data comprises acquisition information and standardized near space environment parameter data under each piece of acquisition information; wherein the standardized near space environment parameters comprise longitude, latitude, altitude and atmospheric parameters, and the acquisition information comprises site names, acquisition equipment names and acquisition time.

Description

Technical Field

[0001] The present invention relates to the technical field of near-space environment detection, and in particular to a method for improving the quality of near-space environment data. Background Art

[0002] Near-space refers to the environmental space at an altitude of 20 to 100 km from the earth's surface, corresponding to the middle atmosphere (10 to 100 km), covering the stratosphere, mesosphere and lower thermosphere. Due to the unique physical properties of near-space, it has become an important strategic battleground in the world in recent years.

[0003] Atmospheric parameters such as near-space atmospheric temperature, pressure, density, wind speed, and wind direction have important applications in the design and testing of aircraft. At present, there are ground-based, air-based, and space-based means to detect near-space atmospheric parameters. Different detection means use different detection equipment and models, and the parameters and formats of the original data generated are different. The availability and accuracy of the data are not high, making it difficult to directly use the collected data for convenient and efficient analysis, research, and application. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide a method for improving the quality of near-space environment data, so as to solve the problem of low availability and accuracy of existing near-space environment data.

[0005] An embodiment of the present invention provides a method for improving the quality of near-space environment data, comprising the following steps:

[0006] Obtaining each near-space environment data file to be processed;

[0007] Analyze and process each near-space environment data file to obtain first near-space environment data;

[0008] Performing abnormal data processing on the first near-space environment data to obtain second near-space environment data;

[0009] performing resolution processing on the second near-space environment data to obtain processed near-space environment data;

[0010] Among them, the first near-space environment data includes collection information and standardized near-space environment parameter data under each collection information; wherein the standardized near-space environment parameters include longitude, latitude, altitude and atmospheric parameters, and the collection information includes site name, collection equipment name and collection time.

[0011] Further, the first near-space environment data is obtained through the following steps:

[0012] Based on the file names of the near space environment data files to be processed, acquisition information of the near space environment data files is obtained;

[0013] Based on the suffix names of the near space environment data files to be processed, the format types of the near space environment data files are obtained, and then the near space environment data files are parsed based on the format types to obtain the corresponding near space environment parameter data, and then the units of the near space environment parameter data are converted to obtain the standardized near space environment parameter data;

[0014] The collected information of each near space environment data file to be processed and the corresponding standardized near space environment parameter data are used as the first near space environment data.

[0015] Furthermore, the format type of each near-space environment data file includes a text type and a binary multi-dimensional type;

[0016] If the format type of the near space environment data file is text type, the corresponding data information is obtained by text parsing;

[0017] If the format type of the near-space environment data file is a binary multi-dimensional type, the corresponding data information is obtained through binary parsing.

[0018] Further, the atmospheric parameters in the near-space environmental parameter data before standardization include temperature, density, air pressure, wind speed and wind direction, or include temperature, density, air pressure, meridional wind and latitudinal wind;

[0019] The unit conversion includes unifying the identifiers, parameter names and parameter units of each near-space environmental parameter; wherein, if the atmospheric parameters in the near-space environmental parameters are wind speed and wind direction, they are converted into meridional wind and latitudinal wind.

[0020] Furthermore, performing abnormal data processing on the first near-space environment data includes:

[0021] The missing values ​​of each near space environment parameter data in the first near space environment data are identified, and the identified missing values ​​are marked as invalid data.

[0022] Furthermore, performing abnormal data processing on the first near-space environment data further includes:

[0023] It is determined whether each near space environment parameter data in the first near space environment data exceeds the value range set by the corresponding parameter. If exceeded, the corresponding data is corrected or marked as invalid data.

[0024] Furthermore, performing abnormal data processing on the first near-space environment data further includes:

[0025] It is determined whether each near space environment parameter data in the first near space environment data falls outside a set interval of a normal distribution of the corresponding parameter data; if so, the corresponding near space environment parameter data is marked as invalid data.

[0026] Furthermore, performing abnormal data processing on the first near-space environment data further includes:

[0027] Determine whether each air pressure data in the first near-space environment data satisfies the following formula:

[0028] |log(P)-log(PL)|≤1

[0029] Where, PL = 100000 × e -z / H ;

[0030] Wherein, P represents the air pressure data in the first adjacent space environment data, z represents the altitude data corresponding to the air pressure data in the first adjacent space environment data, and H represents a constant;

[0031] If not, the corresponding air pressure data and other parameters at the corresponding altitude will be marked as invalid data.

[0032] Further, performing resolution processing on the second near-space environment data includes:

[0033] Each adjacent space environment parameter data in the second adjacent space environment data is interpolated in the vertical direction based on a set height interval to obtain second adjacent space environment data with the same height resolution.

[0034] Furthermore, performing resolution processing on the second near-space environment data further includes:

[0035] For each near space environment parameter data in the second near space environment data, values ​​are taken at the collection time based on a set time interval to obtain second near space environment data with the same time resolution.

[0036] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0037] The present invention provides a method for improving the quality of near space environment data. The method obtains each near space environment data file to be processed, performs parsing processing on each near space environment data file to obtain first near space environment data, performs abnormal data processing on the first near space environment data to obtain second near space environment data, and then performs resolution processing on the second near space environment data to obtain processed near space environment data. The near space environment parameter data is standardized and unified, thereby improving the availability and accuracy of the near space environment data, improving the data quality, and laying a good foundation for subsequent data analysis and application.

[0038] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0040] Figure 1 A schematic flow chart of a method for improving the quality of near-space environment data provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0042] A specific embodiment of the present invention discloses a method for improving the quality of near-space environment data, such as Figure 1 As shown, the following steps are included:

[0043] S1. Obtain each near-space environment data file to be processed.

[0044] Specifically, the near-space environment data file is a data file generated by different detection devices using different space-based, air-based and ground-based detection methods.

[0045] Specifically, the format types of each near space environment data file include text type and binary multidimensional type; wherein, the text type near space environment data file stores the spatial dimension, time dimension, parameter type and parameter value of the near space environment data in the form of plain text, and the binary multidimensional type near space environment data file is the abbreviation of the internationally used NetCDF format, which stores the spatial dimension, time dimension, parameter type and parameter value of the near space environment data in binary form.

[0046] Exemplarily, the text type is in TXT and DAT format, and the binary multidimensional type is in NC format.

[0047] S2. Analyze and process each near-space environment data file to obtain first near-space environment data.

[0048] Specifically, the first near-space environment data includes collection information and standardized near-space environment parameter data under each collection information; wherein the standardized near-space environment parameters include longitude, latitude, altitude and atmospheric parameters, and the collection information includes site name, collection equipment name and collection time.

[0049] During implementation, in step S2, the first near-space environment data is obtained by the following steps:

[0050] S21. Based on the file name of each near space environment data file to be processed, acquisition information of each near space environment data file is obtained.

[0051] Specifically, the file names of the near-space environment data files provided by different detectors have fixed naming rules, and the site name, collection device name and collection time of the file can be parsed according to the naming rules.

[0052] S22. Based on the suffix names of the near space environment data files to be processed, the format types of the near space environment data files are obtained, and then based on the format types, the near space environment data files are parsed to obtain the corresponding near space environment parameter data, and then the units of the near space environment parameter data are converted to obtain standardized near space environment parameter data.

[0053] Specifically, if the format type of the near space environment data file is a text type, the corresponding data information is obtained by parsing the text;

[0054] If the format type of the near-space environment data file is a binary multidimensional model, the corresponding data information is obtained through binary parsing.

[0055] More specifically, when performing text parsing, a near space environment data file whose format is text type reads data line by line to obtain data of each near space environment parameter; when performing binary parsing, a near space environment data file whose format is binary multidimensional type uses a dedicated file reading library to read data to obtain data of each near space environment parameter.

[0056] It should be noted that each format type of near space environment data file has a corresponding data format description, which gives the correspondence between each near space environment parameter and the data in the file, and can be read according to the format description; among them, the near space environment parameters in the text type near space environment data file are separated by spaces or commas, and the binary multidimensional type contains variables and variable descriptions. The near space environment parameters are identified according to the description of the variables and the data values ​​in the variables are read as the data of the near space environment parameters.

[0057] Specifically, the atmospheric parameters in the near space environmental parameter data before standardization include temperature, density, air pressure, wind speed and wind direction, or include temperature, density, air pressure, meridional wind and latitudinal wind; the unit conversion includes unifying the identifiers, parameter names and parameter units of each near space environmental parameter; wherein, if the atmospheric parameters in the near space environmental parameters are wind speed and wind direction, they are converted into meridional wind and latitudinal wind; the unit conversion is specifically shown in Table 1.

[0058] Table 1 Unit conversion table

[0059] Parameter name meaning unit longitude longitude Degree(°) latitude latitude Degree(°) altitude Altitude Kilometer (km) T temperature Calvin (K) D density <![CDATA[Kilogram per cubic meter (kg / m 3 )]]> P Air pressure Pa V Meridional wind Meters per second (m / s) U Zonal wind Meters per second (m / s)

[0060] S23: taking the collection information of each near space environment data file to be processed and the corresponding standardized near space environment parameter data as the first near space environment data.

[0061] It can be understood that by parsing and processing each near-space environment data file, the near-space environment parameter data files collected by different types of detection equipment can be uniformly parsed to obtain longitude, latitude, collection time, altitude and atmospheric parameter data, and the form and unit of the parameters can be unified to facilitate subsequent data reading and data storage.

[0062] S3. Perform abnormal data processing on the first near-space environment data to obtain second near-space environment data.

[0063] During implementation, performing abnormal data processing on the first near-space environment data includes:

[0064] The missing values ​​of each near space environment parameter data in the first near space environment data are identified, and the identified missing values ​​are marked as invalid data.

[0065] Specifically, according to the representation methods of missing values ​​in the near space environment data files obtained by different detection devices, the missing values ​​of each near space environment parameter data are identified. Exemplarily, the representation methods of missing values ​​include NaN, -999, and null.

[0066] Preferably, performing abnormal data processing on the first near-space environment data further includes:

[0067] It is determined whether each near space environment parameter data in the first near space environment data exceeds the value range set by the corresponding parameter. If exceeded, the corresponding data is corrected or marked as invalid data.

[0068] Specifically, if the acquisition equipment name in the acquisition information corresponding to the near-space environmental parameter data is ground-based detection equipment and air-based detection equipment, then

[0069] The data range of temperature parameter setting in near-space environment parameters is 100~300K, and the data range of density parameter setting is 10e-6~10e-0.5kg / m3. Data not within the corresponding range will be marked as invalid data.

[0070] If the acquisition device name in the acquisition information corresponding to the near-space environment parameter data is a space-based detection device, then

[0071] The data range of temperature parameter setting in near-space environment parameters is 80~1000K, and the data range of pressure parameter setting is 0~100000Pa. Data not within the corresponding range is marked as invalid data.

[0072] The range of latitude setting in the near-space environmental parameters is -90° to 90°, and the data outside this range is marked as invalid data; further, the data of longitude, altitude, temperature, density, air pressure, meridional wind and zonal wind parameters collected corresponding to the invalid latitude data are all marked as invalid data;

[0073] The range of longitude setting in the near-space environment parameters is 0 to 360°. For data not within this range, correction is made by adding or subtracting 360° according to the periodicity of longitude to make the longitude data within a reasonable range;

[0074] The collection time corresponding to the near-space environmental parameter data is set to 0 to 24 hours. For data not within this range, it is corrected by adding or subtracting 24 hours to ensure that the collection time is within a reasonable range.

[0075] It can be understood that, through the above processing, the instability factors of the detection equipment operation or the detection results that do not conform to objective laws caused by interference from other external factors can be eliminated, thereby improving the accuracy of the data.

[0076] Preferably, performing abnormal data processing on the first near-space environment data further includes:

[0077] It is determined whether each near space environment parameter data in the first near space environment data falls outside a set interval of a normal distribution of the corresponding parameter data; if so, the corresponding near space environment parameter data is marked as invalid data.

[0078] Specifically, based on the data of each near-space environmental parameter at all acquisition times, the normal distribution of each near-space environmental parameter is obtained, and the data whose residuals fall outside the ±3σ interval are marked as invalid data; where σ represents the standard deviation of the normal distribution.

[0079] It can be understood that according to Gaussian error theory, when the collected data obeys the normal distribution, the probability that the residual falls between ±3σ is 99.7%, and the probability that it falls outside the interval is 0.3%. Therefore, marking the data outside the interval as invalid data can improve the accuracy of the data.

[0080] Preferably, performing abnormal data processing on the first near-space environment data further includes:

[0081] Determine whether each air pressure data in the first near-space environment data satisfies the following formula:

[0082] |log(P)-log(PL)|≤1

[0083] Where, PL = 100000 × e -z / H ;

[0084] Wherein, P represents the air pressure data in the first adjacent space environment data, z represents the altitude data corresponding to the air pressure data in the first adjacent space environment data, and H represents a constant;

[0085] If not, the corresponding air pressure data and other parameters at the corresponding altitude will be marked as invalid data.

[0086] Specifically, the constant H=7.

[0087] Specifically, invalid data is marked as NaN.

[0088] S4. Perform resolution processing on the second near-space environment data to obtain processed near-space environment data.

[0089] During implementation, performing resolution processing on the second near-space environment data includes:

[0090] Each adjacent space environment parameter data in the second adjacent space environment data is interpolated in the vertical direction based on a set height interval to obtain second adjacent space environment data with the same height resolution.

[0091] Specifically, the altitude interval is set at 1 km.

[0092] Specifically, the height resolution is the data interval in the height direction; the data of each near-space environmental parameter are interpolated to an integer kilometer height in the vertical direction for each detection profile using the vertical interpolation method; wherein the detection profile is equivalent to a cross-section in the vertical direction of the ground, and the detection equipment is equivalent to collecting and detecting the data of each near-space environmental parameter in the vertical direction.

[0093] More specifically, linear interpolation is used for temperature parameters, and logarithmic interpolation is used for pressure parameters and density parameters.

[0094] Preferably, performing resolution processing on the second near-space environment data further includes:

[0095] For each near space environment parameter data in the second near space environment data, values ​​are taken at the collection time based on a set time interval to obtain second near space environment data with the same time resolution.

[0096] Specifically, if the original time resolution is less than 1 hour, the time interval is set to 1 hour; if the original time resolution is greater than 1 hour, the time interval is set to 1 day;

[0097] Specifically, the value is obtained by averaging, and the average value is taken for each near-space environmental parameter data within the set time interval, and the time resolution of each near-space environmental parameter is unified; for example, the original time resolution is to record one data per minute, and after being unified to one hour, one value is recorded per hour, wherein the average value of all the data within the original one hour is taken as the data for this hour.

[0098] It can be understood that by performing resolution processing on the second near-space environment data, the problem of inconsistent height resolution and time resolution caused by data generated by different detection means and different detection equipment can be eliminated, which is beneficial to subsequent statistical analysis and comparison of the data.

[0099] Compared with the prior art, this embodiment provides a method for improving the quality of near space environment data, by obtaining each near space environment data file to be processed, parsing each near space environment data file to obtain first near space environment data, then performing abnormal data processing on the first near space environment data to obtain second near space environment data, and then performing resolution processing on the second near space environment data to obtain processed near space environment data, standardizing and unifying the near space environment parameter data, improving the availability and accuracy of the near space environment data, improving the data quality, and laying a good foundation for subsequent data analysis and application.

[0100] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0101] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for improving the quality of near-space environment data, characterized in that: The following steps are involved: Obtaining each near-space environment data file to be processed; Analyze and process each near-space environment data file to obtain first near-space environment data; Performing abnormal data processing on the first near-space environment data to obtain second near-space environment data; performing resolution processing on the second near-space environment data to obtain processed near-space environment data; Among them, the first near-space environment data includes collection information and standardized near-space environment parameter data under each collection information; wherein the standardized near-space environment parameters include longitude, latitude, altitude and atmospheric parameters, and the collection information includes site name, collection equipment name and collection time.

2. The method for improving the quality of near space environment data according to claim 1, characterized in that: The first near-space environment data is obtained by the following steps: Based on the file names of the near space environment data files to be processed, acquisition information of the near space environment data files is obtained; Based on the suffix names of the near space environment data files to be processed, the format types of the near space environment data files are obtained, and then the near space environment data files are parsed based on the format types to obtain the corresponding near space environment parameter data, and then the units of the near space environment parameter data are converted to obtain the standardized near space environment parameter data; The collected information of each near space environment data file to be processed and the corresponding standardized near space environment parameter data are used as the first near space environment data.

3. The method for improving the quality of near space environment data according to claim 2, characterized in that: The format types of each near space environment data file include text type and binary multi-dimensional type; If the format type of the near space environment data file is text type, the corresponding data information is obtained by text parsing; If the format type of the near-space environment data file is a binary multi-dimensional type, the corresponding data information is obtained through binary parsing.

4. The method for improving the quality of near space environment data according to claim 2, characterized in that: The atmospheric parameters in the near-space environmental parameter data before standardization include temperature, density, air pressure, wind speed and wind direction, or include temperature, density, air pressure, meridional wind and latitudinal wind; The unit conversion includes unifying the identifiers, parameter names and parameter units of each near-space environmental parameter; wherein, if the atmospheric parameters in the near-space environmental parameters are wind speed and wind direction, they are converted into meridional wind and latitudinal wind.

5. The method for improving the quality of near space environment data according to claim 1, characterized in that: The abnormal data processing of the first near-space environment data includes: The missing values ​​of each near space environment parameter data in the first near space environment data are identified, and the identified missing values ​​are marked as invalid data.

6. The method for improving the quality of near space environment data according to claim 5, characterized in that: The performing abnormal data processing on the first near-space environment data further includes: It is determined whether each near space environment parameter data in the first near space environment data exceeds the value range set by the corresponding parameter. If exceeded, the corresponding data is corrected or marked as invalid data.

7. The method for improving the quality of near space environment data according to claim 5, characterized in that: The performing abnormal data processing on the first near-space environment data further includes: It is determined whether each near space environment parameter data in the first near space environment data falls outside a set interval of a normal distribution of the corresponding parameter data; if so, the corresponding near space environment parameter data is marked as invalid data.

8. The method for improving the quality of near space environment data according to claim 4, characterized in that: The performing abnormal data processing on the first near-space environment data further includes: Determine whether each air pressure data in the first near-space environment data satisfies the following formula: log(P)-log(PL)|≤1 Where, PL = 100000 × e -zH ; Wherein, P represents the air pressure data in the first adjacent space environment data, z represents the altitude data corresponding to the air pressure data in the first adjacent space environment data, and H represents a constant; If not satisfied, the corresponding air pressure data and other parameters at the corresponding altitude will be marked as invalid data.

9. The method for improving the quality of near space environment data according to claim 1, characterized in that: Performing resolution processing on the second near-space environment data includes: Each adjacent space environment parameter data in the second adjacent space environment data is interpolated in the vertical direction based on a set height interval to obtain second adjacent space environment data with the same height resolution.

10. The method for improving the quality of near space environment data according to claim 1, characterized in that: Performing resolution processing on the second near-space environment data further includes: For each near space environment parameter data in the second near space environment data, values ​​are taken at the collection time based on a set time interval to obtain second near space environment data with the same time resolution.

Images