Method for constructing ultraviolet edge cutting high-precision correction based on machine learning
By building a fine atmospheric background library and using machine learning models, the problem of insufficient accuracy in ultraviolet edge detection technology is solved, and high-precision and rapid spectral correction effect is achieved, which is suitable for scientific research and application needs.
Patent Information
- Application Number
- CN202510380537.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-08-01
AI Technical Summary
The existing ultraviolet edge detection technology has problems such as systematic errors, interference with complex atmospheric physical characteristics and insufficient model assumptions in terms of accuracy correction, resulting in the inability to meet the requirements of scientific research and application.
Using machine learning-based methods, a fine atmospheric background library is built, combined with the atmospheric radiation transmission model SCIATRAN and the XGBoost algorithm of the gradient enhancement decision tree, a rapid spectral calculation model is established, and high-precision correction of ultraviolet edge spectroscopy is achieved through machine learning model training and verification.
It realizes high-precision ultraviolet edge spectral correction under clear sky conditions, which is fast, universal and not affected by the operating system, and improves the calculation speed and correction effect.
Smart Images
Figure CN120412784A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical fields of machine learning and satellite remote sensing, and particularly to a method for constructing high-precision correction of ultraviolet limb tangent based on machine learning. Background Art
[0002] In the field of detecting atmospheric components using ultraviolet spectra, when the limb sounding technique is used to obtain the atmospheric profile, it has the advantages of high vertical resolution, high sensitivity, strong coverage ability, and being independent of ground control. However, current ultraviolet limb sounding techniques have many limitations in precision correction. On the one hand, there are systematic errors in the detector itself, such as non-uniform response of the detector, dark current noise, etc. These factors will cause deviations in the detection data and seriously affect the accurate inversion of atmospheric parameters. On the other hand, during the observation process, the complex physical properties of the atmosphere, such as refraction and scattering of the atmosphere, as well as the small attitude changes of the observation platform, will interfere with the observation data, making it difficult for existing correction methods to meet the requirements of high precision.
[0003] Existing correction techniques are mostly based on simple model assumptions and cannot fully consider the complex influencing factors in actual observations, resulting in the corrected precision still being unable to meet the growing scientific research and application needs. For example, traditional correction methods do not comprehensively consider the absorption and scattering characteristics of different components in the atmosphere, and it is easy to introduce additional errors during the correction process. Therefore, it is of great practical significance and urgent need to develop an ultraviolet limb tangent high-precision correction method that can overcome the above problems. Summary of the Invention
[0004] Aiming at the disadvantages of low correction accuracy, complex calculation, poor universality, and inapplicability to operations of the ultraviolet limb tangent high-precision correction, the present invention provides a simpler, faster, and more accurate calculation method based on a machine learning compensation model.
[0005] To achieve the above object, the present invention provides a method for constructing high-precision correction of ultraviolet limb tangent based on machine learning, including the following steps:
[0006] Construct a fine atmospheric background library to provide a set of fine grid atmospheric background libraries suitable for the atmospheric radiative transfer model;
[0007] Input the atmospheric background library, geometric parameters corresponding to the satellite, time, and longitude and latitude information into the atmospheric radiative transfer model to obtain a fine grid ultraviolet limb spectral sequence library;
[0008] Use the atmospheric background library and simulated ultraviolet limb spectra as the training input of the machine learning model, and use the tangent height as the output label for training to obtain a non-linear relationship, called the spectral calculation fast model, i.e., Spectral calculation fastmodel-SCFM;
[0009] Obtain the observation data sequence of ultraviolet limb sounding, including ultraviolet limb spectral data and corresponding geometric parameters;
[0010] Extract real-time atmospheric parameters according to the time, longitude, and latitude of the ultraviolet limb spectrum;
[0011] Verify and evaluate the machine learning model.
[0012] The observation geometric information includes: solar zenith angle, observation angle, azimuth angle, and field of view angle.
[0013] The fine grid uses date and longitude-latitude as the reference scale, and the atmospheric background library is generated by fusing multi-source satellite profile products and the fifth-generation global climate reanalysis dataset products released by the European Centre for Medium-Range Weather Forecasts. The multi-source satellite products include Microwave Limb Sounder (MLS) and Atmospheric Chemistry Experiment-Fourier Transform Spectrometer (ACE-FTS).
[0014] The specific steps for model training based on the atmospheric radiative transfer model SCIATRAN are as follows:
[0015] Use longitude, latitude, date, solar zenith angle, observation angle, azimuth angle, surface reflectivity, field of view angle, instrument slit function, atmospheric background library, tangent height sequence, and band range as the input data of the forward radiative transfer model;
[0016] According to the input data, use the atmospheric radiative transfer model SCIATRAN and the programming tool matlab. In matlab, loop to call the atmospheric radiative transfer model SCIATRAN and modify the parameter files of the atmospheric radiative transfer model SCIATRAN, including control.inp, control_geom.inp, control_ac.inp, control_la.inp, xsection.inp;
[0017] Output the solar radiation intensity data under different atmospheric background conditions, different surface reflectivities, different solar zenith angles, different regions, and different tangent heights.
[0018] The machine learning model uses the efficient machine learning algorithm XGBoost based on gradient boosting decision tree as the model framework.
[0019] The specific steps for using longitude, latitude, date, solar zenith angle, observation angle, azimuth angle, surface reflectivity, field of view angle, instrument slit function, atmospheric background library, tangent height sequence, and band range as the input data of the forward radiative transfer model are as follows:
[0020] Step 401: Extract the reanalysis ERA5 product and the secondary product of AURAMLS according to the longitude, latitude and the date of the detection data.
[0021] Step 402: Extract the MODIS surface reflectance product according to the longitude, latitude and the date of the detection data.
[0022] Step 403: Extract the satellite payload ultraviolet limb sounding data according to the longitude, latitude and the date of the detection data.
[0023] Step 404: According to Steps 401 to 403, obtain the solar zenith angle, azimuth angle, field of view angle, observation angle, tangent height sequence, surface reflectance, atmospheric absorption profile, and the slit function input to the satellite payload as the input data of the machine learning model, and calculate the solar spectrum at the satellite entrance pupil according to the machine learning model.
[0024] The limb ultraviolet spectral data includes the following data: date, initial tangent height sequence, ultraviolet limb spectral sequence corresponding to different tangent heights, longitude and latitude.
[0025] The real-time atmospheric parameters are the atmospheric temperature, humidity, pressure, and various atmospheric component profile products of MLS, ACE-FTS or ERA5 corresponding to the corresponding time and longitude and latitude.
[0026] According to one aspect of the present invention, a storage medium is provided, in which instructions are stored. When a computer reads the instructions, the computer is made to execute the method for constructing ultraviolet limb tangent height precision correction based on machine learning described in any one of the above.
[0027] According to another aspect of the present invention, an electronic device is provided, including a processor and the above storage medium, and the processor executes the instructions in the storage medium.
[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0029] The present invention adopts the method of machine learning, can calculate the solar radiation under limb observation under clear sky conditions, has the advantages of fast calculation speed, strong universality, not being affected by the operating system, and high portability. Description of the Drawings
[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.
[0031] Unless otherwise defined, the technical terms or scientific terms used herein shall have the ordinary meanings as understood by those of ordinary skill in the art to which the present invention pertains.
[0032] Figure 1 Schematic diagram of the method process for a preferred embodiment of the present invention;
[0033] Figure 2 Schematic diagram of generating training samples using SCIATRAN for a preferred embodiment of the present invention;
[0034] Figure 3 Schematic diagram of model training using machine learning for a preferred embodiment of the present invention;
[0035] Figure 4 Schematic diagram of calculating ultraviolet limb spectra for a preferred embodiment of the present invention. Detailed implementation manners
[0036] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.
[0037] As Figures 1-4 shown, an embodiment of the present invention discloses a method for constructing high-precision correction of ultraviolet limb cut based on machine learning, including the following steps:
[0038] Step 1: Construct a fine atmospheric background library to provide a set of fine-grid atmospheric background information suitable for the atmospheric radiation transfer model.
[0039] Step 1: Construct a fine atmospheric background library to provide a set of fine-grid atmospheric background information suitable for the atmospheric radiation transfer model.
[0040] Preferably, the globe is divided into 360×180 grids according to 1°×1° of longitude and latitude.
[0041] Further, download the atmospheric remote sensing MLS, ACE-FTS or ERA5 secondary products in the past year and generate them according to the following steps:
[0042] Eliminate invalid and low-precision products to ensure that all products have high precision.
[0043] Statistical calculations are performed according to month, longitude, and latitude grids to count the number of products falling within the same grid. For all atmospheric products within the range of 5 - 100 km in altitude, linear interpolation is performed at 1 km altitude intervals, and the mean value is calculated, resulting in a total of 96 layers. Each grid contains the atmospheric layer serial number K, altitude Z, pressure P, temperature T, water vapor H2O, carbon dioxide CO2, nitrous oxide N2O, carbon monoxide CO, methane CH4, oxygen O2, nitric oxide NO, ammonia NH3, nitric acid HNO3, hydroxyl radical OH, hydrogen fluoride HF, hydrogen chloride HCl, hydrogen bromide HBr, and hydrogen iodide HI.
[0044] Statistical calculations are performed on the data for each grid. If there is no atmospheric mean product in a grid, spatial interpolation methods can be used to generate it.
[0045] Furthermore, the Kriging spatial interpolation method can be divided into the following steps:
[0046] Data preparation: Using the already generated grid products, the standard distributions of parameters such as atmospheric temperature T, pressure p, and atmospheric profile product ρ with respect to altitude H are given. These data form the basic data for Kriging interpolation.
[0047] Variogram calculation and fitting: Calculate the variogram values: For each pair of data points (ρ i , H i ), calculate the variogram value based on their altitude difference and attribute value difference. For example, for the atmospheric attribute values H i and H j at two altitudes ρ i and ρ j , the variogram value
[0048] Fitting the theoretical model: Select a suitable theoretical variogram model - the spherical model to fit the calculated variogram values and determine the model parameters. These parameters will be used for subsequent calculation of the weight coefficients.
[0049] Determination of weight coefficients: For the altitude point to be interpolated, based on its spatial distance (here the altitude difference) from the known data points, use the fitted variogram to calculate the weight coefficient w i for each known data point. Data points closer to the point to be interpolated and with strong spatial correlation have larger weights.
[0050] Interpolation calculation: Multiply the atmospheric density of each known data point by the corresponding weight coefficient and then sum them to obtain the estimated value of the atmospheric attribute at the altitude point to be interpolated. For example, if there are n known data points, the estimated value of the atmospheric density point to be interpolated
[0051] Step 2: Input the atmospheric background library, geometric parameters corresponding to the satellite, time, longitude and latitude information into the atmospheric radiative transfer model to obtain a library of ultraviolet limb spectral sequences with a fine grid.
[0052] Preferably, the limb ultraviolet spectral range is 300nm - 500nm, and the spectral interval is 0.4nm;
[0053] Preferably, the software used to simulate the atmospheric radiative transfer model is SCIATRAN version 4.6, and the atmospheric model is a self-built fine-grid atmospheric background library;
[0054] Preferably, according to the atmospheric radiative transfer model, the steps of simulating and constructing the limb ultraviolet spectral library specifically include:
[0055] Use matlab software to compile a program that modifies each of the control.inp, control_geom.inp, control_ac.inp, control_la.inp, xsection.inp files of the atmospheric radiative transfer model SCIATRAN. This program can repeatedly and automatically modify longitude and latitude, date, solar zenith angle, observation angle, azimuth angle, surface reflectivity, field of view angle, atmospheric background library, and tangent height sequence;
[0056] Furthermore, input the modified longitude and latitude, date, solar zenith angle, observation angle, azimuth angle, surface reflectivity, field of view angle, atmospheric background library, and tangent height sequence into the SCIATRAN control.inp, control_geom.inp, control_ac.inp, control_la.inp, xsection.inp configuration files to obtain the simulated spectral values in the limb ultraviolet band at different tangent heights. The number of simulated spectra is the number of tangent height sequences * 402;
[0057] Further, for the longitude and latitude of the limb ultraviolet spectral dataset, the latitude is at intervals of 5°, with values of 5°N, 15°N, 25°N, 35°N, 45°N, 55°N, 65°N, 75°N, 85°N, 5°S, 15°S, 25°S, 35°S, 45°S, 55°S, 65°S, 75°S, 85°S; the longitude is at intervals of 30°, with values of: 30°E, 60°E, 120°E, 150°E, 180°E, 30°W, 60°W, 120°W, 150°W, 180°W. The solar zenith angle is at intervals of 5°, which are 0°, 5°, 10°, 15°, 20°, 25°, 30°, 35°, 40°, 45°, 50°, 55°, 60°, 70°, 80° and 90° respectively; the observation angle is calculated and set according to the position of the longitude and latitude, with a setting interval of 2°; the field of view angle is set according to the instrument, and the surface reflectance values are 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 and 1.0; the range of the tangent height sequence is 5 - 65 km, with a setting interval of 100 m.
[0058] Step 3: Establish a machine learning model for the atmospheric background library, simulated ultraviolet limb spectrum and tangent height, and obtain a non - linear relationship, called the Spectral calculation fast model (SCFM).
[0059] Preferably, the inputs of the spectral calculation fast model include: band, longitude and latitude, date, solar zenith angle, observation angle, azimuth angle, field of view angle, atmospheric background library, tangent height sequence; the output label of the model is the ultraviolet limb spectrum.
[0060] Further, the platform for deep learning using the limb ultraviolet spectral simulation dataset is the Python programming language platform and the Pytorch framework.
[0061] Further, the spectral calculation fast model is XGBoost based on gradient boosting. The steps are as follows: The GridSearch method is used for model training and evaluation, and finally the parameter combination with the best performance on the validation set is selected. The parameter settings are'max_depth': [3, 5, 7], 'learning_rate': [0.1, 0.01, 0.005], 'n_estimators': [50, 100, 150, 200, 250, 300], and a total of 54 parameter combinations are searched.
[0062] Further, feature selection and feature transformation are performed. By setting the threshold to 0.05, features that have a significant impact on the target variable are selected, and irrelevant or redundant features are removed to reduce the computational amount and avoid overfitting; data normalization is used for feature transformation. The numerical features are scaled to a specified range, commonly the interval [0,1]. The formula is used for transformation, where x max and x min are the maximum and minimum values of each feature respectively.
[0063] Step Four: Obtain the observation data sequence s of ultraviolet limb sounding, including the ultraviolet limb spectral sequence and the corresponding geometric parameters. The ultraviolet limb spectral sequence and its corresponding geometric parameters include the following data: longitude and latitude, date, solar zenith angle, observation angle, azimuth angle, field of view angle, tangent height sequence, and ultraviolet spectral sequences at different tangent heights.
[0064] Step Five: Extract the real-time atmospheric parameters according to the time, longitude and latitude of the ultraviolet limb spectrum in Step Four.
[0065] The real-time atmospheric parameters are the atmospheric temperature, humidity, pressure, and various atmospheric component profile products of MLS, ACE-FTS, or ERA5 at the corresponding time and longitude and latitude.
[0066] Step Six: Verify and evaluate the machine learning model in Step Three.
[0067] When verifying the machine learning model, the input of the model is the ultraviolet limb spectrum, time, longitude and latitude, temperature, humidity, pressure, and various atmospheric component profile products extracted in Step Four and Step Five.
[0068] The model for evaluating the machine learning is the Shapley model. The evaluation steps are as follows:
[0069] Determine the dataset of the feature matrix X (with multiple features x1, x2, x3,...., x n ) and the target variable y. The feature matrix X of this model includes (date, longitude, latitude, band, ultraviolet spectrum, atmospheric temperature, humidity, pressure, and 14 atmospheric profiles), and the target variable y is the tangent height.
[0070] Calculate the marginal contribution of the feature combination: Consider all possible combinations of feature subsets. For the matrix X with 26 feature vectors, there are a total of 2^26 feature subset combinations, and calculate the marginal contribution of each feature subset combination.
[0071] Calculate the Shapley value: The Shapley value is obtained by weighted averaging the marginal contributions of each feature under all possible orders of feature subsets.
[0072] Example 2:
[0073] The computer-readable storage medium of this embodiment stores a computer program, and when the program is executed by a processor, it implements the steps in the method for constructing high-precision ultraviolet limb cut correction based on machine learning in Embodiment 1.
[0074] The computer-readable storage medium of this embodiment can be an internal storage unit of the terminal, such as the hard disk or memory of the terminal; the computer-readable storage medium of this embodiment can also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card, a secure digital card, a flash card, etc. equipped on the terminal; further, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the terminal.
[0075] The computer-readable storage medium of this embodiment is used to store the computer program and other programs and data required by the terminal, and the computer-readable storage medium can also be used to temporarily store the data that has been output or will be output.
[0076] Embodiment 3:
[0077] The computer device of this embodiment includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the method for constructing high-precision ultraviolet limb cut correction based on machine learning in Embodiment 1.
[0078] In this embodiment, the processor can be a central processing unit, or can also be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc.; the memory can include a read-only memory and a random access memory, and provides instructions and data to the processor. A part of the memory can also include a non-volatile random access memory. For example, the memory can also store information about the device type.
[0079] Those skilled in the art should understand that the content disclosed in the embodiments can be provided as a method, a system, or a computer program product. Therefore, this solution can adopt the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, this solution can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program codes.
[0080] This solution is described with reference to the flowcharts and / or block diagrams of methods and computer program products according to embodiments of this solution. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions; these computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or in multiple blocks.
[0081] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or in multiple blocks.
[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or in multiple blocks.
[0083] Those of ordinary skill in the art can understand that all or part of the processes in the above-described embodiment methods can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-described method embodiments. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.
[0084] The examples described in the present invention are only descriptions of the preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various deformations and improvements made by those skilled in the art to the technical solutions of the present invention should all fall within the protection scope of the present invention.
Claims
1. A method for constructing high-precision correction of ultraviolet limb tangent based on machine learning, characterized in that, It includes the following steps: Construct a fine atmospheric background library, providing a set of fine-grid atmospheric background libraries suitable for atmospheric radiation transfer models; Input the atmospheric background library, the geometric parameters corresponding to the satellite, time, longitude, and latitude information into the atmospheric radiation transfer model to obtain a fine-grid ultraviolet limb spectral sequence library; Use the atmospheric background library and the simulated ultraviolet limb spectrum as the training input of the machine learning model, and use the tangent height as the output label for training to obtain a non-linear relationship, called the spectral calculation fast model, i.e., Spectral calculation fast model - SCFM; Obtain the observation data sequence of ultraviolet limb detection, including ultraviolet limb spectral data and corresponding geometric parameters; Extract the real-time atmospheric parameters according to the time, longitude, and latitude of the ultraviolet limb spectrum; Verify and evaluate the machine learning model.
2. The method according to claim 1, wherein The observation geometric information includes: solar zenith angle, observation angle, azimuth angle, and field of view angle.
3. The method according to claim 2, wherein The fine grid uses date and longitude and latitude as the reference scale, and the atmospheric background library is generated by fusing multi-source satellite profile products and the fifth-generation global climate reanalysis dataset products released by the European Centre for Medium-Range Weather Forecasts.
4. The method according to claim 2, wherein The specific steps for model training based on the atmospheric radiation transfer model SCIATRAN are as follows: Use longitude, latitude, date, solar zenith angle, observation angle, azimuth angle, surface reflectivity, field of view angle, instrument slit function, atmospheric background library, tangent height sequence, and band range as the input data of the forward radiation transfer model; According to the input data, use the atmospheric radiation transfer model SCIATRAN and the programming tool matlab, and call the atmospheric radiation transfer model SCIATRAN in a matlab loop to modify the parameter files of the atmospheric radiation transfer model SCIATRAN, including control.inp, control_geom.inp, control_ac.inp, control_la.inp, xsection.inp; Output the solar radiation intensity data under different atmospheric background conditions, different surface reflectivities, different solar zenith angles, different regions, and different tangent heights.
5. The method according to claim 1, characterized in that The machine learning model uses the XGBoost model.
6. The method according to claim 5, wherein The specific steps for using longitude, latitude, date, solar zenith angle, observation angle, azimuth angle, surface reflectivity, field of view angle, instrument slit function, atmospheric background library, tangent height sequence, and band range as the input data of the forward radiation transfer model are as follows: Step 401, extract the reanalysis ERA5 product and the secondary product of AURA MLS according to the longitude, latitude, and date of the detection data; Step 402, extract the MODIS surface reflectivity product according to the longitude, latitude, and date of the detection data; Step 403, extract the satellite payload ultraviolet limb detection data according to the longitude, latitude, and date of the detection data; Step 404: According to Steps 401 to 403, obtain the solar zenith angle, azimuth angle, field of view angle, observation angle, tangent height sequence, surface reflectivity, atmospheric absorption profile, and the slit function input to the satellite payload as the input data of the machine learning model, and calculate the solar spectrum at the satellite entrance pupil according to the machine learning model.
7. The method according to claim 1, wherein The limb ultraviolet spectral data includes the following data: date, initial tangent height sequence, ultraviolet limb spectral sequence corresponding to different tangent heights, and longitude and latitude.
8. The method according to claim 1, characterized in that The real-time atmospheric parameters are the atmospheric temperature, humidity, pressure, and various atmospheric component profile products of MLS, ACE-FTS, or ERA5 corresponding to the corresponding time and longitude and latitude.
9. A storage medium, characterized in that, The storage medium stores instructions that, when read by a computer, cause the computer to execute the method for constructing high-precision ultraviolet limb tangent correction based on machine learning according to any one of claims 1-7.
10. An electronic device, characterized in that, It includes a processor and the storage medium according to claim 9, and the processor executes the instructions in the storage medium.