Automatic anomaly detection method and device for sand and dust observation data

Abstract

The invention relates to the technical field of meteorological data processing, and provides an automatic anomaly detection method and device for sand and dust observation data, and the method comprises the steps: obtaining a sand and dust observation data abnormal value of a to-be-observed region; carrying out real-time inversion on an earth surface sand and dust concentration image of the to-be-observed area based on the satellite data of the sand and dust observation time period; determining a target pixel of the sand and dust weather in the to-be-observed area based on the earth surface sand and dust concentration image; comparing the ground surface sand and dust concentration value corresponding to the target pixel with a sand and dust concentration abnormal value in the sand and dust observation data abnormal values at the corresponding position of the target pixel; and when the difference obtained by comparison is smaller than a preset difference threshold, deleting the sand and dust observation data abnormal value corresponding to the sand and dust concentration abnormal value. According to the invention, the problem that the sand and dust observation data is wrongly identified as an abnormal value is avoided, and the credibility of the sand and dust observation data after quality control is improved.

Description

Technical Field

[0001] This invention relates to the field of meteorological data processing technology, and in particular to an automatic anomaly detection method and device for dust observation data. Background Technology

[0002] Dust monitoring data quality control refers to a set of automated and manual verification and labeling processes implemented for multi-source dust monitoring data (such as dust concentration). The aim is to ensure the authenticity, comparability and traceability of the data, and to avoid introducing erroneous data caused by instrument failures into operational analysis and forecasting.

[0003] Currently, existing dust monitoring methods may not accurately detect outliers. For example, a fully automatic outlier detection method based on residual probability (PAOD) can effectively remove outliers from the observation data, but it can also produce some false positives, leading to a decrease in the quality control of the observation data. However, for dust (such as PM10) observations during dusty weather, due to the rapid propagation of dust events, abrupt changes may occur in the observation time series, followed by a rapid drop in concentration. The PAOD method may identify this abrupt change as an outlier and remove it. However, this is actually a normal dusty weather event causing a rise in observed concentration followed by a rapid drop, which reduces the quality of the dust monitoring data.

[0004] The PAOD method described above can be found in the literature: "Wu, HJ, X. Tang, ZF Wang, L. Wu, MMLu, LF Wei, and J. Zhu, 2018: Probabilistic automatic outlier detection for surface air quality measurements from the China National Environmental Monitoring Network. Adv. Atmos. Sci., 35(12), 1522–1532, https: / / doi.org / 10.1007 / s00376-018-8067-9". Summary of the Invention

[0005] This invention provides an automatic anomaly detection method and apparatus for dust observation data, which solves the problem of misidentifying dust observation data as outliers in the prior art.

[0006] This invention provides an automatic anomaly detection method for dust storm observation data, comprising the following steps: Obtain outliers in the dust observation data of the area to be observed; Real-time inversion of surface dust concentration images of the observed area based on satellite data during the dust observation period; Based on the surface dust concentration image, the target pixels of the observed area where dust weather occurred are determined; Compare the surface dust concentration value corresponding to the target pixel with the dust concentration anomaly value in the dust observation data anomaly value at the location corresponding to the target pixel; If the difference obtained from the comparison is less than a preset difference threshold, then the abnormal dust observation data corresponding to the abnormal dust concentration value is deleted.

[0007] An automatic anomaly detection method for dust observation data provided by the present invention, which retrieves surface dust concentration images of the observed area in real time based on satellite data during the dust observation period, includes: Cloud masking is performed on the satellite's thermal infrared remote sensing data during the dust observation period to divide the area to be observed into cloudless and cloudy areas. For cloudless areas, the thermal infrared brightness temperature difference, specific humidity data, planetary boundary layer height and surface temperature data of the area to be observed are input into the first dust concentration prediction model to obtain the first surface dust concentration value of each pixel in the cloudless area output by the first dust concentration prediction model. The first surface dust concentration sub-image is generated based on the first surface dust concentration value. The thermal infrared brightness temperature difference is determined based on thermal infrared remote sensing data. For cloud areas, the first surface dust concentration sub-image and the surface dust concentration observation values ​​observed by the cloud-covered station are input into the second dust concentration prediction model to obtain the second surface dust concentration value of each pixel in the cloud area output by the second dust concentration prediction model, and the second surface dust concentration sub-image is generated based on the second surface dust concentration value. By merging the first and second sub-images of surface dust concentration, a surface dust concentration image of the area to be observed is obtained. The first dust concentration prediction model is trained based on thermal infrared brightness temperature difference samples, specific humidity data samples, planetary boundary layer height samples, and surface temperature data samples under cloudless conditions, as well as the first surface dust concentration value label under cloudless conditions. The second dust concentration prediction model is trained based on simulated surface dust concentration image samples, simulated surface dust concentration value samples, and second surface dust concentration value labels under cloud conditions, using chemical transport model simulation under cloud conditions.

[0008] According to the automatic anomaly detection method for sandstorm observation data provided by the present invention, the determination method of the thermal infrared brightness temperature difference is as follows: Acquire each thermal infrared band from satellite thermal infrared remote sensing data; The thermal infrared brightness temperature difference is calculated based on each thermal infrared band.

[0009] According to the automatic anomaly detection method for dust observation data provided by the present invention, the loss function of the first dust concentration prediction model during training is as follows: ; in, Indicates the first b Group samples, at time step t Pixel p The predicted concentration of dust storms, This represents a cloud mask; a value of 1 indicates no clouds, and a value of 0 indicates clouds. Indicates the first b Group samples, at time step t , pixel p The first surface dust concentration label for the location. B Indicates the number of sample groups. T Indicates the length of time. P This indicates the number of pixels in the area to be observed.

[0010] According to the automatic anomaly detection method for dust observation data provided by the present invention, the loss function of the dust concentration prediction model during training is as follows: ; in, This represents a cloud mask; a value of 1 indicates the presence of clouds, and a value of 0 indicates the absence of clouds. , ε This represents a non-zero coefficient. Indicates in pixel i The predicted concentration of dust storms, Represents the corresponding pixel i The second surface dust concentration value label for the location.

[0011] An automatic anomaly detection method for dust observation data provided by the present invention determines the target pixels of dust weather occurring in the area to be observed based on the surface dust concentration image, including: Determine the pixel dust concentration of any pixel in the surface dust concentration image corresponding to the current time step during the dust observation period; If the dust concentration of a pixel is greater than a preset concentration threshold, the change in the dust concentration of the pixel relative to the previous time step is calculated, and the change is normalized. If the slope of the normalized slope is greater than a preset slope threshold, it is confirmed that dust weather has occurred in the area corresponding to any pixel, and the pixel is determined to be the target pixel; otherwise, no dust weather has occurred. If the dust concentration of the pixel corresponding to the current time step is less than or equal to the preset concentration threshold, it is confirmed that no dust weather has occurred.

[0012] The present invention also provides an automatic anomaly detection device for sandstorm observation data, comprising the following modules: The abnormal data acquisition module is used to acquire abnormal values ​​in the dust observation data of the area to be observed; The inversion module is used to invert the surface dust concentration image of the area to be observed in real time based on satellite data during the dust observation period; The target pixel determination module is used to determine the target pixels in the area to be observed where dust weather occurs, based on the surface dust concentration image. The comparison module is used to compare the surface dust concentration value corresponding to the target pixel with the dust concentration anomaly value in the dust observation data anomaly value at the location corresponding to the target pixel; The abnormal data deletion module is used to delete the abnormal dust observation data corresponding to the abnormal dust concentration value if the difference obtained from the comparison is less than a preset difference threshold.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the automatic anomaly detection method for sand and dust observation data as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the automatic anomaly detection method for dust observation data as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements an automatic anomaly detection method for sand and dust observation data as described above.

[0016] The automatic anomaly detection method and apparatus for dust observation data provided by this invention uses satellite data to invert the surface dust concentration image of the area to be observed in real time, and then uses a pixel-by-pixel comparison method to obtain the target pixel where the dust weather occurred. The surface dust concentration value corresponding to the target pixel is compared with the dust concentration anomaly value in the dust observation data anomaly value at the corresponding position of the target pixel, thereby performing anomaly screening to reduce the misjudgment rate of anomalies, minimize the misidentification of dust observation data as anomalies, and improve the reliability of dust observation data after quality control. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts of the automatic anomaly detection method for sand and dust observation data provided by the present invention.

[0019] Figure 2 This is the second flowchart of the automatic anomaly detection method for sand and dust observation data provided by the present invention.

[0020] Figure 3 This is the third flowchart of the automatic anomaly detection method for sand and dust observation data provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the automatic anomaly detection device for sand and dust observation data provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The automatic anomaly detection method for dust observation data in this invention embodiment, such as... Figure 1 As shown, it includes the following steps: Step S110: Obtain outliers in the dust observation data of the area to be observed. For example, the outliers in the dust observation data of the area to be observed can be obtained using the Probability of Residuals (PAOD) method. It is understood that there are multiple outliers in the dust observation data, forming an outlier set. Due to the special nature of dust storms, their rapid propagation and high intensity can cause the concentration of dust in the atmosphere (e.g., PM10 concentration) to spike to a very high value in a very short time. In such cases, the observed values ​​and outliers have very similar probabilistic characteristics. Therefore, the outliers detected by the PAOD method in dust storms may not actually be outliers. Further screening of the outliers detected by the PAOD method is required in subsequent steps to improve the quality of the dust observation data.

[0025] Step S120: Real-time inversion of the surface dust concentration image of the area to be observed based on satellite data during the dust observation period. That is, in this step, the surface dust concentration image of the area to be observed during the dust observation period can be obtained by inverting satellite data. This surface dust concentration image is generated from the surface dust concentration values ​​of different pixels. It can be understood that: based on this surface dust concentration image, the surface dust concentration value of each pixel in the image can be obtained, and different pixels correspond one-to-one with the actual sub-regions of the area to be observed. The dust observation period refers to the observation period corresponding to the abnormal values ​​of the dust observation data in step S110.

[0026] Step S130: Based on the surface dust concentration image, determine the target pixel of the observation area where dust weather occurs. Specifically, the sub-area corresponding to each pixel in the surface dust concentration image can be used to determine whether dust weather has occurred. If dust weather has occurred, the pixel is determined as the target pixel.

[0027] Step S140: Compare the surface dust concentration value corresponding to the target pixel with the dust concentration anomaly value in the dust observation data anomaly value at the location corresponding to the target pixel.

[0028] Step S150: If the difference obtained from the comparison is less than a preset difference threshold, then delete the dust observation data anomaly corresponding to the dust concentration anomaly, that is, delete a set of dust observation data anomalies corresponding to the dust concentration anomaly from the anomaly set. Specifically, if the difference between the two is small and less than the preset difference threshold, it indicates that step S110 (PAOD method) was misjudged, and the dust observation data anomaly corresponding to the dust concentration anomaly is deleted from the anomaly set to ensure the reliability of the dust observation data; otherwise, it indicates that step S110 (PAOD method) was not misjudged. The preset difference threshold can be set according to actual conditions, and can be 80~110 ug / m³. 3 .

[0029] The automatic anomaly detection method for dust observation data in this embodiment uses satellite data to invert the surface dust concentration image of the area to be observed in real time. Then, it uses a pixel-by-pixel comparison method to obtain the target pixel where the dust weather occurred. It compares the surface dust concentration value corresponding to the target pixel with the dust concentration anomaly value in the dust observation data anomaly value at the corresponding position of the target pixel, thereby performing anomaly screening to reduce the misjudgment rate of anomalies, minimize the misidentification of dust observation data as anomalies, and improve the reliability of the dust observation data after quality control.

[0030] In some embodiments, the specific process of step S120, which involves real-time inversion of the surface dust concentration image of the area to be observed based on satellite data during the dust observation period, is as follows: Figure 2 As shown, it includes the following steps: Step 121: Perform cloud masking on the satellite's thermal infrared remote sensing data during the dust observation period to divide the area to be observed into cloudless and cloudy areas.

[0031] Since the thermal infrared signal of the ground surface is blocked by clouds, it is necessary to divide the area to be observed into cloudless and cloudy areas, and use different inversion methods for the cloudless and cloudy areas, so as to make the final image of the surface dust concentration of the area to be observed more accurate.

[0032] Cloud masking operations can employ existing, mature cloud masking techniques. For example, the cloud masking operation method mentioned in the paper "An Algorithm for Land Surface Temperature Retrieval Using Three Thermal Infrared Bands of Himawari-8" can be used.

[0033] Step 122: For cloudless areas, input the thermal infrared brightness temperature difference, specific humidity data, planetary boundary layer height and surface temperature data of the area to be observed into the first dust concentration prediction model to obtain the first surface dust concentration value of each pixel in the cloudless area output by the first dust concentration prediction model. Generate a first surface dust concentration sub-image based on the first surface dust concentration value. The thermal infrared brightness temperature difference is determined based on thermal infrared remote sensing data.

[0034] The method for determining the thermal infrared brightness temperature difference is as follows: Acquire the thermal infrared bands in the satellite thermal infrared remote sensing data. For example, the thermal infrared bands detected by the satellite are: 11.2µm (BT11), 12.4µm (BT12), 8.6µm (BT8.6) and 3.9µm (BT3.9).

[0035] Based on the aforementioned thermal infrared bands, the thermal infrared brightness temperature difference can be calculated using the following formula: BTD(λ1-λ2)=BT(λ1)-BT(λ2). For example, the following three infrared brightness temperature differences can be calculated: Calculate the first thermal infrared brightness temperature difference: BT11-BT12(BTD(11-12)).

[0036] Calculate the second thermal infrared brightness temperature difference: BT11-BT8.6 (BTD(8.6-11)).

[0037] Calculate the third thermal infrared brightness temperature difference: BT3.9-BT11 (BTD(3.9-11)).

[0038] In the three thermal infrared brightness temperature difference expressions above, the part outside the outer parentheses represents the thermal infrared band, and the part inside the outer parentheses is the label for that thermal infrared brightness temperature difference. BT stands for Brightness Temperature, and BTD stands for Brightness Temperature Difference.

[0039] Specific humidity data and planetary boundary layer height can be obtained from the fifth-generation ECMWF atmospheric reanalysis dataset (ERA5), while surface temperature (LST) can be obtained from satellite data.

[0040] The first dust concentration prediction model is trained based on samples of thermal infrared brightness difference, specific humidity, planetary boundary layer height, and surface temperature under cloudless conditions, as well as labels of first surface dust concentration values ​​under cloudless conditions (actual observed surface dust concentration values ​​under cloudless conditions). During model training, based on the input samples of thermal infrared brightness difference, specific humidity, planetary boundary layer height, and surface temperature under cloudless conditions, a first surface dust concentration prediction sub-image is output. The first surface dust concentration prediction sub-image and its labels are substituted into the corresponding loss function. Model training is complete when the loss function converges. It should be noted that the first dust concentration prediction model outputs the dust concentration for each pixel in the observed area, thus forming either a first surface dust concentration prediction sub-image (during training) or a first surface dust concentration sub-image (during inference). For example, the first dust concentration prediction model can employ the TimeSformer model.

[0041] In this embodiment, the loss function of the first dust concentration prediction model during training is as follows: ; in, Indicates the first b Group samples, at time step tPixel p The predicted concentration of dust storms, This represents a cloud mask; a value of 1 indicates no clouds, and a value of 0 indicates clouds. Indicates the first b Group samples, at time step t , pixel p The first surface dust concentration label for the location. B Indicates the number of sample groups. T Indicates the length of time. P This indicates the number of pixels in the area to be observed.

[0042] Step 123: For cloud areas, input the first surface dust concentration sub-image and the surface dust concentration observation values ​​observed by the cloud-covered stations into the second dust concentration prediction model to obtain the second surface dust concentration value of each pixel in the cloud area output by the second dust concentration prediction model, and generate the second surface dust concentration sub-image based on the second surface dust concentration value.

[0043] The second dust concentration prediction model utilizes the principle of AI image inpainting. It takes as input a first surface dust concentration sub-image and observed surface dust concentration values ​​from under-cloud stations to predict a second surface dust concentration sub-image for cloud-covered areas. For example, the second dust concentration prediction model can use the MAT (Mask-Aware Transformer) model.

[0044] The second dust concentration prediction model is trained based on simulated surface dust concentration image samples from the Chemical Transport Model (CTM) under cloud conditions, simulated surface dust concentration value samples from the CTM, and second surface dust concentration value labels under cloud conditions (actual observed surface dust concentration data under cloud conditions). During model training, based on the input CTM-simulated surface dust concentration image samples and CTM-simulated surface dust concentration value samples, a second surface dust concentration prediction sub-image is output. The second surface dust concentration prediction sub-image and the labels of the second surface dust concentration sub-image are substituted into the corresponding loss function. The model training is completed when the loss function converges.

[0045] In this embodiment, the loss function of the dust concentration prediction model during training is as follows: ; in, This represents a cloud mask; a value of 1 indicates the presence of clouds, and a value of 0 indicates the absence of clouds. , ε To represent non-zero coefficients, in order to avoid =0, ε It can be no more than 10 -6Extremely small coefficients of this order of magnitude. Indicates in pixel i The predicted concentration of dust storms, Represents the corresponding pixel i The second surface dust concentration value label for the location.

[0046] Step 124: Merge the first surface dust concentration sub-image and the second surface dust concentration sub-image to obtain the surface dust concentration image of the area to be observed.

[0047] In this embodiment, by dividing the area to be observed into a cloudless area and a clouded area through cloud masking, it is possible to obtain more accurate sub-images of surface dust concentration in the clouded area and the cloudless area, thereby obtaining a more accurate surface dust concentration image of the area to be observed.

[0048] In some embodiments, step S130 involves determining the target pixels in the area to be observed where dust storms occur based on the surface dust concentration image. The specific process is as follows: Figure 3 As shown, it includes: Determine the pixel dust concentration of any pixel in the surface dust concentration image corresponding to the current time step within the dust observation period. This can be achieved by traversing the dust concentration image pixel by pixel to obtain the pixel dust concentration corresponding to any pixel at the current time.

[0049] When the dust concentration in a pixel is greater than a preset concentration threshold (e.g., a concentration threshold of 200 ug / m³), 3 In the case of a given time step, the change in the dust concentration of the pixel relative to the previous time step (i.e., the pixel dust concentration of the corresponding pixel in the surface dust concentration image corresponding to the previous time step) is calculated. The change in the change is normalized. If the slope of the normalized slope is greater than a preset slope threshold (e.g., the slope threshold is 0.1), it is confirmed that dust weather has occurred in the area corresponding to any pixel, and the pixel is determined to be the target pixel; otherwise, no dust weather has occurred. If the dust concentration of the pixel corresponding to the current time step is less than or equal to the preset concentration threshold, it is confirmed that no dust weather has occurred.

[0050] In this embodiment, through Figure 3 The steps shown can accurately determine whether dust storms have occurred in the area covered by any pixel. Therefore, when comparing the retrieved surface dust concentration with the dust concentration anomalies in the corresponding area's dust observation data anomalies, it is first determined whether the dust observation data anomalies are in the area where dust storms occurred. If they are in the area, the comparison is made; otherwise, no comparison is made.

[0051] Furthermore, in step S150 above, the preset difference threshold can be dynamically set according to the following formula: .

[0052] This indicates a preset difference threshold. α This is the uncertainty coefficient, for example, taken as 1.8. This represents the mean of the uncertainties of the first and second dust concentration prediction models. (The uncertainty of an AI model refers to the model's confidence in its prediction results or a measure of its reliability; it reflects the "uncertainty" of the model's output when making decisions and is a key indicator for evaluating the credibility of AI models.) The uncertainties of the first and second dust concentration prediction models were obtained through cross-validation. This indicates the uncertainty of the instrument, which is used to observe dust data in the PAOD method. The uncertainty of the instrument can be approximated by empirical formulas or by using the values ​​in the instrument manual.

[0053] Using the above formula to determine the preset difference threshold is more adaptable than a fixed preset difference threshold, and can more accurately determine whether the abnormal value of PAOD is misjudged.

[0054] The automatic anomaly detection device for dust observation data provided by the present invention is described below. The automatic anomaly detection device for dust observation data described below can be referred to in correspondence with the automatic anomaly detection method for dust observation data described above.

[0055] The automatic anomaly detection device for dust observation data in this embodiment of the invention, such as Figure 4 As shown, it includes the following modules: The abnormal data acquisition module 410 is used to acquire abnormal values ​​of dust observation data in the area to be observed.

[0056] The inversion module 420 is used to invert the surface dust concentration image of the area to be observed in real time based on satellite data during the dust observation period.

[0057] The target pixel determination module 430 is used to determine the target pixels in the area to be observed where dust weather occurs based on the surface dust concentration image.

[0058] The comparison module 440 is used to compare the surface dust concentration value corresponding to the target pixel with the dust concentration anomaly value in the dust observation data anomaly value at the location corresponding to the target pixel.

[0059] The abnormal data deletion module 450 is used to delete the abnormal dust observation data value corresponding to the abnormal dust concentration value if the difference obtained from the comparison is less than a preset difference threshold.

[0060] The automatic anomaly detection device for dust observation data in this embodiment uses satellite data to invert the surface dust concentration image of the area to be observed in real time, and then uses a pixel-by-pixel comparison method to obtain the target pixel where the dust weather occurred. It compares the surface dust concentration value corresponding to the target pixel with the dust concentration anomaly value in the dust observation data anomaly value at the corresponding position of the target pixel, thereby performing anomaly screening to reduce the misjudgment rate of anomalies, minimize the misidentification of dust observation data as anomalies, and improve the reliability of the dust observation data after quality control.

[0061] In some embodiments, the inversion module 420 specifically includes: The cloud masking operation module is used to perform cloud masking operations on the satellite's thermal infrared remote sensing data during the dust observation period, dividing the area to be observed into cloudless areas and clouded areas.

[0062] The first sub-image prediction module is used to input the thermal infrared brightness temperature difference, specific humidity data, planetary boundary layer height and surface temperature data of the area to be observed into the first dust concentration prediction model for cloudless areas, and obtain the first surface dust concentration value of each pixel in the cloudless area output by the first dust concentration prediction model. Based on the first surface dust concentration value, a first surface dust concentration sub-image is generated. The thermal infrared brightness temperature difference is determined based on thermal infrared remote sensing data.

[0063] The second sub-image prediction module is used to input the first surface dust concentration sub-image and the surface dust concentration observation values ​​observed by the cloud-covered station into the second dust concentration prediction model for the cloud-covered area, and obtain the second surface dust concentration value of each pixel in the cloud-covered area output by the second dust concentration prediction model, and generate the second surface dust concentration sub-image based on the second surface dust concentration value.

[0064] The sub-image merging module is used to merge the first surface dust concentration sub-image and the second surface dust concentration sub-image to obtain the surface dust concentration image of the area to be observed.

[0065] The first dust concentration prediction model is trained based on thermal infrared brightness temperature difference samples, specific humidity data samples, planetary boundary layer height samples, and surface temperature data samples under cloudless conditions, as well as the first surface dust concentration value label under cloudless conditions.

[0066] The second dust concentration prediction model is trained based on simulated surface dust concentration image samples, simulated surface dust concentration value samples, and second surface dust concentration value labels under cloud conditions, using chemical transport model simulation under cloud conditions.

[0067] In some embodiments, the thermal infrared brightness temperature difference is determined as follows: acquiring each thermal infrared band in satellite thermal infrared remote sensing data; and calculating the thermal infrared brightness temperature difference based on each thermal infrared band.

[0068] In some embodiments, the loss function of the first dust concentration prediction model during training is as follows: ; in, Indicates the first b Group samples, at time step t Pixel p The predicted concentration of dust storms, This represents a cloud mask; a value of 1 indicates no clouds, and a value of 0 indicates clouds. Indicates the first b Group samples, at time step t , pixel p The first surface dust concentration label for the location. B Indicates the number of sample groups. T Indicates the length of time. P This indicates the number of pixels in the area to be observed.

[0069] In some embodiments, the loss function of the dust concentration prediction model during training is as follows: ; in, This represents a cloud mask; a value of 1 indicates the presence of clouds, and a value of 0 indicates the absence of clouds. , ε This represents a non-zero coefficient. Indicates in pixel i The predicted concentration of dust storms, Represents the corresponding pixel i The second surface dust concentration value label for the location.

[0070] In some embodiments, the target pixel determination module 430 is specifically used to determine the pixel dust concentration of any pixel in the surface dust concentration image corresponding to the current time step during the dust observation period; if the pixel dust concentration is greater than a preset concentration threshold, calculate the change amplitude of the pixel dust concentration relative to the previous time step, normalize the change amplitude, and if the slope after normalization is greater than a preset slope threshold, confirm that dust weather has occurred in the area corresponding to any pixel, and determine that any pixel is the target pixel; otherwise, no dust weather has occurred; if the pixel dust concentration corresponding to the current time step is less than or equal to the preset concentration threshold, confirm that no dust weather has occurred.

[0071] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an automatic anomaly detection method for sandstorm observation data, the method including: Obtain outlier values ​​in the dust observation data of the area to be observed.

[0072] The surface dust concentration image of the observed area is retrieved in real time based on satellite data during the dust observation period.

[0073] The target pixels for dust storms in the observed area are determined based on the surface dust concentration image.

[0074] The surface dust concentration value corresponding to the target pixel is compared with the dust concentration anomaly value in the dust observation data anomaly value at the location corresponding to the target pixel.

[0075] If the difference obtained from the comparison is less than a preset difference threshold, then the abnormal dust observation data corresponding to the abnormal dust concentration value is deleted.

[0076] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the automatic anomaly detection method for sandstorm observation data provided by the above methods, the method comprising: Obtain outlier values ​​in the dust observation data of the area to be observed.

[0078] The surface dust concentration image of the observed area is retrieved in real time based on satellite data during the dust observation period.

[0079] The target pixels for dust storms in the observed area are determined based on the surface dust concentration image.

[0080] The surface dust concentration value corresponding to the target pixel is compared with the dust concentration anomaly value in the dust observation data anomaly value at the location corresponding to the target pixel.

[0081] If the difference obtained from the comparison is less than a preset difference threshold, then the abnormal dust observation data corresponding to the abnormal dust concentration value is deleted.

[0082] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an automatic anomaly detection method for dust observation data provided by the methods described above, the method comprising: Obtain outlier values ​​in the dust observation data of the area to be observed.

[0083] The surface dust concentration image of the observed area is retrieved in real time based on satellite data during the dust observation period.

[0084] The target pixels for dust storms in the observed area are determined based on the surface dust concentration image.

[0085] The surface dust concentration value corresponding to the target pixel is compared with the dust concentration anomaly value in the dust observation data anomaly value at the location corresponding to the target pixel.

[0086] If the difference obtained from the comparison is less than a preset difference threshold, then the abnormal dust observation data corresponding to the abnormal dust concentration value is deleted.

[0087] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for automatic anomaly detection of sand and dust observation data, characterized by, The method comprises the following steps: acquiring dust observation data outliers of a to-be-observed area; real-time inversion of a surface dust concentration image of the to-be-observed area based on satellite data in a dust observation period; determining a target pixel of the to-be-observed area where dust weather occurs based on the surface dust concentration image; comparing a surface dust concentration value corresponding to the target pixel with a dust concentration outlier in the dust observation data outlier corresponding to the position of the target pixel; in the case that the difference obtained by comparison is less than a preset difference threshold, deleting the dust observation data outlier corresponding to the dust concentration outlier.

2. The automatic outlier detection method of sand and dust observation data according to claim 1, characterized in that, The real-time inversion of the surface dust concentration image of the to-be-observed area based on satellite data in the dust observation period comprises the following steps: carrying out cloud mask operation on thermal infrared remote sensing data of a satellite in the dust observation period, and dividing the to-be-observed area into cloud-free areas and cloud areas; for the cloud-free areas, inputting thermal infrared brightness temperature difference, specific humidity data, planetary boundary layer height and surface temperature data of the to-be-observed area into a first dust concentration prediction model to obtain first surface dust concentration values of each pixel in the cloud-free areas output by the first dust concentration prediction model, and generating a first surface dust concentration sub-image based on the first surface dust concentration values, wherein the thermal infrared brightness temperature difference is determined based on the thermal infrared remote sensing data; for the cloud areas, inputting the first surface dust concentration sub-image and surface dust concentration observation values observed by a station under the cloud into a second dust concentration prediction model to obtain second surface dust concentration values of each pixel in the cloud areas output by the second dust concentration prediction model, and generating a second surface dust concentration sub-image based on the second surface dust concentration values; merging the first surface dust concentration sub-image and the second surface dust concentration sub-image to obtain the surface dust concentration image of the to-be-observed area; wherein the first dust concentration prediction model is trained based on thermal infrared brightness temperature difference samples, specific humidity data samples, planetary boundary layer height samples and surface temperature data samples under cloud-free conditions, and first surface dust concentration value labels under cloud-free conditions; the second dust concentration prediction model is trained based on surface dust concentration simulation image samples simulated by a chemical transport model under cloud conditions, surface dust concentration value samples simulated by the chemical transport model under cloud conditions, and second surface dust concentration value labels under cloud conditions.

3. The automatic outlier detection method of sand and dust observation data according to claim 2, characterized in that, The determination method of the thermal infrared brightness temperature difference is as follows: acquiring each thermal infrared band in the satellite thermal infrared remote sensing data; calculating the thermal infrared brightness temperature difference based on the thermal infrared bands.

4. The automatic outlier detection method of sand and dust observation data according to claim 2, characterized in that, The loss function of the first dust concentration prediction model during training is as follows: ； wherein, represents the b group of samples, at time step t , the pixel p location, represents the cloud mask, 1 for no cloud, 0 for cloud, represents the b group of samples, at time step t , the first ground dust concentration label at pixel p location, B represents the number of sample groups, T represents the time length, P represents the number of pixels of the region to be observed.

5. The automatic outlier detection method of sand and dust observation data according to claim 2, characterized in that, The loss function of the second dust concentration prediction model during training is as follows: ； wherein, represents a cloud mask, 1 for cloud, 0 for no cloud, , ε represents a coefficient other than 0, represents a sand-dust forecast concentration of a pixel i , represents a second ground sand-dust concentration value tag corresponding to the position of the pixel i .

6. The automatic outlier detection method of sand and dust observation data according to any one of claims 1 to 5, characterized in that, The method for determining the target pixel of the to-be-observed area where dust weather occurs based on the surface dust concentration image comprises the following steps: determining a pixel dust concentration of any pixel in the surface dust concentration image corresponding to a current time step in the dust observation period; In a case where the pixel dust concentration is greater than a preset concentration threshold, a change amplitude of the pixel dust concentration relative to a previous time step is calculated, and in a case where a normalized slope of the change amplitude is greater than a preset slope threshold, it is determined that dust weather occurs in a region corresponding to the any pixel, and the any pixel is determined as the target pixel, otherwise, it is determined that dust weather does not occur; in a case where the pixel dust concentration corresponding to the current time step is less than or equal to the preset concentration threshold, it is determined that dust weather does not occur.

7. An automatic anomaly detection device for sandstorm observation data, characterized in that, Comprise: An abnormal data acquisition module configured to acquire dust observation data outliers of a to-be-observed region; An inversion module configured to inversely calculate a ground dust concentration image of the to-be-observed region based on satellite data of a dust observation period in real time; A target pixel determination module configured to determine a target pixel of the to-be-observed region in which dust weather occurs based on the ground dust concentration image; A comparison module configured to compare a ground dust concentration value corresponding to the target pixel with a dust concentration outlier in the dust observation data outliers of a position corresponding to the target pixel; An abnormal data deletion module configured to delete the dust observation data outlier corresponding to the dust concentration outlier in a case where a difference obtained by comparison is less than a preset difference threshold.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to realize the automatic abnormal detection method of dust observation data according to any one of claims 1 to 6. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the automatic abnormal detection method of dust observation data according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the automatic abnormal detection method of dust observation data according to any one of claims 1 to 6.

Images