Precipitation data anomaly recognition model construction method and device, equipment and storage medium
By clustering and filtering precipitation datasets, a precipitation data anomaly identification model is constructed, which solves the problem of high resource input and poor identification effect in existing technologies, and achieves efficient and accurate precipitation data anomaly identification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 河北省气象信息中心
- Filing Date
- 2026-01-23
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies are too complex to identify anomalies in precipitation data, requiring significant resources and yielding poor results.
By acquiring multiple historical precipitation datasets, clustering and filtering are performed to construct a precipitation data anomaly identification model. Anomaly indices are determined by transforming data dimensions, reducing resource input and improving identification accuracy.
It achieves high efficiency and accuracy in identifying precipitation data anomalies without requiring data labeling and extensive computation, reducing resource consumption and improving judgment effectiveness.
Smart Images
Figure CN122020461A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of meteorological data anomaly identification technology, and in particular to a method, apparatus, equipment and storage medium for constructing a precipitation data anomaly identification model. Background Technology
[0002] Anomaly identification in precipitation data is a crucial step in meteorological research and disaster prevention and mitigation. Its core is to accurately filter out "outliers" (including erroneous values and extreme values) that deviate from normal patterns from massive amounts of precipitation observation data (such as station observations, satellite inversion, and radar estimation) or simulated data. These outliers may originate from instrument malfunctions, human recording errors, or real extreme weather events (such as torrential rains or abnormal droughts). Therefore, the identification process requires a comprehensive judgment combining data characteristics, physical laws, and operational needs.
[0003] Current methods for identifying anomalies in precipitation data often involve cross-validation based on the relationships between features. However, precipitation data contains numerous elements (such as temperature, air pressure, humidity, and wind speed), resulting in strong coupling and complex relationships between data points. Therefore, this approach can lead to overly complex models when dealing with large datasets, requiring significant manpower and computing power for model building and application. This severely tests the technical capabilities and resource investment required for modeling.
[0004] Therefore, it is necessary to develop a method for constructing a precipitation data anomaly identification model. Summary of the Invention
[0005] The present invention provides a method, apparatus, equipment and storage medium for constructing a precipitation data anomaly identification model, which solves the problem that the prior art requires a large investment of resources.
[0006] In a first aspect, embodiments of the present invention provide a method for constructing a precipitation data anomaly identification model, comprising: Multiple first historical precipitation datasets were obtained, which included multiple precipitation data and multiple factor data that influenced the precipitation data; Cluster the multiple first historical precipitation datasets to obtain the first clustering result; The plurality of first historical precipitation datasets are filtered according to the first clustering result, and a plurality of first precipitation classes are obtained according to the filtering result and the first clustering result; For each first precipitation category, a precipitation data anomaly identification model is constructed, wherein the precipitation data anomaly identification model determines the anomaly index based on the input precipitation dataset through data dimension transformation.
[0007] In one possible implementation, clustering the plurality of first historical precipitation datasets to obtain a plurality of first precipitation classes includes: Get the neighborhood radius and the number of first friendly neighbors; The first historical precipitation dataset with a greater number of first historical precipitation datasets within its neighborhood radius than the number of first friendly neighbors is used as the initial dataset. Extract a dataset from multiple unclustered initial datasets as the starting dataset, and then perform the following steps after extraction: Centered on the originating dataset, search for the first unclustered historical precipitation dataset within the neighborhood radius of the originating dataset; If a first historical precipitation dataset that is not clustered is found within the neighborhood radius, the found dataset is added to the class of the originating dataset, the found dataset is used as the originating dataset, and the process jumps to the step of searching for the first historical precipitation dataset that is not clustered within the neighborhood radius of the originating dataset. If all initial datasets are clustered, then the first historical precipitation dataset that is not clustered will be marked as not clustered.
[0008] In one possible implementation, the neighborhood radius is determined through multiple clustering operations, including: Obtain the number of multiple first clusters; Based on the number of each first cluster, the multiple first historical precipitation datasets are clustered to obtain multiple second clustering results; For each clustering result, a first aggregation degree index is determined according to a first formula, where the first formula is:
[0009] In the formula, The first aggregation index, The number of the first cluster, The number of first historical precipitation datasets in the class. For the first The first in the class The first historical precipitation dataset, For the first The class center of each class; Plot the first curve of the aggregation degree index as the number of clusters with the number of clusters as the horizontal axis and the first aggregation degree index as the vertical axis. Find the point with the largest curvature from the first curve graph, and take the number of the first cluster corresponding to the point with the largest curvature as the target number of clusters; The second clustering result obtained based on the target number of clusters will be used as the target clustering result; For each class in the target clustering result, the average neighborhood radius is determined, where the average neighborhood radius is the mean of multiple first neighborhood radii, and the first neighborhood radius is the distance between the first historical precipitation dataset in the class and the nearest first historical precipitation dataset. The maximum value among multiple average neighborhood radii is taken as the neighborhood radius.
[0010] In one possible implementation, the step of filtering the plurality of first historical precipitation datasets based on the first clustering result, and obtaining a plurality of first precipitation classes based on the filtering result and the first clustering result, includes: The first clustering result includes multiple second precipitation classes and multiple first historical precipitation datasets marked with non-clustering labels; Delete the first historical precipitation dataset with unclustered labels; The second precipitation class with the most data sets in the class is used as the reference class; The number of thresholds is determined based on the first proportional threshold and the number of datasets in the reference class; Delete the second precipitation class whose number of datasets in the class is less than the threshold number; The remaining multiple second precipitation categories are treated as the multiple first precipitation categories.
[0011] In one possible implementation, the step of constructing a precipitation data anomaly identification model for each first precipitation class includes: Obtain a base model and multiple first coefficient arrays, wherein the base model has multiple first coefficients, and the number of coefficients in the first coefficient arrays is the same as the number of the multiple first coefficients; Substituting the multiple first coefficient arrays into the basic model respectively, multiple process models are obtained; For each process model, the dataset in the first precipitation class is substituted into the process model, and the mean of the multiple deviation indices is used as the model deviation index. If the iteration threshold is not reached, the multiple first coefficient arrays are adjusted according to multiple model deviation indices; Jump to the step of substituting the multiple first coefficient arrays into the basic model to obtain multiple process models; If the iteration threshold is reached, the process model with the smallest model deviation index will be used as the anomaly identification model. The anomaly detection threshold is determined based on the model deviation index of the anomaly detection model.
[0012] In one possible implementation, the basic model is:
[0013] In the formula, For compression vectors, These are the original vectors constructed from the dataset. For the restored vector, This is the compression coefficient matrix. The compression coefficient matrix is the first Line number Column elements, To restore the coefficient matrix, For the restored coefficient matrix, the first... Line number Column elements, This represents the total number of elements in the compressed vector. The total number of data points in the dataset. This is the deviation index.
[0014] In one possible implementation, adjusting the plurality of first coefficient arrays according to a plurality of model bias indices includes: Obtain multiple deviation index queues, where each queue corresponds to a first coefficient array; Add the model deviation index to the deviation index queue of the corresponding coefficient array; Find the index with the smallest value in each deviation index queue, and take the historical first coefficient array corresponding to the smallest index as the process optimal array; The first coefficient array corresponding to the index with the smallest model deviation index value is taken as the current optimal array; For each first coefficient array, adjustments are made according to the second formula, the process-optimal array, and the current-optimal array, wherein the second formula is:
[0015] In the formula, For the first The first coefficient array after the second adjustment One data point, For the first The first coefficient array after the second adjustment One data point, The first coefficient array in history One data point, The first adjustment factor is... The th of the current optimal array One data point, This is the second adjustment factor.
[0016] Secondly, embodiments of the present invention provide a precipitation data anomaly identification model construction apparatus, used to implement the precipitation data anomaly identification model construction method as described in the first aspect or any possible implementation thereof, wherein the precipitation data anomaly identification model construction apparatus includes: The historical precipitation data acquisition module is used to acquire multiple first historical precipitation datasets, which include multiple precipitation data and multiple factor data that affect the precipitation data; The data clustering module is used to cluster the multiple first historical precipitation datasets to obtain the first clustering result; The data filtering module is used to filter the multiple first historical precipitation datasets according to the first clustering result, and obtain multiple first precipitation categories according to the filtering result and the first clustering result; as well as, An anomaly identification model construction module is used to construct a precipitation data anomaly identification model for each first precipitation category, wherein the precipitation data anomaly identification model determines the anomaly index based on the input precipitation dataset through data dimension transformation.
[0017] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect.
[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation thereof.
[0019] The beneficial effects of the embodiments of the present invention compared with the prior art are: This invention discloses a method for constructing a precipitation data anomaly identification model. First, multiple first historical precipitation datasets are acquired, each including multiple precipitation data points and multiple factors influencing the precipitation data. Then, the multiple first historical precipitation datasets are clustered to obtain a first clustering result. Next, the multiple first historical precipitation datasets are filtered based on the first clustering result, and multiple first precipitation classes are obtained based on the filtering result and the first clustering result. Finally, for each first precipitation class, a precipitation data anomaly identification model is constructed, wherein the precipitation data anomaly identification model determines an anomaly index based on the input precipitation dataset through data dimension transformation. This method does not require prior data labeling or extensive training or computation; therefore, both model construction and application consume relatively few resources. Furthermore, because data anomalies are analyzed through both clustering and model analysis, the accuracy of anomaly detection is higher, and the detection effect is guaranteed. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a flowchart of the precipitation data anomaly identification model construction method provided by the embodiments of the present invention; Figure 2 This is a schematic diagram illustrating the principle of filtering data based on clustering results provided by an embodiment of the present invention; Figure 3 This is a functional block diagram of the precipitation data anomaly identification model construction device provided in the embodiments of the present invention; Figure 4 This is a functional block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0022] In the following description, specific details such as particular system structures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.
[0023] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.
[0024] The embodiments of the present invention will be described in detail below. This example is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operation processes. However, the protection scope of the present invention is not limited to the following embodiments.
[0025] Figure 1 A flowchart illustrating the method for constructing a precipitation data anomaly identification model provided in an embodiment of the present invention.
[0026] like Figure 1 As shown, a flowchart illustrating the implementation of the precipitation data anomaly identification model construction method provided by the embodiments of the present invention is illustrated below: In step 101, multiple first historical precipitation datasets are obtained, wherein the first historical precipitation datasets include multiple precipitation data and multiple factor data that affect the precipitation data.
[0027] In step 102, the multiple first historical precipitation datasets are clustered to obtain the first clustering result.
[0028] In some implementations, clustering the plurality of first historical precipitation datasets to obtain a plurality of first precipitation classes includes: Get the neighborhood radius and the number of first friendly neighbors; The first historical precipitation dataset with a greater number of first historical precipitation datasets within its neighborhood radius than the number of first friendly neighbors is used as the initial dataset. Extract a dataset from multiple unclustered initial datasets as the starting dataset, and then perform the following steps after extraction: Centered on the originating dataset, search for the first unclustered historical precipitation dataset within the neighborhood radius of the originating dataset; If a first historical precipitation dataset that is not clustered is found within the neighborhood radius, the found dataset is added to the class of the originating dataset, the found dataset is used as the originating dataset, and the process jumps to the step of searching for the first historical precipitation dataset that is not clustered within the neighborhood radius of the originating dataset. If all initial datasets are clustered, then the first historical precipitation dataset that is not clustered will be marked as not clustered.
[0029] In some implementations, the neighborhood radius is determined through multiple clustering operations, including: Obtain the number of multiple first clusters; Based on the number of each first cluster, the multiple first historical precipitation datasets are clustered to obtain multiple second clustering results; For each clustering result, a first aggregation degree index is determined according to a first formula, where the first formula is:
[0030] In the formula, The first aggregation index, The number of the first cluster, The number of first historical precipitation datasets in the class. For the first The first in the class The first historical precipitation dataset, For the first The class center of each class; Plot the first curve of the aggregation degree index as the number of clusters with the number of clusters as the horizontal axis and the first aggregation degree index as the vertical axis. Find the point with the largest curvature from the first curve graph, and take the number of the first cluster corresponding to the point with the largest curvature as the target number of clusters; The second clustering result obtained based on the target number of clusters will be used as the target clustering result; For each class in the target clustering result, the average neighborhood radius is determined, where the average neighborhood radius is the mean of multiple first neighborhood radii, and the first neighborhood radius is the distance between the first historical precipitation dataset in the class and the nearest first historical precipitation dataset. The maximum value among multiple average neighborhood radii is taken as the neighborhood radius.
[0031] For example, this invention aims to provide a method for accurately identifying precipitation data anomalies that does not require data labeling and has strong flexibility. To achieve the above objective, this invention first clusters the data and performs preliminary data cleaning based on the clustering results. Based on the cleaned data and the clustering results, a data compression and restoration model is constructed. Because the clustered data have similar characteristics, this model can achieve a good restoration effect; however, for data with large deviations, the restoration effect is relatively poor. In other words, the anomaly degree is identified through the restoration effect. In this way, during modeling, the similarity of the clustering results can be verified, that is, whether the data screening is thorough. Furthermore, an anomaly identification model can be constructed.
[0032] To achieve the above objectives, this invention first organizes the data to ensure that the datasets have identical data items. For example, each dataset includes temperature, air pressure, humidity, wind speed, and station precipitation observations during precipitation. These datasets are then clustered to obtain clustering results. One clustering method provided by this invention is density-based clustering. After clustering, multiple precipitation classes are obtained. Simultaneously, some data that cannot be clustered due to sparse density are removed as outliers.
[0033] When performing clustering, we first perform k-means clustering on the multiple datasets obtained: multiple first historical precipitation datasets, in order to determine a more suitable cluster density.
[0034] Specifically, multiple first clustering numbers are preset. Based on each first clustering number, multiple first historical precipitation datasets are clustered to obtain second clustering results. In fact, the second clustering results are multiple dataset classes. Obviously, the larger the clustering number value, the more compact the data in the class, and the higher the overall aggregation degree. To express this compactness, this invention calculates the first aggregation degree index of the clustering results using a first formula:
[0035] In the formula, The first aggregation index, The number of the first cluster, The number of first historical precipitation datasets in the class. For the first The first in the class The first historical precipitation dataset, For the first The class center of each class.
[0036] In other words, each initial clustering count yields a first aggregation index. A graph is then plotted with the initial clustering count on the horizontal axis and the aggregation index on the vertical axis, showing the aggregation index as a function of the clustering count. This graph reveals a point with maximum curvature. Finding this point indicates that the number of clusters at this point provides a good balance between aggregation and the number of clusters. Therefore, the second clustering result obtained based on this initial clustering count is used as the target clustering result. The neighborhood radius of the target clustering result is then calculated as the neighborhood radius for density clustering.
[0037] When calculating the neighborhood radius of the target clustering result, this invention determines the average neighborhood radius for each class in the target clustering result. Note that the average neighborhood radius is the mean of multiple first neighborhood radii, and the first neighborhood radius is the distance between the first historical precipitation dataset and the nearest first historical precipitation dataset in each class. Finally, we take the maximum value among the multiple average neighborhood radii as the neighborhood radius and perform subsequent density clustering.
[0038] After obtaining the neighborhood radius, we can perform density clustering based on this radius. We first obtain the number of first friendly neighbors, which is generally determined by twice the data dimension. For example, if there are N data points in the dataset, then this number is 2N.
[0039] When performing density clustering, we first select the first historical precipitation datasets from multiple first historical precipitation datasets whose number of first historical precipitation datasets within their neighborhood radius is greater than the number of first friendly neighbors, as the initial dataset (the dataset with the number of datasets within its own radius greater than the number of first friendly neighbors is the initial dataset). Then, we iteratively extract the dataset from the multiple unclustered initial datasets and perform the following steps after extraction: Centered on the extracted origin dataset, searching for the first unclustered historical precipitation dataset within the neighborhood radius of the origin dataset will result in two cases: If a first historical precipitation dataset that is not clustered is found within the neighborhood radius, the found dataset is added to the class of the originating dataset. The found dataset is then used as the originating dataset, and the process of searching for a first historical precipitation dataset that is not clustered within the neighborhood radius of the originating dataset is repeated.
[0040] If all initial datasets are clustered, the first historical precipitation dataset that is not clustered will be marked as not clustered.
[0041] In this way, we obtain multiple first precipitation classes, and possibly multiple datasets that cannot be clustered.
[0042] In step 103, the plurality of first historical precipitation datasets are filtered according to the first clustering result, and a plurality of first precipitation classes are obtained according to the filtering result and the first clustering result.
[0043] In some implementations, the first clustering result includes multiple second precipitation classes and multiple first historical precipitation datasets labeled with unclustered identifiers; Delete the first historical precipitation dataset with unclustered labels; The second precipitation class with the most data sets in the class is used as the reference class; The number of thresholds is determined based on the first proportional threshold and the number of datasets in the reference class; Delete the second precipitation class whose number of datasets in the class is less than the threshold number; The remaining multiple second precipitation categories are treated as the multiple first precipitation categories.
[0044] For example, such as Figure 2 As shown, after the aforementioned clustering steps are completed, multiple classes 201 will be obtained. There may also be some datasets 202 that cannot be clustered. These datasets are marked as unclustered. Therefore, in terms of data deletion, this invention first deletes the unclustered ones, that is, those that have not been clustered by density clustering. Then, the precipitation class, which has a significantly different proportion from the largest class in the clustering results, is also deleted. In this way, the abnormal data is cleaned up.
[0045] In step 104, for each first precipitation category, a precipitation data anomaly identification model is constructed, wherein the precipitation data anomaly identification model determines the anomaly index based on the input precipitation dataset through data dimension transformation.
[0046] In some implementations, a base model and a plurality of first coefficient arrays are obtained, wherein the base model has a plurality of first coefficients, and the number of coefficients in the first coefficient arrays is the same as the number of the plurality of first coefficients; Substituting the multiple first coefficient arrays into the basic model respectively, multiple process models are obtained; For each process model, the dataset in the first precipitation class is substituted into the process model, and the mean of the multiple deviation indices is used as the model deviation index. If the iteration threshold is not reached, the multiple first coefficient arrays are adjusted according to multiple model deviation indices; Jump to the step of substituting the multiple first coefficient arrays into the basic model to obtain multiple process models; If the iteration threshold is reached, the process model with the smallest model deviation index will be used as the anomaly identification model. The anomaly detection threshold is determined based on the model deviation index of the anomaly detection model.
[0047] In some implementations, the basic model is:
[0048] In the formula, For compression vectors, These are the original vectors constructed from the dataset. For the restored vector, This is the compression coefficient matrix. The compression coefficient matrix is the first Line number Column elements, To restore the coefficient matrix, For the restored coefficient matrix, the first... Line number Column elements, This represents the total number of elements in the compressed vector. The total number of data points in the dataset. This is the deviation index.
[0049] In some implementations, adjusting the plurality of first coefficient arrays based on a plurality of model bias indices includes: Obtain multiple deviation index queues, where each queue corresponds to a first coefficient array; Add the model deviation index to the deviation index queue of the corresponding coefficient array; Find the index with the smallest value in each deviation index queue, and take the historical first coefficient array corresponding to the smallest index as the process optimal array; The first coefficient array corresponding to the index with the smallest model deviation index value is taken as the current optimal array; For each first coefficient array, adjustments are made according to the second formula, the process-optimal array, and the current-optimal array, wherein the second formula is:
[0050] In the formula, For the first The first coefficient array after the second adjustment One data point, For the first The first coefficient array after the second adjustment One data point, The first coefficient array in history One data point, The first adjustment factor is... The th of the current optimal array One data point, This is the second adjustment factor.
[0051] For example, in constructing the precipitation data anomaly identification model, this invention clusters the first precipitation category obtained in the aforementioned steps. Its basic model is as follows:
[0052] In the formula, For compression vectors, These are the original vectors constructed from the dataset. For the restored vector, This is the compression coefficient matrix. The compression coefficient matrix is the first Line number Column elements, To restore the coefficient matrix, For the restored coefficient matrix, the first... Line number Column elements, This represents the total number of elements in the compressed vector. The total number of data points in the dataset. This is the deviation index.
[0053] As we can see, this model has multiple coefficients, and it ultimately outputs a deviation index. The larger the index, the greater the risk of anomalies. When applying this model for anomaly detection, a threshold needs to be set; data with an output index exceeding this threshold will be considered anomalous.
[0054] In fact, the above-mentioned threshold for judging anomalies and the determination of model coefficients are carried out simultaneously.
[0055] Specifically, we initialized multiple first coefficient arrays for the aforementioned base model. We know that the number of coefficients in these first coefficient arrays is the same as the number of first coefficients in the base model (the first coefficient arrays are set for the coefficients of the base model). Substituting these first coefficient arrays into the base model, we obtain multiple process models. For each process model, we substitute the dataset from the first precipitation class into the process model. Each dataset, after substitution, yields a bias; the mean of these biases is the model bias index.
[0056] If the iteration threshold is not reached, then multiple first coefficient arrays need to be adjusted according to the model bias index.
[0057] Specifically, the construction corresponding to the first coefficient array actually has a deviation index queue. The model deviation index obtained in the previous steps is added to the deviation index queue of the corresponding coefficient array. Then, the index with the smallest value is found from each deviation index queue. The historical first coefficient array corresponding to this smallest index is the process-optimal array in the previous adjustment process of the first coefficient array.
[0058] Among multiple model bias indices, the first coefficient array corresponding to the index with the smallest value is the current optimal array.
[0059] Using these two arrays, for each of the first coefficient arrays, adjust according to the second formula:
[0060] In the formula, For the first The first coefficient array after the second adjustment One data point, For the first The first coefficient array after the second adjustment One data point, The first coefficient array in history One data point, The first adjustment factor is... The th of the current optimal array One data point, This is the second adjustment factor.
[0061] After the adjustment is completed, substitute the adjusted first coefficient arrays into the basic model and repeat the above steps to calculate the model deviation.
[0062] When the iteration threshold is reached, the process model with the smallest model deviation index can be used as the anomaly identification model. The anomaly identification threshold can be obtained by multiplying the model deviation index of the anomaly identification model by a coefficient greater than 1, for example, by multiplying by 1.2.
[0063] When applying the above model, after obtaining the precipitation dataset, when using the project constructed by the method of this invention to perform anomaly judgment, we first find the nearest dataset from multiple first historical precipitation datasets. If the distance between the two is less than the neighborhood radius (the neighborhood radius calculated in the previous steps), then the class of the nearest dataset is taken as the class of the precipitation dataset for anomaly judgment, and the precipitation dataset for anomaly judgment is substituted into the precipitation data anomaly identification model corresponding to the class. The model will output a deviation index. If the deviation index is greater than the anomaly judgment threshold, it indicates that the data is abnormal; otherwise, the data is relatively normal.
[0064] In addition, there is another situation where, among multiple first historical precipitation datasets, the nearest dataset is found, and the distance between the two is greater than the neighborhood radius (the neighborhood radius calculated in the previous steps). This indicates that the data may also have a significant risk of anomalies.
[0065] The present invention discloses a method for constructing a precipitation data anomaly identification model. First, multiple first historical precipitation datasets are acquired, each including multiple precipitation data points and multiple factors influencing the precipitation data. Then, the multiple first historical precipitation datasets are clustered to obtain a first clustering result. Next, the multiple first historical precipitation datasets are filtered based on the first clustering result, and multiple first precipitation classes are obtained based on the filtering result and the first clustering result. Finally, for each first precipitation class, a precipitation data anomaly identification model is constructed, wherein the precipitation data anomaly identification model determines an anomaly index based on the input precipitation dataset through data dimension transformation. This method does not require prior data labeling or extensive training or computation; therefore, both model construction and application consume relatively few resources. Furthermore, because data anomalies are analyzed through both clustering and model analysis, the accuracy of anomaly detection is higher, and the detection effect is guaranteed.
[0066] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0067] The following are embodiments of the apparatus of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.
[0068] Figure 3 This is a functional block diagram of the precipitation data anomaly identification model construction device provided in the embodiments of the present invention, with reference to... Figure 3 The precipitation data anomaly identification model construction device includes: a historical precipitation data acquisition module 301, a data clustering module 302, a data filtering module 303, and an anomaly identification model construction module 304, wherein: The historical precipitation data acquisition module 301 is used to acquire multiple first historical precipitation datasets, wherein the first historical precipitation datasets include multiple precipitation data and multiple factor data that affect the precipitation data; Data clustering module 302 is used to cluster the multiple first historical precipitation datasets to obtain a first clustering result; The data filtering module 303 is used to filter the plurality of first historical precipitation datasets according to the first clustering result, and obtain a plurality of first precipitation categories according to the filtering result and the first clustering result; Anomaly identification model construction module 304 is used to construct a precipitation data anomaly identification model for each first precipitation category, wherein the precipitation data anomaly identification model determines the anomaly index based on the input precipitation dataset through data dimension transformation.
[0069] Figure 4 This is a functional block diagram of the electronic device provided in an embodiment of the present invention. For example... Figure 4 As shown, the electronic device 4 in this embodiment includes a processor 400 and a memory 401, wherein the memory 401 stores a computer program 402 that can run on the processor 400. When the processor 400 executes the computer program 402, it implements the steps in the above-described methods and embodiments for constructing precipitation data anomaly identification models, for example... Figure 1 Steps 101 to 104 are shown.
[0070] For example, the computer program 402 may be divided into one or more modules / units, which are stored in the memory 401 and executed by the processor 400 to complete the present invention.
[0071] The electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The electronic device 4 may include, but is not limited to, a processor 400 and a memory 401. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 4 may also include input / output devices, network access devices, buses, etc.
[0072] The processor 400 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0073] The memory 401 can be an internal storage unit of the electronic device 4, such as a hard disk or memory. The memory 401 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 401 can include both internal and external storage units of the electronic device 4. The memory 401 is used to store the computer program 402 and other programs and data required by the electronic device 4. The memory 401 can also be used to temporarily store data that has been output or will be output.
[0074] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the aforementioned method embodiments, and will not be repeated here.
[0075] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0076] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0077] In the embodiments provided by this invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0079] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0080] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various methods and apparatus embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0081] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A method for constructing a precipitation data anomaly identification model, characterized in that, include: Multiple first historical precipitation datasets were obtained, which included multiple precipitation data and multiple factor data that influenced the precipitation data; Cluster the multiple first historical precipitation datasets to obtain the first clustering result; The plurality of first historical precipitation datasets are filtered according to the first clustering result, and a plurality of first precipitation classes are obtained according to the filtering result and the first clustering result; For each first precipitation category, a precipitation data anomaly identification model is constructed, wherein the precipitation data anomaly identification model determines the anomaly index based on the input precipitation dataset through data dimension transformation.
2. The method for constructing a precipitation data anomaly identification model according to claim 1, characterized in that, The clustering of the multiple first historical precipitation datasets yields multiple first precipitation classes, including: Get the neighborhood radius and the number of first friendly neighbors; The first historical precipitation dataset with a greater number of first historical precipitation datasets within its neighborhood radius than the number of first friendly neighbors is used as the initial dataset. Extract a dataset from multiple unclustered initial datasets as the starting dataset, and then perform the following steps after extraction: Centered on the originating dataset, search for the first unclustered historical precipitation dataset within the neighborhood radius of the originating dataset; If a first historical precipitation dataset that is not clustered is found within the neighborhood radius, the found dataset is added to the class of the originating dataset, the found dataset is used as the originating dataset, and the process jumps to the step of searching for the first historical precipitation dataset that is not clustered within the neighborhood radius of the originating dataset. If all initial datasets are clustered, then the first historical precipitation dataset that is not clustered will be marked as not clustered.
3. The method for constructing a precipitation data anomaly identification model according to claim 2, characterized in that, The neighborhood radius is determined through multiple clustering operations, including: Obtain the number of multiple first clusters; Based on the number of each first cluster, the multiple first historical precipitation datasets are clustered to obtain the second clustering result; For each clustering result, a first aggregation degree index is determined according to a first formula, where the first formula is: In the formula, The first aggregation index, The number of the first cluster, The number of first historical precipitation datasets in the class. For the first The first in the class The first historical precipitation dataset, For the first The class center of each class; Plot the first curve of the aggregation degree index as the number of clusters with the number of clusters as the horizontal axis and the first aggregation degree index as the vertical axis. Find the point with the largest curvature from the first curve graph, and take the number of the first cluster corresponding to the point with the largest curvature as the target number of clusters; The second clustering result obtained based on the target number of clusters will be used as the target clustering result; For each class in the target clustering result, the average neighborhood radius is determined, where the average neighborhood radius is the mean of multiple first neighborhood radii, and the first neighborhood radius is the distance between the first historical precipitation dataset in the class and the nearest first historical precipitation dataset. The maximum value among multiple average neighborhood radii is taken as the neighborhood radius.
4. The method for constructing a precipitation data anomaly identification model according to claim 1, characterized in that, The step involves filtering the plurality of first historical precipitation datasets based on the first clustering result, and obtaining a plurality of first precipitation classes based on the filtering result and the first clustering result, including: The first clustering result includes multiple second precipitation classes and multiple first historical precipitation datasets marked with non-clustering labels; Delete the first historical precipitation dataset with unclustered labels; The second precipitation class with the most data sets in the class is used as the reference class; The number of thresholds is determined based on the first proportional threshold and the number of datasets in the reference class; Delete the second precipitation class whose number of datasets in the class is less than the threshold number; The remaining multiple second precipitation categories are treated as the multiple first precipitation categories.
5. The method for constructing a precipitation data anomaly identification model according to any one of claims 1-4, characterized in that, For each first precipitation category, a precipitation data anomaly identification model is constructed, including: Obtain a base model and multiple first coefficient arrays, wherein the base model has multiple first coefficients, and the number of coefficients in the first coefficient arrays is the same as the number of the multiple first coefficients; Substituting the multiple first coefficient arrays into the basic model respectively, multiple process models are obtained; For each process model, the dataset in the first precipitation class is substituted into the process model, and the mean of the multiple deviation indices is used as the model deviation index. If the iteration threshold is not reached, the multiple first coefficient arrays are adjusted according to multiple model deviation indices; Jump to the step of substituting the multiple first coefficient arrays into the basic model to obtain multiple process models; If the iteration threshold is reached, the process model with the smallest model deviation index will be used as the anomaly identification model. The anomaly detection threshold is determined based on the model deviation index of the anomaly detection model.
6. The method for constructing a precipitation data anomaly identification model according to claim 5, characterized in that, The basic model is as follows: In the formula, For compression vectors, These are the original vectors constructed from the dataset. For the restored vector, This is the compression coefficient matrix. The compression coefficient matrix is the first Line 1 Column elements, To restore the coefficient matrix, For the restored coefficient matrix, the first... Line 1 Column elements, This represents the total number of elements in the compressed vector. The total number of data points in the dataset. This is the deviation index.
7. The method for constructing a precipitation data anomaly identification model according to claim 6, characterized in that, The adjustment of the plurality of first coefficient arrays based on multiple model deviation indices includes: Obtain multiple deviation index queues, where each queue corresponds to a first coefficient array; Add the model deviation index to the deviation index queue of the corresponding coefficient array; Find the index with the smallest value in each deviation index queue, and take the historical first coefficient array corresponding to the smallest index as the process optimal array; The first coefficient array corresponding to the index with the smallest model deviation index value is taken as the current optimal array; For each first coefficient array, adjustments are made according to the second formula, the process-optimal array, and the current-optimal array, wherein the second formula is: In the formula, For the first The first coefficient array after the second adjustment One data point, For the first The first coefficient array after the second adjustment One data point, The first coefficient array in history One data point, The first adjustment factor is... The th of the current optimal array One data point, This is the second adjustment factor.
8. A device for constructing a precipitation data anomaly identification model, characterized in that, For implementing the precipitation data anomaly identification model construction method as described in any one of claims 1-7, the precipitation data anomaly identification model construction device comprises: The historical precipitation data acquisition module is used to acquire multiple first historical precipitation datasets, which include multiple precipitation data and multiple factor data that affect the precipitation data; The data clustering module is used to cluster the multiple first historical precipitation datasets to obtain the first clustering result; The data filtering module is used to filter the plurality of first historical precipitation datasets according to the first clustering result, and to obtain a plurality of first precipitation categories according to the filtering result and the first clustering result; as well as, An anomaly identification model construction module is used to construct a precipitation data anomaly identification model for each first precipitation category, wherein the precipitation data anomaly identification model determines the anomaly index based on the input precipitation dataset through data dimension transformation.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7 above.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7 above.