Geochemical element content simulation and abnormal value detection method

By using a random forest model and a sliding window algorithm to process geochemical element content data, the problems of missing values ​​and outliers were solved, achieving accurate simulation and classification, improving data availability and database quality, and saving collection and analysis costs.

CN119028483APending Publication Date: 2024-11-26XINJIANG INST OF ECOLOGY & GEOGRAPHY CHINESE ACAD OF SCI +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410879483.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-02
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing technologies for processing geochemical element content data suffer from subjective and unreasonable methods for handling missing and outlier values, which affect the effectiveness and accuracy of data analysis.

Method used

We used a random forest model for machine learning to build an elemental content simulation model, and used a sliding window algorithm to identify outliers. We then combined multi-element geochemical elemental content data for simulation and classification.

Benefits of technology

It enables accurate simulation of geochemical element content data and identification of outliers, improves data availability and database construction quality, and saves on sample collection and analysis costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028483A_ABST
    Figure CN119028483A_ABST
Patent Text Reader

Abstract

The invention discloses a geochemical element content simulation and abnormal value detection method, and belongs to the technical field of data information mining, and the method comprises the steps: obtaining and preprocessing multi-element geochemical element content data, inputting the preprocessed geochemical element content data into a random forest model, and carrying out machine learning to obtain a random forest model; and establishing an element content simulation model, and performing content simulation and abnormal value detection on the original geochemical element content data set by using the element content simulation model. According to the method, the element content simulation model is constructed through the geochemical element content data and the random forest model, content simulation can be carried out on major elements and microelements, the overall change trend of the element content is reflected, the problems of missing data and wrong data in the geochemical element content data are solved, and the accuracy of the geochemical element content data is improved. Meanwhile, the cost of geochemical sample collection, transportation, multi-element content analysis and the like can be indirectly saved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data information mining, and in particular relates to a method for simulating geochemical element content and detecting abnormal values. Background Art

[0002] The purpose of producing geochemical element content data is varied, and the analytical test methods used for chemical element content analysis are also different. Therefore, there are often some records in the geochemical element content data that are higher than the upper detection limit or lower than the lower detection limit. Such records are generally referred to as censored values. In addition, due to research purposes or funding restrictions, only some important target elements are analyzed, and other elements are ignored. These unanalyzed data are generally referred to as missing values. When a large amount of original geochemical element content data is collected for certain research purposes, the phenomenon of missing values ​​and censored values ​​becomes more obvious, and the aggregated geochemical element content data appear to be very sparse.

[0003] When conducting data analysis and information mining of geochemical element content, various data analysis methods usually do not accept data containing censored or missing values, which brings great inconvenience to research work. In order to increase the availability of data, the generally used methods include replacing records below the detection limit with a certain proportion of the lower detection limit (such as 0.5 times); for records above the upper detection limit, replacing them with a certain proportion of the upper detection limit (such as 1.5 times). Although the above approach meets the requirements of data analysis methods for data organization, it is obviously an arbitrary method. In order to use a more reasonable method to replace censored values ​​or supplement missing values, the maximum likelihood method is also used for censored and missing value processing. However, this type of method still has a high degree of subjectivity.

[0004] Therefore, there is an urgent need for a widely applicable machine learning algorithm to study the problems of censored and missing values ​​that are common in geochemical element content data, explore the implicit correlation between geochemical elements, and provide a reliable technical method for simulating geochemical element content data. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for simulating geochemical element content, classifying geochemical elements and detecting outliers, which provides solutions to the problems of missing values ​​and outliers appearing in geochemical element content data sets.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a method for simulating the content of geochemical elements and detecting abnormal values, comprising the following steps:

[0007] S1: obtaining geochemical element content data and performing preprocessing to obtain preprocessed geochemical element content data;

[0008] S2: Based on the preprocessed geochemical element content data, the random forest model is used for machine learning to establish an element content simulation model;

[0009] S3: Obtain the original geochemical element content data set, and use the element content simulation model to simulate the content to obtain the geochemical element content simulation results;

[0010] S4: Based on the simulation results of geochemical element content and the original geochemical element content data set, a sliding window is used to traverse and the outliers in the original geochemical data are identified by calculation.

[0011] The beneficial effects of the present invention are as follows: the present invention constructs an element content simulation model through multi-element geochemical element content data and a random forest model, which can simulate the content of major elements and trace elements in most rocks and stream sediments, and can obtain reliable simulation results, which can reflect the overall change trend of element content to a certain extent, solve the problems of missing data and erroneous data in geochemical element content data, help improve the construction quality of geochemical element database, and indirectly save the costs of geochemical sample collection, transportation and multi-element content analysis.

[0012] Further: the specific steps of S2 include:

[0013] S201: dividing the preprocessed geochemical element content data into a training set, a test set and a validation set according to a set ratio;

[0014] S202: Use the training set and the test set to construct and test the random forest model to obtain an unverified element content simulation model;

[0015] S203: Using the verification set to verify the unverified element content simulation model to obtain a verified element content simulation model.

[0016] The beneficial effect of the above further scheme is that the present invention can use the leave-one-out method to train the random forest model, construct an element content simulation model, and can simulate the content of missing values ​​in the geochemical element content data set, and the content simulation results are accurate and reliable.

[0017] Furthermore: the specific steps of S2 also include:

[0018] A1: The pre-processed geochemical element content data is divided into a training set, a test set and a validation set according to a set ratio. The training set and the test set are combined and then divided into k sets of data, where k is a natural number greater than or equal to 2;

[0019] A2: Select one of the k pieces of data as the test data;

[0020] A3: Train the random forest model based on k-1 pieces of data except the test data to obtain an unverified element content simulation model;

[0021] A4: Based on the verification data, the unverified element content simulation model is verified to obtain a verified element content simulation model;

[0022] A5: Determine whether each of the k pieces of data has been used as test data. If so, obtain a verified element content simulation model. Otherwise, return to A2.

[0023] The beneficial effect of the above further scheme is that the present invention can use k-fold cross-validation to verify the effect of the random forest model, construct an element content simulation model, and can simulate the content of missing values ​​in the geochemical element content data set, and the content simulation results are accurate and reliable.

[0024] Further: the specific steps of S3 are as follows:

[0025] S301: Obtaining the original geochemical element content data set;

[0026] S302: Screen out geochemical element content data corresponding to missing values ​​in the original geochemical element content data set, and use it as an input data set for the element content simulation model;

[0027] S303: performing content simulation using an element content simulation model according to the input data set to obtain simulated values ​​of missing values ​​in the input data set;

[0028] S304: Combining the simulated values ​​of the missing values ​​in the input data set with the original geochemical element content data set to obtain the geochemical element content simulation results.

[0029] The beneficial effect of the above further scheme is that the present invention uses the constructed element content simulation model to process the original geochemical element content data set, which can simulate its missing values ​​and obtain the simulation results of geochemical elements with good accuracy and reliability.

[0030] Further: the specific steps of S303 are as follows:

[0031] S3031: performing content simulation through an element content simulation model according to an input data set, and obtaining output results of several decision trees in the element content simulation model;

[0032] S3032: Calculate the average value of the results output by several decision trees to obtain simulated values ​​of missing values ​​in the input data set.

[0033] The beneficial effect of the above further scheme is that through the element content simulation model, the simulation of missing values ​​in the geochemical element data set and the classification of chemical elements can be realized, which is convenient for subsequent calculation to obtain the geochemical element content classification results and geochemical element content distribution map.

[0034] Further: the specific steps of S4 are as follows:

[0035] S401: removing missing values ​​in the original geochemical element content data set, and combining the data after the missing values ​​are removed with the geochemical element content simulation results to obtain an unsorted geochemical element content data set;

[0036] S402: sorting the unsorted geochemical element content data set to obtain a sorted geochemical element content data set;

[0037] S403: traversing the sorted geochemical element content data set using a fixed-size window in a sliding manner to obtain a plurality of data subsets;

[0038] S404: According to the plurality of data subsets, the ratio of each data in each data subset to the median of the data subset is calculated, and then the process proceeds to S405;

[0039] S405: Determine whether the ratio is less than a first specified threshold value. If so, the data corresponding to the ratio is an abnormal value. Otherwise, proceed to S406.

[0040] S406: Determine whether the ratio is greater than a second specified threshold value. If so, the data corresponding to the ratio is an abnormal value. Otherwise, the ratio is normal data.

[0041] The beneficial effect of the above further scheme is that the outliers can be calculated quickly and easily by combining the original geochemical element content data set and the geochemical element content simulation results through the window sliding algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of a geochemical element content simulation and outlier detection method;

[0043] Figure 2 This is a simulation diagram of the geochemical element content. DETAILED DESCRIPTION

[0044] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0045] like Figure 1 As shown, a method for simulating geochemical element content and detecting outliers includes the following steps:

[0046] S1: obtaining geochemical element content data and performing preprocessing to obtain preprocessed geochemical element content data;

[0047] S2: Based on the preprocessed geochemical element content data, the random forest model is used for machine learning to establish an element content simulation model;

[0048] S3: Obtain the original geochemical element content data set, and use the element content simulation model to simulate the content to obtain the geochemical element content simulation results;

[0049] S4: Based on the simulation results of geochemical element content and the original geochemical element content data set, a sliding window is used to traverse and the outliers in the original geochemical data are identified by calculation.

[0050] In one embodiment of the present invention, the multi-element geochemical element content data obtained in S1 include those obtained from various global public databases, publicly published / published literature, and regional geochemical element content data sets. The preprocessing performed by S1 includes: deleting null values, 0 values, and extreme outliers contained in the obtained multi-element geochemical element content data.

[0051] In one embodiment of the present invention, in S2, based on the preprocessed geochemical element content data, the effect of the random forest model can be evaluated using the leave-one-out method or the k-fold cross-validation method to establish an element content simulation model; wherein the specific steps of the leave-one-out method include:

[0052] S201: dividing the preprocessed geochemical element content data into a training set, a test set and a validation set according to a set ratio, wherein the ratio of the validation set to the total data set can be set to 2:8 or 1:9;

[0053] S202: Use the training set and the test set to construct and test the random forest model to obtain an unverified element content simulation model;

[0054] S203: Using the verification set to verify the unverified element content simulation model to obtain a verified element content simulation model.

[0055] The specific steps of the k-fold cross-validation method include:

[0056] A1: The pre-processed geochemical element content data is divided into a training set, a test set and a validation set according to a set ratio. The training set and the test set are combined and then divided into k sets of data, where k is a natural number greater than or equal to 2;

[0057] A2: Select one of the k pieces of data as the test data;

[0058] A3: Train the random forest model based on k-1 pieces of data except the test data to obtain an unverified element content simulation model;

[0059] A4: Based on the verification data, the unverified element content simulation model is verified to obtain a verified element content simulation model;

[0060] A5: Determine whether each of the k pieces of data has been used as test data. If so, obtain a verified element content simulation model. Otherwise, return to A2.

[0061] The present invention evaluates the effect of the random forest model by leave-one-out or k-fold cross validation, and finally realizes the content simulation of the missing values ​​in the geochemical element content data set. At the same time, the content simulation results of the missing values ​​are accurate and reliable. Among them, the present invention can be determined by the coefficient of determination R 2 Or the square of the Pearson correlation coefficient between the predicted value and the reference value is used to evaluate the reliability. That is, when it is below 0.4 or 0.3, multiple different data subsets are randomly selected from the predicted data and the reference data and a line graph is drawn. The reliability and usability of the element content simulation model are qualitatively evaluated in combination with expert knowledge. The calculation expression is as follows:

[0062] R 2 =r×r

[0063]

[0064] Among them, R 2 is the coefficient of determination, r is the Pearson correlation coefficient, x i is the measured value of the i-th data in the geochemical element dataset, is the average value of all data measurements in the geochemical element dataset, y i is the predicted value of the i-th data in the geochemical element dataset, is the average value of all predicted values ​​in the geochemical element dataset, and i is the sequence number of the data in the geochemical element dataset.

[0065] In one embodiment of the present invention, the specific steps of S3 are as follows:

[0066] S301: Obtaining the original geochemical element content data set;

[0067] S302: Screen out geochemical element content data corresponding to missing values ​​in the original geochemical element content data set, and use it as an input data set for the element content simulation model;

[0068] S303: performing content simulation using an element content simulation model according to the input data set to obtain simulated values ​​of missing values ​​in the input data set;

[0069] The specific steps of S303 are as follows:

[0070] S3031: performing content simulation through an element content simulation model according to an input data set, and obtaining output results of several decision trees in the element content simulation model;

[0071] S3032: Calculate the average of the results output by several decision trees to obtain the simulated value of the missing value in the input data set

[0072] S304: combining the simulated values ​​of the missing values ​​in the input data set with the original geochemical element content data set to obtain the geochemical element content simulation results;

[0073] After obtaining the simulation results of geochemical element content, the simulation results of geochemical element content can be corresponded to the geographic space coordinates, and the geochemical element content distribution map can be drawn through the spatial interpolation method; among them, according to actual needs, various GIS software, inverse distance weighted interpolation and Kriging interpolation methods are used to draw the geochemical element content distribution map.

[0074] In one embodiment of the present invention, the specific steps of S4 are as follows:

[0075] S401: removing missing values ​​in the original geochemical element content data set, and combining the data after the missing values ​​are removed with the geochemical element content simulation results to obtain an unsorted geochemical element content data set;

[0076] S402: sorting the unsorted geochemical element content data set to obtain a sorted geochemical element content data set;

[0077] S403: traversing the sorted geochemical element content data set using a fixed-size window in a sliding manner to obtain a plurality of data subsets;

[0078] S404: According to the plurality of data subsets, the ratio of each data in each data subset to the median of the data subset is calculated, and then the process proceeds to S405;

[0079] S405: Determine whether the ratio is less than a first specified threshold value. If so, the data corresponding to the ratio is an abnormal value. Otherwise, proceed to S406.

[0080] S406: Determine whether the ratio is greater than a second specified threshold value. If so, the data corresponding to the ratio is an abnormal value. Otherwise, the ratio is normal data.

[0081] In a specific embodiment of the present invention, according to the sorted geochemical element content data set, all possible data subsets are selected by sliding through a fixed-size window, and the ratio of each data in each data subset to the median of the data subset is calculated. If the ratio is less than the first specified threshold or greater than the second specified threshold, the data corresponding to the ratio is an abnormal value; wherein the window size can be set to 10, 20 or 30, that is, the sliding window contains 10, 20 or 30 data, and the ratio of each data in each data subset to the median of the data subset is calculated by the following formula:

[0082]

[0083] Among them, a n is the ratio of the nth data in the data subset to the median, A n is the nth data in the data subset, and I is the median of the data subset;

[0084] In this embodiment, the first specified threshold is set to 0.01, and the second specified threshold is set to 100. If the ratio of the nth data in the data subset to the median is a n Less than 0.01 or a n If it is greater than 100, then the nth data A in the data subset n The outliers are checked to see if there are any input errors, and the outliers are corrected according to the actual situation to obtain a complete geochemical element content data set without outliers.

[0085] The geochemical element content simulation, geochemical element classification and outlier detection methods provided by the present invention can obtain the following Figure 2 The simulation effect diagram of geochemical element content is shown in the figure; Figure 2 Including 8 compounds, namely Na2O, MgO, Al2O3, Si2O, K2O, CaO, TiO2 and FeOT, corresponding to Figure 2The 8 sub-graphs in the figure have the horizontal axis value of simulation representing the simulated element content value, and the vertical axis measurement value representing the actual element content value. 2 is the determination coefficient, which can reflect the accuracy and reliability of the simulation. The closer its value is to 1, the more accurate the simulation result is. Figure 2 It can be seen that the present invention can fill the missing values ​​in the chemical element data set. At the same time, in most cases, the chemical content simulation results of the present invention have good accuracy and reliability. 2 For chemical elements whose concentrations are far less than 1, the element content simulation model can be trained specifically using a data set that is similar to the data source or sample type of the chemical element being simulated. At the same time, the amount of data can be increased to improve the content simulation performance of the model and enhance the accuracy and reliability of the simulation results. At the same time, it can be checked whether the poor simulation results are caused by a few extreme outliers in the modeling data or verification data.

[0086] In an embodiment of the present invention, based on the random forest model, the target variable of the element content simulation model can be set. If the target variable of the element content simulation classification model is a continuous value, the average value of the results output by several decision trees is calculated to obtain the simulation value of the missing value in the input data set, that is, the geochemical element content simulation result is finally obtained; if the target variable of the element content simulation classification model is not a continuous value, the geochemical element content classification result is obtained by probability calculation to complete the classification of high and low element contents in the geochemical element content data set.

[0087] The beneficial effects of the present invention are as follows: the present invention constructs an element content simulation model through multi-element geochemical element content data and a random forest model, which can simulate and classify the content of major elements and trace elements in most rocks and stream sediments, and can obtain reliable simulation results, which can reflect the overall change trend of element content to a certain extent, and provide a good solution to the problems of missing data and erroneous data in geochemical element content data, which helps to improve the construction quality of geochemical databases, and can indirectly save the costs of geochemical sample collection, transportation and multi-element content analysis.

Claims

1. A method for simulating geochemical element content and detecting outliers, characterized in that: The following steps are involved: S1: obtaining geochemical element content data and performing preprocessing to obtain preprocessed geochemical element content data; S2: Based on the preprocessed geochemical element content data, the random forest model is used for machine learning to establish an element content simulation model; S3: Obtain the original geochemical element content data set, and use the element content simulation model to simulate the content to obtain the geochemical element content simulation results; S4: Based on the simulation results of geochemical element content and the original geochemical element content data set, a sliding window is used to traverse and the outliers in the original geochemical data are identified by calculation.

2. The method for simulating geochemical element content and detecting outliers according to claim 1, characterized in that: The specific steps of S2 include: S201: dividing the preprocessed geochemical element content data into a training set, a test set and a validation set according to a set ratio; S202: Use the training set and the test set to construct and test the random forest model to obtain an unverified element content simulation model; S203: Using the verification set to verify the unverified element content simulation model to obtain a verified element content simulation model.

3. The method for simulating geochemical element content and detecting outliers according to claim 1, characterized in that: The specific steps of S2 also include: A1: The pre-processed geochemical element content data is divided into a training set, a test set and a validation set according to a set ratio. The training set and the test set are combined and then divided into k sets of data, where k is a natural number greater than or equal to 2; A2: Select one of the k pieces of data as the test data; A3: Train the random forest model based on k-1 pieces of data except the test data to obtain an unverified element content simulation model; A4: Based on the verification data, the unverified element content simulation model is verified to obtain a verified element content simulation model; A5: Determine whether each of the k pieces of data has been used as test data. If so, obtain a verified element content simulation model. Otherwise, return to S202.

4. The method for simulating geochemical element content and detecting outliers according to claim 1, characterized in that: The specific steps of S3 are as follows: S301: Obtaining the original geochemical element content data set; S302: Screen out geochemical element content data corresponding to missing values ​​in the original geochemical element content data set, and use it as an input data set for the element content simulation model; S303: performing content simulation using an element content simulation model according to the input data set to obtain simulated values ​​of missing values ​​in the input data set; S304: Combining the simulated values ​​of the missing values ​​in the input data set with the original chemical element content data set to obtain the geochemical element content simulation results.

5. The method for simulating geochemical element content and detecting outliers according to claim 4, characterized in that: The specific steps of S303 are as follows: S3031: performing content simulation through an element content simulation model according to an input data set, and obtaining output results of several decision trees in the element content simulation model; S3032: Calculate the average value of the results output by several decision trees to obtain simulated values ​​of missing values ​​in the input data set.

6. The method for simulating geochemical element content and detecting outliers according to claim 1, characterized in that: The specific steps of S4 are as follows: S401: removing missing values ​​in the original geochemical element content data set, and combining the data after the missing values ​​are removed with the geochemical element content simulation results to obtain an unsorted geochemical element content data set; S402: sorting the unsorted geochemical element content data set to obtain a sorted geochemical element content data set; S403: traversing the sorted geochemical element content data set using a fixed-size window in a sliding manner to obtain a plurality of data subsets; S404: According to the plurality of data subsets, the ratio of each data in each data subset to the median of the data subset is calculated, and then the process proceeds to S405; S405: Determine whether the ratio is less than a first specified threshold value. If so, the data corresponding to the ratio is an abnormal value. Otherwise, proceed to S406. S406: Determine whether the ratio is greater than a second specified threshold value. If so, the data corresponding to the ratio is an abnormal value. Otherwise, the ratio is normal data.

Citation Information

Patent Citations

  • Vibration abnormity positioning method and device

    CN111947903A

  • Geochemical variable space prediction method based on geostatistical weighted random forest

    CN114139819A

  • Method for screening and determining geochemical exploration index elements

    CN116298190A

  • Industrial data feature selection and abnormal value detection method, system, equipment and medium

    CN118194204A