Interpolation method for sparse biased data

Abstract

The invention discloses an interpolation method for sparse biased data, belongs to the technical field of data processing, and can solve the problem of low interpolation precision of an existing interpolation method. The method comprises the following steps: S1, constructing a first adjacent matrix and a second adjacent matrix between point locations according to sampling information of sampling points of a research area; the information transmission rules of the first adjacent matrix and the second adjacent matrix are different; s2, constructing an initial model, training the initial model by using the sampling information of the sampling points and the first adjacent matrix to obtain an intermediate model, and training the intermediate model by using the sampling information of the sampling points and the second adjacent matrix to obtain a prediction model; and S3, inputting the coordinates of the to-be-interpolated points in the research area and the sampling information of the sampling points into the prediction model to obtain predicted values of the to-be-interpolated points. The method is used for interpolation of sparse biased data.

Description

Technical Field

[0001] The invention relates to an interpolation method for sparse biased data, and belongs to the technical field of data processing. Background Art

[0002] At present, the long-term accumulation of waste discharged by high-energy-consuming and high-pollution industrial equipment in the soil has caused serious soil pollution. It is of great practical significance to clarify the spatial distribution of typical pollutants in the site and accurately define the boundary and scope of pollution.

[0003] Due to exploration costs and land use restrictions, the actual available drilling data are often sparse and biased, making it difficult for traditional methods to accurately describe the three-dimensional distribution of polluted areas. At present, traditional spatial interpolation methods are mainly statistical methods based on design (such as IDW) and model (such as Kriging). However, IDW relies on a specific sampling design, and the accuracy of its interpolation results is easily affected by the sample size and soil sample design. The Kriging model describes the spatial autocorrelation between two sampling points by establishing a variogram, and then predicts unknown points. However, the smoothing effect of this method often cannot capture the changes in pollutants at sparse drilling points that are less than the sampling distance. In addition, it is difficult to construct a suitable semi-variogram due to the location of the pollution source, the nature of the pollutants themselves, soil characteristics (such as soil texture, porosity, permeability, etc.) and human activities (such as production processes, functional area distribution, pollution leakage), which ultimately leads to reduced accuracy of the interpolation results. Summary of the invention

[0004] The present invention provides a sparse biased data interpolation method, which can solve the problem of low interpolation accuracy of existing interpolation methods.

[0005] The present invention provides a sparse biased data interpolation method, the method comprising: S1. Constructing a first adjacency matrix and a second adjacency matrix between points according to sampling information of sampling points in the study area; the information transmission rules of the first adjacency matrix and the second adjacency matrix are different; S2, constructing an initial model, training the initial model using the sampling information of the sampling points and the first adjacency matrix to obtain an intermediate model, and training the intermediate model using the sampling information of the sampling points and the second adjacency matrix to obtain a prediction model; S3. Input the coordinates of the points to be interpolated and the sampling information of the sampling points in the study area into the prediction model to obtain the predicted values ​​of the points to be interpolated.

[0006] Optionally, the sampling points are divided into training nodes and testing nodes; The information transmission rule of the first adjacency matrix is: training nodes are only allowed to transmit information to each other and to test nodes; test nodes are only allowed to receive information from training nodes; The information transmission rule of the second adjacency matrix is: information transmission is allowed between training nodes, and information transmission to test nodes and themselves is also allowed; test nodes are allowed to receive information from training nodes, and are also allowed to transmit information to training nodes and themselves.

[0007] Optionally, the S1 specifically includes: Construct a weighted adjacency matrix between points based on the sampling information of the sampling points in the study area; According to different information transmission rules, the weighted adjacency matrix is ​​divided into a first adjacency matrix and a second adjacency matrix.

[0008] Optionally, the weighted adjacency matrix between points is constructed according to the sampling information of the sampling points in the study area, specifically: A distance matrix between points is constructed according to the sampling information of the sampling points in the study area, and the distance matrix is ​​converted into a weighted adjacency matrix.

[0009] Optionally, the constructing of the initial model in S2 specifically includes: Determine an initial weight matrix and an initial edge mask matrix according to sampling information of the sampling points; An initial model is constructed according to the initial weight matrix and the initial edge mask matrix.

[0010] Optionally, the step S2 uses the sampling information of the sampling points and the first adjacency matrix to train the initial model to obtain an intermediate model, specifically including: Performing function activation processing and symmetrization processing on the initial edge mask matrix to obtain an updated edge mask matrix; An updated weight matrix is ​​determined according to sampling information of sampling points, the updated edge mask matrix, the first adjacency matrix and the initial weight matrix; the updated edge mask matrix and the updated weight matrix constitute an intermediate model.

[0011] Optionally, determining an updated weight matrix according to sampling information of the sampling points, the updated edge mask matrix, the first adjacency matrix and the initial weight matrix specifically includes: Multiplying the updated edge mask matrix by the first adjacency matrix to obtain a first mask adjacency matrix; The first mask adjacency matrix is ​​multiplied by the initial weight matrix, and the multiplication result is multiplied by the information matrix of the sampling points to obtain an updated weight matrix.

[0012] Optionally, the step S2 uses the sampling information of the sampling points and the second adjacency matrix to train the intermediate model to obtain a prediction model, specifically including: Multiplying the updated edge mask matrix by the second adjacency matrix to obtain a second mask adjacency matrix; The second mask adjacency matrix is ​​multiplied by the updated weight matrix, and the multiplication result is multiplied by the information matrix of the sampling points to obtain a trained weight matrix; the updated edge mask matrix and the trained weight matrix constitute a prediction model.

[0013] Optionally, after S3, the method further includes: S4, inputting the coordinates of the sampling points in the study area into the prediction model to obtain the predicted values ​​of the sampling points, and determining the residual values ​​of the sampling points according to the sampling information and the predicted values ​​of the sampling points; S5. Correct the predicted value of the point to be interpolated using the residual value.

[0014] Optionally, the S5 specifically includes: Determine the residual prediction value of the point to be interpolated according to the residual value; The predicted value of the point to be interpolated is corrected using the residual predicted value to obtain a corrected prediction result of the point to be interpolated.

[0015] The beneficial effects that the present invention can produce include: The interpolation method for sparse biased data provided by the present invention constructs two adjacency matrices with different information transfer rules through the spatial autocorrelation of sampling points, and performs model training based on the two adjacency matrices in sequence to dynamically capture the concentration information of the sampling points. The trained prediction model has high interpolation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flow chart of a method for interpolating sparse biased data provided by an embodiment of the present invention; Figure 2 A schematic diagram of information transmission of nodes in the first convolutional layer during the training process provided by an embodiment of the present invention; Figure 3 A schematic diagram of information transmission of nodes in the first convolutional layer during the test process provided by an embodiment of the present invention; Figure 4 A first adjacency matrix of nodes in the first convolutional layer during the training process provided by an embodiment of the present invention; Figure 5 A schematic diagram of information transmission of nodes in the second convolutional layer during the training process provided by an embodiment of the present invention; Figure 6 A schematic diagram of information transmission of nodes in the second convolutional layer during the test process provided by an embodiment of the present invention; Figure 7 A second adjacency matrix of nodes in the second convolutional layer during the training process provided by an embodiment of the present invention; Figure 8 A schematic diagram of a process for creating a three-dimensional grid of points to be interpolated provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention is described in detail below in conjunction with embodiments, but the present invention is not limited to these embodiments.

[0018] The embodiment of the present invention provides a method for interpolation of sparse biased data, such as Figure 1 As shown, the method includes: S1. Constructing a first adjacency matrix and a second adjacency matrix between points according to sampling information of sampling points in the study area; the information transmission rules of the first adjacency matrix and the second adjacency matrix are different.

[0019] Specifically include: (1) Construct a weighted adjacency matrix between sampling points based on the sampling information of the study area.

[0020] Specifically, a distance matrix between points is constructed based on the sampling information of the sampling points in the study area, and the distance matrix is ​​converted into a weighted adjacency matrix.

[0021] In practical applications, the weighted adjacency matrix can be constructed according to the following steps.

[0022] Step 1: Determine the study area. Based on the grid point method (e.g., grid interval, length × width is 3 × 3m or length × width × height is 3 × 3 × 3m), the spatial location of the sample points is arranged (the points are arranged at the center of the grid), and the sample collection and analysis are strictly carried out in accordance with the requirements of the "Technical Specifications for Soil Environmental Monitoring (HJ / T166-2004)".

[0023] Step 2: Sampling and collection of pollutant data (polluting elements such as arsenic, beryllium, cadmium, chromium, copper, lead, mercury, nickel, zinc, tin, cyanide, fluoride, polycyclic aromatic hydrocarbons, etc.; point coordinates such as X, Y, Z; concentration values; ID numbers and other related data) to form sampling information of sampling points in the study area.

[0024] Step 3: Construct a weighted adjacency matrix. Calculate the distance between any two points and construct a distance matrix between the points. Convert the distance matrix into a weighted adjacency matrix, where the weight is the square of the inverse of the distance, and the weight of the point to itself is "0". Example: The distance matrix is , calculate the square of the reciprocal of the values ​​in the distance matrix, that is is 0.25, and the weighted adjacency matrix is The corresponding number data can also be obtained as .

[0025] (2) According to different information transmission rules, the weighted adjacency matrix is ​​divided into a first adjacency matrix and a second adjacency matrix.

[0026] Among them, the sampling points are divided into training nodes and testing nodes; The information transmission rule of the first adjacency matrix is: training nodes are only allowed to transmit information to each other and to test nodes; test nodes are only allowed to receive information from training nodes; The information transmission rule of the second adjacency matrix is: training nodes are allowed to transmit information to each other, and to test nodes and themselves; test nodes are allowed to receive information from training nodes, and to transmit information to training nodes and themselves.

[0027] Specifically, the first adjacency matrix information transmission rule: The initial value of the training node is derived from the actual sampling value, so the training node can pass the concentration information to the adjacent nodes (including the training node and the test node). However, the self-loop will weaken the information transmitted by other training nodes, so the information should not be passed to itself. In addition, the initial value of the test node is set to 0, which is an assumed value. Therefore, the test node can only receive concentration information from the adjacent node (training node), and cannot pass the concentration information to the adjacent nodes (including the training node and the test node), nor to itself.

[0028] Second adjacency matrix information transmission rule: The initial values ​​of training nodes and test nodes are derived from the aggregated values ​​obtained after training in the previous layer. Therefore, both training nodes and test nodes can transmit or receive concentration information from adjacent nodes, and can also transmit concentration information to themselves, thus ensuring that the aggregated values ​​from the previous layer are not ignored.

[0029] refer to Figures 2 to 7 As shown, for example, the training node is: X 1 , X 2 , X 4 , X 5 , X 6 The test node is: X 3 , X 7 . Figure 2 and Figure 3 Medium X 1 , X 2 , X 4 , X 5 , X 6 The value of X is the actual observed value. 3 , X 7 The initial value of is 0. Figure 5 and Figure 6 The center represents the aggregated value obtained after training the previous layer. The lines with arrows represent information flow. Figure 2 and Figure 5 Represents the training process. Figure 3 and Figure 6 Indicates the testing process. Figure 4 and Figure 7 Represents the weight change of the weighted adjacency matrix. Figure 2 , Figure 3 and Figure 4 Represents the information transfer of the previous layer (i.e. the first convolutional layer). Figure 5 , Figure 6 and Figure 7 Represents the information transfer of the current layer (i.e., the second convolutional layer).

[0030] S2. construct an initial model, train the initial model using the sampling information of the sampling points and the first adjacency matrix to obtain an intermediate model, and train the intermediate model using the sampling information of the sampling points and the second adjacency matrix to obtain a prediction model.

[0031] The initial model constructed in S2 specifically includes: Firstly, the initial weight matrix and the initial edge mask matrix are determined according to the sampling information of the sampling points; then the initial model is constructed according to the initial weight matrix and the initial edge mask matrix.

[0032] Construct the network structure of the initial model: adaptive layer (i.e., mask layer), the mask layer consists of weight matrix and edge mask matrix. Both weight matrix and edge mask matrix are created by Python's torch.nn package. The dimension of the former is determined by the sampling information (i.e., concentration information) of the input sampling points and the set output dimension, while the dimension of the latter is determined by the input concentration information. Specifically, the weight matrix and edge mask matrix in the network structure are initialized according to the sampling point concentration information and node dimension to obtain the initial weight matrix and initial edge mask matrix, both of which are normalized by Python's torch.nn package.

[0033] Then set the hyperparameters of the initial model, such as Epoch, learning rate, weight decay, node dimension, etc.

[0034] In S2, the initial model is trained using the sampling information of the sampling points and the first adjacency matrix to obtain an intermediate model, which specifically includes: (1) Perform function activation and symmetry processing on the initial edge mask matrix to obtain the updated edge mask matrix.

[0035] (2) Determine an updated weight matrix based on the sampling information of the sampling points, the updated edge mask matrix, the first adjacency matrix and the initial weight matrix; the updated edge mask matrix and the updated weight matrix constitute an intermediate model.

[0036] Specifically: first, the updated edge mask matrix is ​​multiplied by the first adjacency matrix to obtain the first mask adjacency matrix; then, the first mask adjacency matrix is ​​multiplied by the initial weight matrix, and the multiplication result is multiplied by the information matrix of the sampling point to obtain the updated weight matrix.

[0037] In practical applications, the first adjacency matrix is ​​masked using the updated edge mask matrix, that is, the updated edge mask matrix is ​​multiplied by the first adjacency matrix to obtain the first mask adjacency matrix; then the first mask adjacency matrix can be normalized, the normalized first mask adjacency matrix is ​​multiplied by the initial weight matrix, and the multiplication result is multiplied by the information matrix of the sampling points to obtain the updated weight matrix.

[0038] It should be noted that the above step can be repeated according to the number of mask layers set, and the updated weight matrix obtained after each cycle is saved. The multiple updated weight matrices obtained after the cycle are regularized using the F.dropout function in the PyTorch framework to obtain the final updated weight matrix of the first convolutional layer.

[0039] In S2, the sampling information of the sampling points and the second adjacency matrix are used to train the intermediate model to obtain a prediction model, which specifically includes: First, the updated edge mask matrix is ​​multiplied by the second adjacency matrix to obtain the second mask adjacency matrix; then the second mask adjacency matrix is ​​multiplied by the updated weight matrix, and the multiplication result is multiplied by the information matrix of the sampling point to obtain the trained weight matrix, that is, the weight matrix of the second convolutional layer; the updated edge mask matrix and the trained weight matrix constitute the prediction model.

[0040] In practical applications, the intermediate model and the prediction model can be trained cyclically according to the specific Epoch value to obtain the final prediction model. After training, the loss and accuracy results of the prediction model are calculated, and the corresponding curves are drawn. The results of model training are verified using the leave-one-out cross-validation method, the loss and accuracy results of the prediction model are calculated, and the corresponding curves are drawn. When the model training and verification results are consistent with the actual situation, the model can be predicted. When the results of model training and verification are not ideal, reset the hyperparameters and retrain.

[0041] It should be noted that the sampling points can be divided into multiple data according to the depth value (Z) of the sampling points (such as Z=14m, that is, divided into 14 data, such as: 0~1m is the first sampling data, 1~2m is the second sampling data,..., 13~14m is the fourteenth sampling data), and according to the leave-one-out verification method, each piece of data is trained and the data point is marked, and the sampling data is normalized. According to the divided data, the model training is carried out in sequence to obtain the prediction model of the corresponding depth value.

[0042] S3. Input the coordinates of the points to be interpolated and the sampling information of the sampling points in the study area into the prediction model to obtain the predicted values ​​of the points to be interpolated.

[0043] refer to Figure 8 As shown, based on the sampling point coordinate range, three-dimensional grid point coordinates (ie, coordinates of the points to be interpolated) are created, and the initial value "0" and the corresponding number are set for the coordinates of the points to be interpolated. Figure 8 In the 0~1m depth range, V 1 、V 2 、V 3 、V 4 and V 5 is the sampling point; V 6 and V 7 is the point to be interpolated.

[0044] By inputting the sampling information of the sampling point and the coordinates of the point to be interpolated into the trained prediction model, the predicted value of the point to be interpolated, that is, the predicted concentration value of the point to be interpolated, can be obtained.

[0045] After S3, the method further includes: S4. Input the coordinates of the sampling points in the study area into the prediction model to obtain the predicted values ​​of the sampling points, and determine the residual values ​​of the sampling points based on the sampling information and the predicted values ​​of the sampling points.

[0046] The sampling point data is input into the trained prediction model to obtain the predicted value of the sampling point concentration. Then, the residual value of the sampling point is calculated based on the actual value and the predicted value of the sampling point.

[0047] S5. Correct the predicted value of the point to be interpolated using the residual value.

[0048] Specifically include: First, the residual prediction value of the point to be interpolated is determined according to the residual value; then the prediction value of the point to be interpolated is corrected using the residual prediction value to obtain the corrected prediction result of the point to be interpolated.

[0049] The residual value of the sampling point can be input into the trained prediction model to obtain the residual prediction value of the point to be interpolated. It can also be input into Kriging or other interpolation methods to obtain the residual prediction value of the point to be interpolated. The embodiment of the present invention does not limit this.

[0050] The residual prediction value of the point to be interpolated is added to the previously obtained prediction value of the point to be interpolated to obtain the corrected prediction result of the point to be interpolated.

[0051] The advantages of the present invention mainly lie in the following points: Advantage 1: The dual-mode adaptive learning sample feature mechanism deeply mines the feature information of sparse and biased data. First, the transmission rules of sample information are optimized, and different adjacency matrices are constructed based on spatial autocorrelation to dynamically capture sample concentration information. Secondly, the sample structure is dynamically learned, and a dynamic mask matrix and full parameter learning method are constructed to dynamically optimize the weight value of the "edge" (the nonlinear relationship between sample points) in a data-driven manner, making full use of the limited sample spatial distribution data to adapt to the dynamic characteristics of the spatial structure of pollutants.

[0052] Advantage 2: The residual mechanism improves the scalability of the model. The prediction value of the interpolation point and the residual value result of the sampling point obtained by the prediction model of the present invention are interpolated again based on the residual value result of the sampling point, and finally combined with the prediction result obtained by the first interpolation to obtain the final interpolation prediction result. The present invention utilizes the residual mechanism to reduce the impact of outliers on the model, and can be combined with the results of other interpolation methods to obtain the final prediction result, thereby improving the accuracy of the final prediction result.

[0053] Advantage 3: No auxiliary variable information is required. Only by using the three-dimensional point and sampling point pollutant concentration information, a high-precision three-dimensional spatial distribution pattern of pollutants can be achieved.

[0054] The above are only a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A method for interpolation of sparse biased data, characterized in that: The method comprises: S1. Constructing a first adjacency matrix and a second adjacency matrix between points according to sampling information of sampling points in the study area; the information transmission rules of the first adjacency matrix and the second adjacency matrix are different; S2, constructing an initial model, training the initial model using the sampling information of the sampling points and the first adjacency matrix to obtain an intermediate model, and training the intermediate model using the sampling information of the sampling points and the second adjacency matrix to obtain a prediction model; S3. Input the coordinates of the points to be interpolated and the sampling information of the sampling points in the study area into the prediction model to obtain the predicted values ​​of the points to be interpolated.

2. The method according to claim 1, characterized in that The sampling points are divided into training nodes and testing nodes; The information transmission rule of the first adjacency matrix is: training nodes are only allowed to transmit information to each other and to test nodes; The test node is only allowed to receive information from the training node; The information transmission rule of the second adjacency matrix is: information transmission is allowed between training nodes, and information transmission to test nodes and itself is also allowed; The test node is allowed to receive information from the training node, and also allows to pass information to the training node and itself.

3. The method according to claim 2, characterized in that The S1 specifically includes: Construct a weighted adjacency matrix between points based on the sampling information of the sampling points in the study area; According to different information transmission rules, the weighted adjacency matrix is ​​divided into a first adjacency matrix and a second adjacency matrix.

4. The method according to claim 3, characterized in that: The weighted adjacency matrix between points is constructed based on the sampling information of the sampling points in the study area, specifically: A distance matrix between points is constructed according to the sampling information of the sampling points in the study area, and the distance matrix is ​​converted into a weighted adjacency matrix.

5. The method according to claim 1, characterized in that The construction of the initial model in S2 specifically includes: Determine an initial weight matrix and an initial edge mask matrix according to sampling information of the sampling points; An initial model is constructed according to the initial weight matrix and the initial edge mask matrix.

6. The method according to claim 5, characterized in that The step S2 uses the sampling information of the sampling points and the first adjacency matrix to train the initial model to obtain an intermediate model, which specifically includes: Performing function activation processing and symmetrization processing on the initial edge mask matrix to obtain an updated edge mask matrix; An updated weight matrix is ​​determined according to sampling information of sampling points, the updated edge mask matrix, the first adjacency matrix and the initial weight matrix; the updated edge mask matrix and the updated weight matrix constitute an intermediate model.

7. The method according to claim 6, characterized in that Determining an updated weight matrix according to the sampling information of the sampling points, the updated edge mask matrix, the first adjacency matrix and the initial weight matrix specifically includes: Multiplying the updated edge mask matrix by the first adjacency matrix to obtain a first mask adjacency matrix; The first mask adjacency matrix is ​​multiplied by the initial weight matrix, and the multiplication result is multiplied by the information matrix of the sampling points to obtain an updated weight matrix.

8. The method according to claim 6, characterized in that The step S2 uses the sampling information of the sampling points and the second adjacency matrix to train the intermediate model to obtain a prediction model, which specifically includes: Multiplying the updated edge mask matrix by the second adjacency matrix to obtain a second mask adjacency matrix; The second mask adjacency matrix is ​​multiplied by the updated weight matrix, and the multiplication result is multiplied by the information matrix of the sampling points to obtain a trained weight matrix; the updated edge mask matrix and the trained weight matrix constitute a prediction model.

9. The method according to claim 1, characterized in that: After S3, the method further includes: S4, inputting the coordinates of the sampling points in the study area into the prediction model to obtain the predicted values ​​of the sampling points, and determining the residual values ​​of the sampling points according to the sampling information and the predicted values ​​of the sampling points; S5. Correct the predicted value of the point to be interpolated using the residual value.

10. The method according to claim 9, characterized in that The S5 specifically includes: Determine the residual prediction value of the point to be interpolated according to the residual value; The predicted value of the point to be interpolated is corrected using the residual predicted value to obtain a corrected prediction result of the point to be interpolated.

Images