A method for detecting outliers in ship-borne bathymetric data taking into account the three-dimensional distribution characteristics of the data

Abstract

This invention proposes a method for detecting outliers in ship-borne bathymetric data that takes into account the three-dimensional distribution characteristics of the data, belonging to the field of marine surveying and mapping technology. This method not only performs spatial outlier detection in multi-source bathymetric data but also detects anomalous depth points by using the distance relationship between points in three-dimensional space. This overcomes the shortcomings of existing line-based outlier detection methods, which cannot detect anomalous depth points and do not consider the depth data on surrounding lines during the detection process. Experiments have demonstrated that the method described in this invention can automatically detect outliers in ship-borne bathymetric data after adaptively tuning the algorithm parameters.

Description

Technical Field

[0001] The present invention belongs to the technical field of ocean surveying and mapping, and relates to a method for detecting outliers in ship-borne bathymetric data taking into account the three-dimensional distribution characteristics of the data. Background Art

[0002] Passage-based bathymetry is a technique in which a ship uses onboard positioning and sounding equipment to collect water depth data within its track area while at sea. Unlike professional bathymetry, passage-based bathymetry is difficult to strictly adhere to relevant industry standards and regulations regarding the use of data acquisition equipment, vessel speed, and track control. This severely impacts data quality, often resulting in a high number of outliers in the collected data.

[0003] The detection of outliers in bathymetric data has always been a research hotspot in the field of marine surveying and mapping. These studies can be divided into two categories in principle: (1) bathymetric data outlier filtering methods based on survey lines, such as sequence statistical filtering and median filtering; (2) bathymetric data outlier filtering methods based on regional surfaces, such as trend surface filtering and robust estimation filtering. These methods are effective in detecting outliers in traditional professional bathymetric data. However, unlike professional bathymetric data, voyage bathymetric data is water depth data collected by ships during navigation. The voyage bathymetric data collected by different ships and different voyages show obvious irregular distribution characteristics in space, which leads to the poor effect of traditional outlier detection methods in detecting outliers in multi-source bathymetric data. In recent years, with the popularization of the concept of multi-source bathymetric data and the promotion of related technologies, some scholars have begun to explore the detection of outliers in multi-source bathymetric data. Thornton et al. analyzed the processing flow of crowdsource bathymetric data and pointed out that outlier detection and processing of crowdsource bathymetric data are crucial for subsequent data use. Sedaghat et al. combined the local outlier factor algorithm with the DBSCAN algorithm to perform spatial outlier detection on massive regionally distributed crowdsource bathymetric data sets. However, this method is inefficient and time-consuming when processing massive data, and the processing process requires manual parameter tuning, resulting in a low degree of automation. Summary of the Invention

[0004] In order to solve the problem that the existing outlier detection methods do not consider the irregular three-dimensional spatial distribution of ship-borne bathymetric data, resulting in low quality of processing results, the present invention proposes an outlier detection method for ship-borne bathymetric data that takes into account the three-dimensional distribution characteristics of the data.

[0005] The technical solution of the present invention is:

[0006] A method for detecting outliers in sea-going bathymetric data taking into account the three-dimensional distribution characteristics of the data comprises the following steps:

[0007] a. Input the spatial discrete point dataset R of the bathymetric data;

[0008] b. Extract a discrete point r from the data set R i , i=1,2,…n;

[0009] c. Calculate the discrete point r i To the remaining discrete points r in the data set R j Horizontal distance DistH (j=1,2,…,n;j≠i) i,j and vertical distance DistV i,j ;

[0010] d. Statistics satisfy DistH i,j ≤EpsH and DistV i,j Discrete point r ≤EpsV j The number of , recorded as SumPts i ; EpsH and EpsV are the horizontal distance threshold and vertical distance threshold respectively;

[0011] e. If SumPts i ≥MinPts, then mark the discrete point r i is the core point; MinPts represents the minimum number of points in the neighborhood;

[0012] f. Loop through all discrete points in the dataset R, repeat steps b to e, and mark the discrete points that meet the conditions in step e, thereby dividing the dataset R into a core point dataset R′ and a non-core point dataset R″;

[0013] g. Extract a discrete point r from the non-core point dataset R″ u ″;

[0014] h. Calculate discrete points r u ″ to each discrete point r in the core point dataset R′ v 'Horizontal distance DistH u,v and vertical distance DistV u,v ;

[0015] i. If DistH exists u,v ≤EpsH and DistV u,v ≤EpsV, then mark the discrete point r u ″ is a boundary point, otherwise it marks a discrete point r u ″ is a noise point;

[0016] j. Extract all noise points in R″, which are the outlier points of the multi-source bathymetric data.

[0017] In step c, the discrete point r is calculated i To the remaining discrete points r in the data set Rj Horizontal distance DistH i,j and vertical distance DistV i,j The method is:

[0018]

[0019] DistV i,j =|z i -z j |

[0020] Among them, (x i ,y i ,z i ) is a discrete point r i The coordinates of (x j ,y j ,z j ) is a discrete point r j 's coordinates.

[0021] In step d, the method for obtaining the values ​​of EpsH and EpsV is as follows: first, the horizontal distance DistH is calculated respectively. i,j and vertical distance DistV i,j Mean μ of the list H 、μ V and the standard deviation σ H , σ V , and then determine the threshold EpsH, EpsV value ranges are [μ H -σ H ,μ H +2σ H ]、[μ V -σ V ,μ V +2σ V ], and finally the genetic algorithm is used to find the optimal threshold. Among them, the adaptive crossover probability P of the genetic algorithm is c and mutation probability P m , the calculation formula is:

[0022]

[0023]

[0024] Where, Fit′ max is the larger fitness value of the two chromosomes selected by the crossover operation, that is, Fit′ max =max{Fit1,Fit2};Fit max is the maximum value of chromosome fitness in each generation population; avgis the average fitness value of chromosomes in the population; Fit′ is the fitness value of the chromosome selected by the mutation operation; usually the constants k1, k2∈[0.5,1], k3, k4∈[0,0.1].

[0025] The fitness function SSC of the genetic algorithm is designed as follows:

[0026]

[0027]

[0028]

[0029] Where n represents the number of all discrete points in the dataset R, k represents the number of outliers after clustering the dataset; a(m,h) is the intra-cluster dissimilarity, which calculates the average distance between the mth discrete point of the hth cluster in the dataset R and the other discrete points in the cluster; n h represents the number of discrete points in the h-th cluster; b(m,h) is the inter-cluster dissimilarity, which calculates the average distance between the m-th discrete point of the h-th cluster in the data set R and the discrete points of other clusters (each outlier is considered as a cluster for calculation); t and h represent different clusters, respectively, and T represents the total number of clusters; n t represents the number of discrete points in the tth cluster; and Represent the spatial coordinates of the mth discrete point and the qth discrete point of the hth cluster respectively; represents the spatial coordinates of the pth discrete point of the tth cluster; ||·|| represents the distance between two discrete points; SSC∈[-1,1] and the closer this value is to 1, the more reasonable the classification is.

[0030] In step e, the method for determining the value of MinPts is as follows: first, the minimum number of points in the initial neighborhood, MinPts0, is set to 4 by the traditional parameter selection method, and then combined with the filter number 5 set in the existing filtering method, the value range of the minimum number of points in the neighborhood is obtained to be [3, 6], and finally the optimal MinPts value is found by the genetic algorithm.

[0031] The beneficial effects of the present invention are as follows: the present invention can not only complete the spatial detection of outliers in multi-source bathymetric data, but also detect abnormal water depth points through the distance relationship between points in three-dimensional space, overcoming the shortcomings of the existing line-based outlier detection method in that it cannot detect abnormal water depth points in the detection process and cannot take into account the water depth data on the surrounding survey lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a main flow chart of the method for detecting outliers in sea-going bathymetric data taking into account the three-dimensional distribution characteristics of the data, as described in the present invention. DETAILED DESCRIPTION

[0033] The following examples and drawings further explain the specific embodiments of the present invention, but are not intended to limit the present invention.

[0034] The present invention proposes a method for detecting outliers in cross-sea bathymetric data that takes into account the three-dimensional distribution characteristics of the data. The implementation process is to use a computer to automatically calculate the spatial position relationship of each data point in the cross-sea bathymetric data set to automatically detect data outliers. The method includes the following steps: Figure 1 As shown:

[0035] a. Input spatial discrete point dataset of the bathymetric data R = {r1, r2, ..., r n}, where r i (i=1,2,…n) represents the i-th bathymetric data point;

[0036] b. Extract a discrete point r from the data set R i =(x i ,y i ,z i );

[0037] c. Calculate the discrete point r i To the remaining discrete points r in the data set R j Horizontal distance (j=1,2,…,n;j≠i) and vertical distance DistV i,j =|z i -z j |;

[0038] d. Statistics satisfy discrete points r i Horizontal distance DistH i,j ≤ horizontal distance threshold EpsH and vertical distance DistV i,j The number of points ≤ the vertical distance threshold EpsV, recorded as SumPts i ;

[0039] e. If SumPts i ≥MinPts, then mark the discrete point r i is the core point; MinPts represents the minimum number of points in the neighborhood;

[0040] f. Loop through all discrete points in the dataset R, repeat steps b to e, and mark the discrete points that meet the conditions in step e, thereby dividing the dataset R into two datasets: one is the core point dataset R′ consisting of the marked core points, and the other is the non-core point dataset R″ consisting of the remaining points;

[0041] g. Extract a discrete point r from the non-core point dataset R″ u ″;

[0042] h. Calculate discrete points r u ″ to each discrete point r in the core point dataset R′ v 'Horizontal distance DistH u,v and vertical distance DistV u,v ;

[0043] i. If DistH exists u,v ≤EpsH and DistV u,v ≤EpsV, then mark the discrete point r u ″ is a boundary point, otherwise it marks a discrete point r u ″ is a noise point.

[0044] j. Extract all noise points in R″, which are the outlier points of the multi-source bathymetric data.

[0045] In step d, the method for determining the values ​​of the horizontal distance threshold EpsH and the vertical distance threshold EpsV is:

[0046] First, the horizontal distance DistH is counted separately i,j and vertical distance DistV i,j The mean horizontal distance μ H , vertical distance mean μ V , horizontal distance standard deviation σ H , vertical distance standard deviation σ V .

[0047]

[0048]

[0049]

[0050]

[0051] Then, the threshold values ​​EpsH and EpsV are determined to have a range of [μ H -σ H ,μ H +2σ H ]、[μ V -σ V ,μ V +2σ V ].

[0052] Finally, the adaptive genetic algorithm is used to find the optimal threshold. Among them, the adaptive crossover probability P of the genetic algorithm is c and mutation probability Pm , the calculation formula is:

[0053]

[0054]

[0055] Where Fit′ max is the larger fitness value of the two chromosomes selected by the crossover operation, that is, Fit′ max =max{Fit1,Fit2};Fit max is the maximum value of chromosome fitness in each generation population; avg is the average fitness value of chromosomes in the population; Fit′ is the fitness value of the chromosome selected by the mutation operation; usually the constants k1, k2∈[0.5,1], k3, k4∈[0,0.1].

[0056] The fitness function SSC of the genetic algorithm is designed as follows:

[0057]

[0058]

[0059]

[0060] Where n represents the number of all discrete points in the dataset R, k represents the number of outliers after clustering the dataset; a(m,h) is the intra-cluster dissimilarity, which calculates the average distance between the mth discrete point of the hth cluster in the dataset R and the other discrete points in the cluster; n h represents the number of discrete points in the h-th cluster; b(m,h) is the inter-cluster dissimilarity, which calculates the average distance between the m-th discrete point of the h-th cluster in the data set R and the discrete points of other clusters (each outlier is considered as a cluster for calculation); t and h represent different clusters, respectively, and T represents the total number of clusters; n t represents the number of discrete points in the tth cluster; and Represent the spatial coordinates of the mth discrete point and the qth discrete point of the hth cluster respectively; represents the spatial coordinates of the pth discrete point of the tth cluster; ||·|| represents the distance between two discrete points; SSC∈[-1,1] and the closer this value is to 1, the more reasonable the classification is.

[0061] In step e, the method for obtaining the value of MinPts is:

[0062] First, set the minimum number of points in the initial neighborhood, MinPts = 4;

[0063] Then, the Kalman filter method is used to set the filter number to 5, and the range of the minimum number of points in the neighborhood is [3,6];

[0064] Finally, the adaptive genetic algorithm is used to find the optimal MinPts value. When the maximum value of the SSC value is obtained, the MinPts at this time is the optimal value.

[0065] The present invention is described by way of example, and those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments may be modified to suit specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be within the scope of the present invention.

Claims

1. A method for detecting outliers in sounding data taking into account the three-dimensional distribution characteristics of the data, characterized in that: The method comprises the following steps: a. Input the spatial discrete point dataset R of the bathymetric data; b. Extract a discrete point r from the data set R i , i=1,2,…n; c. Calculate the discrete point r i To the remaining discrete points r in the data set R j Horizontal distance DistH i,j and vertical distance DistV i,j ; where j = 1, 2, ..., n; j ≠ i; d. Statistics satisfy DistH i,j ≤EpsH and DistV i,j Discrete point r ≤EpsV j The number of , recorded as SumPts i ; EpsH and EpsV are the horizontal distance threshold and vertical distance threshold respectively; e. If SumPts i ≥MinPts, then mark the discrete point r i is the core point; MinPts represents the minimum number of points in the neighborhood; f. Loop through all discrete points in the dataset R, repeat steps b to e, and mark the discrete points that meet the conditions in step e, thereby dividing the dataset R into a core point dataset R′ and a non-core point dataset R″; g. Extract a discrete point r from the non-core point dataset R″ u ″; h. Calculate discrete points r u ″ to each discrete point r in the core point dataset R′ v 'Horizontal distance DistH u,v and vertical distance DistV u,v ; i. If DistH exists u,v ≤EpsH and DistV u,v ≤EpsV, then mark the discrete point r u ″ is a boundary point, otherwise it marks a discrete point r u ″ is a noise point; j. Extract all noise points in R″, which are the outlier points of the multi-source bathymetric data.

2. The method according to claim 1, characterized in that In step c, the discrete point r is calculated i To the remaining discrete points r in the data set R j Horizontal distance DistH i,j and vertical distance DistV i,j The method is: DistV i,j =|of i -With j | Among them, (x i ,y i ,z i ) is a discrete point r i The coordinates of (x j ,y j ,z j ) is a discrete point r j 's coordinates.

3. The method according to claim 1 or 2, characterized in that In step d, the method for obtaining the values ​​of EpsH and EpsV is as follows: first, the horizontal distance DistH is calculated respectively. i,j and vertical distance DistV i,j Mean μ of the list H 、μ V and the standard deviation σ H , σ V , and then determine the threshold EpsH, EpsV value ranges are [μ H -σ H ,μ H +2σ H ]、[μ V -σ V ,μ V +2σ V ], and finally the genetic algorithm is used to find the optimal threshold; among them, the adaptive crossover probability P of the genetic algorithm is c and mutation probability P m , the calculation formula is: Where, Fit′ max is the larger fitness value of the two chromosomes selected by the crossover operation, that is, Fit′ max =max{Fit1,Fit2};Fit max is the maximum value of chromosome fitness in each generation population; avg is the average fitness value of chromosomes in the population; Fit′ is the fitness value of the chromosome selected by the mutation operation; constants k1, k2∈[0.5,1], k3, k4∈[0,0.1]; The fitness function SSC of the genetic algorithm is designed as follows: Where n represents the number of all discrete points in the dataset R, k represents the number of outliers after clustering the dataset; a(m,h) is the intra-cluster dissimilarity, which calculates the average distance between the mth discrete point of the hth cluster in the dataset R and the other discrete points in the cluster; n h represents the number of discrete points in the h-th cluster; b(m,h) is the inter-cluster dissimilarity, which calculates the average distance between the m-th discrete point of the h-th cluster in the data set R and the discrete points of other clusters; t and h represent different clusters respectively, and T represents the total number of clusters; n t represents the number of discrete points in the tth cluster; and Represent the spatial coordinates of the mth discrete point and the qth discrete point of the hth cluster respectively; represents the spatial coordinates of the pth discrete point of the tth cluster; ||·|| represents the distance between two discrete points; SSC∈[-1,1] and the closer this value is to 1, the more reasonable the classification is.

4. The method according to claim 1 or 2, characterized in that In step e, the method for determining the value of MinPts is as follows: first, the minimum number of points in the initial neighborhood, MinPts0, is set to 4, and then combined with the filter number 5 set in the existing filtering method, the value range of the minimum number of points in the neighborhood is obtained to be [3, 6], and finally the optimal MinPts value is found by the genetic algorithm.

5. The method according to claim 3, characterized in that In step e, the method for determining the value of MinPts is as follows: first, the minimum number of points in the initial neighborhood, MinPts0, is set to 4, and then combined with the filter number 5 set in the existing filtering method, the value range of the minimum number of points in the neighborhood is obtained to be [3, 6], and finally the optimal MinPts value is found by the genetic algorithm.

Images