A method for detecting and eliminating abnormal values ​​in route data and a navigation monitoring system

Abstract

The present invention discloses a method for detecting and eliminating outliers in route data and a navigation monitoring system. The method first obtains ship route trajectory data; then uses a predefined algorithm to detect outliers in the trajectory data and identify the abnormal data interval where the outliers are located; then interpolates the abnormal data interval where the outliers are located, deletes the abnormal data, and obtains replacement data; finally, outputs the ship route trajectory data after eliminating the outliers and draws the route trajectory on a map. It can be seen that the method of the present invention can quickly and accurately eliminate outliers in ship route trajectory data and draw an accurate route trajectory on a map; it can also be easily integrated into a navigation monitoring system to improve the utilization efficiency of navigation data, thereby realizing a simple, fast, and accurate ship navigation data analysis method.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a method for detecting and eliminating abnormal values ​​in route data and a navigation monitoring system. Background Art

[0002] In recent years, with the continuous improvement of ship navigation technology, more and more ship route data has been collected, including the ship's latitude and longitude data. However, in such a large amount of route data, there are often some outliers, which will affect the accuracy of the route track drawn on the map.

[0003] Existing techniques use a moving median outlier detection algorithm, where the moving median is the median of a fixed-length sliding window that moves as the dataset progresses. The moving median is a smoothing operation that can be used to remove high-frequency noise from a dataset. The algorithm first calculates the moving median of the dataset and then the median absolute deviation of the dataset. If the moving median of the dataset is m and the median absolute deviation is d, then any data point that satisfies the condition |xm| > 3*d is considered an outlier.

[0004] The moving median outlier detection algorithm is computationally very time-consuming for large datasets. This is because the algorithm must calculate the median for each window, which is very time-consuming for long datasets. This algorithm is also insensitive to outliers in long-term data, leading to missed outliers. Summary of the Invention

[0005] In view of the above-mentioned shortcomings, the technical problem to be solved by the present invention is: to provide a method for detecting and eliminating outliers in route data and a navigation monitoring system. This method can quickly and accurately eliminate outliers in ship route trajectory data through outlier detection and interpolation elimination, and draw an accurate route trajectory on a map.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A method for detecting and eliminating outliers in route data, comprising the following steps:

[0008] S1. Obtain ship route trajectory data;

[0009] S2. Use a predefined algorithm to detect outliers on the trajectory data and identify the abnormal data interval where the outliers are located;

[0010] S3, interpolating the abnormal data interval where the identified abnormal value is located, deleting the abnormal data, and obtaining replacement data;

[0011] S4. Output the ship route trajectory data after eliminating outliers and draw the route trajectory on the map.

[0012] Preferably, the predefined algorithm comprises the following steps:

[0013] S20, calculate adjacent data X n and data X n+1 The difference Δe n , where n is a positive integer;

[0014] S21. Calculate the difference Δe of all data n The absolute difference IΔe n Ⅰ;

[0015] S22. Calculate all absolute differences IΔe n The median of Ⅰ;

[0016] S23, using the median as the threshold;

[0017] S24, each absolute difference IΔe n Ⅰ is compared with the product of the median and k, where k is a preset value;

[0018] S25. If the absolute difference IΔe n I is greater than the product of the median and k, then the data X n+1 is an outlier, data X n+1 is an abnormal data point;

[0019] S26. Then, based on all abnormal data points, determine the abnormal data interval, and determine the abnormal data interval where each abnormal value is located.

[0020] Preferably, the determination of the abnormal data interval in S26 includes the following steps:

[0021] Traverse all abnormal data points;

[0022] Calculate adjacent abnormal data points X j and abnormal data point X i The difference Δe ji , where j and i are positive integers, and i is greater than j;

[0023] Determine the difference Δe ji Is it less than M, M = median * w * N, where w is the preset value and N is the abnormal data point X j and abnormal data point X i The number of data points between intervals;

[0024] If it is less than, then the two adjacent abnormal data points are in the same abnormal data interval;

[0025] If not less than, the abnormal data point X jis the endpoint of an abnormal data interval, and the abnormal data point X i The starting endpoint of the next abnormal data interval.

[0026] A preferred approach is to replace the abnormal data points in each abnormal data interval by linear interpolation of the data between the two end points of the abnormal data interval, and to perform linear interpolation of the data using the median.

[0027] A preferred manner is that, if the last data point is an abnormal data point, the replacement data of the last data point = the second to last data + (the second to last data - the third to last data).

[0028] Preferably, k and w can be adjusted according to specific data and desired results.

[0029] Preferably, the trajectory data includes longitude and latitude, and the abnormal values ​​of longitude and latitude are analyzed separately.

[0030] A navigation monitoring system includes a control unit and a storage unit electrically connected to the control unit; the storage unit is used to store computer programs, and the control unit is used to execute the programs stored in the storage unit to implement the above method.

[0031] After adopting the above technical solution, the beneficial effects of the present invention are:

[0032] The method for detecting and eliminating outliers in route data and the navigation monitoring system of the present invention include the following steps: obtaining ship route trajectory data; the trajectory data includes longitude and latitude, and the outliers of longitude and latitude are analyzed separately; using a predefined algorithm, detecting outliers in the trajectory data, and identifying the abnormal data interval where the outliers are located; interpolating the abnormal data interval where the outliers are located, deleting the abnormal data, and obtaining replacement data; outputting the ship route trajectory data after eliminating the outliers, and drawing the route trajectory on a map. It can be seen that the method of the present invention can quickly and accurately eliminate outliers in ship route trajectory data and draw an accurate route trajectory on a map; it can also be easily integrated into a navigation monitoring system, thereby improving the utilization efficiency of navigation data; and realizing a simple, fast, and accurate ship navigation data analysis method. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 1 is a flow chart of a method for detecting and eliminating outliers in route data according to the present invention;

[0034] Figure 2 This is a schematic diagram before longitude data processing;

[0035] Figure 3 This is a schematic diagram before longitude data processing;

[0036] Figure 4 This is a schematic diagram before latitude data processing;

[0037] Figure 5 This is a schematic diagram before latitude data processing;

[0038] Figure 6 This is a schematic diagram before route processing;

[0039] Figure 7 This is a schematic diagram of the route after processing. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] Example 1:

[0042] like Figure 1 As shown, a method for detecting and eliminating outliers in route data includes the following steps:

[0043] Step S1, obtain the ship route trajectory data; the trajectory data includes longitude and latitude, and the abnormal values ​​of longitude and latitude are respectively analyzed. Figure 2 and Figure 3 This is a schematic diagram before and after longitude data processing. Figure 4 and Figure 5 The trajectory data can be input into the ship's route trajectory data through the human-computer interaction unit, or automatically input into the ship's route trajectory data through the trajectory data detection unit.

[0044] Step S2: Detect outliers on the trajectory data using a predefined algorithm and identify the abnormal data interval where the outliers are located. The predefined algorithm includes the following steps:

[0045] S20, calculate adjacent data X n and data X n+1 The difference Δe n , where n is a positive integer;

[0046] S21. Calculate the difference Δe of all data n The absolute difference IΔe n Ⅰ;

[0047] S22. Calculate all absolute differences IΔe n The median of Ⅰ;

[0048] S23, using the median as the threshold;

[0049] S24, each absolute difference IΔe n I is compared with the product of the median and k, where k is a preset value; k can be adjusted according to the specific data and expected results.

[0050] S25. If the absolute difference IΔe n I is greater than the product of the median and k, then the data X n+1 is an outlier, data X n+1 is an abnormal data point; if the absolute difference IΔe n If I is not greater than the product of the median and k, the data is normal;

[0051] For example: Assume that the median corresponding to the longitude data is 3, Δe7 = data X7 - data X8 = -2, then IΔe7I = 2, and k is preset to 0.6. The product of the median and k is 3*0.6=1.8. Since 2 is greater than 1.8, that is, IΔe7I is greater than the product of the median and k, then data X8 is an abnormal data point.

[0052] Assume that Δe7 = data X7 - data X8 = 1.6, IΔe7I = 1.6, and preset k = 0.6, then the product of the median and k is 3*0.6 = 1.8. Since 1.6 is less than 1.8, that is, IΔe7I is less than the product of the median and k, then data X8 is a normal data point.

[0053] S26. Then, based on all abnormal data points, determine the abnormal data interval, and determine the abnormal data interval where each abnormal value is located; specifically, there may be multiple abnormal data intervals.

[0054] Step S3: interpolate the abnormal data interval where the abnormal value is identified, delete the abnormal data, obtain replacement data, and use the median to perform linear interpolation of the data.

[0055] If the last data point is an abnormal data point, the replacement data of the last data point = the second to last data + (the second to last data - the third to last data).

[0056] For example: Assume that the two endpoints of the abnormal data interval are data X7 and data X 17， The data X in this abnormal data interval 10 For abnormal data points, when the data is linearly interpolated, the data X 10 The replacement value can be X9 plus the median, or, X 10 Subtract the median, or normal value of data point X n ± Ⅰ n - 10 Ⅰ * median, where n is a positive integer from 7 to 17.

[0057] If the data X 10is the last data, then X 10 The replacement data = X9+(X9-X8).

[0058] Step S4: Output the ship route trajectory data after eliminating the outliers and draw the route trajectory on the map. Figure 6 and Figure 7 .

[0059] In step S26 of this embodiment, determining the abnormal data interval includes the following steps:

[0060] Traverse all abnormal data points;

[0061] Calculate adjacent abnormal data points X j and abnormal data point X i The difference Δe ji , where j and i are positive integers, and i is greater than j;

[0062] Determine the difference Δe ji Is it less than M, M = median * w * N, where w is the preset value and N is the abnormal data point X j and abnormal data point X i The number of data points between them; w can be adjusted according to the specific data and expected results.

[0063] If it is less than, then the two adjacent abnormal data points are in the same abnormal data interval;

[0064] If not less than, the abnormal data point X j is the endpoint of an abnormal data interval, and the abnormal data point X i The starting endpoint of the next abnormal data interval.

[0065] For example: Assume that the abnormal data points are X4, X7, X9, X 11 , X 23 , X 25 , preset w = 0.6, the median is 3;

[0066] Difference Δe 47 =X7-X4=3.3, N=7-4=3, M=3*0.6*3=3.6, 3.3 is less than 3.6;

[0067] Difference Δe 79 =X9-X7=3.2, N=9-7=2, M=3*0.6*2=3.6, 3.2 is less than 3.3;

[0068] Difference Δe 911 =X 11 -X9=3.1, N=11-9=2, M=3*0.6*2=3.6, 3.1 is less than 3.6;

[0069] Difference Δe 1123 =X 23 -X 11 =22.4, N=23-11=12, M=3*0.6*12=21.6, 22.4 is greater than 21.6;

[0070] Difference Δe 2325 =X 25 -X 23 =21.4, N=25-23=2, M=3*0.6*2=21.6, 21.4 is less than 21.6;

[0071] From the above, we can see that X4, X7, X9, X 11 In the same abnormal data interval, X 23 and X 25 In another abnormal data interval, the abnormal data point X 11 is the ending endpoint of the abnormal data interval point, and the abnormal data point X4 is the starting endpoint of the abnormal data interval point.

[0072] By adopting the method of the present invention, outliers in ship route trajectory data can be quickly and accurately eliminated, and accurate route trajectories can be drawn on a map; by adopting the method of the present invention, it can also be conveniently integrated into the ship's navigation monitoring system, thereby improving the utilization efficiency of navigation data; thus, a simple, fast and accurate ship navigation data analysis method is realized.

[0073] The data linear interpolation involved in the present invention is: for known points (x1, y1), (x2, y2), linear interpolation can calculate a straight line, and estimate the value of the unknown point through the straight line.

[0074] Example 2:

[0075] A ship navigation monitoring system includes a control unit and a storage unit electrically connected to the control unit; the storage unit is used to store computer programs, and the control unit is used to execute the programs stored in the storage unit to implement the method for detecting and eliminating abnormal values ​​in route data in embodiment 1.

[0076] Because the method of the first embodiment is integrated into the navigation monitoring system of the present invention, the efficiency of utilizing navigation data is improved, and an accurate route track can be drawn on a map.

[0077] The above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications made within the spirit and principles of the present invention, methods for detecting and eliminating abnormal values ​​in route data, and improvements to the navigation monitoring system, etc., should be included in the scope of protection of the present invention.

Claims

1. A method for detecting and eliminating outliers in route data, characterized in that: The following steps are involved: S1. Obtain ship route trajectory data; S2. Use a predefined algorithm to detect outliers on the trajectory data and identify the abnormal data interval where the outliers are located; S3, interpolating the abnormal data interval where the identified abnormal value is located, deleting the abnormal data, and obtaining replacement data; S4, outputting the ship route trajectory data after eliminating outliers, and drawing the route trajectory on the map; The predefined algorithm comprises the following steps: S20, calculate adjacent data X n and data X n+1 The difference Δe n , where n is a positive integer; S21. Calculate the difference Δe of all data n The absolute difference |Δe n |; S22. Calculate all absolute differences |Δe n The median of | S23, using the median as the threshold; S24, each absolute difference |Δe n |Compare with the product of the median and k respectively, where k is a preset value; S25. If the absolute difference |Δe n | is greater than the product of the median and k, then the data X n+1 is an outlier, data X n+1 is an abnormal data point; S26. Then, based on all abnormal data points, determine the abnormal data interval, and determine the abnormal data interval where each abnormal value is located; The determination of the abnormal data interval in S26 includes the following steps: Traverse all abnormal data points; Calculate adjacent abnormal data points X j and abnormal data point X i The difference Δe ji , where j and i are positive integers, and i is greater than j; Determine the difference Δe ji Is it less than M, M = median * w * N, where w is the preset value and N is the abnormal data point X j and abnormal data point X i The number of data points between intervals; If it is less than, then the two adjacent abnormal data points are in the same abnormal data interval; If not less than, the abnormal data point X j is the endpoint of an abnormal data interval, and the abnormal data point X i The starting endpoint of the next abnormal data interval.

2. The method for detecting and eliminating outliers in route data according to claim 1, characterized in that: The abnormal data points in each abnormal data interval are replaced by linear interpolation of the data between the two end points of the abnormal data interval, and the data are linearly interpolated using the median.

3. The method for detecting and eliminating outliers in route data according to claim 2, characterized in that: If the last data point is an abnormal data point, the replacement data of the last data point = the second to last data + (the second to last data - the third to last data).

4. The method for detecting and eliminating outliers in route data according to claim 1, characterized in that: k and w are adjusted according to the specific data and desired results.

5. The method for detecting and eliminating outliers in route data according to claim 2, characterized in that: The trajectory data includes longitude and latitude, and the outliers of longitude and latitude are analyzed separately.

6. A navigation monitoring system, characterized in that: It comprises a control unit and a storage unit electrically connected to the control unit; the storage unit is used to store computer programs, and the control unit is used to execute the programs stored in the storage unit to implement the method described in any one of claims 1 to 5.

Images