Data optimization method, device and server

Abstract

The application provides a data optimization method and device and a server, and relates to the technical field of latitude and longitude data screening, and comprises the following steps: obtaining original latitude and longitude data and ship heading information of a to-be-optimized ship sailing path; performing preliminary cleaning on abnormal data in the original latitude and longitude data to obtain standard latitude and longitude data; determining a target thinning strategy from a pre-set target thinning strategy set by using the ship heading information; selecting target latitude and longitude data from the standard latitude and longitude data according to a time period specified by the target thinning strategy; and performing thinning processing on the target latitude and longitude data by using the target thinning strategy to obtain optimized latitude and longitude data. By cleaning and thinning the latitude and longitude data, the accuracy of the latitude and longitude data can be significantly improved, and the data amount can be reduced while the data characteristics are retained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of longitude and latitude data screening, and in particular to a data optimization method, device and server. BACKGROUND

[0002] At present, the longitude and latitude data of a ship is mainly collected through a GPS terminal or a Beidou terminal. The longitude and latitude data may be disturbed in the collection process, thereby producing abnormal conditions such as drift, null value and mutation, and thus reducing the accuracy of the longitude and latitude data. In addition, the collection period of the longitude and latitude data is relatively long and the collection frequency is relatively high, so that a large amount of longitude and latitude data is generated. When the longitude and latitude data is processed through software, the processing time is relatively long, thereby affecting the user experience. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a data optimization method, device and server, which can significantly improve the accuracy of the longitude and latitude data by cleaning and thinning the longitude and latitude data, and reduce the data amount while retaining the data characteristics.

[0004] In a first aspect, an embodiment of the present application provides a data optimization method, which is applied to a data optimization system and includes the following steps: obtaining original longitude and latitude data of a to-be-optimized ship navigation path and ship heading information; preliminarily cleaning abnormal data in the original longitude and latitude data to obtain standard longitude and latitude data; determining a target thinning strategy from a pre-set thinning strategy set by using the ship heading information; selecting target longitude and latitude data from the standard longitude and latitude data according to a time period specified by the target thinning strategy; and performing thinning processing on the target longitude and latitude data by using the target thinning strategy to obtain optimized longitude and latitude data.

[0005] In an embodiment, the step of preliminarily cleaning abnormal data in the original longitude and latitude data to obtain standard longitude and latitude data includes the following steps: performing data error cleaning on the abnormal data in the original longitude and latitude data to remove blank data and original longitude and latitude data greater than a pre-set threshold range to obtain first cleaned data; performing data offset cleaning on the first cleaned data to obtain second cleaned data, and determining the second cleaned data as the standard longitude and latitude data.

[0006] In an embodiment, the step of performing data offset cleaning on the first cleaned data to obtain second cleaned data, and determining the second cleaned data as the standard longitude and latitude data includes the following steps: selecting any adjacent time data in the first cleaned data, and determining the distance of the adjacent time data as a distance ideal value; determining twice the product of the time interval of the adjacent time data and the average speed between the adjacent time data as a distance real value; and when the distance ideal value is not greater than the distance real value, determining the adjacent time data as the standard longitude and latitude data.

[0007] In an embodiment, the set of thinning strategies comprises a short-period thinning strategy and a long-period thinning strategy, and the step of determining the target thinning strategy from the set of pre-set thinning strategies using the ship heading information comprises: when the ship heading information is constant, selecting the short-period thinning strategy from the set of thinning strategies and determining the short-period thinning strategy as the target thinning strategy; and when the ship heading information changes, selecting the long-period thinning strategy from the set of thinning strategies and determining the long-period thinning strategy as the target thinning strategy.

[0008] In an embodiment, the step of thinning the target latitude and longitude data using the target thinning strategy to obtain the optimized latitude and longitude data comprises: when the target thinning strategy is the short-period thinning strategy, determining the data at the earliest time in the target latitude and longitude data as the optimized latitude and longitude data.

[0009] In an embodiment, the step of thinning the target latitude and longitude data using the target thinning strategy to obtain the optimized latitude and longitude data further comprises: when the target thinning strategy is the long-period thinning strategy, determining the average value of the ship heading and the variance of the average value within the pre-set time interval using the ship heading information; if the variance is less than a pre-set variance threshold, determining the data at the earliest time and the data at the latest time in the target latitude and longitude data as the optimized latitude and longitude data; and if the variance is not less than the pre-set variance threshold, deeply thinning the target latitude and longitude data.

[0010] In an embodiment, the step of deeply thinning the target latitude and longitude data comprises: determining a target curve between the earliest time and the latest time in the target latitude and longitude data as the target curve; obtaining the shortest distances between each point in the target latitude and longitude data and the target curve; if all the shortest distances are not greater than the ship length, determining the data at the earliest time and the data at the latest time in the target latitude and longitude data as the optimized latitude and longitude data; if the maximum value of the shortest distances is greater than the ship length, determining the target latitude and longitude data corresponding to the maximum value as the target optimized data, and dividing the pre-set time interval into a first pre-set time interval and a second pre-set time interval according to the time corresponding to the maximum value, and deeply thinning the target latitude and longitude data in the first pre-set time interval and the second pre-set time interval, respectively.

[0011] In a second aspect, the embodiments of the present application further provide a data optimization device, which is applied to a data optimization system and comprises: a data acquisition module, configured to acquire original latitude and longitude data of a to-be-optimized ship navigation path and ship heading information; a preliminary cleaning module, configured to preliminarily clean abnormal data in the original latitude and longitude data to obtain standard latitude and longitude data; a target strategy determination module, configured to determine a target thinning strategy from a pre-set thinning strategy set by using the ship heading information; a target data determination module, configured to select target latitude and longitude data from the standard latitude and longitude data according to a time period specified by the target thinning strategy; and a data thinning module, configured to perform thinning processing on the target latitude and longitude data by using the target thinning strategy to obtain optimized latitude and longitude data.

[0012] In a third aspect, the embodiments of the present application further provide a server, comprising a processor and a memory, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method of any one of the first aspect.

[0013] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores computer executable instructions, and the computer executable instructions, when invoked and executed by a processor, cause the processor to implement the method of any one of the first aspect.

[0014] The embodiments of the present application bring the following beneficial effects:

[0015] The data optimization method, device and server provided by the embodiments of the present application, after acquiring original latitude and longitude data of a to-be-optimized ship navigation path and ship heading information, preliminarily clean abnormal data in the original latitude and longitude data to obtain standard latitude and longitude data, determine a target thinning strategy from a pre-set thinning strategy set by using the ship heading information, select target latitude and longitude data from the standard latitude and longitude data according to a time period specified by the target thinning strategy, and perform thinning processing on the target latitude and longitude data by using the target thinning strategy to obtain optimized latitude and longitude data. The embodiments of the present application can significantly improve the accuracy of latitude and longitude data by cleaning and thinning the latitude and longitude data, and reduce the data amount while retaining the data characteristics.

[0016] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by the structures particularly pointed out in the description, claims and drawings.

[0017] In order to make the above objectives, characteristics and advantages of the present application more apparent and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to for details. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the specific embodiments or the prior art of the present application, the drawings required to be used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0019] Figure 1 A schematic diagram of a data optimization algorithm provided for an embodiment of the present application is shown in the figure.

[0020] Figure 2 A flowchart of a data optimization method provided for an embodiment of the present application is shown in the figure.

[0021] Figure 3 A flowchart of another data optimization method provided for an embodiment of the present application is shown in the figure.

[0022] Figure 4 A structural schematic diagram of a data optimization device provided for an embodiment of the present application is shown in the figure.

[0023] Figure 5 A structural schematic diagram of an electronic device provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the present application will be described clearly and completely below in combination with embodiments. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0025] At present, the latitude and longitude data of a ship is collected through a GPS terminal or a Beidou terminal. Due to interference conditions such as ocean weather and satellite signal coverage, the latitude and longitude data may be interfered, resulting in abnormal conditions such as drift, null value and mutation. In addition, the collection frequency of the latitude and longitude data of the ship is usually 1 Hz, and the time of a voyage of the ship can reach several months, which will generate a large amount of latitude and longitude data. Therefore, the prior art cannot correctly clean the original latitude and longitude data, which makes it difficult for application software to directly use the latitude and longitude data, such as drawing the historical track of the ship. In addition, the prior art cannot perform data thinning under the condition of preserving features of the latitude and longitude data, which causes the application software to take too long time to process the large amount of latitude and longitude data, affecting the user experience. Based on this, the data optimization method provided in the embodiments of the present application is described with reference to Figure 1The schematic diagram of one data optimization algorithm is shown, by cleaning the abnormal value of the latitude and longitude data, only normal value can be reserved, the accuracy of the latitude and longitude data can be significantly improved, and the target thinning strategy is used for thinning the latitude and longitude data under the condition of ensuring the characteristic value, so that the data amount is reduced, and the application software is facilitated to process quickly.

[0026] Referring to Figure 2 The flowchart of one data optimization method is shown, and the method mainly includes the following steps S202 to S206:

[0027] In step S202, the original latitude and longitude data of the to-be-optimized ship navigation path and the ship heading information are acquired, wherein the original latitude and longitude information is the latitude and longitude information collected by a GPS terminal or a Beidou terminal in real time, and the ship heading information is the data collected by a sensor, and the range is-180~180.

[0028] In step S204, the abnormal data in the original latitude and longitude data is preliminarily cleaned to obtain standard latitude and longitude data, wherein the standard latitude and longitude data is the latitude and longitude data after the abnormal value is removed.

[0029] In step S206, the ship heading information is used to determine a target thinning strategy from a pre-set thinning strategy set, wherein the thinning strategy set includes short-period thinning and long-period thinning, and in one embodiment, the target thinning strategy can be determined based on the ship heading information, the ship voyage time and the time span of the historical trajectory.

[0030] In step S208, the target latitude and longitude data is selected from the standard latitude and longitude data according to the time period specified by the target extraction strategy, and in one embodiment, the time period specified by the short-period thinning and the long-period thinning can be freely set according to the ship heading information, the ship voyage time and the time span of the historical trajectory.

[0031] In step S210, the target latitude and longitude data is thinned by using the target thinning strategy to obtain optimized latitude and longitude data, wherein the optimized latitude and longitude data is the thinning value corresponding to the target thinning strategy.

[0032] The above data optimization provided by the embodiment of the application can significantly improve the accuracy of the latitude and longitude data by cleaning and thinning the latitude and longitude data, and reduce the data amount while retaining the data characteristics.

[0033] For the foregoing step S204, the embodiment of the application further provides an implementation manner of data cleaning, and specific reference is made to (a) to (b) as follows:

[0034] (a) data error cleaning on abnormal data in original longitude and latitude data, blank data and original longitude and latitude data greater than a preset threshold range are removed to obtain first cleaned data, in an embodiment, for unprocessed historical longitude and latitude data in the database, first, it is checked whether the longitude data or the dimension data is empty, when it is detected that the longitude data or the dimension data is empty at any time, the longitude and dimension data at the time is deleted; second, it is checked whether the longitude data is between [-180°, 180°] and the dimension data is between [-90°, 90°], when it is detected that any longitude data or dimension data is out of the above range, the longitude and latitude data at the time is deleted to obtain the first cleaned data.

[0035] (b) data offset cleaning on the first cleaned data to obtain second cleaned data, and the second cleaned data is determined as standard longitude and latitude data, and the step includes the following (A) to (B):

[0036] (A) selecting data at each time and data of the latest position correct point in the first cleaned data, judging the correctness of the selected time, when at least one correct point exists in the selected two points, the data correctness of the other point is judged according to the algorithm, and the distance between adjacent time data is determined as the distance ideal value, in an embodiment, for the first cleaned data, the great circle route algorithm or the constant heading line algorithm is used to calculate the distance between each time ship position point and the latest longitude and latitude correct time ship position point, and the distance ideal value is determined.

[0037] (B) the product of twice the time interval between two times and the average speed (real-time acquisition of the average speed between two time data, so as to calculate the average speed between two time data) between two time data is determined as the distance real value, when the distance ideal value is not greater than the distance real value, the adjacent time data is determined as the standard longitude and latitude data, wherein the standard longitude and latitude data is the longitude and latitude data after removing data abnormal value, in an embodiment, the time interval Δt between 2 times and the average speed v are calculated at the same time, and 2*Δt*v is determined as the distance real value.

[0038] The embodiment of the application further provides an embodiment of data thinning, specifically referring to the following (1) to (3):

[0039] (1) When the ship heading information is unchanged, a short-period thinning is selected from the thinning strategy set, and the short-period thinning is determined as the target thinning strategy; when the ship heading information changes, a long-period thinning is selected from the thinning strategy set, and the long-period thinning is determined as the target thinning strategy. In an implementation, the ship turning radius is large, and it usually takes tens of minutes to complete one turn. Therefore, when the ship speed is low and the distance traveled within the thinning time interval corresponding to the thinning strategy is short, the short-period thinning is used. When the ship changes the heading, such as turning, if the short-period thinning is used, the turning feature will be lost, resulting in distortion of the ship historical trajectory drawing. At this time, the long-period thinning is used.

[0040] (2) When the target thinning strategy is the short-period thinning, the earliest time data in the target latitude and longitude data is determined as the optimized latitude and longitude data. In an implementation, when the target thinning strategy is the short-period thinning and the thinning time interval is 1 min, all latitude and longitude data in each minute (00-59) is obtained from the cleaned latitude and longitude data according to the timing period of 1 min, and the latitude and longitude data at 00 seconds is taken as the thinning value of the 1 min. If the latitude and longitude data at 00 seconds is empty, the earliest time latitude and longitude data in the minute is taken as the thinning value of the 1 min.

[0041] (3) When the target thinning strategy is the long-period thinning, the average value of the ship heading in the preset time interval and the variance of the average value are determined by using the ship heading information. If the variance is less than a preset variance threshold, the earliest time data and the latest time data in the target latitude and longitude data are determined as the optimized latitude and longitude data. In an implementation, when the target thinning strategy is the long-period thinning and the thinning time interval is 10 min, all latitude and longitude data in each 10 min (00:00-09:59) is obtained from the cleaned latitude and longitude data according to the timing period of 10 min, to obtain a data set 1. According to the time range in the data set 1, the ship heading data is obtained, the average value of the heading in each 30 s is calculated, and the variance of the above ship heading average value is calculated. When the variance is less than a preset threshold, the latitude and longitude data corresponding to the earliest time and the latest time in the data set 1 is directly selected as the thinning value of the 10 min.

[0042] In an embodiment, if the variance is not less than the preset variance threshold, the target latitude and longitude data is deeply decimated, a curve between the earliest time and the latest time in the target latitude and longitude data is determined as a target curve, the shortest distances between each point in the target latitude and longitude data and the target curve are obtained, if all the shortest distances are not greater than the ship length, the data of the earliest time and the latest time in the target latitude and longitude data is determined as the optimized latitude and longitude data, if the maximum value in the shortest distances is greater than the ship length, the target latitude and longitude data corresponding to the maximum value is determined as the target optimized data, and the preset time interval is divided into a first preset time interval and a second preset time interval according to the time corresponding to the maximum value, the target latitude and longitude data in the first preset time interval and the second preset time interval is decimated, respectively, in actual application, when the variance is not less than the preset threshold, the data set 1 is obtained by deeply decimating the earliest point and the latest point, the great circle route algorithm or the constant heading line algorithm is used to sequentially calculate the shortest distances from the remaining points in the data set to the curve formed by the two points, for the plurality of distances calculated, the maximum value is obtained, if the maximum distance is less than or equal to the ship length, only the latitude and longitude data corresponding to the earliest time and the latest time in the data set is reserved as the decimation value of the 10 min, wherein, if the maximum distance is greater than the ship length, the latitude and longitude data corresponding to the time of the maximum distance is reserved, the data between the earliest time and the time corresponding to the maximum distance in the data set 1 is formed into a data set 2, the data between the time corresponding to the maximum distance and the latest time in the data set 1 is formed into a data set 3, the data set 2 and the data set 3 are repeatedly decimated, until only the latitude and longitude data corresponding to the earliest time and the latest time in each data set is reserved, and the latitude and longitude data corresponding to the earliest time and the latest time in each data set is collected to form the decimation value of the 10 min.

[0043] In order to facilitate the understanding of the data optimization method provided in the above embodiment, the application example of the data optimization method is provided by the embodiments of the application, referring to Figure 3 The flow diagram of another data optimization method is shown in FIG. 3, which mainly includes the following steps S302 to S306:

[0044] In an embodiment, when the algorithm is initially deployed and runs, the data cleaning enabling identification bit is 0, at this time, the latitude and longitude data is not processed, and the collected latitude and longitude data is directly stored in the database. After the latitude and longitude data is stored in the database, the correctness of the latitude and longitude data at the earliest time is verified by manual operation. If the verification is passed, the data cleaning enabling identification bit is set to 1, if the verification is not passed, the data is manually deleted, and the correctness of the latitude and longitude data at the next time is verified until the correct latitude and longitude data at the earliest time is obtained.

[0045] In an embodiment, for the unprocessed historical longitude and latitude data in the database, first, it is checked whether the longitude data or the latitude data is empty, when it is detected that the longitude data or the latitude data is empty at any time, the longitude and latitude data at the time is deleted; secondly, it is checked whether the longitude data is within [-180°, 180°] and whether the latitude data is within [-90°, 90°], when it is detected that any longitude data or latitude data is out of the above range, the longitude and latitude data at the time is deleted, thereby obtaining preliminary cleaning results, in another embodiment, for the preliminary cleaning results, the great circle route algorithm or the rhumb line algorithm is used to calculate the distance between the position point of the ship at each time and the position point of the ship at the latest correct longitude and latitude time, and the time interval Δt and the average speed v between the two times are calculated, if the distance between the two position points is less than or equal to 2*Δt*v, it is considered that the longitude and latitude data at the time is normal, otherwise, the longitude and latitude data at the time is deleted, according to the above cleaning logic, the cleaning of all historical longitude and latitude data is completed.

[0046] In an embodiment, for the real-time collected longitude and latitude data, first, it is checked whether the longitude data or the latitude data is empty, when it is detected that the longitude data or the latitude data is empty at any time, the longitude and latitude data at the time is deleted; secondly, it is checked whether the longitude data is within [-180°, 180°] and whether the latitude data is within [-90°, 90°], when it is detected that any longitude data or latitude data is out of the above range, the longitude and latitude data at the time is deleted; finally, the great circle route algorithm or the rhumb line algorithm is used to calculate the distance between the position point of the ship at the current time and the position point of the ship at the latest correct longitude and latitude time, and the time interval Δt and the average speed v between the two times are calculated, if the distance between the two position points is less than or equal to 2*Δt*v, it is considered that the longitude and latitude data at the time is normal, otherwise, the longitude and latitude data at the time is deleted, based on the above processing logic, the cleaning of all historical longitude and latitude data and real-time longitude and latitude data is completed.

[0047] In an embodiment, the target thinning strategy is determined based on the ship heading information, the ship voyage time and the time span of the historical trajectory, wherein the target thinning strategy includes short-period thinning and long-period thinning, in an embodiment, the ship turning radius is large, and it usually takes tens of minutes to complete a turn, therefore, when the ship speed is low and the distance traveled within the thinning time interval corresponding to the thinning strategy is short, short-period thinning is adopted, when the ship changes the heading, if short-period thinning is adopted, the turning feature will be lost, leading to distortion of the historical trajectory of the ship, therefore, long-period thinning is adopted, in another embodiment, the thinning time intervals of the short-period thinning and the long-period thinning can be freely set according to the ship heading information, the ship voyage time and the time span of the historical trajectory.

[0048] In an embodiment, when the target thinning strategy is short-period thinning, and the thinning time interval is 1 min, according to the timing period of 1 min, all the longitude and latitude data in each minute (00-59) is obtained from the cleaned longitude and latitude data, and the longitude and latitude data at 00 seconds is taken as the thinning value of the 1 min, if the longitude and latitude data at 00 seconds is empty, the longitude and latitude data at the earliest time in the minute is obtained as the thinning value of the 1 min.

[0049] In an embodiment, when the target thinning strategy is long-period thinning, and the thinning time interval is 10 min, according to the timing period of 10 min, all the longitude and latitude data in each 10 min (00:00-09:59) is obtained from the cleaned longitude and latitude data, to obtain a data set 1, according to the time range in the data set 1, the ship heading data is obtained, the average value of the heading in each 30 s is calculated, and the variance of the above ship heading average value is calculated, in an embodiment, when the variance is less than a preset threshold, the longitude and latitude data corresponding to the earliest time and the latest time in the data set 1 is directly selected as the thinning value of the 10 min, in another embodiment, when the variance is not less than the preset threshold, deep thinning is performed, the point at the earliest time and the point at the latest time in the data set 1 is obtained, the great circle route algorithm or the constant heading line algorithm is used to calculate the shortest distance from the remaining points in the data set to the curve formed by the above two points in turn, for the above calculated multiple distances, the maximum value is obtained, if the maximum distance is less than or equal to the ship length, only the longitude and latitude data corresponding to the earliest time and the latest time in the data set is retained as the thinning value of the 10 min, wherein, if the maximum distance is greater than the ship length, the longitude and latitude data corresponding to the time of the maximum distance is retained, the data between the earliest time in the data set 1 and the time corresponding to the maximum distance is formed into a data set 2, and the data between the time corresponding to the maximum distance and the latest time in the data set 1 is formed into a data set 3, the deep thinning step is repeated for the data set 2 and the data set 3, until only the longitude and latitude data corresponding to the earliest time and the latest time in each data set is retained, the longitude and latitude data corresponding to the earliest time and the latest time in each data set is collected to form the thinning value of the 10 min.

[0050] In an embodiment, when the target thinning strategy is long-period thinning, the thinning time interval is 1h, and according to the timing period of 1h, all longitude and latitude data in every 10 minutes (00:00:00-00:59:59) is obtained from the cleaned longitude and latitude data, to obtain a data set 1, according to the time range in the data set 1, the ship heading data is obtained, the average value of the heading in every 30s is calculated, and the variance of the above-mentioned average value of the ship heading is calculated, in an embodiment, when the variance is less than a preset threshold, the longitude and latitude data corresponding to the earliest time and the latest time in the data set 1 is directly selected as the thinning value of 1h, in another embodiment, when the variance is not less than the preset threshold, deep thinning is performed, the point at the earliest time and the point at the latest time in the data set 1 is obtained, the great circle route algorithm or the constant heading line algorithm is used to calculate the shortest distance of the remaining points in the data set to the curve formed by the above-mentioned two points in turn, for the plurality of distances calculated, the maximum value is obtained, if the maximum distance is less than or equal to the ship length, only the longitude and latitude data corresponding to the earliest time and the latest time in the data set is retained as the thinning value of 1h, wherein, if the maximum distance is greater than the ship length, the longitude and latitude data corresponding to the time of the maximum distance is retained, the data between the earliest time in the data set 1 and the time corresponding to the maximum distance is formed into a data set 2, the data between the time corresponding to the maximum distance and the latest time in the data set 1 is formed into a data set 3, the deep thinning step is repeated for the data set 2 and the data set 3, until only the longitude and latitude data corresponding to the earliest time and the latest time in each data set is retained, the longitude and latitude data corresponding to the earliest time and the latest time in each data set is summarized to form the thinning value of 1h.

[0051] In summary, by cleaning the abnormal values of the longitude and latitude data and retaining only the normal values, the accuracy of the longitude and latitude data can be significantly improved, and by using the target thinning strategy to thin the longitude and latitude data while ensuring the feature values, the data amount can be reduced, and the application software can be quickly processed.

[0052] For the data optimization method provided by the foregoing embodiments, an embodiment of the present application provides a data optimization device, which is applied to a data optimization system, and refers to a structure schematic diagram of a data optimization system device as shown in Figure 4 The device comprises the following parts:

[0053] The data acquisition module 402 acquires the original longitude and latitude data and the ship heading information of the ship navigation path to be optimized;

[0054] The preliminary cleaning module 404 performs preliminary cleaning on the abnormal data in the original longitude and latitude data to obtain standard longitude and latitude data;

[0055] The target strategy determination module 406 determines the target thinning strategy from a pre-set thinning strategy set by using the ship heading information.

[0056] The target data determination module 408 selects target longitude and latitude data from the standard longitude and latitude data according to a time period specified by a target extraction strategy;

[0057] The data extraction module 410 extracts the target longitude and latitude data by using a target extraction strategy to obtain optimized longitude and latitude data.

[0058] The data processing device provided by the embodiments of the present application can significantly improve the accuracy of the longitude and latitude data by cleaning the abnormal values of the longitude and latitude data and retaining only the normal values, and can reduce the data volume and facilitate the rapid processing of the application software by extracting the longitude and latitude data under the condition of ensuring the characteristic values by using the target extraction strategy.

[0059] In one embodiment, when the step of performing preliminary cleaning on the abnormal data in the original longitude and latitude data to obtain the standard longitude and latitude data is performed, the preliminary cleaning module 404 is further configured to: perform data error cleaning on the abnormal data in the original longitude and latitude data, and remove the blank data and the original longitude and latitude data greater than a preset threshold range to obtain first cleaned data; perform data offset cleaning on the first cleaned data to obtain second cleaned data, and determine the second cleaned data as the standard longitude and latitude data.

[0060] In one embodiment, when the step of performing data offset cleaning on the first cleaned data to obtain second cleaned data, and determining the second cleaned data as the standard longitude and latitude data is performed, the preliminary cleaning module 404 is further configured to: select the data of each time point and the data of the latest correct point in the first cleaned data, and judge the correctness of the selected time point, when at least one of the two selected points is a correct point, judge the correctness of the data of the other point according to an algorithm, and determine the distance between the adjacent time point data as a distance ideal value; determine twice the product of the time interval of the adjacent time point data and the average speed between the adjacent time point data as a distance real value; when the distance ideal value is not greater than the distance real value, determine the adjacent time point data as the standard longitude and latitude data.

[0061] In one embodiment, the extraction strategy set includes short-period extraction and long-period extraction, and when the step of determining the target extraction strategy from the pre-set extraction strategy set by using the ship heading information is performed, the target strategy determination module 406 is further configured to: when the ship heading information is unchanged, select the short-period extraction from the extraction strategy set, and determine the short-period extraction as the target extraction strategy; and when the ship heading information changes, select the long-period extraction from the extraction strategy set, and determine the long-period extraction as the target extraction strategy.

[0062] In one embodiment, when the step of performing the thinning processing on the target latitude and longitude data by using the target thinning strategy to obtain the optimized latitude and longitude data is performed, the data thinning module 410 is further configured to: when the target thinning strategy is short-period thinning, determine the data at the earliest time in the target latitude and longitude data as the optimized latitude and longitude data.

[0063] In one embodiment, when the step of performing the thinning processing on the target latitude and longitude data by using the target thinning strategy to obtain the optimized latitude and longitude data is performed, the data thinning module 410 is further configured to: when the target thinning strategy is long-period thinning, determine the average value of the ship heading in the preset time interval and the variance of the average value by using the ship heading information; if the variance is less than a preset variance threshold, determine the data at the earliest time and the data at the latest time in the target latitude and longitude data as the optimized latitude and longitude data; and if the variance is not less than the preset variance threshold, perform deep thinning on the target latitude and longitude data.

[0064] In one embodiment, when the step of performing deep thinning on the target latitude and longitude data is performed, the data thinning module 410 is further configured to: determine a curve between the earliest time and the latest time in the target latitude and longitude data as a target curve; obtain the shortest distances between each point in the target latitude and longitude data and the target curve; if all the shortest distances are not greater than the ship length, determine the data at the earliest time and the data at the latest time in the target latitude and longitude data as the optimized latitude and longitude data; if the maximum value in the shortest distances is greater than the ship length, determine the target latitude and longitude data corresponding to the maximum value as the target optimized data, and divide the preset time interval into a first preset time interval and a second preset time interval according to the time corresponding to the maximum value, and perform deep thinning on the target latitude and longitude data in the first preset time interval and the second preset time interval, respectively.

[0065] The device provided in the embodiments of the present application has the same implementation principle and technical effects as the foregoing method embodiments, and for brevity of description, the part not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments.

[0066] The electronic device provided in the embodiments of the present application comprises a processor and a storage device; the storage device stores a computer program, and the computer program performs the method according to any one of the foregoing embodiments when executed by the processor.

[0067] Figure 5 The electronic device provided in the embodiments of the present application comprises a processor 50, a memory 51, a bus 52, and a communication interface 53, wherein the processor 50, the communication interface 53, and the memory 51 are connected through the bus 52; the processor 50 is configured to execute the executable modules stored in the memory 51, such as a computer program.

[0068] The memory 51 can include a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 53 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0069] The bus 52 can be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0070] The memory 51 is used to store programs, and the processor 50 executes the programs after receiving execution instructions. The method executed by the device defined by the flow process disclosed in any of the embodiments of the present application can be applied to the processor 50 or implemented by the processor 50.

[0071] The processor 50 can be an integrated circuit chip with signal processing capability. In implementation, each step of the above method can be completed by integrated logic circuit of hardware in the processor 50 or by instructions in the form of software. The processor 50 described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art. The storage medium is located in the memory 51, and the processor 50 reads the information in the memory 51, and combines the hardware to complete the steps of the above method.

[0072] The computer program product of the readable storage medium provided by the embodiments of the present application includes a computer readable storage medium storing program codes, and the program codes include instructions for executing the method described in the foregoing method embodiments. For specific implementation, reference can be made to the foregoing method embodiments, which will not be described here.

[0073] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0074] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit them. The protection scope of the present application is not limited to this. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present application, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A data optimization method, characterized by, The method is applied to a data optimization system, and comprises: obtaining original latitude and longitude data and ship heading information of a ship sailing path to be optimized; performing preliminary cleaning on abnormal data in the original latitude and longitude data to obtain standard latitude and longitude data; determining a target thinning strategy from a pre-set thinning strategy set by using the ship heading information; selecting target latitude and longitude data from the standard latitude and longitude data according to a time period specified by the target thinning strategy; performing thinning processing on the target latitude and longitude data by using the target thinning strategy to obtain optimized latitude and longitude data, including: when the target thinning strategy is short-period thinning, determining data at an earliest time in the target latitude and longitude data as the optimized latitude and longitude data; when the target thinning strategy is long-period thinning, determining an average value of the ship heading in a preset time interval and a variance of the average value by using the ship heading information; if the variance is less than a preset variance threshold, determining data at the earliest time and the latest time in the target latitude and longitude data as the optimized latitude and longitude data; if the variance is not less than the preset variance threshold, performing deep thinning on the target latitude and longitude data; wherein the step of performing deep thinning on the target latitude and longitude data comprises: determining a curve between the earliest time and the latest time in the target latitude and longitude data as a target curve; obtaining shortest distances between each point in the target latitude and longitude data and the target curve; if all the shortest distances are not greater than a ship length, determining data at the earliest time and the latest time in the target latitude and longitude data as the optimized latitude and longitude data; if a maximum value in the shortest distances is greater than the ship length, determining the target latitude and longitude data corresponding to the maximum value as target optimization data, and dividing the preset time interval into a first preset time interval and a second preset time interval according to the time corresponding to the maximum value, and repeatedly performing deep thinning on the target latitude and longitude data in the first preset time interval and the second preset time interval, respectively.

2. The method of claim 1, wherein, The step of performing preliminary cleaning on abnormal data in the original latitude and longitude data to obtain standard latitude and longitude data comprises: performing data error cleaning on the abnormal data in the original latitude and longitude data to remove blank data and the original latitude and longitude data greater than a preset threshold range to obtain first cleaning data; performing data offset cleaning on the first cleaning data to obtain second cleaning data, and determining the second cleaning data as the standard latitude and longitude data.

3. The method of claim 2, wherein, The step of performing data offset cleaning on the first cleaning data to obtain second cleaning data, and determining the second cleaning data as the standard latitude and longitude data comprises: selecting any adjacent time data in the first cleaning data, and determining a distance ideal value as a distance between the adjacent time data; determining a distance real value as twice a product of a time interval of the adjacent time data and an average speed between the adjacent time data; when the distance ideal value is not greater than the distance real value, determining the adjacent time data as the standard latitude and longitude data.

4. The method of claim 1, wherein, The set of thinning strategies includes a short-period thinning and a long-period thinning, and the step of determining the target thinning strategy from the set of pre-set thinning strategies by using the ship heading information includes: when the ship heading information is unchanged, selecting the short-period thinning from the set of thinning strategies and determining the short-period thinning as the target thinning strategy; when the ship heading information is changed, selecting the long-period thinning from the set of thinning strategies and determining the long-period thinning as the target thinning strategy.

5. A data optimization device, characterized by, The device is applied to a data optimization system and includes: a data acquisition module configured to acquire original latitude and longitude data and ship heading information of a ship sailing path to be optimized; a preliminary cleaning module configured to perform preliminary cleaning on abnormal data in the original latitude and longitude data to obtain standard latitude and longitude data; a target strategy determination module configured to determine a target thinning strategy from a set of pre-set thinning strategies by using the ship heading information; a target data determination module configured to select target latitude and longitude data from the standard latitude and longitude data according to a time period specified by the target thinning strategy; a data thinning module configured to perform thinning processing on the target latitude and longitude data by using the target thinning strategy to obtain optimized latitude and longitude data, including: when the target thinning strategy is the short-period thinning, determining data at the earliest time in the target latitude and longitude data as the optimized latitude and longitude data; when the target thinning strategy is the long-period thinning, determining an average value of the ship heading in a preset time interval and a variance of the average value by using the ship heading information; if the variance is less than a preset variance threshold, determining data at the earliest time and the latest time in the target latitude and longitude data as the optimized latitude and longitude data; if the variance is not less than the preset variance threshold, performing deep thinning on the target latitude and longitude data; wherein the deep thinning on the target latitude and longitude data includes: determining a curve between the earliest time and the latest time in the target latitude and longitude data as a target curve; acquiring shortest distances between each point in the target latitude and longitude data and the target curve; if all the shortest distances are not greater than a ship length, determining data at the earliest time and the latest time in the target latitude and longitude data as the optimized latitude and longitude data; if a maximum value of the shortest distances is greater than the ship length, determining the target latitude and longitude data corresponding to the maximum value as target optimization data, and dividing the preset time interval into a first preset time interval and a second preset time interval according to the time corresponding to the maximum value, and repeatedly performing deep thinning on the target latitude and longitude data in the first preset time interval and the second preset time interval, respectively.

6. A server, characterized by The device includes a processor and a memory, the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, which when invoked and executed by the processor, cause the processor to implement the method of any one of claims 1 to 4.

Images