Ship date data processing method and device, electronic equipment and readable storage medium

Abstract

The invention provides a ship date data processing method and device, electronic equipment and a readable storage medium. The shipping date data processing method comprises the following steps: acquiring historical shipping date data and real arrival and departure information; associating the historical shipping date data with the real arrival and departure information; cleaning and de-weighting the historical shipping date data according to the associated historical shipping date data and the real arrival and departure information to obtain first effective shipping date data; performing aggregation processing on the first effective shipping date data to obtain a shipping date data statistical table; virtual arrival and departure information is generated according to the shipping date data statistical table; associating the virtual arrival and departure information with the current shipping date data; and according to the associated virtual arrival and departure information and the current shipping date data, cleaning and de-weighting the current shipping date data to obtain second effective shipping date data. Therefore, the virtual arrival and departure information is generated based on the historical shipping date data and the real arrival and departure information, and the current shipping date data is cleaned and de-duplicated, so that the credibility of the shipping date data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for processing ship schedule data, an electronic device and a readable storage medium. Background Art

[0002] For cargo owners, accurate shipping schedule data plays a vital role in inventory management and production. Relatively stable shipping schedule data can reduce the amount of surplus cargo that needs to be prepared in advance to cope with fluctuations in the transportation cycle, thereby significantly reducing costs and improving capital utilization.

[0003] However, at present, users obtain sailing schedule data directly from shipping companies, but each shipping company has inconsistent descriptions of the departure time and port stops of the same ship, which makes it difficult for users to accurately judge which data is available and which data is misleading. Looking back at history, it is found that the port routes corresponding to a large number of ports published in sailing schedules are not real routes, and the difference in departure time is even more than 2 weeks, so the credibility of sailing schedule data is low. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0005] To this end, the first aspect of the present invention is to propose a method for processing ship schedule data.

[0006] A second aspect of the present invention is to provide a device for processing ship schedule data.

[0007] A third aspect of the present invention is to provide an electronic device.

[0008] A fourth aspect of the present invention is to provide a readable storage medium.

[0009] In view of this, according to one aspect of the present invention, a method for processing ship schedule data is proposed, the method comprising: obtaining historical ship schedule data and real arrival and departure information; associating the historical ship schedule data with the real arrival and departure information; cleaning and deduplicating the historical ship schedule data according to the associated historical ship schedule data and the real arrival and departure information to obtain first valid ship schedule data; aggregating the first valid ship schedule data to obtain a ship schedule data statistics table; generating virtual arrival and departure information according to the ship schedule data statistics table; associating the virtual arrival and departure information with current ship schedule data; cleaning and deduplicating the current ship schedule data according to the associated virtual arrival and departure information and current ship schedule data to obtain second valid ship schedule data.

[0010] Specifically, in the method for processing the ship schedule data provided by the present invention, when cleaning the current ship schedule data, the historical ship schedule data that has occurred and its corresponding real arrival and departure information are obtained, and the acquired historical ship schedule data and the real arrival and departure information are associated. Further, according to the associated historical ship schedule data and the real arrival and departure information, the historical ship schedule data is cleaned and deduplicated, and invalid data such as abnormal data and duplicate data are removed therefrom to obtain the first valid ship schedule data. Further, the first valid ship schedule data is aggregated to obtain a ship schedule data statistics table, and then virtual arrival and departure information that has not occurred is estimated and generated according to the ship schedule data statistics table. Further, the generated virtual arrival and departure information and the current ship schedule data that has not occurred are associated, and according to the associated virtual arrival and departure information and the current ship schedule data, the current ship schedule data is cleaned and deduplicated, and invalid data such as abnormal data and duplicate data in the current ship schedule data are removed to obtain the second valid ship schedule data. In this way, by summarizing and counting the historical ship schedule data of actual arrival and departure, recursively inferring the virtual arrival and departure information to be occurred, and then cleaning the current ship schedule data that has not occurred. Specifically, the ship schedule data statistics table is determined based on the historical ship schedule data and the actual arrival and departure information, and then the virtual arrival and departure information is determined based on the ship schedule data statistics table, and the current ship schedule data is cleaned, so that useful and reliable ship schedule data can be retained, thereby improving the credibility of the ship schedule data.

[0011] The method for processing the above-mentioned ship schedule data according to the present invention may also have the following additional technical features: In some technical solutions, historical ship schedule data and actual arrival and departure information are optionally associated, including: standardizing the historical ship schedule data; sorting the actual arrival and departure information; and associating the standardized historical ship schedule data with the sorted actual arrival and departure information.

[0012] In this technical solution, in the process of associating the historical ship schedule data with the real arrival and departure information, specifically, the historical ship schedule data is standardized, and the real arrival and departure information is sorted in positive order according to the order of the real arrival and departure time indicated by the real arrival and departure information. On this basis, the standardized historical ship schedule data and the sorted real arrival and departure information are associated. In this way, the accuracy of data association is guaranteed, and then the accuracy of subsequent data cleaning is guaranteed.

[0013] In some technical solutions, historical shipping schedule data can be optionally standardized, including: retaining the first type of characters and the second type of characters in the ship name information in the historical shipping schedule data, and unifying the first type of characters into a target writing format; retaining the ship company identification information in the ship name information; obtaining the longitude and latitude information corresponding to the port of call place name in the historical shipping schedule data; and determining the port of call name corresponding to the port of call place name based on the longitude and latitude information.

[0014] In this technical solution, in the process of standardizing the historical ship schedule data, specifically, the ship name information and the port names in the historical ship schedule data are standardized respectively. Specifically, for the ship name information in the historical ship schedule data, only the first type of characters and the second type of characters in the ship name information are retained, and the first type of characters are unified into the target writing format. On this basis, the ship operator keywords in the ship name information are standardized, and only the ship operator identification information in the ship name information is retained. Furthermore, for the port names in the historical ship schedule data, the map search engine is called to obtain the longitude and latitude information corresponding to the port names, and then the port name closest to the port name is determined according to the longitude and latitude information. In this way, the ship name information and the port names in the historical ship schedule data are standardized to ensure the accuracy of subsequent data association, and then ensure the accuracy of subsequent data cleaning.

[0015] In some technical solutions, optionally, the standardized historical ship schedule data and the sorted actual arrival and departure information are associated, including: according to the ship arrival and departure time indicated by the historical ship schedule data, the actual arrival and departure time indicated by the actual arrival and departure information, the standardized ship name information and the name of the port of call, the actual arrival and departure information is associated with the port of call that is closest in time sequence in the historical ship schedule data.

[0016] In this technical solution, in the process of associating the standardized historical ship schedule data with the sorted real arrival and departure information, specifically, according to the ship arrival and departure time indicated by the historical ship schedule data, the real arrival and departure time indicated by the real arrival and departure information, the standardized ship name information and the port of call name, the real arrival and departure information is associated with the port of call that is closest in time sequence in the historical ship schedule data. In this way, the accuracy of data association is guaranteed, and then the accuracy of subsequent data cleaning is guaranteed.

[0017] In some technical solutions, optionally, the historical ship schedule data is cleaned and deduplicated based on the associated historical ship schedule data and the actual arrival and departure information, including: determining the departure time deviation based on the difference between the actual departure time indicated by the actual arrival and departure information and the ship arrival time indicated by the historical ship schedule data; determining the arrival time deviation based on the difference between the actual arrival time indicated by the actual arrival and departure information and the ship arrival time; removing the historical ship schedule data with a departure time deviation or an arrival time deviation greater than a preset time length; removing the historical ship schedule data with an associated actual arrival time of the destination port that is earlier than the actual departure time of the departure port; and deduplicating the historical ship schedule data with the same associated actual arrival and departure information.

[0018] In this technical solution, in the process of cleaning and deduplicating the historical ship schedule data according to the associated historical ship schedule data and the real arrival and departure information, specifically, the departure time deviation is determined according to the difference between the real departure time indicated by the real arrival and departure information and the ship arrival time indicated by the historical ship schedule data, and the arrival time deviation is determined according to the difference between the real arrival time indicated by the real arrival and departure information and the ship arrival time. Further, the abnormal ship schedule data and repeated ship schedule data in the historical ship schedule data are marked. Specifically, the historical ship schedule data whose departure time deviation or arrival time deviation is greater than the preset time length is marked as abnormal ship schedule data, and the historical ship schedule data whose real arrival time associated with the destination port is earlier than the real departure time associated with the departure port is marked as abnormal ship schedule data. Further, the historical ship schedule data with the same associated real arrival and departure information is marked as repeated ship schedule data. On this basis, the abnormal schedule data in the historical schedule data is deleted, and for the repeated schedule data in the historical schedule data, only the historical schedule data with the earliest release time is retained, so as to obtain the first valid schedule data. In this way, the historical schedule data is cleaned and deduplicated before subsequent processing, which ensures the accuracy of the virtual departure information generated subsequently, and further ensures the accuracy of the subsequent processing of the current schedule data.

[0019] In some technical solutions, optionally, the first valid ship schedule data is aggregated to obtain a ship schedule data statistical table, including: performing a first aggregation process on the first valid ship schedule data according to the ship name information, ship number information and route code in the first valid ship schedule data to obtain multiple port string information; performing a second aggregation process on the multiple port string information according to the route code to obtain a ship schedule data statistical table.

[0020] In this technical solution, in the process of aggregating the first valid ship schedule data to obtain the ship schedule data statistics table, the first valid ship schedule data is respectively subjected to first aggregation processing according to the ship name information, ship number information and route code in the first valid ship schedule data to obtain multiple port string information, each port string information includes multiple sub-ports in each port string and their corresponding sailing time. Further, the multiple port string information is subjected to second aggregation processing according to the route code to obtain the ship schedule data statistics table. In this way, the accuracy of summarizing and statistically analyzing the historical ship schedule data that has actually arrived at and left the port is ensured, and the accuracy of the subsequent cleaning of the current ship schedule data based on the ship schedule data statistics table is ensured.

[0021] In some technical solutions, optionally, virtual arrival and departure information is generated according to the ship schedule data statistics table, including: generating a port string corresponding to each route code according to the ship schedule data statistics table; determining the virtual port of call and its virtual arrival and departure time according to the current ship position, port string and sailing time in the ship schedule data statistics table, and obtaining the virtual arrival and departure information.

[0022] In this technical solution, in the process of generating virtual arrival and departure information according to the ship schedule data statistics table, specifically, a port string corresponding to each route code is generated according to the ship schedule data statistics table, and then the virtual port of call and its virtual arrival and departure time are determined according to the generated port string, the current ship position and the sailing time corresponding to the sub-ports in each port string in the ship schedule data statistics table, so as to obtain the virtual arrival and departure information. In this way, the accuracy of the virtual arrival and departure information is guaranteed, thereby ensuring the accuracy of the subsequent cleaning of the current ship schedule data based on the virtual arrival and departure information.

[0023] According to a second aspect of the present invention, a device for processing ship schedule data is proposed, which includes: an acquisition unit, used to acquire historical ship schedule data and real arrival and departure information; a processing unit, used to associate the historical ship schedule data with the real arrival and departure information; the processing unit is also used to clean and remove duplicates from the historical ship schedule data based on the associated historical ship schedule data and the real arrival and departure information, so as to obtain first valid ship schedule data; the processing unit is also used to aggregate the first valid ship schedule data to obtain a ship schedule data statistics table; the processing unit is also used to generate virtual arrival and departure information based on the ship schedule data statistics table; the processing unit is also used to associate the virtual arrival and departure information with the current ship schedule data; the processing unit is also used to clean and remove duplicates from the current ship schedule data based on the associated virtual arrival and departure information and the current ship schedule data, so as to obtain second valid ship schedule data.

[0024] The processing device of the ship schedule data provided by the present invention comprises an acquisition unit and a processing unit. When cleaning the current ship schedule data, the acquisition unit acquires the historical ship schedule data that has occurred and its corresponding real arrival and departure information, and the processing unit associates the acquired historical ship schedule data with the real arrival and departure information. Further, the processing unit cleans and removes the historical ship schedule data according to the associated historical ship schedule data and the real arrival and departure information, removes the invalid data such as abnormal data and duplicate data therein, and obtains the first valid ship schedule data. Further, the processing unit aggregates the first valid ship schedule data to obtain a ship schedule data statistics table, and then estimates and generates virtual arrival and departure information that has not occurred according to the ship schedule data statistics table. Further, the processing unit associates the generated virtual arrival and departure information with the current ship schedule data that has not occurred yet, and cleans and removes the current ship schedule data according to the associated virtual arrival and departure information and the current ship schedule data, removes the invalid data such as abnormal data and duplicate data in the current ship schedule data, and obtains the second valid ship schedule data. In this way, by summarizing and counting the historical ship schedule data of actual arrival and departure, recursively inferring the virtual arrival and departure information to be occurred, and then cleaning the current ship schedule data that has not occurred. Specifically, the ship schedule data statistics table is determined based on the historical ship schedule data and the actual arrival and departure information, and then the virtual arrival and departure information is determined based on the ship schedule data statistics table, and the current ship schedule data is cleaned, so that useful and reliable ship schedule data can be retained, thereby improving the credibility of the ship schedule data.

[0025] According to the third aspect of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for processing the ship schedule data in any of the above technical solutions are implemented. Therefore, the electronic device provided in the third aspect of the present invention has all the beneficial effects of the method for processing the ship schedule data in any of the above technical solutions in the first aspect, and will not be described in detail here.

[0026] According to a fourth aspect of the present invention, a readable storage medium is provided, on which a program or instruction is stored, and when the program or instruction is executed by a processor, a method for processing ship schedule data as in any of the above technical solutions is implemented. Therefore, the readable storage medium provided in the fourth aspect of the present invention has all the beneficial effects of the method for processing ship schedule data in any of the above technical solutions in the first aspect, and no further description is given here.

[0027] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 A schematic diagram showing a flow chart of a method for processing ship schedule data according to an embodiment of the present invention; Figure 2 A second flow chart of a method for processing ship schedule data according to an embodiment of the present invention is shown; Figure 3 A third flow chart of a method for processing ship schedule data according to an embodiment of the present invention is shown; Figure 4 A fourth flow chart of a method for processing ship schedule data according to an embodiment of the present invention is shown; Figure 5 FIG5 is a flowchart of a method for processing ship schedule data according to an embodiment of the present invention; Figure 6 A structural block diagram showing a device for processing ship schedule data according to an embodiment of the present invention is shown; Figure 7 A structural block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0031] Combine the following Figures 1 to 7 , the method and device for processing the ship schedule data, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

[0032] In one embodiment of the present invention, Figure 1 As shown, the method for processing the ship schedule data may specifically include the following steps 102 to 114: Step 102, obtaining historical ship schedule data and actual arrival and departure information; Step 104, associating historical ship schedule data with actual arrival and departure information; Step 106, according to the associated historical ship schedule data and the actual arrival and departure information, the historical ship schedule data is cleaned and deduplicated to obtain the first valid ship schedule data; Step 108, performing aggregation processing on the first valid sailing schedule data to obtain a sailing schedule data statistics table; Step 110, generating virtual arrival and departure information according to the ship schedule data statistics table; Step 112, associating the virtual arrival and departure information with the current ship schedule data; Step 114, based on the associated virtual arrival and departure information and the current sailing schedule data, the current sailing schedule data is cleaned and deduplicated to obtain the second valid sailing schedule data.

[0033] When cleaning the current ship schedule data, the historical ship schedule data that has occurred and its corresponding real arrival and departure information are obtained, and the obtained historical ship schedule data and the real arrival and departure information are associated. Further, based on the associated historical ship schedule data and the real arrival and departure information, the historical ship schedule data is cleaned and deduplicated, and invalid data such as abnormal data and duplicate data are removed to obtain the first valid ship schedule data. Further, the first valid ship schedule data is aggregated to obtain a ship schedule data statistics table, and then virtual arrival and departure information that has not occurred is estimated and generated based on the ship schedule data statistics table. Further, the generated virtual arrival and departure information is associated with the current ship schedule data that has not occurred yet, and based on the associated virtual arrival and departure information and the current ship schedule data, the current ship schedule data is cleaned and deduplicated, and invalid data such as abnormal data and duplicate data in the current ship schedule data is removed to obtain the second valid ship schedule data. In this way, by summarizing and counting the historical ship schedule data of actual arrival and departure, recursively inferring the virtual arrival and departure information to be occurred, and then cleaning the current ship schedule data that has not occurred. Specifically, the ship schedule data statistics table is determined based on the historical ship schedule data and the actual arrival and departure information, and then the virtual arrival and departure information is determined based on the ship schedule data statistics table, and the current ship schedule data is cleaned, so that useful and reliable ship schedule data can be retained, thereby improving the credibility of the ship schedule data.

[0034] In some embodiments of the present invention, optionally, Figure 2 As shown, the above step 104 may specifically include the following steps 104a to 104c: Step 104a, standardizing the historical ship schedule data; Step 104b, sorting the actual arrival and departure information; Step 104c, associating the standardized historical ship schedule data with the sorted actual arrival and departure information.

[0035] In this embodiment, in the process of associating the historical ship schedule data with the real arrival and departure information, specifically, the historical ship schedule data is standardized, and the real arrival and departure information is sorted in positive order according to the sequence of the real arrival and departure time indicated by the real arrival and departure information. On this basis, the standardized historical ship schedule data and the sorted real arrival and departure information are associated. In this way, the accuracy of data association is guaranteed, and then the accuracy of subsequent data cleaning is guaranteed.

[0036] In some embodiments of the present invention, optionally, the above step 104a may specifically include the following steps 116 to 122: Step 116, retaining the first type of characters and the second type of characters in the ship name information in the historical sailing schedule data, and unifying the first type of characters into a target writing format; Step 118, retaining the ship company identification information in the ship name information; Step 120, obtaining the longitude and latitude information corresponding to the port of call in the historical ship schedule data; Step 122, determining the name of the port of call corresponding to the place name of the port of call according to the latitude and longitude information.

[0037] In this embodiment, in the process of standardizing the historical ship schedule data, specifically, the ship name information and the port of call place names in the historical ship schedule data are standardized respectively. Specifically, for the ship name information in the historical ship schedule data, only the first type of characters and the second type of characters in the ship name information are retained, and the first type of characters are unified into the target writing format. On this basis, the ship operator keywords in the ship name information are standardized, and only the ship operator identification information in the ship name information is retained. Further, for the port of call place names in the historical ship schedule data, the map search engine is called to obtain the longitude and latitude information corresponding to the port of call place names, and then the port of call name closest to the port of call place name is determined according to the longitude and latitude information. In this way, the ship name information and the port of call place names in the historical ship schedule data are standardized, which ensures the accuracy of subsequent data association, and then ensures the accuracy of subsequent data cleaning.

[0038] The first type of characters are English characters, the second type of characters are numeric characters, and the target writing format is uppercase English characters.

[0039] Furthermore, the ship company identification information is information in the ship name information that can effectively indicate the name of the ship company. For example, "COSCO" and "COSCO SHIPPING" are unified into "COSCO".

[0040] In some embodiments of the present invention, optionally, the above step 104c may specifically include the following step 124: Step 124, according to the ship arrival and departure time indicated by the historical ship schedule data, the actual arrival and departure time indicated by the actual arrival and departure information, the standardized ship name information and the name of the port of call, the actual arrival and departure information is associated with the port of call that is closest in time sequence in the historical ship schedule data.

[0041] In this embodiment, in the process of associating the standardized historical ship schedule data with the sorted real arrival and departure information, specifically, according to the ship schedule arrival and departure time indicated by the historical ship schedule data, the real arrival and departure time indicated by the real arrival and departure information, the standardized ship name information and the port of call name, the real arrival and departure information is associated with the port of call that is closest in time sequence in the historical ship schedule data. In this way, the accuracy of data association is guaranteed, and then the accuracy of subsequent data cleaning is guaranteed.

[0042] In the actual application process, the time series fuzzy association function of the Pandas library can be used to perform time series fuzzy association on the standardized historical ship schedule data and the sorted real arrival and departure information. Specifically, the "on" parameter of the Pandas library is set to the ship schedule arrival and departure time and the real arrival and departure time, the "by" parameter is set to the standardized ship name information and the standardized port name, and the "direction" parameter is set to "nearest" so that the data is associated with the port of call that is closest in time series.

[0043] Among them, the Pandas library is a powerful open source data analysis and processing library.

[0044] Furthermore, in the Pandas library, the "on" parameter is used to specify the column or index level name used to merge two data frames; the "by" parameter is an important parameter in the groupby function, which is used to group data according to one or more columns for further data processing and analysis. The "by" parameter determines how to group the data; the "direction" parameter is used to specify the direction that should be considered when deleting missing values.

[0045] Similarly, in the process of associating the virtual arrival and departure information with the current ship schedule data, the "on" parameter of the Pandas library can be set to the ship arrival and departure time and the virtual arrival and departure time, the "by" parameter can be set to the standardized ship name information and the standardized port name, and the "direction" parameter can be set to "nearest". That is, according to the ship arrival and departure time indicated by the current ship schedule data, the virtual arrival and departure time indicated by the virtual arrival and departure information, and the standardized ship name information and port name in the current ship schedule data, the virtual arrival and departure information time sequence is fuzzily associated with the port of call that is closest in time sequence in the current ship schedule data.

[0046] In some embodiments of the present invention, optionally, the step of cleaning and deduplicating the historical ship schedule data according to the associated historical ship schedule data and the actual arrival and departure information may specifically include the following steps 126 to 134: Step 126, determining the departure time deviation according to the difference between the actual departure time indicated by the actual departure information and the ship arrival time indicated by the historical ship schedule data; Step 128, determining the arrival time deviation according to the difference between the actual arrival time indicated by the actual arrival and departure information and the scheduled arrival time of the vessel; Step 130, removing the historical ship schedule data with a departure time deviation or an arrival time deviation greater than a preset time length; Step 132, removing the historical ship schedule data whose associated actual arrival time at the destination port is earlier than the actual departure time at the origin port; Step 134, de-duplication processing is performed on the historical ship schedule data with the same associated real arrival and departure information.

[0047] In this embodiment, in the process of cleaning and deduplication of the historical ship schedule data according to the associated historical ship schedule data and the real arrival and departure information, specifically, the departure time deviation is determined according to the difference between the real departure time indicated by the real arrival and departure information and the ship arrival time indicated by the historical ship schedule data, and the arrival time deviation is determined according to the difference between the real arrival time indicated by the real arrival and departure information and the ship arrival time. Further, the abnormal ship schedule data and repeated ship schedule data in the historical ship schedule data are marked. Specifically, the historical ship schedule data whose departure time deviation or arrival time deviation is greater than the preset time length is marked as abnormal ship schedule data, and the historical ship schedule data whose real arrival time associated with the destination port is earlier than the real departure time associated with the departure port is marked as abnormal ship schedule data. Further, the historical ship schedule data with the same associated real arrival and departure information is marked as repeated ship schedule data. On this basis, the abnormal ship schedule data in the historical ship schedule data is deleted, and for the repeated ship schedule data in the historical ship schedule data, only the historical ship schedule data with the earliest release time is retained, so as to obtain the first valid ship schedule data. In this way, the historical ship schedule data is cleaned and deduplicated before subsequent processing, which ensures the accuracy of the subsequently generated virtual arrival and departure information, and further ensures the accuracy of the subsequent processing of the current ship schedule data.

[0048] In actual application, the above preset time period can be specifically 2 weeks, 15 days, etc. For the specific value of the above preset time period, technicians in this field can set it according to actual conditions, and no specific limitation is made here.

[0049] Similarly, in the process of cleaning and deduplicating the current ship schedule data according to the associated virtual departure information and the current ship schedule data to obtain the second valid ship schedule data, the departure time deviation is determined according to the difference between the virtual departure time indicated by the virtual departure information and the ship arrival time indicated by the current ship schedule data, and the arrival time deviation is determined according to the difference between the virtual arrival time indicated by the virtual departure information and the ship arrival time. Further, the current ship schedule data whose departure time deviation or arrival time deviation is greater than the preset time length is marked as abnormal ship schedule data, and the current ship schedule data whose virtual arrival time associated with the destination port is earlier than the virtual departure time associated with the departure port is marked as abnormal ship schedule data. Further, the current ship schedule data with the same associated virtual departure information is marked as duplicate ship schedule data. On this basis, the abnormal ship schedule data in the current ship schedule data is deleted, and for the duplicate ship schedule data in the current ship schedule data, only the current ship schedule data with the earliest release time is retained, so as to obtain the second valid ship schedule data.

[0050] In some embodiments of the present invention, optionally, Figure 3 As shown, the above step 108 may specifically include the following steps 108a and 108b: Step 108a, performing first aggregation processing on the first valid sailing schedule data according to the ship name information, ship number information and route code in the first valid sailing schedule data, to obtain a plurality of port string information; Step 108b, performing a second aggregation process on the multiple port string information according to the route code to obtain a ship schedule data statistics table.

[0051] In this embodiment, in the process of performing aggregation processing on the first valid ship schedule data to obtain the ship schedule data statistics table, the first valid ship schedule data is respectively subjected to first aggregation processing according to the ship name information, ship number information and route code in the first valid ship schedule data to obtain multiple port string information, each port string information includes multiple sub-ports in each port string and their corresponding sailing time. Further, the multiple port string information is subjected to second aggregation processing according to the route code to obtain the ship schedule data statistics table. In this way, the accuracy of summarizing and statistically analyzing the historical ship schedule data that has actually arrived at or left the port is ensured, and the accuracy of the subsequent cleaning of the current ship schedule data based on the ship schedule data statistics table is ensured.

[0052] The above-mentioned schedule data statistics table includes the frequency statistics of each port string in different route codes, the median of the sailing time corresponding to multiple sub-ports in the port string, and a set of deciles. Thus, by inputting the port code and the port string, the frequency of the port string and the median and deciles of the sailing time corresponding to the sub-ports in the port string can be obtained.

[0053] In some embodiments of the present invention, optionally, Figure 4 As shown, the above step 110 may specifically include the following steps 110a and 110b: Step 110a, generating a port string corresponding to each route code according to the ship schedule data statistics table; Step 110b, according to the current ship position, port string and sailing time in the ship schedule data statistics table, determine the virtual port of call and its virtual arrival and departure time, and obtain virtual arrival and departure information.

[0054] In this embodiment, in the process of generating virtual arrival and departure information according to the ship schedule data statistics table, specifically, a port string corresponding to each route code is generated according to the ship schedule data statistics table, and then the virtual port of call and its virtual arrival and departure time are determined according to the generated port string, the current ship position and the sailing time corresponding to the sub-ports in each port string in the ship schedule data statistics table, so as to obtain the virtual arrival and departure information. In this way, the accuracy of the virtual arrival and departure information is guaranteed, thereby ensuring the accuracy of the subsequent cleaning of the current ship schedule data based on the virtual arrival and departure information.

[0055] In summary, if Figure 5 As shown, the method for processing the ship schedule data provided by the present invention may specifically include the following steps 202 to 226: Step 202, standardizing the ship name information in the historical ship schedule data; Step 204, standardizing the names of ports of call in the historical ship schedule data; Step 206, performing time series fuzzy association between historical ship schedule data and actual arrival and departure information; Step 208, calculating the departure time deviation and the arrival time deviation; Step 210, marking abnormal data in the historical sailing schedule data; Step 212, marking duplicate data in the historical sailing schedule data; Step 214, cleaning and deduplication of historical sailing schedule data to generate first valid sailing schedule data; Step 216, performing aggregation operation on the first valid sailing schedule data to obtain a sailing schedule data statistics table; Step 218, generating a virtual port of call and its virtual arrival and departure time according to the ship schedule data statistics table; Step 220, performing time-series fuzzy association between the current ship schedule data and the virtual arrival and departure information; Step 222, marking abnormal data in the current ship schedule data; Step 224, marking duplicate data in the current sailing schedule data; Step 226, clean and remove duplicates from the current sailing schedule data to generate second valid sailing schedule data.

[0056] Based on this, the method for processing ship schedule data provided by the present invention realizes the generation of reliable ship schedule data statistics tables by looking back at history, determines future virtual arrival and departure information based on the statistical results of historical ship schedule data that have occurred, and detects current repeated ship schedules and abnormal ship schedules based on virtual arrival and departure behaviors, thereby improving the credibility of ship schedule data.

[0057] In one embodiment of the present invention, a device for processing ship schedule data is also provided. Figure 6 As shown, Figure 6 The structural block diagram of the device 300 for processing the ship schedule data according to the embodiment of the present invention is shown. The device 300 for processing the ship schedule data may specifically include the following acquisition unit 302 and processing unit 304: An acquisition unit 302 is used to acquire historical ship schedule data and actual arrival and departure information; Processing unit 304, used to associate historical ship schedule data with actual arrival and departure information; The processing unit 304 is further used to clean and remove duplicates from the historical ship schedule data according to the associated historical ship schedule data and the actual arrival and departure information to obtain first valid ship schedule data; The processing unit 304 is further used to perform aggregation processing on the first valid sailing schedule data to obtain a sailing schedule data statistical table; The processing unit 304 is further used to generate virtual arrival and departure information according to the ship schedule data statistics table; The processing unit 304 is further used to associate the virtual arrival and departure information with the current ship schedule data; The processing unit 304 is further used to clean and remove duplicates of the current ship schedule data according to the associated virtual arrival and departure information and the current ship schedule data to obtain the second valid ship schedule data.

[0058] The processing device 300 of the ship schedule data provided by the embodiment of the present invention includes an acquisition unit 302 and a processing unit 304. When cleaning the current ship schedule data, the acquisition unit 302 acquires the historical ship schedule data that has occurred and its corresponding real arrival and departure information, and the processing unit 304 associates the acquired historical ship schedule data with the real arrival and departure information. Further, the processing unit 304 cleans and de-duplicates the historical ship schedule data according to the associated historical ship schedule data and the real arrival and departure information, removes invalid data such as abnormal data and duplicate data therein, and obtains the first valid ship schedule data. Further, the processing unit 304 aggregates the first valid ship schedule data to obtain a ship schedule data statistical table, and then generates virtual arrival and departure information that has not occurred according to the ship schedule data statistical table. Furthermore, the processing unit 304 associates the generated virtual arrival and departure information with the current ship schedule data that has not occurred yet, and cleans and deduplicates the current ship schedule data according to the associated virtual arrival and departure information and the current ship schedule data, removes invalid data such as abnormal data and duplicate data in the current ship schedule data, and obtains the second valid ship schedule data. In this way, by summarizing and counting the historical ship schedule data that have actually arrived at and left the port, recursively inferring the virtual arrival and departure information to be occurred, and then cleaning the current ship schedule data that has not occurred, specifically, determining the ship schedule data statistics table based on the historical ship schedule data and the real arrival and departure information, and then determining the virtual arrival and departure information based on the ship schedule data statistics table, and cleaning the current ship schedule data, it is possible to retain useful and reliable ship schedule data, thereby improving the credibility of the ship schedule data.

[0059] In some embodiments of the present invention, optionally, the processing unit 304 is specifically used to: standardize the historical ship schedule data; sort the actual arrival and departure information; and associate the standardized historical ship schedule data with the sorted actual arrival and departure information.

[0060] In some embodiments of the present invention, optionally, the processing unit 304 is specifically used to: retain the first type of characters and the second type of characters in the ship name information in the historical shipping schedule data, and unify the first type of characters into a target writing format; retain the ship company identification information in the ship name information; obtain the longitude and latitude information corresponding to the port of call place name in the historical shipping schedule data; determine the port of call name corresponding to the port of call place name based on the longitude and latitude information.

[0061] In some embodiments of the present invention, optionally, the processing unit 304 is specifically used to: associate the actual arrival and departure information with the port of call that is closest in time sequence in the historical ship schedule data based on the ship arrival and departure time indicated by the historical ship schedule data, the actual arrival and departure time indicated by the actual arrival and departure information, the standardized ship name information and the name of the port of call.

[0062] In some embodiments of the present invention, optionally, the processing unit 304 is specifically used to: determine the departure time deviation according to the difference between the actual departure time indicated by the actual arrival and departure information and the ship arrival time indicated by the historical ship schedule data; determine the arrival time deviation according to the difference between the actual arrival time indicated by the actual arrival and departure information and the ship arrival time; remove the historical ship schedule data with a departure time deviation or an arrival time deviation greater than a preset time length; remove the historical ship schedule data with an associated actual arrival time of the destination port that is earlier than the actual departure time of the departure port; and deduplicate the historical ship schedule data with the same associated actual arrival and departure information.

[0063] In some embodiments of the present invention, optionally, the processing unit 304 is specifically used to: perform a first aggregation processing on the first valid shipping schedule data according to the ship name information, ship number information and route code in the first valid shipping schedule data to obtain multiple port string information; perform a second aggregation processing on the multiple port string information according to the route code to obtain a shipping schedule data statistical table.

[0064] In some embodiments of the present invention, optionally, the processing unit 304 is specifically used to: generate a port string corresponding to each route code according to the ship schedule data statistics table; determine the virtual port of call and its virtual arrival and departure time according to the current ship position, port string and sailing time in the ship schedule data statistics table, and obtain virtual arrival and departure information.

[0065] In one embodiment of the present invention, an electronic device is also provided. Figure 7 As shown, Figure 7 FIG. 4 is a block diagram of an electronic device 400 provided in an embodiment of the present invention. The electronic device 400 includes: Memory 402, where programs or instructions are stored; Processor 404, when executing the above program or instruction, implements the steps of the method for processing the ship schedule data in any of the above embodiments.

[0066] The electronic device 400 provided in this embodiment includes a memory 402 and a processor 404. When the program or instructions in the memory 402 are executed by the processor 404, the steps of the method for processing the ship schedule data in any of the above embodiments are implemented. Therefore, the electronic device 400 has all the beneficial effects of the method for processing the ship schedule data in any of the above embodiments, which will not be repeated here.

[0067] Specifically, the memory 402 and the processor 404 may be connected via a bus or other means. The processor 404 may include one or more processing units, and the processor 404 may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and other chips.

[0068] In one embodiment of the present invention, a readable storage medium is also provided, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method for processing the ship schedule data in any of the above embodiments are implemented.

[0069] The readable storage medium provided by the embodiment of the present invention can implement the steps of the method for processing the ship schedule data in any of the above embodiments when the program or instruction stored therein is executed by the processor. Therefore, the readable storage medium has all the beneficial effects of the method for processing the ship schedule data in any of the above embodiments, which will not be described in detail here.

[0070] Specifically, the above-mentioned readable storage medium may include any medium capable of storing or transmitting information. Examples of readable storage media include electronic circuits, semiconductor memory devices, read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), flash memory, erasable ROM (EROM), magnetic tape, floppy disk, optical disk, hard disk, optical fiber medium, radio frequency (RF) link, optical data storage device, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0071] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance unless otherwise clearly specified and limited; the terms "connection", "installation", "fixation" and the like should be understood in a broad sense, for example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0072] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0073] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for processing ship schedule data, characterized in that: include: Obtain historical ship schedule data and real arrival and departure information; Associating the historical ship schedule data with the actual arrival and departure information; According to the associated historical ship schedule data and the real arrival and departure information, the historical ship schedule data is cleaned and deduplicated to obtain first valid ship schedule data; Aggregating the first valid sailing schedule data to obtain a sailing schedule data statistics table; Generate virtual arrival and departure information according to the ship schedule data statistics table; Associating the virtual arrival and departure information with current ship schedule data; According to the associated virtual arrival and departure information and the current ship schedule data, the current ship schedule data is cleaned and deduplicated to obtain second valid ship schedule data.

2. The method for processing ship schedule data according to claim 1, characterized in that: The associating the historical ship schedule data with the actual arrival and departure information includes: Performing standardization on the historical ship schedule data; sorting the actual arrival and departure information; The standardized historical ship schedule data and the sorted actual arrival and departure information are associated.

3. The method for processing ship schedule data according to claim 2, characterized in that: The standardization of the historical ship schedule data includes: retaining the first type of characters and the second type of characters in the ship name information in the historical sailing schedule data, and unifying the first type of characters into a target writing format; Retaining the ship company identification information in the ship name information; Obtain the longitude and latitude information corresponding to the port of call in the historical ship schedule data; The name of the port of call corresponding to the place name of the port of call is determined according to the latitude and longitude information.

4. The method for processing ship schedule data according to claim 3, characterized in that: The associating the standardized historical ship schedule data with the sorted actual arrival and departure information includes: According to the ship arrival and departure time indicated by the historical ship schedule data, the actual arrival and departure time indicated by the actual arrival and departure information, the standardized ship name information and the name of the port of call, the actual arrival and departure information is associated with the port of call that is closest in time sequence in the historical ship schedule data.

5. The method for processing ship schedule data according to any one of claims 1 to 4, characterized in that: The step of cleaning and deduplicating the historical ship schedule data according to the associated historical ship schedule data and the actual arrival and departure information includes: Determine the departure time deviation according to the difference between the actual departure time indicated by the actual arrival and departure information and the ship arrival time indicated by the historical ship schedule data; Determine the arrival time deviation according to the difference between the actual arrival time indicated by the actual arrival and departure information and the scheduled arrival time of the vessel; Remove the historical ship schedule data whose departure time deviation or arrival time deviation is greater than a preset time length; Remove the historical ship schedule data whose associated actual arrival time at the destination port is earlier than the actual departure time at the origin port; Deduplication processing is performed on the historical ship schedule data with the same associated real arrival and departure information.

6. The method for processing ship schedule data according to any one of claims 1 to 4, characterized in that: The aggregating process is performed on the first valid sailing schedule data to obtain a sailing schedule data statistical table, including: According to the ship name information, ship number information and route code in the first valid ship schedule data, the first valid ship schedule data is respectively subjected to first aggregation processing to obtain a plurality of port string information; A second aggregation process is performed on the plurality of port string information according to the route code to obtain the ship schedule data statistics table.

7. The method for processing ship schedule data according to any one of claims 1 to 4, characterized in that: The generating virtual arrival and departure information according to the ship schedule data statistics table includes: Generate a port string corresponding to each route code according to the shipping schedule data statistics table; According to the current ship position, the port string and the sailing time in the ship schedule data statistics table, a virtual port of call and its virtual arrival and departure time are determined to obtain the virtual arrival and departure information.

8. A device for processing ship schedule data, characterized in that: include: Acquisition unit, used to obtain historical ship schedule data and actual arrival and departure information; A processing unit, used for associating the historical ship schedule data with the actual arrival and departure information; The processing unit is further used to clean and remove duplicates from the historical ship schedule data according to the associated historical ship schedule data and the actual arrival and departure information to obtain first valid ship schedule data; The processing unit is further used to perform aggregation processing on the first valid sailing schedule data to obtain a sailing schedule data statistical table; The processing unit is further used to generate virtual arrival and departure information according to the ship schedule data statistical table; The processing unit is further used to associate the virtual arrival and departure information with current ship schedule data; The processing unit is further used to clean and remove duplicates of the current ship schedule data according to the associated virtual arrival and departure information and the current ship schedule data to obtain second valid ship schedule data.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method for processing ship schedule data as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method for processing ship schedule data according to any one of claims 1 to 7 are implemented.

Images