Marine transportation data periodic acquisition and scheduling system applied to international logistics

Abstract

According to the marine transportation data periodic acquisition and scheduling system applied to international logistics provided by the invention, the operation end can set the corresponding scheduling rule for each tenant according to the data acquisition requirements of different tenants, and store the set scheduling rule in the scheduling rule table; in this way, different data acquisition frequencies can be set for different tenants, the tenants can set different acquisition frequency requirements for different data sources under multiple acquisition data sources, the expansibility is high, different data acquisition frequencies can be set according to logistics nodes, and then the cost of a server is reduced to a great extent; the user side can form a data acquisition task by using the user information of the tenant and the related information and then send the data acquisition task to the data processing side, and the data processing side can acquire and clean the logistics data according to the data acquisition task after receiving the data acquisition task sent by the user side.

Description

Technical Field

[0001] The present application relates to the technical field of international logistics, and in particular to a periodic collection and scheduling system for shipping data applied to international logistics. Background Art

[0002] In international logistics SaaS applications, logistics tracking data is very important data. As a comprehensive logistics tracking data query platform, logistics data must be updated periodically when goods are in transit. The underlying logic of data update is achieved through data collection, and the collection methods can generally be divided into one-time and periodic. Since logistics data changes regularly, the use of periodic collection can better actively perceive data changes. This is particularly evident in the international shipping logistics industry. In terms of technical implementation, periodic data collection is often simply and crudely sent at a certain time through a scheduled task, and then the downstream data collection server receives the collection task and performs subsequent data collection operations.

[0003] For example, the current process of periodic collection of logistics data in the industry is as follows:

[0004] 1. Design a table to store user collection requirements (tasks);

[0005] 2. Set the cron expression for the scheduled task, such as triggering it once every morning at 8 o'clock;

[0006] 3. After the timing framework is started, the task records that need to be collected in the user collection task are scanned;

[0007] 4. Send the collection task to the downstream data collection service application through interface call;

[0008] 5. The data collection application accepts the collection task and calls the data source interface to collect data;

[0009] 6. After the collection is completed, the data will be stored in the corresponding data table, and the data collection results will be sent back to the collection sender.

[0010] The above periodic collection process has the following defects:

[0011] 1. In the Sass multi-tenant scenario, it cannot meet the inconsistent data collection frequency requirements of different tenants;

[0012] 2. It cannot meet the collection frequency requirements of tenants for different data sources under multiple data sources (such as different shipping companies);

[0013] 3. It cannot meet the needs of setting different data collection frequencies according to the nodes in different logistics node scenarios of shipping data (such as the nodes of picking up empty containers, loading ships, sailing, arriving at ports, unloading, picking up heavy containers, and returning empty containers), resulting in high server costs;

[0014] 4. It can only be triggered at a certain point in time and cannot be evenly dispersed for scheduling, resulting in a large peak problem, which leads to high resource costs for the server and high system pressure. Summary of the invention

[0015] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defects that the periodic collection process of logistics data in the prior art cannot meet the collection frequency requirements of multi-tenant and multi-data sources, and there is a large peak problem, resulting in high resource costs of the server and high system pressure.

[0016] The present application provides a periodic collection and scheduling system for shipping data applied to international logistics, the system comprising:

[0017] The operation side is used to set corresponding scheduling rules for each tenant according to the data collection requirements of different tenants, and save the set scheduling rules into the scheduling rule table;

[0018] The user end is used to obtain the relevant information input by the tenant when querying the logistics data, and send the user information of the tenant and the relevant information to the data processing end after forming a data collection task, and after querying the scheduling rule table corresponding to the data collection task, determine the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule and store it;

[0019] The scheduling system end is used to regularly select the order set for data collection from the next scheduling time corresponding to each stored logistics order, send the corresponding collection task to the data processing end, and query the scheduling rule table corresponding to each logistics order in the order set, and update the next scheduling time of each logistics order according to each scheduling rule and the current scheduling time;

[0020] The data processing end is used to receive the data collection tasks sent by the user end and the collection tasks corresponding to the order set sent by the scheduling system end, collect and clean the logistics data according to the data collection tasks and the collection tasks corresponding to the order set, and after querying the scheduling rules corresponding to the cleaned logistics data in the scheduling rule table, update the stored next scheduling time according to the scheduling rules and the current collection time.

[0021] Optionally, the operation end sets corresponding scheduling rules for each tenant according to the data collection requirements of different tenants, including:

[0022] The operation side sets scheduling rules corresponding to different logistics nodes under different data sources in different categories for each tenant based on the different tenants' demands for logistics scheduling frequency.

[0023] Optionally, the user terminal determines and stores the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule, including:

[0024] The user terminal determines the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule, converts the next scheduling time into a millisecond value, and stores it using the zadd command of the zset data structure of Redis.

[0025] Optionally, the scheduling system end periodically selects an order set for data collection from the next scheduling time corresponding to each stored logistics order and sends the corresponding collection task to the data processing end, including:

[0026] When executing the preset timed task, the scheduling system uses the zrange command to scan the logistics orders for the next scheduling time corresponding to the stored logistics orders to be collected, and after the scanned logistics orders are formed into an order set, the corresponding collection task is sent to the data processing end.

[0027] Optionally, the data processing end collects and cleans the logistics data according to the data collection task and the collection task corresponding to the order set, including:

[0028] The data processing end collects logistics data by calling the data collection interface provided by the data source corresponding to each logistics order according to the data collection task and the collection task corresponding to the order set, and cleans the logistics data using a preset data cleaning program.

[0029] The present application also provides an operation end, which is applied to a periodic collection and scheduling system for shipping data applied to international logistics as described in any one of the above embodiments, including:

[0030] According to the data collection requirements of different tenants, corresponding scheduling rules are set for each tenant, and the set scheduling rules are saved in the scheduling rule table.

[0031] The present application also provides a user terminal, which is applied to a periodic collection and scheduling system for shipping data applied to international logistics as described in any one of the above embodiments, including:

[0032] Obtain relevant information input by the tenant when querying logistics data, and send the tenant's user information and the relevant information to the data processing end after forming a data collection task;

[0033] And, after querying the scheduling rule corresponding to the data collection task in the scheduling rule table, the next scheduling time corresponding to the logistics order queried by the tenant is determined according to the current query time and the scheduling rule and stored.

[0034] Optionally, determining and storing the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule includes:

[0035] The next scheduling time corresponding to the logistics order queried by the tenant is determined according to the current query time and the scheduling rules, and the next scheduling time is converted into a millisecond value and stored using the zadd command of the zset data structure of Redis.

[0036] The present application also provides a scheduling system end, which is applied to a periodic collection and scheduling system for shipping data applied to international logistics as described in any one of the above embodiments, including:

[0037] Regularly filter out the order set to be collected from the next scheduling time corresponding to each stored logistics order, and then send the corresponding collection task to the data processing end;

[0038] After querying the dispatching rule corresponding to each logistics order in the order set in the dispatching rule table, the next dispatching time of each logistics order is updated according to each dispatching rule and the current dispatching time.

[0039] The present application also provides a data processing end, which is applied to a periodic collection and scheduling system for shipping data applied to international logistics as described in any one of the above embodiments, including:

[0040] Receive data collection tasks sent by the user end and collection tasks corresponding to the order set sent by the scheduling system end;

[0041] Collect and clean logistics data according to the data collection task and the collection task corresponding to the order set;

[0042] And, after querying the scheduling rule corresponding to the cleaned logistics data in the scheduling rule table, the stored next scheduling time is updated according to the scheduling rule and the current collection time.

[0043] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0044] The present application provides a periodic collection and scheduling system for shipping data applied to international logistics, the system includes an operation end, a user end, a scheduling system and a data processing end, wherein the operation end can set corresponding scheduling rules for each tenant according to the data collection requirements of different tenants, and save the set scheduling rules into a scheduling rule table, so that different data collection frequencies can be set for different tenants, and under multiple collection data sources, tenants can set different collection frequency requirements for different data sources, with strong scalability, and different data collection frequencies can be set according to logistics nodes, thereby greatly reducing server costs; the user end of the present application can obtain the relevant information input by the tenant when querying logistics data, and send the tenant's user information and related information to the data processing end after forming a data collection task. The user end can also query the scheduling rule table corresponding to the data collection task, and determine the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule and store it, so that the data collection tasks can be evenly distributed. The scheduling is broken up at various time points, thereby avoiding the system's large peak problem and greatly reducing the server resource cost and system pressure; the scheduling system end of the present application can regularly filter out the order set for data collection from the next scheduling time corresponding to each stored logistics order, and then send the corresponding collection task to the data processing end, and query the scheduling rule table corresponding to each logistics order in the order set. After that, the next scheduling time of each logistics order is updated according to each scheduling rule and the current scheduling time, thereby further reducing the server resource cost and system pressure; after receiving the data collection task sent by the user end and the collection task corresponding to the order set sent by the scheduling system, the data processing end can collect and clean the logistics data according to the data collection task and the collection task corresponding to the order set, and query the scheduling rule table The scheduling rule corresponding to the cleaned logistics data, update the stored next scheduling time according to the scheduling rule and the current collection time, thereby further reducing the server resource cost and system pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 An architecture diagram of a periodic collection and scheduling system for shipping data applied to international logistics provided in an embodiment of the present application;

[0047] Figure 2A process diagram showing multi-terminal interaction provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a user terminal process provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a flow chart of a scheduling system end provided in an embodiment of the present application;

[0050] Figure 5 A flowchart of a data processing terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] In one embodiment, Figure 1 As shown, Figure 1 The present application provides an architecture diagram of a periodic collection and scheduling system for shipping data applied to international logistics, which is provided in an embodiment of the present application. The present application provides a periodic collection and scheduling system for shipping data applied to international logistics, and the system may include:

[0053] The operation side is used to set corresponding scheduling rules for each tenant according to the data collection needs of different tenants, and save the set scheduling rules into the scheduling rule table.

[0054] In this application, the operation end can set corresponding scheduling rules for each tenant according to the data collection needs of different tenants. For example, if tenant A wants to set his own scheduling frequency and does not follow the system default frequency, the corresponding scheduling rules can be set for him according to the data collection needs of tenant A; in addition, if tenant B does not have his own requirements, the system default scheduling rules can be used. After collecting the data collection needs of each tenant, the set scheduling rules can be saved in the scheduling rule table, so that the user end or the scheduling system and data processing end can call the scheduling rule table, and use the scheduling rule table to determine the data collection scheduling rules corresponding to different logistics orders.

[0055] For example, the operation end in this application can generate the following two scheduling rules according to the tenant's data collection requirements, and save them to the scheduling rule table, and then the operation end process ends:

[0056] 1) Tenant A + ship data source MSK + sailing + data collection every 4 hours;

[0057] 2) Tenant A + ship data source MSK+ collects data every hour.

[0058] The user end is used to obtain the relevant information input by the tenant when querying logistics data, and send the tenant's user information and the relevant information to the data processing end after forming a data collection task, and after querying the scheduling rule table corresponding to the data collection task, determine the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule and store it.

[0059] In this application, the tenant can enter the order number + shipping company, such as MSK, on ​​the user side, and query the logistics data through the input relevant information. At this time, the user side can generate a tenant query record and save it to the corresponding query record table. In addition, the user side can also query the dispatch rule table for dispatch rules that meet the conditions based on the tenant ID and the input shipping company data source, and determine the next dispatch time corresponding to the logistics order queried by the tenant based on the current query time and the dispatch rule, and store the next dispatch time through the cache system, so that other clients can update the stored next dispatch time.

[0060] For example, assuming that the current query time T1 is 16:05, if the scheduling rule indicates that the data collection frequency of the logistics order is once an hour, then the next scheduling time T2 of the logistics order will be 17:05. In this way, the query time of each tenant can be dispersed, thereby avoiding the system's large peak problem and greatly reducing server resource costs and system pressure.

[0061] The scheduling system end is used to periodically filter out the order set for data collection from the next scheduling time corresponding to each stored logistics order, and then send the corresponding collection task to the data processing end, and after querying the scheduling rules corresponding to each logistics order in the order set in the scheduling rule table, update the next scheduling time of each logistics order according to each scheduling rule and the current scheduling time.

[0062] In this application, the scheduling system can pre-develop a xxl-job scheduled task and set it to execute the scheduled task every X minutes (X can be configurable). Of course, this application can also develop a scheduled task in other ways, which can be set according to the actual situation and is not limited here.

[0063] Specifically, since the user end determines the next scheduling time corresponding to the logistics order queried by the tenant based on the current query time and scheduling rules and then stores it, the scheduling system end can periodically filter out the order set to be collected from the next scheduling time corresponding to each stored logistics order and send the corresponding collection task to the data processing end, so that the data processing end can obtain the logistics data corresponding to the logistics order.

[0064] In addition, since the logistics cycle of sea freight logistics is relatively long, the scheduling system can also query the scheduling rules corresponding to each logistics order in the order set in the scheduling rule table, and update the next scheduling time of each logistics order according to each scheduling rule and the current scheduling time. In this way, the logistics data of each logistics order can be continuously followed up through scheduled tasks, thereby improving the user experience while meeting the personalized needs of tenants.

[0065] The data processing end is used to receive the data collection tasks sent by the user end and the collection tasks corresponding to the order set sent by the scheduling system, collect and clean the logistics data according to the data collection tasks and the collection tasks corresponding to the order set, and after querying the scheduling rules corresponding to the cleaned logistics data in the scheduling rule table, update the stored next scheduling time according to the scheduling rules and the current collection time.

[0066] In the present application, the data processing end can receive data collection tasks sent by the user end through the message middleware, and can also receive collection tasks corresponding to order sets sent by the scheduling system through the message middleware. The data collection tasks and order sets include but are not limited to user information (tenant id, user id), query order number, query data source code (such as shipping company MSK or port area code CNSHA), unique identifier of this query, etc. In this way, the corresponding logistics data can be queried according to the data collection tasks and the collection tasks corresponding to the order set, and the queried logistics data can be cleaned and saved in the corresponding table. When the tenant queries later, the corresponding logistics data can be displayed through this table.

[0067] Furthermore, since the present application collects logistics data in a periodic manner, after the data collection is completed, the data processing end can also query the scheduling rules corresponding to the cleaned logistics data in the scheduling rules table, and update the stored next scheduling time according to the scheduling rules and the current collection time, thereby continuously following up on the logistics data of each logistics order.

[0068] In the above embodiment, the system includes an operation end, a user end, a scheduling system and a data processing end, wherein the operation end can set corresponding scheduling rules for each tenant according to the data collection requirements of different tenants, and save the set scheduling rules into a scheduling rule table, so that different data collection frequencies can be set for different tenants, and different collection frequency requirements can be set for different data sources under multiple collection data sources, which has strong scalability, and different data collection frequencies can be set according to logistics nodes, thereby greatly reducing server costs; the user end of the present application can obtain the relevant information input by the tenant when querying logistics data, and send the tenant's user information and related information to the data processing end after forming a data collection task. The user end can also query the scheduling rule table corresponding to the data collection task, and determine the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule and store it, so that the data collection tasks can be evenly dispersed and scheduled at various time points, so as to This avoids the problem of large system peaks and greatly reduces server resource costs and system pressure. The scheduling system end of the present application can regularly screen out the order set to be collected from the next scheduling time corresponding to each stored logistics order, and then send the corresponding collection task to the data processing end. After querying the scheduling rule table for the scheduling rule corresponding to each logistics order in the order set, the next scheduling time of each logistics order is updated according to each scheduling rule and the current scheduling time, thereby further reducing server resource costs and system pressure. After receiving the data collection task sent by the user end and the collection task corresponding to the order set sent by the scheduling system, the data processing end can collect and clean the logistics data according to the data collection task and the collection task corresponding to the order set, and after querying the scheduling rule table for the scheduling rule corresponding to the cleaned logistics data, the stored next scheduling time is updated according to the scheduling rule and the current collection time, thereby further reducing server resource costs and system pressure.

[0069] In one embodiment, the operation end sets corresponding scheduling rules for each tenant according to the data collection requirements of different tenants, which may include:

[0070] The operation side sets scheduling rules corresponding to different logistics nodes under different data sources in different categories for each tenant based on the different tenants' demands for logistics scheduling frequency.

[0071] In this embodiment, the operation side can set the scheduling rules corresponding to different logistics nodes of different data sources under different categories according to the requirements of different tenants for logistics scheduling frequency. In this way, different data collection frequencies can be set for different tenants, and under multiple collection data sources, tenants can set different collection frequency requirements for different data sources. It has strong scalability and can also set different data collection frequencies according to logistics nodes, thereby greatly reducing server costs.

[0072] In a specific implementation, Figure 2 As shown, Figure 2 A process display diagram of multi-terminal interaction provided for an embodiment of the present application; in the present application, the operation end can select the major category for which rules need to be set according to the needs of different tenants for logistics scheduling frequency, such as selecting a shipping company or a port area, and then determine the tenant for setting the scheduling rules, such as: setting it for tenant A; then, based on the needs of tenant A, determine whether to set scheduling rules for a certain shipping company data source, such as: choose to set scheduling rules for MSK; continue to determine whether to set different scheduling rules according to logistics nodes based on the needs of tenant A, such as choosing to set different scheduling rules for the MSK sailing node and other nodes other than the MSK sailing node, so that scheduling rules corresponding to different tenants can be formed.

[0073] In one embodiment, the user terminal determines and stores the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule, which may include:

[0074] The user terminal determines the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule, converts the next scheduling time into a millisecond value, and stores it using the zadd command of the zset data structure of Redis.

[0075] In this embodiment, Figure 2 As shown, when the user determines and stores the next scheduling time corresponding to the logistics order queried by the tenant based on the current query time and scheduling rules, the user can first determine the next scheduling time corresponding to the logistics order queried by the tenant based on the current query time and scheduling rules, and then convert the next scheduling time into a millisecond value, and then use the zadd command of the zset data structure of Redis to store it.

[0076] In a specific implementation, assuming that the current query time T1 is 16:05, the next scheduling time T2 calculated according to the scheduling rule is 17:05, and the next scheduling time T2 is converted into millisecond value and stored using the zadd command of the zset data structure of Redis.

[0077] For example, the key of zset = data_collect_schedule: tenant unique identifier: data source (data_collect_schedule:A:MSK), score = the millisecond value corresponding to the next scheduling time T2 of the logistics order, value = the order number of the logistics order.

[0078] In addition, when updating the next scheduling time, the scheduling system end and the data processing end of the present application can also use the above calculation method to generate the latest next scheduling time, and use the latest next scheduling time to replace the original next scheduling time for updating.

[0079] In one embodiment, the scheduling system end periodically selects the order set to be collected from the next scheduling time corresponding to each stored logistics order and sends the corresponding collection task to the data processing end, which may include:

[0080] When executing the preset timed task, the scheduling system uses the zrange command to scan the logistics orders for the next scheduling time corresponding to the stored logistics orders to be collected, and after the scanned logistics orders are formed into an order set, the corresponding collection task is sent to the data processing end.

[0081] In this embodiment, Figure 2 As shown, when the scheduling system end periodically filters out the order set for data to be collected from the next scheduling time corresponding to each stored logistics order and sends the corresponding collection task to the data processing end, you can first set the scheduled task, and then when executing the scheduled task, use the zrange command to scan out the logistics orders for data to be collected in the next scheduling time corresponding to each stored logistics order, and after the scanned logistics orders form an order set, send the corresponding collection task to the data processing end.

[0082] For example, when executing a scheduled task, the scheduling system of the present application can use the zrange command to scan the value of score (millisecond value of the next scheduling time)>=current time-X minutes, and then obtain the order number set B that currently needs to perform data collection and update. The zrange command is used as follows: zrange data_collect_schedule:A:MSK current time-X minutes timestamp T10 current time timestamp T11.

[0083] In one embodiment, the data processing end collects and cleans the logistics data according to the data collection task and the collection task corresponding to the order set, which may include:

[0084] The data processing end collects logistics data by calling the data collection interface provided by the data source corresponding to each logistics order according to the data collection task and the collection task corresponding to the order set, and cleans the logistics data using a preset data cleaning program.

[0085] In this embodiment, Figure 2 As shown, when the data processing end collects and cleans the logistics data according to the collection tasks corresponding to the data collection tasks and the order set, the data processing end can first determine the logistics order according to the collection tasks corresponding to the data collection tasks and the order set, and then call the data collection interface provided by the data source corresponding to each logistics order to collect the logistics data. In this way, the collected logistics data can be cleaned using the preset data cleaning program, so that after the cleaning is completed according to the product requirements, the current latest logistics node of the order can be calculated.

[0086] In one embodiment, the present application further provides an operation end, which is applied to a periodic collection and scheduling system for shipping data applied to international logistics as described in any one of the above embodiments, and may include:

[0087] According to the data collection requirements of different tenants, corresponding scheduling rules are set for each tenant, and the set scheduling rules are saved in the scheduling rule table.

[0088] In this embodiment, the operation end can set corresponding dispatching rules for each tenant according to the data collection requirements of different tenants. For example, if tenant A wants to set his own dispatching frequency and does not follow the system default frequency, the corresponding dispatching rules can be set for him according to the data collection requirements of tenant A; in addition, if tenant B does not have his own requirements, the system default dispatching rules can be followed. After collecting the data collection requirements of each tenant, the set dispatching rules can be saved in the dispatching rule table, so that the user end or the dispatching system and the data processing end can call the dispatching rule table, and determine the data collection dispatching rules corresponding to different logistics orders through the dispatching rule table.

[0089] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of a process flow of a user terminal provided in an embodiment of the present application; the present application also provides a user terminal, which is applied to a periodic collection and scheduling system for shipping data applied to international logistics as described in any one of the above embodiments, and may include:

[0090] S110: Obtain relevant information input by the tenant when querying logistics data, and form the tenant's user information and related information into a data collection task and send it to the data processing end.

[0091] S120: After querying the scheduling rule corresponding to the data collection task in the scheduling rule table, determine the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule and store it.

[0092] In this embodiment, the tenant can enter the order number + shipping company, such as MSK, on ​​the user side, and query the logistics data through the input relevant information. At this time, the user side can generate a tenant query record and save it to the corresponding query record table. In addition, the user side can also query the scheduling rules that meet the conditions in the scheduling rule table based on the tenant ID and the input shipping company data source, and determine the next scheduling time corresponding to the logistics order queried by the tenant based on the current query time and the scheduling rules, and store the next scheduling time through the cache system, so that other clients can update the stored next scheduling time.

[0093] In one embodiment, determining and storing the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule in S120 may include:

[0094] S121: Determine the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule, convert the next scheduling time into a millisecond value, and store it using the zadd command of the zset data structure of Redis.

[0095] In this embodiment, when the user end determines and stores the next scheduling time corresponding to the logistics order queried by the tenant based on the current query time and scheduling rules, it can first determine the next scheduling time corresponding to the logistics order queried by the tenant based on the current query time and scheduling rules, and then convert the next scheduling time into a millisecond value, and then use the zadd command of the Redis zset data structure to store it.

[0096] In a specific implementation, assuming that the current query time T1 is 16:05, the next scheduling time T2 calculated according to the scheduling rule is 17:05, and the next scheduling time T2 is converted into millisecond value and stored using the zadd command of the zset data structure of Redis.

[0097] For example, the key of zset = data_collect_schedule: tenant unique identifier: data source (data_collect_schedule:A:MSK), score = the millisecond value corresponding to the next scheduling time T2 of the logistics order, value = the order number of the logistics order.

[0098] In one embodiment, Figure 4 As shown, Figure 4A flow chart of a scheduling system end provided in an embodiment of the present application; the present application also provides a scheduling system end, which is applied to a periodic collection and scheduling system for shipping data applied to international logistics as described in any one of the above embodiments, and may include:

[0099] S210: Periodically filter out the order set for data collection from the next scheduling time corresponding to each stored logistics order, and then send the corresponding collection task to the data processing end.

[0100] S220: After querying the scheduling rule corresponding to each logistics order in the order set in the scheduling rule table, the next scheduling time of each logistics order is updated according to each scheduling rule and the current scheduling time.

[0101] In this embodiment, the scheduling system can pre-develop a xxl-job scheduled task and set it to execute the scheduled task every X minutes (X can be configurable). Of course, this application can also develop a scheduled task in other ways, which can be set according to the actual situation and is not limited here.

[0102] Specifically, since the user end determines the next scheduling time corresponding to the logistics order queried by the tenant based on the current query time and scheduling rules and then stores it, the scheduling system end can periodically filter out the order set to be collected from the next scheduling time corresponding to each stored logistics order and send the corresponding collection task to the data processing end, so that the data processing end can obtain the logistics data corresponding to the logistics order.

[0103] In addition, since the logistics cycle of sea freight logistics is relatively long, the scheduling system can also query the scheduling rules corresponding to each logistics order in the order set in the scheduling rule table, and update the next scheduling time of each logistics order according to each scheduling rule and the current scheduling time. In this way, the logistics data of each logistics order can be continuously followed up through scheduled tasks, thereby improving the user experience while meeting the personalized needs of tenants.

[0104] In one embodiment, Figure 5 As shown, Figure 5 A flow chart of a data processing end provided in an embodiment of the present application; the present application further provides a data processing end, which is applied to a periodic collection and scheduling system for shipping data applied to international logistics as described in any one of the above embodiments, and may include:

[0105] S310: Receive the data collection task sent by the user end and the collection task corresponding to the order set sent by the scheduling system end.

[0106] S320: Collect and clean logistics data according to the data collection tasks and the collection tasks corresponding to the order set.

[0107] S330: After querying the scheduling rule table for the scheduling rule corresponding to the cleaned logistics data, the stored next scheduling time is updated according to the scheduling rule and the current collection time.

[0108] In this embodiment, the data processing end can receive data collection tasks sent by the user end through the message middleware, and can also receive collection tasks corresponding to order sets sent by the scheduling system through the message middleware. The data collection tasks and order sets include but are not limited to user information (tenant id, user id), query order number, query data source code (such as shipping company MSK or port area code CNSHA), unique identifier of this query, etc., so that the corresponding logistics data can be queried according to the data collection tasks and order sets, and the queried logistics data can be cleaned and saved in the corresponding table. When the tenant queries later, the corresponding logistics data can be displayed through this table.

[0109] Furthermore, since the present application collects logistics data in a periodic manner, after the data collection is completed, the data processing end can also query the scheduling rules corresponding to the cleaned logistics data in the scheduling rules table, and update the stored next scheduling time according to the scheduling rules and the current collection time, thereby continuously following up on the logistics data of each logistics order.

[0110] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0111] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0112] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A periodic collection and scheduling system for shipping data applied to international logistics, characterized in that: The system comprises: The operation side is used to set corresponding scheduling rules for each tenant according to the data collection requirements of different tenants, and save the set scheduling rules into the scheduling rule table; The user end is used to obtain the relevant information input by the tenant when querying the logistics data, and send the user information of the tenant and the relevant information to the data processing end after forming a data collection task, and after querying the scheduling rule table corresponding to the data collection task, determine the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule and store it; The scheduling system end is used to regularly select the order set for data collection from the next scheduling time corresponding to each stored logistics order, send the corresponding collection task to the data processing end, and query the scheduling rule table corresponding to each logistics order in the order set, and update the next scheduling time of each logistics order according to each scheduling rule and the current scheduling time; The data processing end is used to receive the data collection tasks sent by the user end and the collection tasks corresponding to the order set sent by the scheduling system end, collect and clean the logistics data according to the data collection tasks and the collection tasks corresponding to the order set, and after querying the scheduling rules corresponding to the cleaned logistics data in the scheduling rule table, update the stored next scheduling time according to the scheduling rules and the current collection time.

2. The periodic collection and scheduling system for shipping data applied to international logistics according to claim 1 is characterized in that: The operation end sets corresponding scheduling rules for each tenant according to the data collection requirements of different tenants, including: The operation side sets scheduling rules corresponding to different logistics nodes under different data sources in different categories for each tenant based on the different tenants' demands for logistics scheduling frequency.

3. The periodic collection and scheduling system for ocean shipping data applied to international logistics according to claim 1 is characterized in that: The user terminal determines and stores the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule, including: The user terminal determines the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule, converts the next scheduling time into a millisecond value, and stores it using the zadd command of the zset data structure of Redis.

4. The periodic collection and scheduling system for ocean shipping data applied to international logistics according to claim 1 is characterized in that: The scheduling system end regularly selects the order set to be collected from the next scheduling time corresponding to each stored logistics order and sends the corresponding collection task to the data processing end, including: When executing the preset timed task, the scheduling system uses the zrange command to scan the logistics orders for the next scheduling time corresponding to the stored logistics orders to be collected, and after the scanned logistics orders are formed into an order set, the corresponding collection task is sent to the data processing end.

5. The periodic collection and scheduling system for ocean shipping data applied to international logistics according to claim 1 is characterized in that: The data processing end collects and cleans the logistics data according to the data collection task and the collection task corresponding to the order set, including: The data processing end collects logistics data by calling the data collection interface provided by the data source corresponding to each logistics order according to the data collection task and the collection task corresponding to the order set, and cleans the logistics data using a preset data cleaning program.

6. An operation terminal, applied to a periodic collection and scheduling system for shipping data applied to international logistics as claimed in any one of claims 1 to 5, characterized in that: include: According to the data collection requirements of different tenants, corresponding scheduling rules are set for each tenant, and the set scheduling rules are saved in the scheduling rule table.

7. A user terminal, applied to a periodic collection and scheduling system for shipping data applied to international logistics as claimed in any one of claims 1 to 5, characterized in that: include: Obtain relevant information input by the tenant when querying logistics data, and send the tenant's user information and the relevant information to the data processing end after forming a data collection task; And, after querying the scheduling rule corresponding to the data collection task in the scheduling rule table, the next scheduling time corresponding to the logistics order queried by the tenant is determined according to the current query time and the scheduling rule and stored.

8. The user terminal according to claim 7, characterized in that: The determining and storing the next scheduling time corresponding to the logistics order queried by the tenant according to the current query time and the scheduling rule includes: The next scheduling time corresponding to the logistics order queried by the tenant is determined according to the current query time and the scheduling rules, and the next scheduling time is converted into a millisecond value and stored using the zadd command of the zset data structure of Redis.

9. A scheduling system end, applied to a periodic collection and scheduling system for shipping data applied to international logistics as claimed in any one of claims 1 to 5, characterized in that: include: Regularly filter out the order set to be collected from the next scheduling time corresponding to each stored logistics order, and then send the corresponding collection task to the data processing end; After querying the dispatching rule corresponding to each logistics order in the order set in the dispatching rule table, the next dispatching time of each logistics order is updated according to each dispatching rule and the current dispatching time.

10. A data processing terminal, applied to a periodic collection and scheduling system for shipping data applied to international logistics as claimed in any one of claims 1 to 5, characterized in that: include: Receive data collection tasks sent by the user end and collection tasks corresponding to the order set sent by the scheduling system end; Collect and clean logistics data according to the data collection task and the collection task corresponding to the order set; And, after querying the scheduling rule corresponding to the cleaned logistics data in the scheduling rule table, the stored next scheduling time is updated according to the scheduling rule and the current collection time.

Images