Big data-based distribution matching method and system

Abstract

The invention relates to a big data-based distribution matching method and system, and the method specifically comprises the steps: starting a distribution matching program, recognizing a job sent to a big data platform in the distribution matching program, obtaining a plurality of intervals from the job, and enabling adjacent operators in each interval to be in narrow dependence; for each operator in the interval, giving execution time to the operator according to the time complexity of the operator, the data volume to be processed and the degree of parallelism, determining a light operator in each interval, and determining subtasks to which the light operator is divided based on the execution time of front and back operators of the light operator; determining the interval of each group of sub-tasks according to the operation distance of the group of sub-tasks and the operator between the two sub-tasks, and adjusting the coroutine execution sequence of the sub-tasks by using the execution time of the sub-tasks and the interval of the group of sub-tasks; and returning a job operation result of the big data platform to the distribution matching program, and completing distribution matching by the distribution matching program.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a distribution matching method and system based on big data. Background Art

[0002] E-commerce, food delivery, and express delivery have brought a lot of convenience to people's lives and have become an indispensable part of people's daily lives. The popularity of these services has greatly facilitated people's lives, allowing goods and food to reach consumers quickly. With the continuous growth of demand, the number of delivery orders has also shown a blowout outbreak, which has put higher requirements on logistics and delivery systems. Delivering goods to consumers quickly and well is the key to delivery services. Traditional delivery matching is mostly based on some rules, such as the closest distance. These delivery methods do not consider traffic information, such as congestion, or weather conditions. It is inevitable that there will be problems such as low delivery efficiency and low customer satisfaction.

[0003] With the rapid development of big data technology, massive amounts of order data, traffic data, user behavior data, etc. can be collected and analyzed in real time, providing new possibilities for delivery matching. The delivery matching system based on big data can make full use of this data to build more accurate and efficient delivery plans, improve resource utilization efficiency and customer experience. However, the real-time analysis of massive data requires high computing power. How to improve the speed of big data analysis has become the key to accurate delivery and improving the delivery service experience. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a delivery matching method based on big data, which comprises the following steps:

[0005] Obtain the delivery request submitted by the user, start the delivery matching program at the same time, identify the job sent to the big data platform in the delivery matching program, and obtain multiple intervals from the job. The adjacent operators in each interval are narrowly dependent.

[0006] For each operator in the interval, assign execution time to the operator according to the operator's time complexity, the amount of data to be processed, and the degree of parallelism, determine the lightweight operator in each interval, and determine the subtask to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator;

[0007] The subtasks in a task slot are grouped according to the jobs to which they belong. The interval between subtasks in each group is determined based on the distance between the subtasks in the job and the operator between two subtasks. The execution time of the subtasks and the interval between the subtasks in the group are used to adjust the execution order of the coroutines in which the subtasks are located.

[0008] The job running results of the big data platform are returned to the delivery matching program, and the delivery matching program completes the delivery matching.

[0009] Preferably, the determining of the lightweight operator in each interval determines the subtasks to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator, specifically:

[0010] If the ratio of the execution time of an operator to the time it takes to create a coroutine is less than the threshold, the operator is considered a lightweight operator;

[0011] Divide the lightweight operator and the operators with the minimum execution time into one subtask.

[0012] Preferably, the interval between subtasks in each group is determined according to the distance between the subtasks in the group and the operator between two subtasks, specifically:

[0013] Sort the subtasks according to the order of their subtasks in the same group, and calculate all operators between the last operator of the previous subtask and the first operator of the next subtask in the sorted adjacent subtasks;

[0014] The sum of the execution times of all the operators is taken as the interval between two adjacent subtasks.

[0015] Preferably, the method of adjusting the execution order of the coroutines in which the subtasks are located by using the execution time of the subtasks and the intervals between the subtasks in the group is specifically as follows:

[0016] The sum of the execution time of operators in a subtask is taken as the execution time of the subtask;

[0017] Sort the groups according to the priority of the jobs they belong to, and insert the subtasks in the next group into the previous task group according to the interval and execution time of the subtasks in the group, and repeat until the last task group;

[0018] Sort and execute the coroutines to which the subtasks belong according to the subtask sorting results.

[0019] Preferably, the subtasks in the next group are inserted into the previous task group cyclically according to the interval of the subtasks in the group and the subtask execution time, and the process is repeated until the last task group, specifically:

[0020] Find the first interval in the previous task group that can accommodate the first subtask in the next task group, add the first subtask of the next task group to the previous task group, move the newly added subtask from front to back within the interval of the previous task group where the first subtask in the next task group is located, and keep the interval between the first subtask and the second subtask in the next task group unchanged, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; if not, increase the interval between the first subtask and the second subtask in the next task group, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; and so on, until all the subtasks in the next group are added to the previous group, and the previous group is updated to have all the subtasks in the next group added;

[0021] Repeat the above process until the last task grouping is added to the previous grouping.

[0022] On the other hand, the present invention provides a delivery matching system based on big data, the system comprising the following modules:

[0023] The job interval division module is used to obtain the delivery request submitted by the user, start the delivery matching program at the same time, identify the job sent to the big data platform in the delivery matching program, and obtain multiple intervals from the job. The adjacent operators in each interval have narrow dependencies.

[0024] The subtask division module is used to assign execution time to each operator in the interval according to the operator's time complexity, the amount of data to be processed, and the degree of parallelism, determine the lightweight operator in each interval, and determine the subtask to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator;

[0025] The coroutine sorting module is used to group the subtasks in a task slot according to the jobs to which the subtasks belong, determine the interval between the subtasks in each group according to the distance between the subtasks in the job and the operator between two subtasks, and adjust the execution order of the coroutines in which the subtasks belong using the execution time of the subtasks and the interval between the subtasks in the group;

[0026] The delivery matching module is used to return the job running results of the big data platform to the delivery matching program, and the delivery matching program completes the delivery matching.

[0027] Preferably, the determining of the lightweight operator in each interval determines the subtasks to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator, specifically:

[0028] If the ratio of the execution time of an operator to the time it takes to create a coroutine is less than the threshold, the operator is considered a lightweight operator;

[0029] Divide the lightweight operator and the operators with the minimum execution time into one subtask.

[0030] Preferably, the interval between subtasks in each group is determined according to the distance between the subtasks in the group and the operator between two subtasks, specifically:

[0031] Sort the subtasks according to the order of their subtasks in the same group, and calculate all operators between the last operator of the previous subtask and the first operator of the next subtask in the sorted adjacent subtasks;

[0032] The sum of the execution times of all the operators is taken as the interval between two adjacent subtasks.

[0033] Preferably, the method of adjusting the execution order of the coroutines in which the subtasks are located by using the execution time of the subtasks and the intervals between the subtasks in the group is specifically as follows:

[0034] The sum of the execution time of operators in a subtask is taken as the execution time of the subtask;

[0035] Sort the groups according to the priority of the jobs they belong to, and insert the subtasks in the next group into the previous task group according to the interval and execution time of the subtasks in the group, and repeat until the last task group;

[0036] Sort and execute the coroutines to which the subtasks belong according to the subtask sorting results.

[0037] Preferably, the subtasks in the next group are inserted into the previous task group cyclically according to the interval of the subtasks in the group and the subtask execution time, and the process is repeated until the last task group, specifically:

[0038] Find the first interval in the previous task group that can accommodate the first subtask in the next task group, add the first subtask of the next task group to the previous task group, move the newly added subtask from front to back within the interval of the previous task group where the first subtask in the next task group is located, and keep the interval between the first subtask and the second subtask in the next task group unchanged, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; if not, increase the interval between the first subtask and the second subtask in the next task group, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; and so on, until all the subtasks in the next group are added to the previous group, and the previous group is updated to have all the subtasks in the next group added;

[0039] Repeat the above process until the last task grouping is added to the previous grouping.

[0040] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0041] In view of the problem that it takes a long time to call the big data platform in the matching program to complete real-time data analysis during delivery matching, the present invention, on the one hand, merges simple operators into a task in the task slot, and on the other hand, sorts the subtasks in the task slot to realize the scheduling of the coroutines where the subtasks are located, improves the coordination between the execution of subtasks, avoids frequent switching of coroutines, improves the execution efficiency of the big data platform, and thereby improves the speed of the delivery matching process. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of Embodiment 1;

[0043] Figure 2 This is a schematic diagram of the task slot;

[0044] Figure 3 This is a schematic diagram of tasks and coroutines in a task slot;

[0045] Figure 4 A schematic diagram for inserting subtasks into a group;

[0046] Figure 5 This is a structural diagram of the second embodiment. DETAILED DESCRIPTION

[0047] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Embodiment 1, the present invention provides a delivery matching method based on big data, such as Figure 1 As shown, the method comprises the following steps:

[0050] S1, obtains the delivery request submitted by the user, starts the delivery matching program at the same time, identifies the job sent to the big data platform in the delivery matching program, and obtains multiple intervals from the job. The adjacent operators in each interval are narrowly dependent.

[0051] On the delivery platform, users submit delivery requests, which include but are not limited to information such as the delivery address, delivery time, and product type. Delivery requests are submitted through mobile terminals such as mobile phones, pads, and other devices, or through PCs. After the delivery request is sent to the delivery platform, the delivery platform will start the delivery matching program, which includes multiple modules, such as route optimization, vehicle or rider scheduling, and traffic condition judgment. The delivery matching program includes some operations that require calling the big data platform, such as vehicle or rider statistics, user statistics, and real-time traffic analysis. These operations need to be completed in a short period of time to quickly complete the delivery matching task. Especially during the peak order period, the speed of big data analysis often becomes a bottleneck for the delivery matching speed.

[0052] After identifying the job to be processed by the big data platform in the distribution program, the job will be divided into multiple intervals. In one embodiment, the basis for dividing the intervals is based on the relationship between the operators in the job. Specifically, adjacent operators with narrow dependencies are divided into one interval. For example, operators 5-8 are adjacent operators in a job, and operators 5-7 are narrow dependencies, so operators 5-7 are divided into one interval.

[0053] S2: For each operator in the interval, assign execution time to the operator according to the operator's time complexity, the amount of data to be processed, and the degree of parallelism, determine the lightweight operator in each interval, and determine the subtask to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator;

[0054] The execution time of different operators is different. Some operators correspond to a small amount of data and have low time complexity, so they can be executed quickly. In addition, the execution time of an operator is also related to the amount of data to be processed by the operator. For example, for the same operator, the degree of parallelism is different, and each operator has a different amount of data to be processed and a different execution time. Different execution times are assigned to operators according to the operator's time complexity, the amount of data to be processed, and the operator's degree of parallelism. In a more detailed embodiment, the execution time is calculated as follows: weight × time complexity × amount of data to be processed / degree of parallelism, where the weight is related to the resource utilization of the core to which the thread of the operator's coroutine is scheduled. The greater the resource utilization, the greater the weight; in another embodiment, when the core to which the thread is scheduled is uncertain, the weight is uniformly set to 1. The amount of data to be processed is the sum of the amount of data to be processed by all parallel coroutines of an operator. For example, if a map operator has to process 100M of data and the degree of parallelism is 10, then the amount of data to be processed by each parallel operator is 10M. In another embodiment, the execution time is obtained by counting the execution time of each operator in a trial execution of the job. In the trial execution, the data volume of the job is not the actual data volume of the job to be called in the program, for example, it can be one tenth of the actual data volume.

[0055] Some operators, especially those with parallelism enabled, execute very quickly, even faster than creating a coroutine. It is obviously not cost-effective to create a coroutine for these operators. For each interval, find the lightweight operator in the interval. If the ratio of the execution time of an operator to the time to create a coroutine is less than the threshold, the operator is considered a lightweight operator. For example, if the execution time of an operator is 2s and the time to create a coroutine is 1s, if the threshold is 3, the operator is considered a lightweight operator.

[0056] After determining the lightweight operator, the lightweight operator and the preceding or following operator are divided into the subtask to be assigned. Preferably, the lightweight operator and the operator with the shortest execution time among the operators before and after the lightweight operator are divided into one subtask. For example, if a lightweight operator is the third operator in an interval, the execution time of the second and fourth operators in the same interval is determined. Assuming that the execution time of the second operator is the shortest, the second and third operators are merged into one subtask.

[0057] S3, group the subtasks in a task slot according to the jobs to which they belong, determine the interval between subtasks in each group according to the distance between the subtasks in the job and the operator between two subtasks, and adjust the execution order of the coroutines in which the subtasks are located using the execution time of the subtasks and the interval between the subtasks in the group;

[0058] In big data platforms such as Flink, each task manager (TaskManager) has multiple task slots (TaskSlot), and each task slot has at least one task, such as Figure 2 As shown, in the present invention, there is at least one subtask in the task slot, and each subtask is executed by a coroutine, such as Figure 3 As shown, each subtask has at least one operator.

[0059] After the subtasks are grouped according to the jobs to which they belong, all the subtasks in the group are subtasks in one job. In one job, the execution of subtasks has a sequence, and since not all tasks of a job are assigned to one task slot, if two non-adjacent subtasks are in the same task slot, there is a time interval between their execution timings, and other subtasks in the task slot can be executed within this time interval. In order to avoid frequent switching of the coroutine context, the coroutine switching will be performed after each execution of a subtask in the task slot. In the present invention, the interval between subtasks in each group is determined according to the distance between the subtasks in the group and the operator between the two subtasks, and then the coroutine execution order is adjusted according to the execution time of the subtasks and the interval. Preferably, the interval between subtasks in each group is determined according to the distance between the subtasks in the group and the operator between the two subtasks, specifically:

[0060] Sort the subtasks according to the order of their subtasks in the same group, and calculate all operators between the last operator of the previous subtask and the first operator of the next subtask in the sorted adjacent subtasks;

[0061] The sum of the execution times of all the operators is taken as the interval between two adjacent subtasks.

[0062] Group the subtasks in the task slots, assign the subtasks belonging to the same job to a group, and then sort the subtasks in the group according to the execution order of the subtasks in the job. The earlier the subtask is in the job, the earlier it is sorted. For each group, calculate the interval between two adjacent subtasks in the group. Since a subtask may have more than one operator, calculate the operators between the last operator in the previous subtask and the first operator in the next subtask. The operators in between do not include the last operator and the first operator. The sum of the execution time of the operators in between is taken as the interval between two adjacent subtasks.

[0063] After obtaining the execution order, execution time and interval of the subtasks, the subtasks in the same task slot are sorted. Since the subtasks and coroutines correspond one to one, sorting the subtasks is also sorting the coroutines. There are many ways to adjust the execution order of the coroutines in which the subtasks are located by using the execution time of the subtasks and the intervals of the subtasks in the group, such as linear programming, genetic algorithms, etc. In another embodiment, a simple method is provided, specifically:

[0064] The sum of the execution time of operators in a subtask is taken as the execution time of the subtask;

[0065] Sort the groups according to the priority of the jobs they belong to, and insert the subtasks in the next group into the previous task group according to the interval and execution time of the subtasks in the group, and repeat until the last task group;

[0066] Sort and execute the coroutines to which the subtasks belong according to the subtask sorting results.

[0067] If a subtask has only one operator, the execution time of this operator is the execution time of this subtask. If a subtask has multiple operators, the execution time of this subtask is the sum of the execution times of multiple operators.

[0068] Different jobs have different priorities. For high-priority jobs, priority is given to execution. For example, the result of sorting the groups according to the job priority is group 1, group 2, and group 3. There are 2 subtasks in group 1, 2 subtasks in group 2, and 1 subtask in group 3. Then, according to the execution time of the subtasks in groups 1 and 2 and the interval between the subtasks, the subtasks in group 2 are continuously inserted into group 1 until group 3 is also inserted into the updated group 1. The update means that in the process of continuously inserting the subtasks of other groups into the first group, the subtasks in the first group are continuously updated. For example, after inserting the first subtask in group 2, there are 3 subtasks in group 1. The insertion process needs to satisfy that the execution time of the inserted subtask is less than the interval between the two subtasks on the left and right of the insertion position. After the last subtask of the last task group is reached, assuming that the first task group becomes subtasks 11, 21, 12, 31, and 22, the subtasks are executed according to the above results, that is, the scheduling coroutine. Preferably, the priority is determined according to the position of the subtask in the job. The closer the position is, the higher the priority.

[0069] In a more detailed embodiment, the subtasks in the next group are inserted into the previous task group according to the interval of the subtasks in the group and the subtask execution time, and the process is repeated until the last task group. Specifically,

[0070] Find the first interval in the previous task group that can accommodate the first subtask in the next task group, add the first subtask of the next task group to the previous task group, move the newly added subtask from front to back within the interval of the previous task group where the first subtask in the next task group is located, and keep the interval between the first subtask and the second subtask in the next task group unchanged, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; if the first subtask of the next task moves to the end of the interval of the first subtask of the next task in the previous task and still overlaps, increase the interval between the first subtask and the second subtask in the next task group, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group;

[0071] This process continues in this way until all subtasks in the next group are added to the previous group, and the previous group is updated to include all subtasks in the next group.

[0072] Repeat the above process until the last task grouping is added to the previous grouping.

[0073] like Figure 4 As shown, subtasks 1, 2, and 3 belong to the previous task group, and subtasks 4 and 5 belong to the next task group. When inserting subtask 4 into the previous task group, find the first interval whose interval is greater than the execution time of subtask 4, and then insert subtask 5. Since there is an interval between subtasks 4 and 5, the subtasks are split into the previous task group according to the interval between subtasks 4 and 5. Subtask 2 and subtask 5 overlap. At this time, move task 4 backward until subtask 5 does not overlap with other subtasks in the previous task group. If subtask 5 overlaps with other subtasks in the previous task group no matter how subtask 4 is moved within the interval where subtask 4 is located, then increase the interval between subtask 4 and subtask 5 and move subtask 4 again.

[0074] After subtask 5 is inserted into the previous subtask group, assuming that there is a subtask 6 after subtask 5 in the next subtask group, then similarly, determine the interval of subtask 5 in the previous group, and move subtask 5 from front to back within the interval until there is no overlap between subtask 5 and other subtasks in the previous task group; if subtask 6 overlaps with other subtasks in the previous task group no matter how subtask 5 is moved within the interval where subtask 5 is located, then increase the interval between subtask 5 and subtask 6 and move subtask 6 again.

[0075] After all subtasks in the latter task group are added to the former subtask group, the two are merged as the new former group, and the task group to be inserted is taken as the latter group. The above process is repeated until all task groups are merged into one.

[0076] Assuming that all groups are merged, that is, all subtasks in the task slot are sorted into subtasks 1, 4, 2, 5, and 3, the coroutines will be executed in this order. On the one hand, this will prevent the subsequent coroutines that depend on the predecessor tasks from being scheduled in advance, that is, overcome the data dependency problem. On the other hand, it will avoid the competition of coroutines for shared resources, reduce the possibility of concurrency errors, and improve resource utilization.

[0077] S4, returns the job running results of the big data platform to the delivery matching program, and the delivery matching program completes the delivery matching.

[0078] After processing these tasks, the big data platform will generate a result, which can be data on the optimal delivery path, optimal delivery time, inventory analysis, etc. Then, the big data platform returns the results of these calculations and analyses to the delivery matching program. The delivery matching program uses these results to complete the final delivery matching task.

[0079] Embodiment 2: The present invention provides a delivery matching system based on big data, such as Figure 5 As shown, the system includes the following modules:

[0080] The job interval division module is used to obtain the delivery request submitted by the user, start the delivery matching program at the same time, identify the job sent to the big data platform in the delivery matching program, and obtain multiple intervals from the job. The adjacent operators in each interval have narrow dependencies.

[0081] The subtask division module is used to assign execution time to each operator in the interval according to the operator's time complexity, the amount of data to be processed, and the degree of parallelism, determine the lightweight operator in each interval, and determine the subtask to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator;

[0082] The coroutine sorting module is used to group the subtasks in a task slot according to the jobs to which the subtasks belong, determine the interval between the subtasks in each group according to the distance between the subtasks in the job and the operator between two subtasks, and adjust the execution order of the coroutines in which the subtasks belong using the execution time of the subtasks and the interval between the subtasks in the group;

[0083] The delivery matching module is used to return the job running results of the big data platform to the delivery matching program, and the delivery matching program completes the delivery matching.

[0084] Preferably, the determining of the lightweight operator in each interval determines the subtasks to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator, specifically:

[0085] If the ratio of the execution time of an operator to the time it takes to create a coroutine is less than the threshold, the operator is considered a lightweight operator;

[0086] Divide the lightweight operator and the operators with the minimum execution time into one subtask.

[0087] Preferably, the interval between subtasks in each group is determined according to the distance between the subtasks in the group and the operator between two subtasks, specifically:

[0088] Sort the subtasks according to the order of their subtasks in the same group, and calculate all operators between the last operator of the previous subtask and the first operator of the next subtask in the sorted adjacent subtasks;

[0089] The sum of the execution times of all the operators is taken as the interval between two adjacent subtasks.

[0090] Preferably, the method of adjusting the execution order of the coroutines in which the subtasks are located by using the execution time of the subtasks and the intervals between the subtasks in the group is specifically as follows:

[0091] The sum of the execution time of operators in a subtask is taken as the execution time of the subtask;

[0092] Sort the groups according to the priority of the jobs they belong to, and insert the subtasks in the next group into the previous task group according to the interval and execution time of the subtasks in the group, and repeat until the last task group;

[0093] Sort and execute the coroutines to which the subtasks belong according to the subtask sorting results.

[0094] Preferably, the subtasks in the next group are inserted into the previous task group cyclically according to the interval of the subtasks in the group and the subtask execution time, and the process is repeated until the last task group, specifically:

[0095] Find the first interval in the previous task group that can accommodate the first subtask in the next task group, add the first subtask of the next task group to the previous task group, move the newly added subtask from front to back within the interval of the previous task group where the first subtask in the next task group is located, and keep the interval between the first subtask and the second subtask in the next task group unchanged, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; if not, increase the interval between the first subtask and the second subtask in the next task group, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; and so on, until all the subtasks in the next group are added to the previous group, and the previous group is updated to have all the subtasks in the next group added;

[0096] Repeat the above process until the last task grouping is added to the previous grouping.

[0097] Embodiment 3, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in embodiment 1 is implemented.

[0098] Embodiment 4, the present invention further provides a computer device, the computer device comprising at least a memory and a processor, a computer program is stored in the memory, and the computer program implements the method described in embodiment 1 when executed by the processor.

[0099] Through the description of the above implementation methods, technicians in this field can clearly understand that each implementation method can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distribution matching method based on big data, characterized in that: The method comprises the following steps: Obtain the delivery request submitted by the user, start the delivery matching program at the same time, identify the job sent to the big data platform in the delivery matching program, and obtain multiple intervals from the job. The adjacent operators in each interval are narrowly dependent. For each operator in the interval, assign execution time to the operator according to the operator's time complexity, the amount of data to be processed, and the degree of parallelism, determine the lightweight operator in each interval, and determine the subtask to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator; The subtasks in a task slot are grouped according to the jobs to which they belong. The interval between subtasks in each group is determined based on the distance between the subtasks in the job and the operator between two subtasks. The execution time of the subtasks and the interval between the subtasks in the group are used to adjust the execution order of the coroutines in which the subtasks are located. The job running results of the big data platform are returned to the delivery matching program, and the delivery matching program completes the delivery matching.

2. The method according to claim 1, characterized in that The step of determining the lightweight operator of each interval and dividing the lightweight operator into subtasks based on the execution time of the operators before and after the lightweight operator is as follows: If the ratio of the execution time of an operator to the time it takes to create a coroutine is less than the threshold, the operator is considered a lightweight operator; Divide the lightweight operator and the operators with the minimum execution time into one subtask.

3. The method according to claim 1, characterized in that The interval between subtasks in each group is determined according to the distance between the subtasks in the group and the operator between two subtasks, specifically: Sort the subtasks according to the order of their subtasks in the same group, and calculate all operators between the last operator of the previous subtask and the first operator of the next subtask in the sorted adjacent subtasks; The sum of the execution times of all the operators is taken as the interval between two adjacent subtasks.

4. The method according to claim 1, characterized in that The method of adjusting the execution order of the coroutines in which the subtasks are located by using the execution time of the subtasks and the intervals between the subtasks in the group is specifically as follows: The sum of the execution time of operators in a subtask is taken as the execution time of the subtask; Sort the groups according to the priority of the jobs they belong to, and insert the subtasks in the next group into the previous task group according to the interval and execution time of the subtasks in the group, and repeat until the last task group; Sort and execute the coroutines to which the subtasks belong according to the subtask sorting results.

5. The method according to claim 4, characterized in that The subtasks in the next group are inserted into the previous task group according to the interval of the subtasks in the group and the execution time of the subtasks, and the process is repeated until the last task group. Specifically, Find the first interval in the previous task group that can accommodate the first subtask in the next task group, add the first subtask of the next task group to the previous task group, move the newly added subtask from front to back within the interval of the previous task group where the first subtask in the next task group is located, and keep the interval between the first subtask and the second subtask in the next task group unchanged, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; if not, increase the interval between the first subtask and the second subtask in the next task group, until the second subtask in the next task group does not overlap with all the subtasks of the previous task group after being added to the previous task group; and so on, until all the subtasks in the next group are added to the previous group, and the previous group is updated to have all the subtasks in the next group added; Repeat the above process until the last task grouping is added to the previous grouping.

6. A delivery matching system based on big data, characterized in that: The system includes the following modules: The job interval division module is used to obtain the delivery request submitted by the user, start the delivery matching program at the same time, identify the job sent to the big data platform in the delivery matching program, and obtain multiple intervals from the job. The adjacent operators in each interval have narrow dependencies. The subtask division module is used to assign execution time to each operator in the interval according to the operator's time complexity, the amount of data to be processed, and the degree of parallelism, determine the lightweight operator in each interval, and determine the subtask to which the lightweight operator is divided based on the execution time of the operators before and after the lightweight operator; The coroutine sorting module is used to group the subtasks in a task slot according to the jobs to which the subtasks belong, determine the interval between the subtasks in each group according to the distance between the subtasks in the job and the operator between two subtasks, and adjust the execution order of the coroutines in which the subtasks belong using the execution time of the subtasks and the interval between the subtasks in the group; The delivery matching module is used to return the job running results of the big data platform to the delivery matching program, and the delivery matching program completes the delivery matching.

7. The system according to claim 6, characterized in that The step of determining the lightweight operator of each interval and dividing the lightweight operator into subtasks based on the execution time of the operators before and after the lightweight operator is as follows: If the ratio of the execution time of an operator to the time it takes to create a coroutine is less than the threshold, the operator is considered a lightweight operator; Divide the lightweight operator and the operators with the minimum execution time into one subtask.

8. The system according to claim 6, characterized in that The interval between subtasks in each group is determined according to the distance between the subtasks in the group and the operator between two subtasks, specifically: Sort the subtasks according to the order of their subtasks in the same group, and calculate all operators between the last operator of the previous subtask and the first operator of the next subtask in the sorted adjacent subtasks; The sum of the execution times of all the operators is taken as the interval between two adjacent subtasks.

9. The system according to claim 6, characterized in that The method of adjusting the execution order of the coroutines in which the subtasks are located by using the execution time of the subtasks and the intervals between the subtasks in the group is specifically as follows: The sum of the execution time of operators in a subtask is taken as the execution time of the subtask; Sort the groups according to the priority of the jobs they belong to, and insert the subtasks in the next group into the previous task group according to the interval and execution time of the subtasks in the group, and repeat until the last task group; Sort and execute the coroutines to which the subtasks belong according to the subtask sorting results.

10. A computer storage device, wherein a computer program is stored on the storage device, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.

Images