Detection task allocation and progress tracking method based on big data management system

Abstract

The invention belongs to the technical field of task allocation and progress tracking, and particularly discloses and provides a detection task allocation and progress tracking method based on a big data management system, which comprises the following steps: firstly, calculating the matching degree of task and processing end multi-aspect information through a weighted fusion algorithm, and selecting a processing end with the highest matching degree as a main node; judging whether a constraint processing task is met or not, collecting related data during task execution, performing coupling analysis to obtain a progress standard-reaching rate, marking the processing end of which the standard-reaching rate is smaller than a set value as abnormal, and calculating a secondary allocation abnormal processing end task through a matching degree in combination with the progress and residual resource information of the processing end; according to the method, the parameters of the to-be-detected task and the processing end are synthesized through the weighted fusion algorithm, the matching degree including calculation, storage, time matching degree and task type correlation degree is analyzed, global optimal matching of the task and the processing end in the aspects of computing power, storage, task types and adaptability is accurately evaluated, and the problem that the task and the processing end are not matched is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of task allocation and progress tracking, and relates to a detection task allocation and progress tracking method based on a big data management system. Background Art

[0002] Ensuring the stability and reliability of big data management systems is crucial for their operation. Testing tasks encompass comprehensive checks on data integrity, consistency, performance efficiency, security, and other aspects of the system. However, with the continuous expansion of big data management systems and the dramatic increase in data volume, the allocation and progress tracking of testing tasks present significant challenges. Reasonable task allocation and effective progress tracking are crucial for optimizing the allocation of human and technical resources and avoiding duplication of work or missed tasks.

[0003] For example, the Chinese invention patent with publication number CN119668773A discloses a method, system and related equipment for CPU load balancing. This solution cuts the continuous execution time of the CPU into several time slices, and configures each time slice as a task container of a fixed time size. Furthermore, based on the task execution time and cycle of the task to be assigned, task allocation is attempted for each time slice, and the load of the time slice is calculated at the same time until the load rate is less than the target load rate to complete the task allocation. Finally, each task is polled until all tasks are successfully assigned. The solution of the present invention can quickly determine whether the task allocation scheme can meet the requirements of CPU load balancing through simulation, without the need for manual repetitive task allocation. The balancing effect is predictable and quantifiable, the rate is fast and the efficiency is high, and it can effectively overcome the problems existing in the prior art.

[0004] For example, the Chinese invention patent with publication number CN119759538A discloses a method, device, storage medium and electronic device for scheduling job tasks, which relates to the field of big data technology. Among them, the method includes: determining the arrival rate and average workload of the job tasks of the big data platform. According to the arrival rate and average workload of the job tasks, the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities are determined. According to the reserved processing time and reserved resource amount allocated by the big data platform to each job task corresponding to different priorities, resource scheduling is performed on each job task received by the big data platform within a preset time period. This application solves the technical problem in the prior art that, due to the characteristics of big data job scheduling with large data volume, large number of tasks and high timeliness requirements, there is uneven resource allocation in actual job scheduling scenarios, which leads to longer job task processing time.

[0005] The above existing technologies have the following deficiencies: 1. Currently, they only focus on CPU load balancing in the task allocation stage, and do not further monitor and analyze the processing end that executes the task. As a result, the allocation strategy fails to adapt to the actual processing capacity of the processing end in real time. When faced with tasks to be detected of different types, computational loads, deadlines, and storage requirements, it is easy for the tasks to be mismatched with the processing end, making it difficult to achieve optimal matching and dynamic coordination between detection tasks and processing end resources.

[0006] 2. Currently, reserved processing time and resources are preset based solely on task priority, a pre-allocation model that doesn't factor in dynamic adjustments such as real-time processing utilization and remaining storage space. As tasks execute, the resource status of the processing end constantly changes, leading to wasted or under-allocated resources and failing to fully meet actual task demands. Summary of the Invention

[0007] In view of this, in order to solve the problems raised in the above background technology, a detection task allocation and progress tracking method based on a big data management system is proposed.

[0008] The purpose of the present invention can be achieved through the following technical solutions: The present invention provides a detection task allocation and progress tracking method based on a big data management system, including: S1, based on the task type, computing amount, deadline, storage requirement space of the task to be detected, and the historical processing data, occupied computing power, and current remaining storage space of each processing end, the matching degree between each processing end and the task to be detected is obtained through multi-dimensional matching analysis.

[0009] S2. Select the processing end with the highest matching degree as the main processing end, and verify whether it meets the computing power, storage and time constraints through the dynamic programming algorithm. If not, split the task to be tested into subtasks and assign them to other processing ends according to the matching degree priority, and recursively verify the resource constraints of each subtask.

[0010] S3. When executing a task, a coupling analysis is performed based on the amount of completed tasks, consumed computing resources, consumed storage resources, and current task execution time of each processing end to obtain the task progress compliance rate of each processing end.

[0011] S4. The processing end whose task progress compliance rate is less than the set reference value is recorded as an abnormal processing end, and the abnormal processing end task is secondary allocated based on the current detection task progress data and remaining resource information of the abnormal processing end.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention integrates the parameters of the task to be detected and the processing end through a weighted fusion algorithm, analyzes the matching degree including computing, storage, time matching and task type matching, accurately evaluates the global optimal matching between the task and the processing end in computing power, storage, task type and adaptability, and solves the problem of mismatch between the task and the processing end.

[0013] (2) The present invention collects dynamic data of each processing end in real time for coupling analysis, obtains the task progress compliance rate of each processing end, and performs secondary allocation on the exception processing end, thereby realizing the closed-loop management of "allocation-execution-adjustment" of the exception processing end, effectively avoiding the problem of resource idleness or overload caused by static allocation.

[0014] (3) The present invention defines state variables through a dynamic programming algorithm. When the resources of the main processing end are insufficient, the task is automatically split into subtasks and recursively assigned to the processing end according to the matching degree. This solves the problem of task timeout caused by the resource bottleneck of a single processing end and improves the decomposition and collaborative execution efficiency of complex tasks.

[0015] (4) The present invention conducts coupled analysis on the amount of completed tasks, consumed computing resources, consumed storage resources and current task execution time to obtain the task progress compliance rate of each processing end, quantitatively evaluate the progress of the processing end in real time, accurately identify abnormal processing ends and perform secondary allocation based on the remaining resource information, thereby comprehensively improving the execution reliability and stability of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a schematic diagram of the connection of various modules of the system of the present invention.

[0018] Figure 2 Schematic diagram of the connection between the processing terminals and the matching degree calculation steps of the task to be detected in the present invention.

[0019] Figure 3 Schematic diagram of the connection steps of the coupling analysis of the task progress achievement rate of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] See also Figure 1 As shown, the present invention provides a detection task allocation and progress tracking method based on a big data management system, the method comprising: S1, based on the task type, computational load, deadline, storage requirement space of the task to be detected, and the historical processing data, occupied computing power, and current remaining storage space of each processing end, a multi-dimensional matching analysis is performed to obtain the matching degree between each processing end and the task to be detected.

[0022] It should be noted that the task type is obtained by parsing the input data file of the task to be tested. For example, if the input file is in .dcm format, it is identified as a "medical image processing task," and if the input data contains timestamp and sensor_id fields, it is identified as an "IoT data cleaning task." The computational cost is determined by analyzing the complexity of the task code. The cyclomatic complexity (CCN) of the task code is calculated using the radon library. The computational cost formula is: computational cost = number of lines of code × CCN × historical task calibration factor. For example, if the task code has 200 lines, CCN = 5, and the historical task calibration factor is 0.8, the computational cost = 200 × 5 × 0.8 = 800 units. The deadline is obtained by reading the task configuration file input to the task to be tested, and the storage space requirement is determined by static analysis of the size of the task's dependency files.

[0023] It should be added that the historical task calibration coefficient is calculated by statistically calculating the ratio of the actual computational amount of similar historical tasks to the number of lines of code × CCN, and taking the average value as the historical task calibration coefficient.

[0024] It should be noted that historical processing data is collected in real time by the log collection agent deployed on the processing end. The occupied computing power is obtained by using the monitoring tool to obtain the real-time CPU resource utilization rate, combined with the total computing power parameters of the device to calculate: occupied computing power = total computing power parameter × real-time resource utilization rate. The current remaining storage space is obtained by executing the disk space query command and parsing the available space field.

[0025] It should be added that the historical processing data includes the total computing power parameters of each processing end, the historical processing times of the task type to which the task to be detected belongs, the total historical processing times, and the qualified historical processing times. Among them, the total computing power parameters are retrieved from the log files of the processing end through the log analysis tool, and the log entries containing the total computing power parameters are located according to specific keywords, such as "hardware information record" or "total computing power parameters", and then the specific total computing power parameters are extracted from the log entries. The method for obtaining the historical processing times of the task type to which the task to be detected belongs is: each time the processing end completes a task, it records the relevant information of the task in the log, including the task type, and uses the log analysis tool to traverse the log files of the processing end and filter out the log entries of all task completion records. Then, for the task type to which the task to be detected belongs, the number of log entries containing the task type is counted, which is the historical processing times of the task type.

[0026] The total historical processing count for the task type to be tested is obtained by counting all task completion log entries in the processing end's logs. The qualified historical processing count for the task type to be tested is obtained by evaluating the task's processing results after completion and logging the qualified or unqualified results in the logs. The qualified historical processing count is obtained by traversing the processing end's log files and selecting log entries for the task type to be tested with qualified results.

[0027] See also Figure 2 As shown, exemplarily, the matching degree between each processing end and the task to be detected is obtained through multi-dimensional matching analysis, including: Y1, extracting the total computing power parameters of each processing end from the historical processing data, and taking the difference between the total computing power parameters and the occupied computing power as the available computing power of each processing end.

[0028] Y2. Normalize the computational effort of the task to be detected with the available computing power of each processing terminal to obtain the computational matching degree between each processing terminal and the task to be detected.

[0029] Furthermore, the analysis of the computational matching degree between each processing terminal and the task to be detected includes: Y2-1, performing a ratio analysis between the available computing power of each processing terminal and the computational amount of the task to be detected to obtain the original computational matching degree between each processing terminal and the task to be detected.

[0030] Y2-2. Filter the maximum value from the original calculated matching degrees as the benchmark calculated matching degree.

[0031] Y2-3. Perform a ratio operation on the original calculated matching degree and the reference calculated matching degree to obtain a normalized calculated matching degree between each processing end and the task to be detected.

[0032] Y3. The ratio of the current remaining storage space of the processing end to the storage space required by the task to be detected is used as the storage requirement matching degree, thereby obtaining the storage requirement matching degree between each processing end and the task to be detected.

[0033] Y4. Comprehensively analyze the computational effort and deadline of the task to be detected, the available computing power of each processing terminal, and the current time to obtain the time matching degree between each processing terminal and the task to be detected.

[0034] Furthermore, the analysis of the time matching degree between each processing end and the task to be detected includes: Y4-1, extracting the processing rate of each processing end corresponding to the task type to which the task to be detected belongs from the historical processing data of each processing end.

[0035] It should be added that the method for obtaining the processing rate is as follows: historical task records of the same type as the task to be detected are screened from the historical processing data of the processing end, and the processing rate of each historical task record is calculated, that is, the ratio of the computational amount of this processing to the actual processing time. If there are multiple historical task records, the arithmetic average of all processing rates is taken as the processing rate of the corresponding task type of the processing end.

[0036] Y4-2. The ratio of the computational complexity of the task to be detected to the processing rate of each processing end is used as the theoretical time consumption of the task at each processing end.

[0037] Y4-3. The difference between the deadline of the task to be detected and the current time is calculated as the permitted processing time of the task to be detected, and the ratio of the difference to the theoretical time consumption of the task is used as the time matching degree, thereby obtaining the time matching degree between each processing end and the task to be detected.

[0038] Y5. Extract relevant parameters of the task type to which the task to be detected belongs from the historical processing data of each processing end, perform comparative analysis, and obtain the matching degree between each processing end and the task type of the task to be detected.

[0039] Furthermore, the analysis of the matching degree between each processing end and the task type to be detected includes: Y5-1, taking the ratio of the historical processing times of each processing end in the relevant parameters to the total historical processing times as the historical processing frequency.

[0040] Y5-2. The ratio of the number of qualified historical processing times of each processing end in the relevant parameters to the number of historical processing times is used as the historical processing qualified rate.

[0041] Y5-3. The average value of the historical processing frequency and the historical processing pass rate is calculated as the matching degree between each processing end and the task type of the task to be detected.

[0042] Y6. Perform weighted sum calculation on the computational matching degree, storage requirement matching degree, time matching degree, and task type matching degree between each processing end and the task to be detected to obtain the matching degree between each processing end and the task to be detected.

[0043] The embodiment of the present invention uses a weighted fusion algorithm to comprehensively analyze the parameters of the task to be detected and the processing end, analyzes the matching degree including calculation, storage, time matching and task type matching, and accurately evaluates the global optimal matching between the task and the processing end in computing power, storage, task type and adaptability, thereby solving the problem of mismatch between the task and the processing end.

[0044] S2. Select the processing end with the highest matching degree as the main processing end, and verify whether it meets the computing power, storage and time constraints through the dynamic programming algorithm. If not, split the task to be tested into subtasks and assign them to other processing ends according to the matching degree priority, and recursively verify the resource constraints of each subtask.

[0045] Exemplarily, the verification of whether computing power, storage and time constraints are met by a dynamic programming algorithm includes: taking the available computing power of the main processing terminal to be greater than the computational amount of the task to be detected as condition 1.

[0046] Condition 2 is that the remaining storage space of the main processing end is greater than the storage space required by the task to be detected.

[0047] Condition 3 is that the theoretical processing time of the task on the main processing end is less than or equal to the permitted processing time of the task to be detected.

[0048] When conditions 1, 2, and 3 are all met, it indicates that the main processing end meets the conditional constraints; otherwise, it indicates that the main processing end does not meet the conditional constraints.

[0049] It should be added that dynamic programming is an efficient algorithm design method that decomposes complex problems into overlapping sub-problems and stores the optimal solutions of the sub-problems to avoid repeated calculations. The application of dynamic programming in resource verification is to verify whether the main processing end meets the computing power, storage and time constraints of the task. If not, the task is intelligently split and allocated to other processing ends. The verification process is as follows: (1) State definition: The state variable triple is available computing power, remaining storage, and remaining time, which represents the available computing power, remaining storage space and remaining permitted processing time in the initial state of the main processing end minus the computing amount, storage requirement and theoretical time consumption of the task to be tested.

[0050] (2) Constraint Verification and State Transfer: Directly verify whether the main processing end resources meet the constraints. If the available computing power ≥ the task computing capacity, the remaining storage ≥ the storage requirement, and the remaining time ≥ the permitted processing time, then the main processing end resources are determined to be sufficient. If any state variable does not meet the constraints, the task splitting is triggered and indicated as insufficient resources. The task to be tested is split into multiple subtasks, and a priority queue is generated according to the matching degree between the processing end and the subtask. The resource constraints of each processing end are recursively verified.

[0051] (3) Recursive verification: During the recursive process, the available computing power, remaining storage, and remaining time of the subtasks are inherited from the resource status of the main processing end after the subtasks are split. When the subtasks are assigned to the processing end, the resource status of the processing end is updated and a global constraint check is performed. If all subtasks are assigned and the total time consumed is ≤ the original task's permitted processing time, the verification is passed. Otherwise, a resource shortage warning is issued.

[0052] In a specific embodiment, the task to be detected and the resource parameters of the processing end are shown in Table 1.

[0053] Table 1: Parameters of tasks to be detected and processing resources

[0054]

[0055] The dynamic programming verification process is as follows: (1) Initial state calculation: available computing power = 80-100 = -20, remaining storage = 200-250 = -50, theoretical task time = 100 / 80 = 1.25 hours, remaining time = 6-1.25 = 4.75 hours, then the computing power and storage are insufficient, triggering task splitting.

[0056] (2) Task splitting and subtask allocation: Split into two subtasks based on resource requirements: Subtask 1: Computational effort 60, storage requirement 150, license processing time 3 hours. Subtask 2: Computational effort 40, storage requirement 100, license processing time 3 hours.

[0057] (3) Matching degree calculation: Through the matching degree calculation, the matching degree ranking of subtask 1 is candidate processing end A (0.85) > main processing end (0.75) > candidate processing end B (0.65). The matching degree ranking of subtask 2 is candidate processing end B (0.80) > candidate processing end A (0.70) > main processing end (0.60).

[0058] Recursive allocation verification: If subtask 1 is assigned to candidate processor A, and its available computing power is 50 < 60, the allocation fails and is deferred to the primary processor. If the primary processor has available computing power of 80 ≥ 60, remaining storage of 200 ≥ 150, and remaining time of 6 - 3 = 3 hours, the allocation succeeds and the primary processor updates its status to remaining computing power of 80 - 60 = 20, remaining storage of 200 - 150 = 50, and remaining time of 6 - 3 = 3 hours.

[0059] Subtask 2 is assigned to candidate processing terminal B, whose available computing power is 30 < 40. If the assignment fails, it is postponed to candidate processing terminal A. Candidate processing terminal A's available computing power is 50 ≥ 40, its remaining storage is 150 ≥ 100, and its remaining time is 4 - 3 = 1 hour. Therefore, the assignment is successful, and candidate processing terminal A updates its status to remaining computing power 50 - 40 = 10, remaining storage 150 - 100 = 50, and remaining time 4 - 3 = 1 hour.

[0060] (4) Global constraint check: total time = max(main processing end 3 hours, candidate processing end A 3 hours), then the total time is 3 hours, which is less than the permitted time of 6 hours, and the verification is passed.

[0061] (5) Confirmation of the optimal solution: The total matching degree = 0.75 + 0.70 = 1.45. The resource utilization rate is 20 for the remaining computing power of the main processing end and 10 for the remaining computing power of the candidate processing end A. The total remaining computing power is 30, and the original total remaining computing power is 80 + 50 + 30 = 160. Therefore, the utilization rate is approximately equal to 0.81, indicating that the solution is effective.

[0062] Exemplarily, the allocation to other processing terminals according to matching priority includes: calculating the matching degree between each processing terminal and each subtask according to the above matching degree calculation method, and constructing a priority sequence for each subtask according to the matching degree from high to low.

[0063] Extract the processing end ranked first from the priority sequence of the subtasks and calculate its resource carrying capacity. If the resource carrying capacity of the processing end is sufficient, assign the subtask to the processing end. Otherwise, verify in order according to the priority sequence until a processing end that meets the conditions is found.

[0064] It should be added that the process of determining resource carrying capacity is: comprehensively consider the current status of the processing end and the subtask requirements, and conduct a quantitative assessment from the three aspects of computing power, storage, and time.

[0065] Computing power capacity: Based on the occupied computing power of the processing end, such as the remaining computing power ratio, historical processing efficiency, such as the task processing volume per unit time, and the computing volume of the subtask, it is determined whether the remaining computing power of the processing end meets the computing requirements of the subtask, that is, remaining computing power ≥ subtask computing volume.

[0066] Storage capacity: Compare the current remaining storage space on the processing end with the storage space required by the subtask, and the remaining storage space must meet the requirement of ≥ the storage space required by the subtask.

[0067] Time carrying capacity: Based on the time window between the current time and the subtask deadline, combined with the estimated time required to process the subtask on the processing end, which is the average time required for similar tasks in the past, the resource carrying capacity is sufficient if all three factors are met. Otherwise, it is insufficient.

[0068] When the priority sequences of multiple subtasks all start with the same processing end, calculate the permitted processing time of each subtask, sort them from short to long according to the permitted processing time, and assign the subtasks to the processing end in sequence until its resource carrying capacity is insufficient.

[0069] Return the priority sequence of the unassigned subtasks, skip the processing end with insufficient resource carrying capacity, and continue to match the suboptimal processing end according to the matching degree until all assignments are completed.

[0070] The embodiment of the present invention defines state variables through a dynamic programming algorithm, automatically splits tasks into subtasks when the resources of the main processing end are insufficient, and recursively distributes them to the processing end according to the matching degree, thereby solving the task timeout problem caused by the resource bottleneck of a single processing end and improving the decomposition and collaborative execution efficiency of complex tasks.

[0071] S3. When executing a task, a coupling analysis is performed based on the amount of completed tasks, consumed computing resources, consumed storage resources, and current task execution time of each processing end to obtain the task progress compliance rate of each processing end.

[0072] It should be added that the amount of completed tasks is recorded in real time through the local task counter on the processing end, and the consumed computing resources are obtained by real-time collection of CPU utilization, CPU core usage, memory usage, GPU memory usage, and computing usage through operating system tools, programming interfaces, and hardware monitoring tools. The consumed storage resources are obtained by obtaining the local used storage space on the processing end through system commands and subtracting it from the storage space before executing the subtask.

[0073] See also Figure 3 As shown, exemplarily, the coupling analysis of the task progress achievement rate includes: Z1, taking the ratio of the completed task amount of each processing end to the assigned task amount as the task completion degree of each processing end.

[0074] Z2. Compare and analyze the consumed computing resources and consumed storage resources with the current task execution time to obtain the resource consumption rate of each processing end.

[0075] Furthermore, the calculation of the resource consumption rate includes: Z2-1, taking the ratio of the consumed computing resources of each processing end to the current task execution time as the computing resource consumption rate of each processing end, and analyzing the storage resource consumption rate of each processing end in the same way as the analysis method of the computing resource consumption rate.

[0076] Z2-2. Filter out the maximum value from the computing resource consumption rate and the storage resource consumption rate of each processing end as the resource consumption rate of each processing end.

[0077] Z3. Subtract the assigned task amount from the completed task amount, and use the quotient of the difference and the resource consumption rate as the predicted remaining processing time of each processing end.

[0078] Z4. Couple the task completion degree, resource consumption rate, and predicted remaining processing time of each processing end with the reference task completion degree, reference resource consumption rate, and reference remaining time to obtain the task progress compliance rate of each processing end.

[0079] It should be added that the reference task completion degree of each processing end is obtained based on the comparative analysis of the current task execution time and the permitted processing time. The reference resource consumption rate of each processing end is obtained by extracting historical tasks of similar task type to the current task from the historical processing data of each processing end. For the screened historical tasks, the computing resources and storage resources consumed by them are counted respectively. At the same time, combined with the task volume of the historical tasks, the total resource consumption of the historical tasks is divided by the task volume to obtain the computing resource consumption and storage resource consumption of the unit task. Then, the assigned task volume of the current task is multiplied by the unit task resource consumption, and then divided by the permitted processing time to obtain the theoretical resource consumption rate. The reference remaining time of each processing end is calculated based on the difference between the deadline and the current task execution time.

[0080] It should be added that the specific coupling analysis method is as follows: Assume that the task completion degree is , the reference task completion is , the resource consumption rate is , the reference resource consumption rate is , the remaining processing time is predicted to be , refer to the remaining time , through the formula Calculate the task progress compliance rate of each processing end ,in, 、 、 are the weights of task completion ratio, resource consumption rate ratio and processing remaining time ratio, .

[0081] It should be added that It is the ratio of the actual task completion to the reference task completion, reflecting the relative situation of task completion. It is the ratio of the actual resource consumption rate to the reference resource consumption rate, reflecting the relative situation of the resource consumption rate. It is the ratio of the predicted remaining time to the reference remaining time, reflecting the relative situation of the remaining time. By multiplying the ratios of the above three dimensions by the corresponding weights and summing them up, it comprehensively measures the task progress achievement rate of each processing end, ensuring a normalized comprehensive evaluation from three aspects: task completion, resource consumption rate, and remaining time.

[0082] It should be added that the above weights 、 、 The size relationship is not fixed, but is determined by the specific characteristics and requirements of the task. The specific analysis is as follows: If the task has extremely high time constraints, such as a real-time task, then will be given a greater weight, i.e. and To ensure that the time schedule meets the requirements. If the task is executed in a resource-constrained environment, such as limited hardware configuration, the resource consumption rate becomes the key. will be given a greater weight, i.e. and , giving priority to ensuring the rational use of resources. If the task emphasizes progressing according to plan, such as a project that strictly follows milestones, the task completion weight is Dominate, that is and , ensuring that actual progress is consistent with expectations.

[0083] The embodiment of the present invention performs a coupled analysis of the amount of completed tasks, consumed computing resources, consumed storage resources, and the current task execution time to obtain the task progress compliance rate of each processing end, quantitatively evaluate the progress of the processing end in real time, accurately identify abnormal processing ends, and perform secondary allocation based on the remaining resource information, thereby comprehensively improving the execution reliability and stability of the task.

[0084] S4. The processing end whose task progress compliance rate is less than the set reference value is recorded as an abnormal processing end, and the abnormal processing end task is secondary allocated based on the current detection task progress data and remaining resource information of the abnormal processing end.

[0085] Exemplarily, the secondary allocation of tasks on the exception handling end includes: calculating the amount of tasks that can be processed based on the current remaining available computing power and remaining storage space in the remaining resource information of the exception handling end, combined with the computing amount and storage requirement space of the remaining tasks in the exception handling end.

[0086] It should be noted that the number of tasks that can be processed = min(remaining available computing power / computational capacity, remaining storage space / storage requirement). Remaining available computing power / computational capacity represents the maximum number of tasks that the processing end's remaining computing power can support, considering only computing power resources. Remaining storage space / storage requirement represents the maximum number of tasks that the processing end's remaining storage can accommodate, considering only storage space. The processing end's number of tasks is limited by the more scarce resource and must meet both computing power and storage constraints. Therefore, the minimum of the two is taken to ensure that task allocation does not exceed the limit of either resource.

[0087] If the amount of processable tasks is greater than 0, the exception processing end will continue to process this part of the tasks, calculate the matching degree between each non-exceptional processing end and the remaining tasks, and select the non-exceptional processing end for allocation in descending order based on the matching degree. At the same time, it will verify whether the selected processing end meets the computing power, storage and time constraints until all remaining tasks are allocated.

[0088] If the amount of processable tasks is equal to 0, the remaining tasks of the exception handling end are allocated according to the above allocation method.

[0089] The embodiment of the present invention collects dynamic data of each processing end in real time for coupling analysis, obtains the task progress compliance rate of each processing end, and performs secondary allocation on the exception processing end to realize the closed-loop management of "allocation-execution-adjustment" of the exception processing end, effectively avoiding the resource idleness or overload problem caused by static allocation.

[0090] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0091] The above embodiments may be implemented in whole or in part through software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0092] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0093] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0094] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0095] Finally, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for assigning inspection tasks and tracking progress based on a big data management system, characterized by: The method includes: S1. Based on the task type, computational load, deadline, storage requirements, and historical processing data, occupied computing power, and current remaining storage space of each processing terminal, a multi-dimensional matching analysis is performed to determine the matching degree between each processing terminal and the task to be tested. S2. Select the processing end with the highest matching degree as the main processing end. Use a dynamic programming algorithm to verify whether it meets the computing power, storage, and time constraints. If not, split the task to be tested into subtasks and assign them to other processing ends according to matching degree priority. Recursively verify the resource constraints of each subtask. S3. During task execution, a coupled analysis is performed based on the amount of completed tasks, consumed computing resources, consumed storage resources, and current task execution time of each processing end to obtain the task progress compliance rate of each processing end; S4. The processing end whose task progress compliance rate is less than the set reference value is recorded as an abnormal processing end, and the abnormal processing end task is secondary allocated based on the current detection task progress data and remaining resource information of the abnormal processing end.

2. The method for assigning and tracking testing tasks based on a big data management system according to claim 1, characterized in that: The matching degree between each processing terminal and the task to be detected is obtained through multi-dimensional matching analysis, including: Y1. Extract the total computing power parameters of each processing terminal from the historical processing data, and use the difference between the total computing power parameters and the occupied computing power as the available computing power of each processing terminal; Y2. Normalize the computational effort of the task to be detected with the available computing power of each processing terminal to obtain the computational matching degree between each processing terminal and the task to be detected; Y3. The ratio of the current remaining storage space of the processing end to the storage space required by the task to be detected is used as the storage requirement matching degree, thereby obtaining the storage requirement matching degree between each processing end and the task to be detected; Y4. Comprehensively analyze the computational effort and deadline of the task to be detected, the available computing power of each processing terminal, and the current time to obtain the time matching degree between each processing terminal and the task to be detected; Y5. Extract relevant parameters of the task type of the task to be detected from the historical processing data of each processing terminal, conduct comparative analysis, and obtain the matching degree between each processing terminal and the task type of the task to be detected; Y6. Perform weighted sum calculation on the computational matching degree, storage requirement matching degree, time matching degree, and task type matching degree between each processing end and the task to be detected to obtain the matching degree between each processing end and the task to be detected.

3. The method for assigning and tracking testing tasks based on a big data management system according to claim 2, characterized in that: The analysis of the matching degree between each processing terminal and the task to be detected includes: Analyze the ratio of the available computing power of each processing terminal to the computing amount of the task to be detected, and obtain the original computing matching degree between each processing terminal and the task to be detected; Selecting the maximum value from the original calculated matching degrees as a benchmark calculated matching degree; A ratio operation is performed on the original calculated matching degree and the reference calculated matching degree to obtain a normalized calculated matching degree between each processing end and the task to be detected.

4. The method for assigning and tracking testing tasks based on a big data management system according to claim 2, characterized in that: The analysis of the time matching between each processing terminal and the task to be detected includes: Extracting the processing rate of each processing end corresponding to the task type to be detected from the historical processing data of each processing end; The ratio of the computational effort of the task to be detected to the processing rate of each processing end is taken as the theoretical time consumption of the task at each processing end; The difference between the deadline of the task to be detected and the current time is calculated as the permitted processing time of the task to be detected, and the ratio of it to the theoretical time consumption of the task is used as the time matching degree, thereby obtaining the time matching degree between each processing end and the task to be detected.

5. The method for assigning and tracking testing tasks and progress based on a big data management system according to claim 2, characterized in that: The analysis of the matching degree between each processing terminal and the task type to be detected includes: The ratio of the historical processing times of each processing terminal to the total historical processing times in the relevant parameters is used as the historical processing frequency; The ratio of the number of qualified historical processing times of each processing terminal in the relevant parameters to the number of historical processing times is used as the historical processing qualified rate; The average calculation result of the historical processing frequency and the historical processing qualification rate is used as the matching degree between each processing end and the task type of the task to be detected.

6. The method for assigning and tracking testing tasks and progress based on a big data management system according to claim 2, characterized in that: Verifying whether computing power, storage, and time constraints are met through a dynamic programming algorithm includes: Condition 1 is that the available computing power of the main processing end is greater than the computing power of the task to be detected; The remaining storage space of the main processing end is greater than the storage space required by the task to be detected as condition 2; Condition 3: The theoretical processing time of the task on the main processing end is less than or equal to the permitted processing time of the task to be detected; When conditions 1, 2, and 3 are all met, it indicates that the main processing end meets the conditional constraints; otherwise, it indicates that the main processing end does not meet the conditional constraints.

7. The method for assigning and tracking testing tasks and progress based on a big data management system according to claim 1, characterized in that: The allocation to other processing terminals according to matching priority includes: Calculate the matching degree between each processing terminal and each subtask according to the above matching degree calculation method, and build a priority sequence for each subtask from high to low according to the matching degree; Extract the first-ranked processing terminal from the subtask priority sequence and calculate its resource carrying capacity. If the resource carrying capacity of the processing terminal is sufficient, assign the subtask to it. Otherwise, verify it in order according to the priority sequence until a processing terminal that meets the conditions is found. When the priority sequence of multiple subtasks all starts with the same processing end, calculate the permitted processing time of each subtask, sort them from shortest to longest according to the permitted processing time, and assign the subtasks to the processing end in sequence until its resource carrying capacity is insufficient; Return the priority sequence of the unassigned subtasks, skip the processing end with insufficient resource carrying capacity, and continue to match the suboptimal processing end according to the matching degree until all assignments are completed.

8. The method for assigning and tracking testing tasks and progress based on a big data management system according to claim 1, characterized in that: The coupling analysis of the task progress achievement rate includes: Z1. The ratio of the completed tasks of each processing terminal to the assigned tasks is used as the task completion degree of each processing terminal; Z2. Compare and analyze the consumed computing resources and consumed storage resources with the current task execution time to obtain the resource consumption rate of each processing end; Z3. Subtract the assigned task amount from the completed task amount, and use the quotient of the difference and the resource consumption rate as the predicted remaining processing time for each processing terminal; Z4. Couple the task completion degree, resource consumption rate, and predicted remaining processing time of each processing end with the reference task completion degree, reference resource consumption rate, and reference remaining time to obtain the task progress compliance rate of each processing end.

9. The method for assigning and tracking testing tasks and progress based on a big data management system according to claim 8, characterized in that: The calculation of the resource consumption rate includes: The ratio of the consumed computing resources of each processing end to the current task execution time is used as the computing resource consumption rate of each processing end. The storage resource consumption rate of each processing end is obtained by the same analysis method as the computing resource consumption rate. The maximum value is selected from the computing resource consumption rate and the storage resource consumption rate of each processing end as the resource consumption rate of each processing end.

10. The method for assigning inspection tasks and tracking progress based on a big data management system according to claim 1, characterized in that: The secondary distribution of the exception handling end tasks includes: Based on the remaining available computing power and storage space in the remaining resource information of the exception handling end, combined with the computing amount and storage space required by the remaining tasks in the exception handling end, the amount of tasks that can be processed is calculated; If the amount of processable tasks is greater than 0, the exception processing end will continue to process the tasks in that part, calculate the matching degree between each non-exceptional processing end and the remaining tasks, and select the non-exceptional processing end for allocation in descending order based on the matching degree. At the same time, verify whether the selected processing end meets the computing power, storage and time constraints until all remaining tasks are allocated; If the amount of processable tasks is equal to 0, the remaining tasks of the exception handling end are allocated according to the above allocation method.

Images