Water conservancy big data efficient processing method and system based on multi-stage parallel strategy

Abstract

The invention provides a water conservancy big data efficient processing method and system based on a multi-stage parallel strategy, and the method comprises the steps: S1, dividing water conservancy data into a plurality of sub-data sets according to a preset time window, and packaging each sub-data set into an independent task package; s2, based on the task package, dynamic load balancing calculation is carried out through a CPU-GPU collaborative architecture, and the task package completing calculation is output; and S3, analyzing the calculated task package, and outputting the analyzed task package to a service end. According to the method, an improved seed spreading algorithm is adopted, the GPU computing resource utilization rate is increased through a CPU-GPU collaborative architecture and by utilizing a global task pool and a dynamic load migration technology, and overload or idling of the GPU is avoided by adopting a three-stage dynamic pulling control strategy. According to the method, a complete water conservancy big data processing system is constructed from four dimensions of algorithm design, hardware acceleration, data organization and business adaptation, the processing efficiency is improved, and reliable technical support is provided for intelligent water conservancy construction.

Description

Technical Field

[0001] This invention relates to the field of water conservancy big data processing technology, specifically to a method and system for efficient processing of water conservancy big data based on a multi-stage parallel strategy. Background Technology

[0002] With the rapid development of water conservancy informatization, various water conservancy monitoring equipment continuously generates massive amounts of real-time monitoring data and historical hydrological data, such as water level, flow rate, and water quality. This data is characterized by its large volume, diverse types, and rapid generation, posing a significant challenge to data processing. Traditional water conservancy big data processing methods are inefficient and fail to meet the requirements of real-time performance and accuracy.

[0003] Traditional seed propagation algorithms suffer from long computation times and low efficiency when processing large-scale water conservancy data due to their serial processing characteristics. Furthermore, their convergence speed slows significantly when the data distribution is uneven, resulting in poor real-time performance. Relying solely on CPUs to process large-scale water conservancy data is insufficient due to the limited number of CPU cores and inadequate parallel processing capabilities, failing to fully utilize the advantages of multi-core processors and resulting in processing speeds that cannot meet practical requirements.

[0004] Although there are some existing methods for processing big data in water conservancy using parallel computing, most of them do not fully consider the characteristics and processing needs of water conservancy data, and are insufficient in terms of algorithm optimization and hardware coordination, thus failing to achieve truly efficient processing.

[0005] A search of patent documents revealed an invention patent with publication number CN113888039A, which discloses a smart water management system based on big data. This system includes a flood data analysis module, a flood impact analysis module, a material dispatch management module, and a water management platform. The flood data analysis module acquires rainfall information within a region and analyzes the probability of flooding based on this information. The flood impact analysis module analyzes the degree of flood impact on a designated area, thereby determining the probability of flooding in that area. If the impact is high, the module notifies the designated area to take timely flood control measures. The material dispatch management module allocates materials to the designated area based on the flood's impact on it. However, this patent has shortcomings in data processing efficiency and lacks load balancing when analyzing data such as flood probability.

[0006] In summary, given the problems of the existing technologies, researching a method and system for efficient processing of water conservancy big data based on a multi-stage parallel strategy has become a critical task that urgently needs to be addressed. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method and system for efficient processing of water conservancy big data based on a multi-stage parallel strategy.

[0008] The present invention provides a method for efficient processing of water conservancy big data based on a multi-stage parallel strategy, comprising the following steps: Step S1: Divide the water conservancy data into multiple subsets according to a preset time window, and encapsulate each subset into an independent task package; Step S2: Based on the task package, perform dynamic load balancing calculations through a CPU-GPU collaborative architecture, and output the completed task package; Step S3: Parse the completed task package and output it to the business terminal.

[0009] Preferably, step S1 includes the following sub-steps: Step S1.1: The CPU divides the water conservancy data into N subsets according to a preset time window; generates an independent task package for each subset, which includes data content, calculation parameters and priority labels; constructs a KD-tree or Octtree index for each subset, and uses differential encoding and LZ4 compression for the coordinate data in the data content. Step S1.2: Obtain the hardware parameters of all GPU devices through the CUDA API. The hardware parameters include the number of streaming multiprocessors, the number of cores per streaming multiprocessor, the GPU core clock frequency, and the concurrency coefficient. Calculate the single-precision floating-point computing power of each GPU device based on the hardware parameters, and classify the GPU devices according to the single-precision floating-point computing power. Step S1.3: Assign the task package to the corresponding level of GPU device according to the priority label; Step S1.4: Dynamically configure the thread block size of the GPU device based on the spatial distribution density of the subset dataset; Step S1.5: Divide the shared memory of the GPU device into three regions according to function. The three regions store the neighborhood water level value, visited points, and whether the spread is complete, respectively.

[0010] Preferably, step S2 includes the following sub-steps: Step S2.1: On the CPU side, construct a global task pool and initialize task packages according to priority; Step S2.2: On the GPU side, dynamic balancing is achieved through task stealing, task splitting, and load migration, and the completed task package is output. Step S2.3: On the CPU side, dynamically adjust the priority of the global task pool based on GPU load metrics.

[0011] Preferably, in step S2.1, on the CPU side, a global task pool based on a lock-free queue is established, and task packages are initialized according to four priority levels: P0-P3. Among them, P0 level tasks are stored in a fixed memory area and are used to process real-time risk alarm tasks; P1 level tasks are stored in an ordered linked list based on timestamps and are used to process urgent business tasks; P2 level tasks are stored in a paged storage structure and are used to process routine analysis tasks; and P3 level tasks are stored in a disk cache area and are used to process background preprocessing tasks.

[0012] Preferably, step S2.2 includes the following sub-steps: In step S2.2.1, the GPU thread block monitors the global task pool in real time, steals tasks according to priority and minimum load principle, and allocates thread blocks to perform computation. Step S2.2.2: For the stolen task package, split the large task into smaller tasks according to the task type and spatial region to ensure that the computational load of each thread block is balanced. In step S2.2.3, load information is exchanged between thread blocks in real time. When the load difference is detected to exceed the preset migration threshold, some tasks are migrated from the high-load thread block to the low-load thread block.

[0013] Preferably, in step S2.2.1, the average task time is calculated using a two-layer trigger threshold method based on the exponential moving average algorithm. The exponential moving average formula is:

[0014] in: α is the smoothing factor; This represents the average task time after the current update, i.e., the EMA result. This indicates the actual time taken for the task that has just been completed; This represents the average task time calculated in the last time.

[0015] Preferably, in step S2.2.3, the formula for calculating the migration task volume is as follows:

[0016] in: local_load represents the remaining load of the current thread block; neighbor_load represents the load metric of adjacent thread blocks.

[0017] Preferably, step S2.3 includes the following sub-steps: Step S2.3.1: Collect the load metrics of the GPU in real time and obtain a comprehensive load score through weighted calculation; Step S2.3.2: When the overall load score exceeds the overload threshold, a dynamic degradation mechanism is triggered until the load returns to a normal level. Step S2.3.3: When the overall load score falls back to the preset normal range, gradually restore the task priority.

[0018] Preferably, in step S2.3.1, four load metrics of the GPU are collected: active thread block ratio, memory bandwidth utilization, task queue length, and task completion time. By standardizing the data of each metric to the 0-1 range and performing weighted calculations, the real-time load status of the GPU is quantified to obtain a comprehensive load score.

[0019] This invention also provides a high-efficiency water conservancy big data processing system based on a multi-stage parallel strategy, comprising: Module M1 divides the water conservancy data into multiple subsets according to a preset time window, and each subset is packaged into an independent task package; Module M2, based on task packages, performs dynamic load balancing computation through a CPU-GPU collaborative architecture and outputs the completed task packages; Module M3 parses the completed task package and outputs it to the business side.

[0020] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention employs an improved seed spread algorithm, which reduces the complexity of neighborhood query from O(n) to O(logn) through spatial region partitioning and dynamic load balancing strategies.

[0021] 2. This invention improves the utilization of GPU computing resources through a CPU-GPU collaborative architecture, utilizing a global task pool and dynamic load migration technology, and adopts a three-level dynamic pull control strategy to avoid GPU overload or idleness.

[0022] 3. This invention constructs a complete water conservancy big data processing system from four dimensions: algorithm design, hardware acceleration, data organization, and business adaptation, which improves processing efficiency and provides reliable technical support for the construction of smart water conservancy. Attached Figure Description

[0023] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a method for efficient processing of water conservancy big data based on a multi-stage parallel strategy, as described in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0025] This invention provides a method and system for efficient processing of water conservancy big data based on a multi-stage parallel strategy. It employs an improved seed spread algorithm, reducing the complexity of neighborhood queries from O(n) to O(logn) through spatial region partitioning and dynamic load balancing strategies. A CPU-GPU collaborative architecture, utilizing a global task pool and dynamic load migration technology, improves GPU computing resource utilization. A three-level dynamic pull control strategy avoids GPU overload or idleness.

[0026] Example 1: Figure 1 This is a flowchart of a method for efficient processing of water conservancy big data based on a multi-stage parallel strategy, as described in an embodiment of the present invention.

[0027] like Figure 1 As shown, this embodiment provides a method for efficient processing of water conservancy big data based on a multi-stage parallel strategy, including the following steps: Step S1: Divide the water conservancy data into multiple subsets according to a preset time window, and encapsulate each subset into an independent task package.

[0028] Specifically, step S1 includes the following sub-steps: In step S1.1, the CPU divides the water conservancy data into N subsets according to a preset time window, and generates an independent task package for each subset. The task package includes data content, calculation parameters, and priority labels. A KD-tree or Octree index is constructed for each subset to reduce the neighborhood query complexity from O(n) to O(logn), thereby optimizing the computational efficiency on the GPU. At the same time, the coordinate data in the data content is subjected to differential encoding and LZ4 compression to achieve a compression ratio of 3:1.

[0029] In this embodiment, the preset time window is 1 hour, and the calculation parameters include the spread threshold and the number of iterations.

[0030] Step S1.2: Obtain the hardware parameters of all GPU devices through the CUDA API. The hardware parameters include the number of streaming multiprocessors (SMs), the number of cores per streaming multiprocessor (SM), and the GPU core clock frequency. And the concurrency coefficient, based on the hardware parameters, calculate the single-precision floating-point computing power of each GPU device. The formula for calculating single-precision floating-point computing power is:

[0031] Wherein: the value 2 allows for 2 floating-point operations to be performed per clock cycle.

[0032] Furthermore, GPU devices are categorized based on their single-precision floating-point computing power.

[0033] In this embodiment, GPU devices with a single-precision floating-point computing power of ≥10 TFLOPS are classified as Level 1 devices, which are used to prioritize high-priority tasks; GPU devices with a single-precision floating-point computing power between 5 TFLOPS and 10 TFLOPS are classified as Level 2 devices, which are responsible for handling medium-priority tasks; and GPU devices with a single-precision floating-point computing power <5 TFLOPS are classified as Level 3 devices, which execute low-priority or background tasks.

[0034] Step S1.3: Assign the task package to the corresponding level of GPU device according to the priority label.

[0035] In this embodiment, the first-level device processes high-priority task packages, the second-level device processes medium-priority task packages, and the third-level device processes low-priority task packages.

[0036] Step S1.4: Dynamically configure the thread block size of the GPU device based on the spatial distribution density of the subset dataset.

[0037] In this embodiment, if the spatial distribution density of the subset is sparse, 512 threads / block are configured; if the spatial distribution density of the subset is dense, 256 threads / block are configured.

[0038] Step S1.5: Divide the shared memory of the GPU device into three regions according to function. The three regions store the neighborhood water level value, visited points, and whether the spread is complete, respectively.

[0039] In this embodiment, the capacity of a single shared memory block is calculated using the following formula:

[0040] The total capacity of shared memory does not exceed the hardware limit of the GPU device.

[0041] Step S2: Based on the task package, perform dynamic load balancing calculations through a CPU-GPU collaborative architecture and output the completed task package.

[0042] Specifically, step S2 includes the following sub-steps: Step S2.1: On the CPU side, construct a global task pool and initialize priorities.

[0043] Specifically, on the CPU side, a global task pool based on lock-free queues is established, and task packages are initialized according to four priority levels: P0-P3. Among them, P0 level tasks are stored in a fixed memory area and are used to handle real-time risk alarm tasks; P1 level tasks are stored in an ordered linked list based on timestamps and are used to handle urgent business tasks; P2 level tasks are stored in a paged storage structure and are used to handle routine analysis tasks; and P3 level tasks are stored in a disk cache area and are used to handle background preprocessing tasks.

[0044] On the CPU side, each compute core maintains a local cache queue with a fixed capacity of 32 tasks. When the number of tasks to be processed in the queue falls below 8, a batch prefetching mechanism is triggered, fetching new tasks from the global queue in batches of 16. This design reduces the performance overhead caused by cross-core cache consistency. For low-priority tasks at the P3 level, an intelligent merging optimization function is implemented, which can automatically package 128 small compute tasks into one large task package, thereby effectively reducing the scheduling overhead of the GPU.

[0045] Step S2.2: On the GPU side, dynamic balancing is achieved through task stealing, task splitting, and load migration, and the completed task package is output.

[0046] Specifically, step S2.2 includes the following sub-steps: In step S2.2.1, the GPU thread block monitors the global task pool in real time, steals tasks based on priority and minimum load principles, and allocates thread blocks to perform computations.

[0047] In this embodiment, when a GPU thread block is idle, it enters a pull mode, where the idle thread retrieves tasks from the global task pool. Task allocation follows the principle of least load priority, dynamically assigning new tasks to the GPU processing unit with the lowest current load by monitoring the utilization metrics of each GPU in real time.

[0048] To optimize task scheduling efficiency in high-concurrency scenarios, a three-level dynamic task retrieval control strategy is adopted. When the GPU is detected to have taken more than 5ms to retrieve tasks three consecutive times, the system switches to push mode, where the CPU allocates task packages to the GPU with the lowest load. The specific scheduling process includes: first, calculating the current GPU load score based on four monitoring metrics (active thread block ratio, memory bandwidth utilization, task queue length, and last task latency); then, determining the task retrieval scale based on the score: a maximum of 64 tasks are retrieved at a time in a light-load state (LoadScore < 0.3), limited to 16 tasks in a heavy-load state (0.3 ≤ LoadScore < 0.7), and only 4 tasks are allowed to be retrieved in a very heavy-load state (LoadScore ≥ 0.7); finally, atomic operations are used to complete task retrieval and queue status updates. This intelligent scheduling mechanism ensures efficient utilization of GPU resources, avoiding both resource idleness and overload.

[0049] Specifically, after completing the current task package, the GPU thread block determines whether to trigger the task stealing mechanism. When it is detected that the task completion time of the thread block is less than 0.8 times the average task time, it indicates that the computing power of the thread block is not being fully utilized. In this case, an unallocated task package is stolen from the global task pool through atomic operations to optimize the utilization of computing resources.

[0050] The average task time is calculated using a two-layer trigger threshold method based on the Exponential Moving Average (EMA) algorithm. The formula for Exponential Moving Average (EMA) is:

[0051] in: α is a smoothing factor that is dynamically adjusted according to the task stage; Initial task phase (first 100 tasks): α=0.8, to accelerate the convergence speed of average time consumption and quickly adapt to the time fluctuations of the initial tasks; Stable operation phase (after 100 tasks): α=0.2, reduce update weight to reduce the impact of occasional time fluctuations on the average value.

[0052] This represents the average task execution time (EMA) after the current update, serving as a benchmark for measuring task execution efficiency and used for subsequent load assessment.

[0053] This represents the actual time taken for the task that has just been completed (latest data), and is used to provide the latest feedback on task execution efficiency, so that the average value can reflect changes in system status in a timely manner.

[0054] This represents the average task time calculated in the last time (historical data). It is used to save historical statistical information, avoid drastic changes in the average value due to abnormal fluctuations in a single task, and ensure the continuity of results.

[0055] To ensure the rationality of task scheduling, a differentiated stealing strategy is implemented: when a high-priority task is completed, the stealing trigger threshold is increased to 1.2 times the average time to ensure that computing resources are prioritized for urgent tasks; when a low-priority task is completed, the standard threshold setting of 0.8 times is maintained to avoid preempting resources for urgent tasks.

[0056] Step S2.2.2: For the stolen task package, split the large task into smaller tasks according to the task type and spatial region to ensure that the computational workload of each thread block is balanced.

[0057] In this embodiment, a differentiated spatial segmentation method is implemented for task packages of different task types. For regular computational regions, an 8×8 uniform grid is used to ensure that the data volume of each subtask does not exceed a preset threshold (e.g., spatial range ≤ 100km × 100km or number of seed points ≤ 32768). For high-density regions (e.g., seed point density > 1000 points / km²), a 4×4 fine grid is used; for low-density regions (< 100 points / km²), a 16×16 coarse grid is used. The task segmentation grid is determined using the following formula:

[0058] The load balancing coefficient is dynamically adjusted based on the real-time GPU utilization: when the GPU utilization exceeds 80%, the load balancing coefficient is set to 1.2 to force a reduction in the task block size.

[0059] In step S2.2.3, load information is exchanged between thread blocks in real time. When the load difference is detected to exceed the preset migration threshold, some tasks are migrated from the high-load thread block to the low-load thread block.

[0060] In this embodiment, a real-time load monitoring and migration mechanism is established to achieve dynamic balancing through the following process: When each thread block completes 1 / 3 of the allocated tasks, the formula for calculating the local load metric is as follows:

[0061] In this embodiment, the migration threshold is 30%. Adjacent thread blocks periodically exchange load information through shared memory or global memory atomic operations. When the load difference is detected to exceed 30%, task migration is triggered.

[0062] The formula for calculating migration task volume is as follows:

[0063] Where: local_load represents the remaining load of the current thread block (unit: points / time), reflecting the task execution efficiency; neighbor_load represents the load index of adjacent thread blocks, which is exchanged through atomic operations of shared memory or global memory; The migration logic is as follows: high-load thread blocks are migrated to low-load thread blocks at half the difference in load, ensuring bidirectional balance. This bidirectional balancing strategy effectively alleviates the problem of uneven load distribution while avoiding the performance overhead caused by excessive migration.

[0064] The load migration mechanism in this embodiment is illustrated through the following specific examples: Suppose there are three adjacent thread blocks A, B, and C, each in a different load state: Thread block A has 1000 data points remaining and is estimated to take 10ms to complete (load metric load = 100 points / ms). Thread block B has 500 data points remaining and is estimated to complete in 5ms (load metric = 100 points / ms). Thread block C has 800 data points remaining and is estimated to complete in 4ms (load metric = 200 points / ms). Real-time calculations show that the load difference between thread blocks A and B is 0% [(100-100) / 100], which does not meet the migration trigger condition. However, the load difference between thread blocks C and A reaches 100% [(200-100) / 100], exceeding the 30% migration threshold. Based on the preset migration algorithm, the amount of task to be migrated is calculated to be (200-100) / 2 = 50 points / millisecond, and accordingly, part of the workload of thread block C is migrated to thread block A.

[0065] Step S2.3: On the CPU side, dynamically adjust the priority of the global task pool based on GPU load metrics.

[0066] Specifically, step S2.3 includes the following sub-steps: Step S2.3.1: Collect the load metrics of the GPU in real time and obtain a comprehensive load score through weighted calculation.

[0067] This embodiment constructs a GPU load quantification and evaluation system based on multi-dimensional indicators. This system includes four core evaluation indicators: active thread block percentage, memory bandwidth utilization, task queue length, and task completion time. By standardizing the data of each indicator to the 0-1 range and performing weighted calculations, the real-time load status of the GPU is quantified.

[0068] Table: GPU Load Evaluation Metrics System

[0069] GPU load status is assessed using quantitative metrics; the load value is calculated using the following formula:

[0070] in: The weight represents the contribution of each indicator to the overall load and is dynamically adjusted according to the characteristics of the GPU architecture. For example, the number of computing cores has a greater impact on the load and therefore has a higher weight.

[0071] Standardized indicators refer to mapping raw data to a uniform 0-1 range to facilitate horizontal comparison.

[0072] In this embodiment, a multi-dimensional priority calculation mechanism is adopted to achieve intelligent task scheduling by comprehensively evaluating risk level R, data volume weight D, and time urgency T.

[0073] in, , , All of these represent weighting coefficients.

[0074] The risk level R uses a quantitative value of 0-10, determined by business logic, and is calculated using the following formula:

[0075] If a flood warning is issued, and the inundation depth exceeds twice the warning value and affects 30% of the population, the R value is... .

[0076] The data volume weight D is standardized to 0-1, and the calculation formula is as follows:

[0077] With 10MB as the baseline, D=1 when the task data size is 20MB.

[0078] The time urgency T is also standardized to 0-1, as shown in the following formula.

[0079] If the remaining time for a task is less than one-third of the planned timeframe (e.g., 10 minutes remaining for a 30-minute task), then the weight of this parameter should be increased. .

[0080] The weighting coefficients are dynamically configured according to the following table:

[0081] A resource affinity factor K is introduced to evaluate the matching degree between the task and computing resources. The final priority calculation formula is as follows:

[0082] Among them, the resource affinity factor (K)

[0083] in, This indicates the basic priority calculated based on the task's inherent attributes (such as risk level, time urgency, etc.). This indicates the actual scheduling priority after considering the influence of data location.

[0084] By combining the final priority formula, the system can find a balance between "the importance of the task itself" and "data transfer efficiency", which can significantly reduce the efficiency loss caused by data transfer, allowing the GPU to spend more time on computation rather than waiting for data, thereby improving the overall system throughput.

[0085] Step S2.3.2: When the overall load score exceeds the overload threshold, a dynamic degradation mechanism is triggered until the load returns to a normal level.

[0086] Specifically, the GPU load status is monitored in real time. When the overall load score exceeds the overload threshold, a dynamic degradation mechanism is triggered, moving some non-urgent tasks to a low-priority queue and prioritizing high-priority tasks. The degradation decision is based on a dynamic threshold calculation, as follows:

[0087] The baseline value for the downgrade threshold is 70%. If the proportion of high-priority tasks reaches 80%, the threshold is raised to 70% + 0.1 × (80% / 50%) = 86%.

[0088] The formula for calculating the downgrade factor is as follows:

[0089] When the GPU load is 95%, the threshold is 80%, and the maximum load is 100%, the degradation factor is: (95-80) / (100-80)=0.75.

[0090] Non-urgent task screening criteria:

[0091] in, The emergency task threshold is typically set to 6. This threshold serves as a selection criterion, and non-emergency tasks are prioritized according to the following formula:

[0092] If a task with an original priority of 5 is downgraded to 5×(1-0.75×0.5)=3.125 due to a downgrade factor of 0.75.

[0093] Step S2.3.3: When the overall load score falls back to the preset normal range, gradually restore the task priority.

[0094] In this embodiment, the formula for calculating the coefficient of recovery is:

[0095] When the GPU load drops from 95% overload to 85% of the normal range, the warning threshold in this embodiment is 80%, and the overload threshold is 90%. The task priority is gradually restored using a recovery coefficient of 1-(85-80) / (90-80)=0.5.

[0096] When the GPU load is detected to be 85% (warning threshold 80%), the calculated degradation factor is (85-80) / (100-80) = 0.25. If the original task priority was 7, the new priority after adjustment is... The value is 6.125, which is still higher than the set threshold of 6. Therefore, the system determines that the task will not be downgraded for the time being.

[0097] In this embodiment, through the aforementioned load monitoring and dynamic adjustment mechanism, the triple goals of zero latency for urgent tasks, no preemption for non-urgent tasks, and continuous and stable system load are achieved, enabling efficient collaboration of the GPU cluster.

[0098] Step S3: Parse the completed task package and output it to the business terminal.

[0099] Example 2: The present invention also provides a high-efficiency water conservancy big data processing system based on a multi-stage parallel strategy. The high-efficiency water conservancy big data processing system based on a multi-stage parallel strategy can be implemented by executing the process steps of the high-efficiency water conservancy big data processing method based on a multi-stage parallel strategy. That is, those skilled in the art can understand the high-efficiency water conservancy big data processing method based on a multi-stage parallel strategy as a preferred embodiment of the high-efficiency water conservancy big data processing system based on a multi-stage parallel strategy.

[0100] Specifically, this high-efficiency water conservancy big data processing system based on a multi-stage parallel strategy includes: Module M1 divides the water conservancy data into multiple subsets according to a preset time window, and each subset is packaged into an independent task package; Module M2, based on task packages, performs dynamic load balancing computation through a CPU-GPU collaborative architecture and outputs the completed task packages; Module M3 parses the completed task package and outputs it to the business side.

[0101] Specifically, module M1 includes the following sub-modules: Module M1.1 divides the water conservancy data into N subsets according to a preset time window. It generates an independent task package for each subset, which includes data content, calculation parameters, and priority labels. It constructs a KD-tree or Octree index for each subset, reducing the neighborhood query complexity from O(n) to O(logn) and optimizing the computational efficiency on the GPU. At the same time, it uses differential encoding and LZ4 compression on the coordinate data in the data content to achieve a compression ratio of 3:1.

[0102] In this embodiment, the preset time window is 1 hour, and the calculation parameters include the spread threshold and the number of iterations.

[0103] Module M1.2 obtains the hardware parameters of all GPU devices through the CUDA API. The hardware parameters include the number of streaming multiprocessors (SMs), the number of cores per streaming multiprocessor (SM), and the GPU core clock frequency. And the concurrency coefficient, based on the hardware parameters, calculate the single-precision floating-point computing power of each GPU device. The formula for calculating single-precision floating-point computing power is:

[0104] Wherein: the value 2 allows for 2 floating-point operations to be performed per clock cycle.

[0105] Furthermore, GPU devices are categorized based on their single-precision floating-point computing power.

[0106] In this embodiment, GPU devices with a single-precision floating-point computing power of ≥10 TFLOPS are classified as Level 1 devices, which are used to prioritize high-priority tasks; GPU devices with a single-precision floating-point computing power between 5 TFLOPS and 10 TFLOPS are classified as Level 2 devices, which are responsible for handling medium-priority tasks; and GPU devices with a single-precision floating-point computing power <5 TFLOPS are classified as Level 3 devices, which execute low-priority or background tasks.

[0107] Module M1.3 assigns task packages to GPU devices of the corresponding level based on priority tags.

[0108] In this embodiment, the first-level device processes high-priority task packages, the second-level device processes medium-priority task packages, and the third-level device processes low-priority task packages.

[0109] Module M1.4 dynamically configures the thread block size of the GPU device based on the spatial distribution density of the subset dataset.

[0110] In this embodiment, if the spatial distribution density of the subset is sparse, 512 threads / block are configured; if the spatial distribution density of the subset is dense, 256 threads / block are configured.

[0111] Module M1.5 divides the shared memory of the GPU device into three regions according to its function. The three regions store the neighborhood water level value, the visited points, and the spread status indicator, respectively.

[0112] In this embodiment, the capacity of a single shared memory block is calculated using the following formula:

[0113] The total capacity of shared memory does not exceed the hardware limit of the GPU device.

[0114] Specifically, module M2 includes the following sub-modules: Module M2.1, on the CPU side, builds a global task pool and initializes priorities.

[0115] Specifically, on the CPU side, a global task pool based on lock-free queues is established, and task packages are initialized according to four priority levels: P0-P3. Among them, P0 level tasks are stored in a fixed memory area and are used to handle real-time risk alarm tasks; P1 level tasks are stored in an ordered linked list based on timestamps and are used to handle urgent business tasks; P2 level tasks are stored in a paged storage structure and are used to handle routine analysis tasks; and P3 level tasks are stored in a disk cache area and are used to handle background preprocessing tasks.

[0116] On the CPU side, each compute core maintains a local cache queue with a fixed capacity of 32 tasks. When the number of tasks to be processed in the queue falls below 8, a batch prefetching mechanism is triggered, fetching new tasks from the global queue in batches of 16. This design reduces the performance overhead caused by cross-core cache consistency. For low-priority tasks at the P3 level, an intelligent merging optimization function is implemented, which can automatically package 128 small compute tasks into one large task package, thereby effectively reducing the scheduling overhead of the GPU.

[0117] Module M2.2, on the GPU side, achieves dynamic balancing through task stealing, task splitting, and load migration, and outputs the completed task package.

[0118] Specifically, module M2.2 includes the following sub-modules: Module M2.2.1 uses GPU thread blocks to monitor the global task pool in real time, steal tasks based on priority and minimum load principles, and allocate thread blocks to perform computations.

[0119] In this embodiment, when a GPU thread block is idle, it enters a pull mode, where the idle thread retrieves tasks from the global task pool. Task allocation follows the principle of least load priority, dynamically assigning new tasks to the GPU processing unit with the lowest current load by monitoring the utilization metrics of each GPU in real time.

[0120] To optimize task scheduling efficiency in high-concurrency scenarios, a three-level dynamic task retrieval control strategy is adopted. When the GPU is detected to have taken more than 5ms to retrieve tasks three consecutive times, the system switches to push mode, where the CPU allocates task packages to the GPU with the lowest load. The specific scheduling process includes: first, calculating the current GPU load score based on four monitoring metrics (active thread block ratio, memory bandwidth utilization, task queue length, and last task latency); then, determining the task retrieval scale based on the score: a maximum of 64 tasks are retrieved at a time in a light-load state (LoadScore < 0.3), limited to 16 tasks in a heavy-load state (0.3 ≤ LoadScore < 0.7), and only 4 tasks are allowed to be retrieved in a very heavy-load state (LoadScore ≥ 0.7); finally, atomic operations are used to complete task retrieval and queue status updates. This intelligent scheduling mechanism ensures efficient utilization of GPU resources, avoiding both resource idleness and overload.

[0121] Specifically, after completing the current task package, the GPU thread block determines whether to trigger the task stealing mechanism. When it is detected that the task completion time of the thread block is less than 0.8 times the average task time, it indicates that the computing power of the thread block is not being fully utilized. In this case, an unallocated task package is stolen from the global task pool through atomic operations to optimize the utilization of computing resources.

[0122] The average task time is calculated using a two-layer trigger threshold method based on the Exponential Moving Average (EMA) algorithm. The formula for Exponential Moving Average (EMA) is:

[0123] in: α is a smoothing factor that is dynamically adjusted according to the task stage; Initial task phase (first 100 tasks): α=0.8, to accelerate the convergence speed of average time consumption and quickly adapt to the time fluctuations of the initial tasks; Stable operation phase (after 100 tasks): α=0.2, reduce update weight to reduce the impact of occasional time fluctuations on the average value.

[0124] This represents the average task execution time (EMA) after the current update, serving as a benchmark for measuring task execution efficiency and used for subsequent load assessment.

[0125] This represents the actual time taken for the task that has just been completed (latest data), and is used to provide the latest feedback on task execution efficiency, so that the average value can reflect changes in system status in a timely manner.

[0126] This represents the average task time calculated in the last time (historical data). It is used to save historical statistical information, avoid drastic changes in the average value due to abnormal fluctuations in a single task, and ensure the continuity of results.

[0127] To ensure the rationality of task scheduling, a differentiated stealing strategy is implemented: when a high-priority task is completed, the stealing trigger threshold is increased to 1.2 times the average time to ensure that computing resources are prioritized for urgent tasks; when a low-priority task is completed, the standard threshold setting of 0.8 times is maintained to avoid preempting resources for urgent tasks.

[0128] Module M2.2.2 splits stolen task packages into smaller tasks based on task type and spatial region to ensure balanced computation across thread blocks.

[0129] In this embodiment, a differentiated spatial segmentation method is implemented for task packages of different task types. For regular computational regions, an 8×8 uniform grid is used to ensure that the data volume of each subtask does not exceed a preset threshold (e.g., spatial range ≤ 100km × 100km or number of seed points ≤ 32768). For high-density regions (e.g., seed point density > 1000 points / km²), a 4×4 fine grid is used; for low-density regions (< 100 points / km²), a 16×16 coarse grid is used. The task segmentation grid is determined using the following formula:

[0130] The load balancing coefficient is dynamically adjusted based on the real-time GPU utilization: when the GPU utilization exceeds 80%, the load balancing coefficient is set to 1.2 to force a reduction in the task block size.

[0131] Module M2.2.3 allows for real-time exchange of load information between thread blocks. When a load difference is detected to exceed a preset migration threshold, some tasks are migrated from the high-load thread block to the low-load thread block.

[0132] In this embodiment, a real-time load monitoring and migration mechanism is established to achieve dynamic balancing through the following process: When each thread block completes 1 / 3 of the allocated tasks, the formula for calculating the local load metric is as follows:

[0133] In this embodiment, the migration threshold is 30%. Adjacent thread blocks periodically exchange load information through shared memory or global memory atomic operations. When the load difference is detected to exceed 30%, task migration is triggered.

[0134] The formula for calculating migration task volume is as follows:

[0135] Where: local_load represents the remaining load of the current thread block (unit: points / time), reflecting the task execution efficiency; neighbor_load represents the load index of adjacent thread blocks, which is exchanged through atomic operations of shared memory or global memory; The migration logic is as follows: high-load thread blocks are migrated to low-load thread blocks at half the difference in load, ensuring bidirectional balance. This bidirectional balancing strategy effectively alleviates the problem of uneven load distribution while avoiding the performance overhead caused by excessive migration.

[0136] The load migration mechanism in this embodiment is illustrated through the following specific examples: Suppose there are three adjacent thread blocks A, B, and C, each in a different load state: Thread block A has 1000 data points remaining and is estimated to take 10ms to complete (load metric load = 100 points / ms). Thread block B has 500 data points remaining and is estimated to complete in 5ms (load metric = 100 points / ms). Thread block C has 800 data points remaining and is estimated to complete in 4ms (load metric = 200 points / ms). Real-time calculations show that the load difference between thread blocks A and B is 0% [(100-100) / 100], which does not meet the migration trigger condition. However, the load difference between thread blocks C and A reaches 100% [(200-100) / 100], exceeding the 30% migration threshold. Based on the preset migration algorithm, the amount of task to be migrated is calculated to be (200-100) / 2 = 50 points / millisecond, and accordingly, part of the workload of thread block C is migrated to thread block A.

[0137] Module M2.3 dynamically adjusts the priority of the global task pool on the CPU side based on GPU load metrics.

[0138] Specifically, module M2.3 includes the following sub-modules: Module M2.3.1 collects load metrics from the GPU in real time and calculates a comprehensive load score through weighted average.

[0139] This embodiment constructs a GPU load quantification and evaluation system based on multi-dimensional indicators. This system includes four core evaluation indicators: active thread block percentage, memory bandwidth utilization, task queue length, and task completion time. By standardizing the data of each indicator to the 0-1 range and performing weighted calculations, the real-time load status of the GPU is quantified.

[0140] Table: GPU Load Evaluation Metrics System

[0141] GPU load status is assessed using quantitative metrics; the load value is calculated using the following formula:

[0142] in: The weight represents the contribution of each indicator to the overall load and is dynamically adjusted according to the characteristics of the GPU architecture. For example, the number of computing cores has a greater impact on the load and therefore has a higher weight.

[0143] Standardized indicators refer to mapping raw data to a uniform 0-1 range to facilitate horizontal comparison.

[0144] In this embodiment, a multi-dimensional priority calculation mechanism is adopted to achieve intelligent task scheduling by comprehensively evaluating risk level R, data volume weight D, and time urgency T.

[0145] in, , , All of these represent weighting coefficients.

[0146] The risk level R uses a quantitative value of 0-10, determined by business logic, and is calculated using the following formula:

[0147] If a flood warning is issued, and the inundation depth exceeds twice the warning value and affects 30% of the population, the R value is... .

[0148] The data volume weight D is standardized to 0-1, and the calculation formula is as follows:

[0149] With 10MB as the baseline, D=1 when the task data size is 20MB.

[0150] The time urgency T is also standardized to 0-1, as shown in the following formula.

[0151] If the remaining time for a task is less than one-third of the planned timeframe (e.g., 10 minutes remaining for a 30-minute task), then the weight of this parameter should be increased. .

[0152] The weighting coefficients are dynamically configured according to the following table:

[0153] A resource affinity factor K is introduced to evaluate the matching degree between the task and computing resources. The final priority calculation formula is as follows:

[0154] Among them, the resource affinity factor (K)

[0155] in, This indicates the basic priority calculated based on the task's inherent attributes (such as risk level, time urgency, etc.). This indicates the actual scheduling priority after considering the influence of data location.

[0156] By combining the final priority formula, the system can find a balance between "the importance of the task itself" and "data transfer efficiency", which can significantly reduce the efficiency loss caused by data transfer, allowing the GPU to spend more time on computation rather than waiting for data, thereby improving the overall system throughput.

[0157] Module M2.3.2 triggers a dynamic degradation mechanism when the overall load score exceeds the overload threshold, until the load returns to a normal level.

[0158] Specifically, the GPU load status is monitored in real time. When the overall load score exceeds the overload threshold, a dynamic degradation mechanism is triggered, moving some non-urgent tasks to a low-priority queue and prioritizing high-priority tasks. The degradation decision is based on a dynamic threshold calculation, as follows:

[0159] The baseline value for the downgrade threshold is 70%. If the proportion of high-priority tasks reaches 80%, the threshold is raised to 70% + 0.1 × (80% / 50%) = 86%.

[0160] The formula for calculating the downgrade factor is as follows:

[0161] When the GPU load is 95%, the threshold is 80%, and the maximum load is 100%, the degradation factor is: (95-80) / (100-80)=0.75.

[0162] Non-urgent task screening criteria:

[0163] in, The emergency task threshold is typically set to 6. This threshold serves as a selection criterion, and non-emergency tasks are prioritized according to the following formula:

[0164] If a task with an original priority of 5 is downgraded to 5×(1-0.75×0.5)=3.125 due to a downgrade factor of 0.75.

[0165] Module M2.3.3 gradually restores task priorities when the overall load score falls back to the preset normal range.

[0166] In this embodiment, the formula for calculating the coefficient of recovery is:

[0167] When the GPU load drops from 95% overload to 85% of the normal range, the warning threshold in this embodiment is 80%, and the overload threshold is 90%. The task priority is gradually restored using a recovery coefficient of 1-(85-80) / (90-80)=0.5.

[0168] When the GPU load is detected to be 85% (warning threshold 80%), the calculated degradation factor is (85-80) / (100-80) = 0.25. If the original task priority was 7, the new priority after adjustment is... The value is 6.125, which is still higher than the set threshold of 6. Therefore, the system determines that the task will not be downgraded for the time being.

[0169] In this embodiment, through the aforementioned load monitoring and dynamic adjustment mechanism, the triple goals of zero latency for urgent tasks, no preemption for non-urgent tasks, and continuous and stable system load are achieved, enabling efficient collaboration of the GPU cluster.

[0170] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0171] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for efficient processing of water conservancy big data based on a multi-stage parallel strategy, characterized in that, Includes the following steps: Step S1: Divide the water conservancy data into multiple subsets according to a preset time window, and encapsulate each subset into an independent task package; Step S2: Based on the task package, perform dynamic load balancing calculations through a CPU-GPU collaborative architecture, and output the completed task package; Step S3: Parse the completed task package and output it to the business terminal.

2. The efficient water conservancy big data processing method based on a multi-stage parallel strategy according to claim 1, characterized in that, Step S1 includes the following sub-steps: Step S1.1: The CPU divides the water conservancy data into N subsets according to a preset time window; generates an independent task package for each subset, the task package including data content, calculation parameters and priority labels; constructs a KD-tree or Octtree index for each subset, and uses differential encoding and LZ4 compression on the coordinate data in the data content; Step S1.2: Obtain the hardware parameters of all GPU devices through the CUDA API. The hardware parameters include the number of streaming multiprocessors, the number of cores per streaming multiprocessor, the GPU core clock frequency, and the concurrency coefficient. Calculate the single-precision floating-point computing power of each GPU device based on the hardware parameters, and classify the GPU devices according to the single-precision floating-point computing power. Step S1.3: Assign the task package to the corresponding level of GPU device according to the priority label; Step S1.4: Dynamically configure the thread block size of the GPU device according to the spatial distribution density of the subset dataset; Step S1.5: Divide the shared memory of the GPU device into three regions according to function. The three regions store the neighborhood water level value, the visited points, and the spread indicator, respectively.

3. The efficient water conservancy big data processing method based on a multi-stage parallel strategy according to claim 1, characterized in that, Step S2 includes the following sub-steps: Step S2.1: On the CPU side, construct a global task pool and initialize task packages according to priority; Step S2.2: On the GPU side, dynamic balancing is achieved through task stealing, task splitting, and load migration, and the completed task package is output. Step S2.3: On the CPU side, the priority of the global task pool is dynamically adjusted based on the GPU load metric.

4. The efficient water conservancy big data processing method based on a multi-stage parallel strategy according to claim 3, characterized in that, In step S2.1, on the CPU side, a global task pool based on a lock-free queue is established, and task packages are initialized according to four priority levels: P0-P3. Among them, P0 level tasks are stored in a fixed memory area and are used to process real-time risk alarm tasks; P1 level tasks are stored in an ordered linked list based on timestamps and are used to process urgent business tasks; P2 level tasks are stored in a paged storage structure and are used to process routine analysis tasks; and P3 level tasks are stored in a disk cache area and are used to process background preprocessing tasks.

5. The efficient water conservancy big data processing method based on a multi-stage parallel strategy according to claim 3, characterized in that, Step S2.2 includes the following sub-steps: Step S2.2.1: The GPU thread block monitors the global task pool in real time, steals tasks according to priority and minimum load principle, and allocates thread blocks to perform computation. Step S2.2.2: For the stolen task package, split the large task into smaller tasks according to the task type and spatial region to ensure that the computational load of each thread block is balanced. In step S2.2.3, load information is exchanged between thread blocks in real time. When the load difference is detected to exceed the preset migration threshold, some tasks are migrated from the high-load thread block to the low-load thread block.

6. The efficient water conservancy big data processing method based on a multi-stage parallel strategy according to claim 5, characterized in that, In step S2.2.1, the average task time is calculated using a two-layer trigger threshold method based on the exponential moving average algorithm. The exponential moving average formula is as follows: in: α is the smoothing factor; This represents the average task time after the current update, i.e., the EMA result. This indicates the actual time taken for the task that has just been completed; This represents the average task time calculated in the last time.

7. The efficient water conservancy big data processing method based on a multi-stage parallel strategy according to claim 5, characterized in that, In step S2.2.3, the formula for calculating the migration task volume is as follows: in: local_load represents the remaining load of the current thread block; neighbor_load represents the load metric of adjacent thread blocks.

8. The efficient water conservancy big data processing method based on a multi-stage parallel strategy according to claim 3, characterized in that, Step S2.3 includes the following sub-steps: Step S2.3.1: Collect the load metrics of the GPU in real time and obtain a comprehensive load score through weighted calculation; Step S2.3.2: When the overall load score exceeds the overload threshold, a dynamic degradation mechanism is triggered until the load returns to a normal level. Step S2.3.3: When the overall load score falls back to the preset normal range, gradually restore the task priority.

9. The efficient water conservancy big data processing method based on a multi-stage parallel strategy according to claim 8, characterized in that, In step S2.3.1, four load metrics of the GPU are collected: active thread block percentage, memory bandwidth utilization, task queue length, and task completion time. By standardizing the data of each metric to the 0-1 range and performing weighted calculations, the real-time load status of the GPU is quantified to obtain a comprehensive load score.

10. A high-efficiency water conservancy big data processing system based on a multi-stage parallel strategy, characterized in that, include: Module M1 divides the water conservancy data into multiple subsets according to a preset time window, and each subset is packaged into an independent task package; Module M2, based on the task package, performs dynamic load balancing calculations through a CPU-GPU collaborative architecture and outputs the completed task package; Module M3 parses the completed task package and outputs it to the business terminal.

Images