A load balancing-based multi-threaded commodity sorting task scheduling and data processing method

Abstract

The application provides a kind of based on load balancing multi-threaded commodity sorting task scheduling and data processing method, it is related to data processing technical field, the method includes: obtaining the real-time load index of each working thread in multi-thread pool, and obtaining the commodity dataset to be processed, commodity dataset is decomposed into multiple sorting tasks according to the initial task granularity of pre-set, real-time load index includes current task queue depth and unit time task processing rate;According to the real-time load index of each working thread, the load metric vector of each working thread at the current time is extracted respectively, the load metric vector is mapped to the pre-set load feature space, and a dynamic load distribution surface is constructed in the load feature space according to the spatial distribution of each load metric vector.The application realizes the load balancing of multi-thread parallel processing, improves the processing efficiency and real-time stability of commodity sorting.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multi-threaded product sorting task scheduling and data processing method based on load balancing. Background Technology

[0002] In high-concurrency access scenarios such as e-commerce platform promotions and flash sales, the platform needs to perform real-time calculation and sorting of massive product datasets based on multiple dimensions such as sales volume, user ratings, price, and popularity. Typically, a multi-threaded pool is used to perform product sorting tasks in parallel to improve data processing efficiency.

[0003] Existing multi-threaded product sorting task scheduling and data processing methods mostly use fixed task granularity to distribute sorting tasks evenly or simple round-robin, without real-time collection of dynamic load indicators such as task queue depth and task processing rate per unit time for each worker thread. It is difficult to dynamically adapt the task allocation based on the real-time load status of the threads, resulting in an unbalanced load problem where some worker threads are overloaded and experience task backlog and delays due to excessive task allocation, while other worker threads are idle and underloaded due to insufficient task allocation. At the same time, the local sorting result generation rate of each thread varies greatly, resulting in low efficiency in merging global results. It is difficult to achieve efficient and balanced processing of massive product data sorting tasks, and it is difficult to meet the real-time and stability requirements of product sorting in e-commerce scenarios. Summary of the Invention

[0004] This invention provides a load-balanced multi-threaded product sorting task scheduling and data processing method, which realizes load balancing of multi-threaded parallel processing and improves the processing efficiency and real-time stability of product sorting.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, a method for scheduling and processing multi-threaded product sorting tasks based on load balancing, the method comprising: Get the real-time load metrics of each worker thread in the multi-thread pool, and get the product dataset to be processed. Decompose the product dataset into multiple tasks to be sorted according to the preset initial task granularity. The real-time load metrics include the current task queue depth and the task processing rate per unit time. Based on the real-time load metrics of each worker thread, the load metric vector of each worker thread at the current moment is extracted, and the load metric vector is mapped to a preset load feature space. In the load feature space, a dynamic load distribution surface is constructed according to the spatial distribution of each load metric vector. The dynamic load distribution surface is spatially partitioned into multiple geometric sub-regions; the spatial volume ratio of each geometric sub-region is calculated, and the spatial volume ratio of each geometric sub-region is used as the theoretical load-bearing ratio of the corresponding working thread. The theoretical task acceptance ratio is numerically fitted to the total number of tasks to be sorted to obtain the task acceptance weight of each worker thread; and multiple tasks to be sorted are scheduled to the corresponding worker threads according to the task acceptance weight. Each worker thread executes the sorting task in parallel, and performs multi-dimensional weight calculation and sorting on the product subset assigned to each worker thread to obtain the local sorting result of each worker thread. By collecting the local sorting results of each worker thread, a merging strategy is determined based on the initial task granularity used when decomposing the tasks to be sorted; the local sorting results are then merged in an orderly manner according to the merging strategy to obtain the global sorting result.

[0006] Furthermore, based on the real-time load metrics of each worker thread, the load metric vector of each worker thread at the current moment is extracted, and the load metric vector is mapped to a preset load feature space. A dynamic load distribution surface is then constructed in the load feature space based on the spatial distribution of each load metric vector, including: Based on the current task queue depth and task processing rate per unit time of each worker thread, the current task queue depth of each worker thread is used as the first dimension component and the task processing rate per unit time is used as the second dimension component, and the two-dimensional load metric vector of each worker thread at the current moment is obtained by combining them. The two-dimensional load metric vector of each worker thread is used as a spatial data point and mapped to a preset two-dimensional load feature space to form a set of spatial data points corresponding to each worker thread in the two-dimensional load feature space. Based on the distribution of spatial data points in the two-dimensional load feature space, a spatial surface fitting method is adopted, using the spatial data points as surface interpolation nodes. By constructing a continuous surface passing through each spatial data point, a dynamic load distribution surface is obtained. The dynamic load distribution surface is used to characterize the comprehensive load distribution state of each working thread at the current moment.

[0007] Furthermore, the dynamic load distribution surface is spatially partitioned into multiple geometric sub-regions; the spatial volume ratio of each geometric sub-region is calculated, and this ratio is used as the theoretical load-bearing ratio for the corresponding working thread, including: In the two-dimensional load feature space, the domain range of the dynamic load distribution surface is determined, and the domain range is limited by the distribution boundary of the spatial data points corresponding to each working thread; The dynamic load distribution surface is spatially partitioned, dividing it into multiple geometric sub-regions along the coordinate axes. Calculate the spatial volume of each geometric sub-region under the dynamic load distribution surface, and calculate the ratio of the spatial volume of each geometric sub-region to the total volume of the dynamic load distribution surface to obtain the spatial volume ratio of each geometric sub-region. The spatial volume ratio of each geometric sub-region is determined as the theoretical load-bearing ratio of the corresponding worker thread. The theoretical load-bearing ratio is used to characterize the proportion of the total workload that each worker thread should undertake under load balancing conditions.

[0008] Furthermore, the theoretical task acceptance ratio is numerically fitted to the total number of tasks to be sorted to obtain the task acceptance weight of each worker thread; and multiple tasks to be sorted are assigned to the corresponding worker threads according to the task acceptance weight, including: The initial task allocation for each worker thread is obtained by multiplying the theoretical capacity of each worker thread by the total number of tasks to be sorted. The initial task allocation quantity of each worker thread is rounded and corrected, and the remaining tasks after rounding and correction are allocated one by one according to the theoretical undertaking ratio of each worker thread until all tasks to be sorted are allocated, thus obtaining the task undertaking weight of each worker thread. Furthermore, based on task weights, multiple tasks to be sorted are scheduled to their corresponding worker threads, including: Obtain the task acceptance weight of each worker thread. The task acceptance weight represents the specific number of tasks to be scheduled by each worker thread in the form of task quantity. Based on the task acceptance weight of each worker thread, extract the number of unsorted tasks that are equal to the task acceptance weight of each worker thread from multiple unsorted tasks, and obtain the subset of unscheduled tasks corresponding to each worker thread. Each subset of tasks to be scheduled is dispatched to the task queue of the corresponding worker thread, thus completing dynamic task scheduling based on task acceptance weight.

[0009] Furthermore, each worker thread executes the sorting task in parallel, performing multi-dimensional weight calculations and sorting on the product subset assigned to each worker thread, obtaining the local sorting results for each worker thread, including: Each worker thread retrieves the scheduled tasks from its corresponding task queue and extracts the assigned subset of products from the product dataset based on the product identifier corresponding to the task to be sorted. Each worker thread performs multi-dimensional weight calculations for each product in the extracted product subset in parallel to obtain the comprehensive weight value of each product; Each worker thread sorts the products in the extracted product subset in descending order based on the comprehensive weight value of each product, thus obtaining the local sorting result of each worker thread.

[0010] Furthermore, by collecting the local sorting results of each worker thread, a merging strategy is determined based on the initial task granularity used when decomposing the tasks to be sorted; the local sorting results are then merged in an ordered manner according to the merging strategy to obtain the global sorting result, including: By collecting the local sorting results of each worker thread, each local sorting result is an ordered sequence of goods arranged in descending order according to the comprehensive weight value; the merging order required for the merge operation is determined according to the initial task granularity used when the sorting task is performed. Based on the merging order, a multi-way merging method is used to perform parallel merging of each local sorting result. The comprehensive weight value of the currently unmerged products in each local sorting result is compared one by one, and the products with the highest comprehensive weight value are taken out and merged into the global sorting result sequence. The process iteratively compares the overall weight values ​​of unmerged items in each local sorting result. During the comparison, the item with the highest overall weight value is extracted and merged into the global sorting result sequence. This continues until all items in the local sorting results have been merged, resulting in a complete global sorting result.

[0011] Secondly, a load-balanced multi-threaded product sorting task scheduling and data processing system includes: The acquisition module is used to acquire the real-time load metrics of each worker thread in the multi-thread pool and acquire the product dataset to be processed. The product dataset is decomposed into multiple tasks to be sorted according to the preset initial task granularity. The real-time load metrics include the current task queue depth and the task processing rate per unit time. The extraction module is used to extract the load metric vector of each worker thread at the current moment based on the real-time load index of each worker thread, map the load metric vector to the preset load feature space, and construct a dynamic load distribution surface in the load feature space according to the spatial distribution of each load metric vector. The partitioning module is used to perform spatial partitioning on the dynamic load distribution surface, dividing the dynamic load distribution surface into multiple geometric sub-regions; calculating the spatial volume ratio of each geometric sub-region, and using the spatial volume ratio of each geometric sub-region as the theoretical load-bearing ratio of the corresponding working thread. The fitting module is used to numerically fit the theoretical acceptance ratio with the total number of tasks to be sorted to obtain the task acceptance weight of each worker thread; and to schedule multiple tasks to be sorted to the corresponding worker threads according to the task acceptance weight. The sorting module is used for each worker thread to execute the sorting task in parallel. It performs multi-dimensional weight calculation and sorting on the product subsets assigned to each worker thread to obtain the local sorting results of each worker thread. The processing module is used to collect the local sorting results of each worker thread, determine the merging strategy based on the initial task granularity used when decomposing the task to be sorted, and merge the local sorting results in an orderly manner according to the merging strategy to obtain the global sorting result.

[0012] Thirdly, a computing device, comprising: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0013] Fourthly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0014] The above-described solution of the present invention has at least the following beneficial effects: This method employs a technique that uses real-time load metrics from each worker thread in a multi-threaded pool to construct a two-dimensional load metric vector. This vector is then mapped to a preset load feature space and fitted to generate a dynamic load distribution surface. By dividing the surface space, the volume ratio of sub-regions is calculated to determine the theoretical load-bearing ratio of each thread. Combined with the total number of tasks, a task-bearing weight is fitted to achieve dynamic task scheduling. After parallel completion of local product sorting, multi-way merging is performed according to the initial task granularity. This overcomes the technical problems of traditional multi-threaded product sorting scheduling, which uses fixed task distribution and simple polling methods that cannot adapt to the real-time load status of threads. These problems lead to uneven load between threads, some threads accumulating tasks with delays or low idle loads, and large differences in the generation rate of local sorting results, resulting in low global merging efficiency. This method achieves load balancing in multi-threaded parallel processing, improves the processing efficiency of sorting massive amounts of product data, and meets the real-time and stability requirements of product sorting in high-concurrency e-commerce scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a multi-threaded product sorting task scheduling and data processing method based on load balancing, provided by an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of a multi-threaded product sorting task scheduling and data processing system based on load balancing, provided by an embodiment of the present invention. Detailed Implementation

[0017] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0018] like Figure 1 As shown, an embodiment of the present invention proposes a multi-threaded product sorting task scheduling and data processing method based on load balancing, the method comprising the following steps: Step 1: Obtain the real-time load metrics of each worker thread in the multi-thread pool and obtain the product dataset to be processed. Decompose the product dataset into multiple tasks to be sorted according to the preset initial task granularity. The real-time load metrics include the current task queue depth and the task processing rate per unit time. Step 2: Based on the real-time load metrics of each worker thread, extract the load metric vector of each worker thread at the current moment, map the load metric vector to the preset load feature space, and construct a dynamic load distribution surface in the load feature space according to the spatial distribution of each load metric vector. Step 3: Perform spatial partitioning on the dynamic load distribution surface, dividing the dynamic load distribution surface into multiple geometric sub-regions; calculate the spatial volume ratio of each geometric sub-region, and use the spatial volume ratio of each geometric sub-region as the theoretical load-bearing ratio of the corresponding working thread. Step 4: Perform numerical fitting between the theoretical acceptance ratio and the total number of tasks to be sorted to obtain the task acceptance weight of each worker thread; and schedule multiple tasks to be sorted to their corresponding worker threads according to the task acceptance weight. Step 5: Each worker thread executes the sorting task in parallel, performing multi-dimensional weight calculation and sorting on the product subset assigned to each worker thread to obtain the local sorting result of each worker thread. Step 6: By collecting the local sorting results of each worker thread, determine the merging strategy based on the initial task granularity used when decomposing the tasks to be sorted; merge the local sorting results in an orderly manner according to the merging strategy to obtain the global sorting result.

[0019] In this embodiment of the invention, a dynamic load distribution surface is constructed by collecting real-time load metrics of multi-threaded pool worker threads, extracting load metric vectors and mapping them to a preset feature space. The theoretical thread load ratio is determined by calculating the volume ratio of geometric sub-regions through surface space subdivision. Dynamic task scheduling is achieved by fitting task load weights based on the total task volume. After parallel completion of local product sorting, the global result is obtained by merging according to the initial task granularity. Therefore, this method overcomes the technical problems of traditional multi-threaded product sorting, which uses fixed granularity equal distribution and simple round-robin task allocation, making it difficult to adapt to the real-time load status of threads, resulting in load imbalance, task backlog delays or thread idleness, and low efficiency in merging global results. This method achieves load balancing in multi-threaded parallel processing, improves the processing efficiency of sorting massive amounts of product data, and meets the real-time and stability requirements of product sorting in e-commerce high-concurrency scenarios.

[0020] In a preferred embodiment of the present invention, step 1 involves obtaining the real-time load metrics of each worker thread in the multi-threaded pool and obtaining the product dataset to be processed. The product dataset is then decomposed into multiple tasks to be sorted according to a preset initial task granularity. The real-time load metrics include the current task queue depth and the task processing rate per unit time. Specifically, this includes: starting a preset thread load monitoring component during high-concurrency periods such as e-commerce promotions and flash sales; traversing all working threads in the multi-threaded pool in real time at fixed short intervals; collecting the current backlog of tasks in the task queue of each worker thread, i.e., the task queue depth; simultaneously calculating the total amount of product sorting tasks processed by each worker thread within a unit time window; accurately calculating the task processing rate per unit time of the corresponding worker thread based on the time window duration; and verifying the validity of the collected load data before completion. The real-time load metrics of each worker thread are collected and stored in full. Through the e-commerce platform's dedicated product data call interface, a data request is sent to the backend distributed data storage module to pull the product dataset to be processed, which includes attributes such as sales volume, user rating, price, and popularity. Missing values, duplicate values, and outliers in the dataset are removed simultaneously. The integrity and timeliness of the dataset are verified to ensure that the product data participating in the sorting meets the processing standards for high-concurrency scenarios. Based on the pre-set initial task granularity adapted to massive data processing, the total data volume of the product dataset to be processed is evenly split and calculated. The overall product dataset is divided into multiple equally sized and clearly defined sorting tasks according to a fixed number of data entries. Each sorting task is bound with a unique identifier to form a standardized task set that can be directly scheduled, avoiding subsequent processing deviations caused by uneven task splitting.

[0021] In this embodiment of the invention, because the invention adopts the technical means of accurately collecting real-time load indicators of each working thread in the multi-thread pool, including the current task queue depth and the task processing rate per unit time, and synchronously acquiring the dataset of goods to be processed and decomposing it into multiple tasks to be sorted according to the preset initial task granularity, it overcomes the technical problems of traditional solutions that do not collect basic data on the dynamic load of threads and lack standardized granularity in task decomposition, resulting in a lack of basis for subsequent task allocation and failure of load prediction. Thus, it provides accurate data support for load feature modeling and dynamic task scheduling, and ensures the regularity of task decomposition and the real-time effectiveness of load monitoring.

[0022] In a preferred embodiment of the present invention, step 2 above may include: Step 2.1: Based on the current task queue depth and task processing rate per unit time of each worker thread, the current task queue depth of each worker thread is used as the first dimension component and the task processing rate per unit time is used as the second dimension component to obtain a two-dimensional load metric vector for each worker thread at the current moment. Specifically, this includes: retrieving standardized worker thread load metric data that has undergone real-time acquisition, noise filtering, outlier removal, and timeliness validity verification from the real-time cache area of ​​the thread load monitoring module; in the high-concurrency processing scenario of e-commerce platform flash sales, all worker threads are traversed one by one according to the pre-set worker thread numbers in the multi-thread pool in ascending order; for each independent target worker thread, the thread is extracted after dual calibration of time-series calibration and numerical calibration. The current task queue depth value is verified by comparing it with the values ​​of adjacent collection periods to ensure there are no abnormal fluctuations or deviations. This value is then officially determined as the first dimension component of the two-dimensional load metric vector. Next, the task processing rate value per unit time obtained by the thread within the same statistical time period is extracted. After verifying the processing continuity and data authenticity of this rate value within the statistical period, it is officially determined as the second dimension component of the two-dimensional load metric vector. Following the system's pre-set two-dimensional vector construction specifications, the two dimension components are combined in an orderly manner according to a fixed sequence. Simultaneously, the generated two-dimensional load metric vector is bound to the unique thread identifier of the corresponding worker thread and the current collection timestamp. Finally, a standard two-dimensional load metric vector is obtained that corresponds to a single worker thread and fits the real-time load status at the current collection moment.

[0023] Step 2.2: The two-dimensional load metric vectors of each worker thread are used as spatial data points and mapped to a preset two-dimensional load feature space. This forms a set of spatial data points corresponding to each worker thread within the two-dimensional load feature space. Specifically, this involves: Based on a pre-built standard two-dimensional load feature space designed for high-concurrency e-commerce product sorting scenarios, clearly defining the horizontal axis as task queue depth and the vertical axis as task processing rate per unit time, while limiting reasonable value ranges, the standard two-dimensional load metric vector corresponding to each worker thread is converted into an independent spatial data point with precise coordinate parameters within the two-dimensional load feature space according to the system's established coordinate mapping transformation rules. Throughout the mapping process, the accuracy of the spatial data point coordinates and the uniqueness of the binding between the spatial data point and the corresponding worker thread are verified. Invalid data points with mapping offsets, coordinates exceeding the value range, or disordered mapping relationships are automatically filtered out. All verified spatial data points are then organized, collected, and sorted in an orderly manner according to the worker thread number, forming a complete set of spatial data points within the two-dimensional load feature space that covers all worker threads in the multi-threaded pool, with clear and accurate coordinates, no duplicate points, and no missing data.

[0024] Step 2.3: Based on the distribution of spatial data points in the two-dimensional load feature space, a spatial surface fitting method is used. The spatial data points are used as interpolation nodes. By constructing a continuous surface passing through each spatial data point, a dynamic load distribution surface is obtained. The dynamic load distribution surface is used to characterize the comprehensive load distribution state of each working thread at the current moment. Specifically, it includes: sequentially traversing the coordinate parameters of each spatial data point in the spatial data point set, comparing the values ​​of each coordinate parameter one by one to determine the overall distribution range of all spatial data points in the two-dimensional load feature space, and determining the minimum and maximum boundary values ​​of the horizontal axis and the vertical axis in the two-dimensional load feature space according to the overall distribution range. Based on the above boundary values, the effective domain boundary of the dynamic load distribution surface is accurately defined, and an interpolation method that can adapt to the real-time processing needs of massive e-commerce product data is selected. The spatial surface fitting algorithm initiates surface construction operations, using each verified spatial data point in the spatial data point set as a core interpolation node for constructing the dynamic load distribution surface. Following preset interpolation rules, step-by-step interpolation processing is performed sequentially based on the precise coordinate parameters of each interpolation node. Then, by splicing segmented surfaces, the interpolated surface fragments are integrated into a continuous whole, gradually generating a smooth and complete surface structure that passes through all interpolation nodes. After surface construction, the smoothness of the surface is verified item by item, and the matching degree between the load state represented by the surface and the actual load state of each worker thread is checked to ensure that the surface has no abrupt inflection points and that the overall trend of the surface can truly reflect the differences in load levels and the overall distribution trend among the worker threads. Finally, a dynamic load distribution surface that accurately and intuitively represents the comprehensive load distribution state of all worker threads in the multi-thread pool at the current acquisition time is obtained.

[0025] In this embodiment of the invention, a two-dimensional load metric vector is constructed by extracting the task queue depth and processing rate per unit time of each working thread. The vector is then mapped to spatial data points to form a corresponding point set. The point set is used as interpolation nodes to fit and construct a continuous dynamic load distribution surface. This overcomes the technical problems of existing technologies that cannot accurately quantify the real-time load of threads and that it is difficult to intuitively perceive the load differences between threads and the overall load distribution. As a result, standardized modeling and visualization of the load status of the multi-thread pool are realized, providing accurate and reliable load reference for subsequent dynamic task scheduling.

[0026] In a preferred embodiment of the present invention, step 3 above may include: Step 3.1: Determine the domain of the dynamic load distribution surface in the two-dimensional load feature space. The domain is defined by the distribution boundary of the spatial data points corresponding to each worker thread. Specifically, this includes: retrieving the complete data of the generated dynamic load distribution surface that has completed smoothness verification and load matching verification; synchronously retrieving a qualified set of spatial data points covering all worker threads in the two-dimensional load feature space with clear and complete coordinates; traversing the horizontal axis coordinate parameters of all spatial data points according to the preset worker thread number order of the multi-thread pool; comparing the values ​​of each horizontal axis coordinate; selecting the horizontal axis coordinate with the smallest value as the left horizontal axis boundary of the domain of the dynamic load distribution surface; and selecting the horizontal axis coordinate with the largest value as the left horizontal axis boundary of the dynamic load distribution surface. The right boundary of the horizontal axis of the load distribution surface domain is determined by iterating through the vertical axis coordinates of all spatial data points in the same thread number order. The values ​​of each vertical axis coordinate are compared one by one, and the vertical axis coordinate with the smallest value is selected as the lower boundary of the vertical axis of the dynamic load distribution surface domain. The vertical axis coordinate with the largest value is selected as the upper boundary of the vertical axis of the dynamic load distribution surface domain. The left boundary of the horizontal axis, the right boundary of the horizontal axis, the lower boundary of the vertical axis, and the upper boundary of the vertical axis form a closed interval, which accurately locks the complete domain range of the dynamic load distribution surface. The fit between the domain boundary and the spatial data point distribution boundary is verified throughout the process to ensure that the domain has no outward expansion redundancy and no inward contraction omission, and completely fits the real-time load distribution range of all working threads.

[0027] Step 3.2 involves spatially partitioning the dynamic load distribution surface, dividing it into multiple geometric sub-regions along the coordinate axes. Specifically, this includes: counting the total number of active worker threads currently in the ready and running states in the multi-threaded pool; using the defined domain of the dynamic load distribution surface as the core partitioning benchmark; synchronously performing regular partitioning along the horizontal and vertical axes of the two-dimensional load feature space; determining the final number of partitions based on the total number of active worker threads during the partitioning process, ensuring the number of partitions perfectly matches the number of active worker threads; performing proportional and unbiased spatial division of the dynamic load distribution surface; synchronously binding the corresponding single worker thread after each sub-region partition; verifying the uniqueness and accuracy of the binding relationship in real time to prevent duplicate or misaligned bindings; and finally dividing the complete dynamic load distribution surface into multiple independent geometric sub-regions with clear boundaries, no overlap, no spatial gaps, and a number perfectly matching the number of worker threads.

[0028] Step 3.3: Calculate the spatial volume of each geometric sub-region under the dynamic load distribution surface. Ratio the spatial volume of each geometric sub-region to the total volume of the dynamic load distribution surface to obtain the spatial volume percentage of each geometric sub-region. Specifically, this includes: assuming the total number of independent geometric sub-regions obtained from the subdivision is the same as the total number of effective worker threads, denoted as... ;No. The spatial volume enclosed by the geometric sub-regions below the dynamic load distribution surface is denoted as . ,in , representing the first thread sorted by thread number A geometric subregion; the total volume of the dynamic load distribution surface over the entire domain is denoted as . ;No. The spatial volume percentage corresponding to each geometric sub-region is denoted as . For each independent geometric sub-region obtained by subdivision, combined with the continuous and smooth trend of the dynamic load distribution surface, a definite integral volume calculation method adapted to the real-time processing of massive e-commerce product data is adopted to calculate the spatial volume enclosed by a single geometric sub-region below the dynamic load distribution surface. To perform precise calculations, first complete the volume calculation of the first geometric sub-region, and then complete the second to third sub-regions sequentially according to the preset work thread numbering order. The volume calculation of each geometric sub-region is performed to ensure that the volume calculation of each sub-region is complete and without deviation, and the spatial volume values ​​obtained from the calculation of all geometric sub-regions are then presented. , ... The summation is performed sequentially, and the total volume is calculated using the formula... Perform continuous addition operations to obtain the total volume value of the dynamic load distribution surface over the entire domain. For each geometric sub-region, the spatial volume ratio is calculated using the formula. To perform the division operation, that is, to use the spatial volume value of a single geometric sub-region. Divide by the total volume of the dynamic load distribution surface This yields the spatial volume percentage corresponding to the geometric sub-region. For each calculated percentage value, the accuracy is verified, the decimal retention deviation and calculation error in the calculation process are checked, and deviation values ​​that exceed the reasonable range are eliminated. Finally, the standard spatial volume percentage of all geometric sub-regions is obtained.

[0029] Step 3.4: Determine the spatial volume ratio of each geometric sub-region as the theoretical workload ratio of the corresponding worker thread. The theoretical workload ratio is used to characterize the proportion of the total workload that each worker thread should undertake under load balancing conditions. Specifically, this includes: based on the established binding relationship between the geometric sub-region and the worker thread, i.e., the... The geometric subregion uniquely corresponds to the first... Each working thread will calculate and verify the spatial volume percentage of each geometric sub-region, which has passed accuracy verification. According to the corresponding relationship, the data is precisely matched to the corresponding individual worker thread. The matched relationship is then verified a second time, checking the consistency between the geometric sub-region number and the worker thread number one by one. It is confirmed that there is no misalignment, omission, or duplicate matching of the percentage value and the worker thread. The verified spatial volume percentage value is then... The theoretical capacity ratio for each worker thread is officially determined. This theoretical capacity ratio precisely quantifies the proportion of product sorting tasks that a single worker thread can handle out of the total number of tasks to be sorted under load balancing conditions. Simultaneously, the theoretical capacity ratio values ​​corresponding to each worker thread are stored in the cache unit of the task scheduling module, and the values ​​are backed up and marked to provide accurate quantitative reference for the dynamic allocation and precise scheduling of tasks to be sorted.

[0030] In this embodiment of the invention, by precisely defining the domain, it is ensured that the real-time load data of all worker threads are included in the calculation scope, avoiding allocation deviations caused by missing load data. Spatial partitioning transforms the abstract load characteristic space into a quantifiable geometric region, and volume ratio calculation transforms the comprehensive load capacity of threads into a specific allocation ratio. This abandons the traditional one-size-fits-all mode of fixed equal distribution and round-robin allocation, ensuring that the task allocation is highly adapted to the actual load capacity of the threads. This fundamentally avoids the problem of some threads being overloaded and having too many tasks, or being idle and underloaded due to too few tasks. The theoretical load ratio is calculated based on the actual load capacity of the threads; threads with stronger load capacity undertake more tasks, and threads with weaker load capacity undertake fewer tasks. The processing resources of all worker threads are balanced. All resources are fully utilized, with no idle or excessive consumption, improving the overall efficiency of the multi-threaded pool in processing product sorting tasks. Because the task load of each thread is adapted to its processing rate and queue status, the time for each thread to complete the local sorting results tends to be consistent, preventing situations where some threads finish early and wait while others are slowed down, thus reducing the waiting time in the global merging phase and accelerating the generation speed of global sorting results. Scenarios such as e-commerce promotions and flash sales have extremely high requirements for the real-time performance and stability of product sorting. This step, through quantitative load balancing calculations, makes the processing time of massive product sorting tasks controllable and the allocation process quantifiable, ensuring that product sorting results can be generated and displayed in real time under high-concurrency scenarios, meeting the platform's requirements for real-time data processing and stability.

[0031] In a preferred embodiment of the present invention, step 4 above may include: Step 4.1 involves multiplying the theoretical load ratio of each worker thread by the total number of tasks to be sorted to obtain the initial task allocation for each worker thread. This includes determining the theoretical load ratio of all worker threads through prior load calculations, and ensuring that the dataset of products to be sorted has been split, the total number of tasks to be sorted is clear, and each task is an indivisible independent integer unit. The core of this step is converting the thread load ratio into a preliminary task allocation. The specific operation is as follows: iterate through each worker thread in the multi-threaded pool, multiplying the theoretical load ratio of each individual worker thread by the total number of tasks to be sorted. The resulting value is the initial task allocation for that worker thread. Since the theoretical load ratio is a floating-point quantized value, the initial task allocation after the product operation will likely contain a decimal part rather than an integer. This value is only a theoretical reference value for load adaptation and cannot be directly used for actual task scheduling.

[0032] Using data from previous implementation examples, let's calculate as follows: Assuming the total number of tasks to be sorted is 1000, and the theoretical handling ratios of the four worker threads are 0.25, 0.4, 0.3, and 0.05 respectively, then the initial task allocation for each worker thread is 0.25 multiplied by 1000, resulting in 250 tasks; 0.4 multiplied by 1000, resulting in 400 tasks; 0.3 multiplied by 1000, resulting in 300 tasks; and 0.05 multiplied by 1000, resulting in 50 tasks. If the scenario changes to 998 tasks, the initial task allocation for each worker thread is 0.05 multiplied by 998, resulting in 49.9 tasks. The corresponding initial values ​​for the remaining threads can be calculated sequentially.

[0033] Step 4.2 involves rounding down the initial task allocation for each worker thread and then allocating the remaining tasks according to the theoretical capacity of each worker thread until all tasks to be sorted are allocated. This process yields the task capacity weight for each worker thread. Specifically, this includes correcting for decimals in the initial task allocation and ensuring that no tasks are omitted or over-allocated. The final result is an integer task capacity weight that can be directly used for scheduling. The entire process consists of three core steps, with a continuous and uninterrupted workflow. The first step is rounding down, which uses a floor rule to process the initial task allocation for each worker thread, retaining the integer part of the initial allocation and discarding all decimals to obtain the temporary task allocation for each thread. The core of this rule is to ensure that the total number of temporarily allocated tasks does not exceed the total number of tasks to be sorted, thus avoiding over-allocation.

[0034] Using the scenario of 998 tasks in total as an example: the initial task allocation for worker threads is 49.9, which, after rounding down, results in 49 temporary tasks. The initial number of tasks for the remaining threads is an integer, and after rounding down, the temporary task allocation remains unchanged at 250, 400, and 300 respectively. To calculate the remaining task quantity, the temporary task allocation quantities for all worker threads are first summed to obtain the total number of temporarily allocated tasks. Then, the total number of tasks to be sorted is subtracted from the total number of temporarily allocated tasks. The value represents the number of remaining tasks. The total number of tasks to be sorted is 1001. The theoretical load ratio of the four threads remains unchanged. The initial task allocation quantities are 250.25, 400.4, 300.3, and 50.05 respectively. After rounding down, the temporary quantities are 250, 400, 300, and 50. The total number of temporarily allocated tasks is 250 + 400 + 300 + 50, which equals 1000. The number of remaining tasks is 1001 minus 1000, resulting in a final number of remaining tasks of 1.

[0035] To allocate the remaining tasks sequentially, all worker threads are first sorted from largest to smallest according to their theoretical capacity, forming a priority allocation sequence. Threads with higher theoretical capacity are ranked higher and given priority in allocating remaining tasks. Starting from the first worker thread in the sequence, one remaining task is allocated to each thread in turn. After each allocation, the number of remaining tasks is reduced by 1. This allocation process is repeated until the number of remaining tasks becomes 0, and all tasks to be sorted are allocated. The final number of tasks allocated to each worker thread after rounding and adjustment is the task capacity weight of that thread. In a scenario with 1 remaining task, the threads are sorted from highest to lowest theoretical capacity as the second thread, the third thread, the first thread, and the fourth thread. The first thread is the second thread, so one remaining task is allocated to the second thread. The final task capacity weights of each thread are 250, 401, 300, and 50, respectively, and the sum is exactly the same as the total number of tasks to be sorted.

[0036] In this embodiment of the invention, task quantification and allocation are carried out based on the theoretical load ratio calculated in the early stage of load measurement. This not only conforms to the real-time load status of threads, but also solves the problem of numerical adaptation after ratio conversion. The core beneficial effects are reflected in four aspects: First, the abstract load ratio is transformed into a specific task allocation value through product operation, realizing the precise link between task allocation and thread load capacity. Second, the core logic of load balancing in the early stage is continued, avoiding the load imbalance problem caused by fixed equal distribution and round-robin allocation. Third, a correction rule of rounding down is adopted to prevent thread queue overflow and task execution abnormalities caused by task over-allocation, ensuring the compliance and stability of task allocation. Fourth, the remaining tasks are allocated sequentially according to the theoretical load ratio priority, ensuring that all tasks are allocated without omission, so that the remaining resources are tilted towards the threads with stronger load capacity, maximizing the processing efficiency of the thread pool. Finally, the task load weight is an integer value, which is fully adapted to the actual execution requirements of multi-threaded task scheduling without additional secondary processing, improving the execution efficiency of task scheduling. At the same time, the balanced task allocation makes the local sorting progress of each thread tend to be synchronized, reducing the waiting time for global result merging, and further meeting the real-time requirements of product sorting in e-commerce high-concurrency scenarios.

[0037] In a preferred embodiment of the present invention, step 4 above may include: Step 4.3: Obtain the task acceptance weight of each worker thread. The task acceptance weight represents the specific number of tasks to be scheduled by each worker thread in the form of task quantity. Specifically, this includes: starting the weight data traversal program of the multi-thread pool, accessing each worker thread in the multi-thread pool one by one according to the preset thread traversal rules. The thread traversal rules can be in ascending order of thread number. Retrieve the final task acceptance weight value determined by the thread after rounding correction and residual allocation. During the retrieval process, a structured worker thread unique identifier-task acceptance weight mapping table is constructed synchronously. The mapping table must contain three levels of core information: the first level is the globally unique identifier of the worker thread, which can be in the form of thread ID or thread number; the second level is the task acceptance weight value corresponding to the thread, which is in the form of an integer number of tasks; and the third level is the source label of the weight value, which can be clearly determined after rounding correction and residual allocation, to ensure that each weight value can be accurately associated with the corresponding worker thread and to prevent the problem of thread and weight mismatch.

[0038] After the weight data of all threads is collected, the weight sum verification operation is immediately performed. The task assignment weight values ​​of all worker threads in the mapping table are accumulated and summed. The calculation formula is: total weight value = weight value of thread 1 + weight value of thread 2 + ... + weight value of thread N, where N is the total number of worker threads.

[0039] The calculated total weight value is compared with the total number of tasks to be sorted to verify whether they are completely equal. If the total number of tasks to be sorted is 1001, and step 4.2 determines that the task weights for threads 1, 2, 3, and 4 are 250, 401, 300, and 50 respectively, then the total weight value = 250 + 401 + 300 + 50 = 1001, which perfectly matches the total number of tasks, and the verification passes. If the cumulative result is 1000 or 1002, it indicates that there is an over-allocation or missing weights, and the backtracking mechanism needs to be triggered immediately to re-examine the initial task allocation calculation, rounding correction, and surplus allocation process to locate the cause of the deviation. The reasons for deviation may include rounding 49.9 to 50 or omitting one task when allocating the margin. These deviations are corrected, and the weight data is re-collected and verified until the total weight value is completely consistent with the total number of tasks to be sorted. After verification, the unique identifier of the worker thread and the task assignment weight mapping table are persisted. The persistent storage can be done by writing to a memory cache or a temporary database, and a status flag indicating that the weight acquisition is complete is generated. At this point, the task assignment weights of all worker threads are acquired. The weight is represented by a specific, executable integer number of tasks, which directly and uniquely represents the specific number of tasks to be sorted that should be scheduled for each worker thread.

[0040] Step 4.4: Based on the task acceptance weight of each worker thread, extract a number of unsorted tasks from the multiple unsorted tasks that are equal to the task acceptance weight of each worker thread, obtaining a subset of unscheduled tasks corresponding to each worker thread. Specifically, this includes: globally standardizing and marking the entire set of unsorted tasks, assigning a unique and consecutive sequential index number to each independent unsorted task, with each unsorted task corresponding to a product sorting calculation unit. The numbering rule starts from 1 and increases sequentially according to the storage order of the tasks in the overall task set until all unsorted tasks are covered, for example, the index numbers of 1001 tasks. The index number, ranging from 1 to 1001, serves as a unique identifier for each task, ensuring the extraction process is traceable and verifiable, and preventing duplicate or missed extractions. Based on the fixed mapping table of unique worker thread identifiers and task assignment weights, the corresponding number of tasks are extracted sequentially from the overall unsorted task set according to the thread traversal order. The extraction rule is that the starting point of the current thread is the extraction endpoint of the previous thread + 1, and the starting point + weight value - 1 is the extraction endpoint of the current thread. For example, with a total of 1001 tasks and thread weights of 250, 401, 300, and 50: Thread 1: Extraction starting point is 1, extraction endpoint... =1+250-1=250, that is, extract the unsorted tasks with index numbers 1 to 250, the number of extractions = 250-1+1=250, which is exactly the same as the task weight of thread 1, forming a subset of the unscheduled tasks of thread 1; Thread 2: extraction start point = 250+1=251, extraction end point = 251+401-1=651, that is, extract the unsorted tasks with index numbers 251 to 651, the number of extractions = 651-251+1=401, which is exactly the same as the task weight of thread 2, forming a subset of the unscheduled tasks of thread 2; Thread 3: extraction start point = 651+ Thread 1: Extraction start point = 652, extraction end point = 652 + 300 - 1 = 951, that is, extract the unsorted tasks with index numbers 652 to 951, the number of extractions = 951 - 652 + 1 = 300, which is completely consistent with the task acceptance weight of thread 3, forming a subset of the unscheduled tasks of thread 3; Thread 4: Extraction start point = 951 + 1 = 952, extraction end point = 952 + 50 - 1 = 1001, extract the unsorted tasks with index numbers 952 to 1001, the number of extractions = 1001 - 952 + 1 = 50, which is consistent with the task acceptance weight of thread 4, resulting in a subset of the unscheduled tasks of thread 4.

[0041] After extracting task subsets for all threads, perform a count check on each scheduled task subset one by one, counting the number of tasks within each subset and comparing it with the task weight value of the corresponding thread. If the number of tasks in a subset does not match the weight, for example, if the number of tasks in the subset for thread 2 is 400, immediately check whether the starting and ending indices of the extraction for that thread are calculated incorrectly, such as mistakenly setting the ending index to 650. Correct the index range, extract again, and check the count again until the number of tasks in all scheduled task subsets accurately matches the task weight of the corresponding thread.

[0042] Step 4.5: Schedule and deliver each subset of tasks to the corresponding worker thread's task queue, completing dynamic task scheduling based on task acceptance weight. This includes: retrieving the queue configuration information of the multi-threaded pool, confirming that each worker thread is bound to an independent, dedicated task queue (a blocking queue with capacity limit configuration and a timeout delivery mechanism), and that all queues are in a normal state capable of receiving tasks. If a thread does not have a dedicated queue configured, it must be created and bound immediately. If a queue is in an abnormally locked state, the abnormality must be resolved first. This can be done by clearing queue deadlocks and repairing queue connections to ensure the queue can receive tasks normally. Establish the correspondence between threads and the subsets of tasks to be scheduled, and start the task delivery program to deliver the complete subsets of tasks to be scheduled for each thread to their dedicated task queues in index order: the 250 task subsets of thread 1 are delivered one by one to the dedicated task queue of thread 1 in the order of index 1 to 250; the 401 task subsets of thread 2 are delivered one by one to the dedicated task queue of thread 2 in the order of index 251 to 651; the 300 task subsets of thread 3 are delivered one by one to the dedicated task queue of thread 3 in the order of index 652 to 951; and the 50 task subsets of thread 4 are delivered one by one to the dedicated task queue of thread 4 in the order of index 952 to 1001.

[0043] During the delivery process, the delivery status of each task is recorded in real time, including delivered, in progress, and failed. Two core metrics are monitored simultaneously: 1. Number of delivered tasks: The number of delivered tasks for each subset is counted in real time to ensure that the final number of delivered tasks is completely consistent with the number of tasks in the subset. For example, thread 2 needs to confirm that all 401 tasks are shown as delivered. 2. Current task queue depth: The current number of tasks in the corresponding thread's queue is read in real time. If the queue depth reaches 90% of the capacity limit (e.g., the depth reaches 450 when the queue capacity is 500), or if a delivery timeout occurs (i.e., a single task is not delivered within the preset 500ms), the thread's task delivery is immediately paused, triggering queue expansion or cache release operations. Queue expansion can adjust the capacity from 500 to 600, and cache release can clear the cache of completed tasks in the queue. Once the queue depth drops to a safe threshold (70% of the capacity limit) and the timeout issue is resolved, delivery resumes and unfinished tasks are delivered.

[0044] Once all threads have completed the delivery of their respective task subsets, and the delivery records show no task loss, no duplicate delivery, and no unhandled exceptions, a scheduling completion check is performed. This check verifies the number of tasks in all queues, for example, if thread 1 has 250 tasks, thread 2 has 401, thread 3 has 300, and thread 4 has 50, confirming that the task weights of each thread are consistent. After the check passes, the scheduling status of the multi-threaded pool is updated to "task scheduling complete." At this point, each worker thread can retrieve tasks from its own dedicated task queue in a first-in-first-out order, initiating the parallel processing flow of multi-dimensional weight calculation and local sorting of products.

[0045] In this embodiment of the invention, task quantification and allocation are carried out based on the theoretical load ratio calculated in the early stage of load measurement. This not only conforms to the real-time load status of threads, but also solves the problem of numerical adaptation after ratio conversion. The core beneficial effects are reflected in four aspects: First, the abstract load ratio is transformed into a specific task allocation value through product operation, realizing the precise link between task allocation and thread load capacity. Second, the core logic of load balancing in the early stage is continued, avoiding the load imbalance problem caused by fixed equal distribution and round-robin allocation. Third, a correction rule of rounding down is adopted to prevent thread queue overflow and task execution abnormalities caused by task over-allocation, ensuring the compliance and stability of task allocation. Fourth, the remaining tasks are allocated sequentially according to the theoretical load ratio priority, which not only ensures that all tasks are allocated without omission, but also tilts the remaining resources to threads with stronger load capacity, maximizing the processing efficiency of the thread pool. Finally, the task load weight is an integer value, which is fully adapted to the actual execution requirements of multi-threaded task scheduling without additional secondary processing, improving the execution efficiency of task scheduling. At the same time, the balanced task allocation makes the local sorting progress of each thread tend to be synchronized, reducing the waiting time for global result merging, and further meeting the real-time requirements of product sorting in e-commerce high-concurrency scenarios.

[0046] In a preferred embodiment of the present invention, step 5 above may include: Step 5.1: Each worker thread retrieves the scheduled tasks from its corresponding task queue. Based on the product identifiers corresponding to the tasks to be sorted, it extracts the assigned subset of products from the product dataset. Specifically, after each worker thread in the multi-threaded pool enters the ready state, it continuously polls its own bound task queue according to the first-in-first-out execution rule, sequentially retrieving the tasks to be sorted that were previously scheduled based on load weights. During the retrieval process, the thread verifies the integrity of each task, discarding abnormal tasks with corrupted formats or missing identifiers, and retaining only compliant tasks to be sorted for subsequent operations. Each compliant task to be sorted has a unique set of product identifiers, which consists of the unique number of the product to be processed and global index information. This set is the core basis for accurate retrieval of the global product dataset, with no duplicate or redundant identifiers. After the worker thread completes the compliant task retrieval, it immediately parses the set of product identifiers carried by the task, decomposes it into individual product identifier information, and uses each product identifier as an accurate retrieval condition to initiate a batch retrieval request to the global product dataset.

[0047] Upon receiving a search request, the global product dataset matches product identifiers one by one, extracting complete business data for the corresponding product. This data includes core fields such as product sales volume, user ratings, product price, and product popularity. The extraction process strictly adheres to the corresponding identifier matching rules, ensuring that product data not assigned to the current thread is not mixed in, and that no core field of any assigned product is omitted. After extraction, the worker thread aggregates, deduplicates, and encapsulates all successfully matched complete product data, forming a product subset specifically for the current worker thread. Subsequently, the thread automatically verifies the total number of products in the product subset, confirming that the total number is completely consistent with the sorting task undertaken by the thread. Once verification is successful, the product subset is officially locked, providing a dedicated and unbiased data foundation for subsequent parallel weight calculations and local sorting.

[0048] Step 5.2: Each worker thread performs multi-dimensional weight calculations on each product in the extracted product subset in parallel to obtain the comprehensive weight value of each product. Specifically, after all worker threads complete the extraction of the product subset, they simultaneously start independent parallel computing processes. Each thread operates independently based on its own computing resources, without data interaction, sharing of computing cache, or contention for hardware resources. Each thread processes its own product subset completely independently, ensuring the stability and efficiency of parallel computing. For each product in the product subset, four core evaluation dimensions that influence ranking in e-commerce scenarios are selected for weight calculation: product sales volume, user rating, product price, and product popularity. Standardized fixed weight coefficients are pre-set for each dimension, and the sum of the weight coefficients for all dimensions is strictly equal to 1, ensuring the rationality and consistency of the weight calculation. The specific dimensional weight coefficients are set as follows: the weight coefficient for product sales volume is 0.3, the weight coefficient for user rating is 0.2, the weight coefficient for product price is 0.2, and the weight coefficient for product popularity is 0.3. The sum of the four coefficients is 0.3 + 0.2 + 0.2 + 0.3, which equals 1, meeting the standardized requirements for weight calculation.

[0049] To address the issues of unbalanced weighting of high-priced items and calculation anomalies caused by prices of 0 in the product price dimension, a reverse normalization process is performed on the price dimension before calculation. A fixed value of 100 divided by the product price plus 1 is used as the final value of the price dimension. The calculation formula is that the price dimension value equals 100 divided by the sum of the product price and 1. This avoids excessive weighting of high-priced items and completely eliminates the anomaly of a denominator of 0 caused by a product price of 0. After completing the dimension preprocessing, the formal calculation of the comprehensive product weight value is carried out. The specific calculation process is executed step by step: Calculate the product sales volume dimension score by multiplying the product sales volume value by the sales volume dimension weight coefficient of 0.3; calculate the user rating dimension score by multiplying the user rating value by the user rating dimension weight coefficient of 0.2; calculate the product price dimension score using the processed price... The dimension value is multiplied by the price dimension weight coefficient of 0.2; the product popularity dimension score is calculated by multiplying the product popularity value by the popularity dimension weight coefficient of 0.3; the scores of the four dimensions are added together in sequence, and the final result is the comprehensive weight value of the product. The complete calculation formula is: Comprehensive weight value = Product sales × 0.3 + User rating × 0.2 + (100 ÷ (Product price + 1)) × 0.2 + Product popularity × 0.3. For each product in the product subset, the worker thread repeatedly executes the above dimension preprocessing and step-by-step weight calculation process to generate a unique comprehensive weight value for each product. The comprehensive weight value is then bound and stored with the corresponding product to ensure that the weight value corresponds to the product, without mismatch or omission. This comprehensive weight value directly quantifies the product's ranking priority; the larger the value, the higher the product's ranking.

[0050] Step 5.3: Each worker thread sorts the products in the extracted product subset in descending order based on the comprehensive weight value of each product, obtaining the local sorting result of each worker thread. Specifically, after a single worker thread completes the comprehensive weight calculation of all products in its own product subset, it immediately starts the local sorting process. The sorting process is only carried out on the product subset of the current thread and does not call or touch the product data of other threads. The sorting logic and calculation rules are kept consistent throughout the process. The sorting is based solely on the comprehensive weight value bound to the product and is executed according to the descending order of comprehensive weight values. An efficient internal sorting algorithm is used to compare and reposition all products in the product subset one by one. The specific sorting execution process is as follows: First, select the benchmark product in the product subset and compare its comprehensive weight value with the other products in turn. If the comprehensive weight value of the compared product is greater than that of the benchmark product, then swap their positions. Repeat this comparison and repositioning operation until all products in the product subset are arranged in order from high to low according to their comprehensive weight values, without any positional errors or weight inversions. After sorting, the thread verifies the ordered product sequence, confirming that the number of products in the sequence matches the total number of product subsets and that the overall weight value is continuously decreasing. Once verification is successful, the thread generates the local sorting result for the current worker thread. This local sorting result is a well-ordered product sequence, with the product sorting priority perfectly matching the overall weight value rules, and can be directly used for global merge processing. All worker threads in the multi-threaded pool execute the above sorting operation synchronously. Thanks to the pre-load-balanced task allocation, the size of the product subset processed by each thread is highly adapted to its own load capacity and processing speed, resulting in synchronized generation of local sorting results without any individual threads experiencing prolonged delays.

[0051] In this embodiment of the invention, this step relies on the results of previous dynamic load balancing task allocation to achieve parallel and precise processing of massive amounts of product data. A closed-loop process is formed from data extraction and weight calculation to local sorting, specifically addressing the core pain points of traditional multi-threaded product sorting. This ensures the accuracy and independence of data processing, extracting a unique subset of products through precise matching of product identifiers and total count verification, eliminating data confusion, mis-taking, and omissions between threads, and preventing data corruption that could distort the sorting results. This meets the accuracy requirements of full-volume product sorting in e-commerce. It maximizes the efficient use of multi-threaded resources, with each thread independently performing parallel weight calculation and local sorting without resource contention or data interaction conflicts. This abandons the inefficient serial processing mode, fully leveraging the parallel computing advantages of the multi-threaded pool, significantly shortening the time required for weight calculation and local sorting of massive amounts of products, and solving the processing speed issues in high-concurrency scenarios such as e-commerce promotions and flash sales. The system addresses the challenge of performance lag; it standardizes and makes weight calculation rules transparent, with unified dimension weight coefficients and price preprocessing logic, ensuring consistent weight calculation across all products. This avoids unfair sorting due to differences in calculation rules, while the refined step-by-step calculation process reduces computational errors and enhances the reliability of the overall weight value. Continuing the core value of load balancing, the matching of task load with thread capacity ensures synchronized local sorting progress across threads, overcoming the problems of large differences in local result generation rates and thread blocking slowing down the overall progress in traditional scheduling. It strengthens the execution foundation of global sorting, ensuring that the local results output by each thread are standard descending ordered sequences with no overlap or omissions in the data range. This avoids global merge failures caused by disordered local results and reduces the time spent on data processing during the merge phase, further improving the stability and real-time performance of the overall sorting process and perfectly meeting the reliability requirements of product sorting in high-concurrency e-commerce scenarios.

[0052] In a preferred embodiment of the present invention, step 6 above may include: Step 6.1 involves collecting the local sorting results from each worker thread. Each local sorting result is an ordered sequence of goods arranged in descending order of the comprehensive weight value. Based on the initial task granularity used in the sorting task, the required merging order for the merging operation is determined. Specifically, this includes: starting the global merging pre-management program; relying on the multi-threaded communication mechanism to collect the local sorting results output by all worker threads in the multi-threaded pool in batches; each local sorting result is accompanied by metadata such as a unique thread identifier, result generation timestamp, and total number of goods to ensure traceability and verification of the results; and performing a double integrity check on each local sorting result during the collection process: verifying whether the result is an ordered sequence of goods arranged in descending order of the comprehensive weight value, with no abnormalities such as ascending weight value, disordered order, or missing weight value; and verifying that the total number of goods in the local result is completely consistent with the amount of sorting tasks previously undertaken by the corresponding thread, with no missing or redundant goods. After all compliant local sorting results are collected, they are uniformly stored in the global merging data buffer to form a structured set of local results to be merged.

[0053] After collecting local results, the initial task granularity used when allocating tasks to be sorted is retrieved. Combined with the number of local results to be merged, the merging order required for the merging operation is calculated and determined. The core calculation rule for the merging order is: the merging order is equal to the ratio of the number of local results to be merged to the initial task granularity, rounded up; if the number of local results to be merged is divisible by the initial task granularity, the merging order is the quotient of the two; if it is not divisible, the merging order is the quotient plus 1. For example: If there are 8 local results to be merged and the initial task granularity is 3, then 8 divided by 3 equals 2 with a remainder of 2, and after rounding up, the merging order is 3; if there are 6 local results to be merged and the initial task granularity is 3, then 6 divided by 3 equals 2, and the merging order is 2; if there are 5 local results to be merged and the initial task granularity is 2, then 5 divided by 2 equals 2 with a remainder of 1, and the merging order is 3. The merging order directly determines the parallel grouping method of subsequent multi-way merging, ensuring that the efficiency of the merging operation is highly adapted to the initial task allocation granularity.

[0054] Step 6.2: Based on the merging order, a multi-way merging method is used to perform parallel merging of each local sorting result. The comprehensive weight value of the currently unmerged items in each local sorting result is compared one by one. The items with the highest comprehensive weight value are extracted and merged into the global sorting result sequence. Specifically, after completing the first round of parallel merging operation, the global merging engine scans all local results to be merged in real time. By marking the unmerged position, it is confirmed whether there are still unmerged items in each local result. If there are unmerged items, the multi-way merging operation is repeated: each parallel merging group relocates the first unmerged item in each local result, extracts and compares the comprehensive weight value, selects the item with the highest weight value and merges it into the global sorting result sequence, and updates the unmerged position of the corresponding local result. This cyclical merging process continues until there are no unmerged items in all local sorting results in all parallel merging groups. At this time, the global sorting result sequence has integrated the items in all local sorting results, and all items are arranged in descending order of comprehensive weight value from high to low. Finally, the integrity of the global sorting result is verified: First, the total number of goods in the global sequence is verified to be completely consistent with the total number of goods corresponding to the previous sorting task, with no goods missing or duplicated; Second, multiple consecutive groups of goods are randomly selected from the sequence to verify whether their comprehensive weight values ​​are arranged in a continuous descending order, with no weight values ​​reversed or mismatched. After the verification is passed, the complete global sorting result is determined to be generated.

[0055] Step 6.3: Iteratively perform the comparison operation of the comprehensive weight values ​​of currently unmerged items in each local sorting result. During the comparison, the item with the highest comprehensive weight value is extracted and merged into the global sorting result sequence until all items in all local sorting results have been merged, resulting in a complete global sorting result. Specifically, after completing the first round of parallel merging operations, the global merging engine scans all local results to be merged in real time. It confirms whether there are still unmerged items in each local result by marking the unmerged position. If unmerged items exist, the multi-way merging operation is repeated: each parallel merging group relocates the first unmerged item in each local result, extracts and compares the comprehensive weight values ​​again, and selects the item with the highest weight value. The products are then merged into the global sorting result sequence, while the unmerged positions of the corresponding local results are updated. This cyclical merging process continues until all local sorting results in all parallel merging groups have no unmerged products. At this point, the global sorting result sequence has integrated all products from the local sorting results, and all products are arranged in descending order of their comprehensive weight values. The integrity of the global sorting result is then verified: the total number of products in the global sequence is verified to be completely consistent with the total number of products corresponding to the previous expected sorting task, with no products missing or duplicated; multiple consecutive groups of products are randomly selected from the sequence, and their comprehensive weight values ​​are verified to be arranged in continuous descending order, with no weight values ​​reversed or mismatched. After the verification is passed, the complete global sorting result is determined to be generated.

[0056] In this embodiment of the invention, the following technical means is adopted: first, collect the compliant descending local sorting results of each working thread; then, dynamically determine the merging order based on the initial task granularity of the task to be sorted; and finally, perform parallel merging of the local results using a multi-way merging method according to the merging order. The comprehensive weight values ​​of the unmerged products in each local result are compared cyclically, and the highest value is extracted and merged into the global sequence until all products are merged. Therefore, this overcomes the technical problems of the traditional global product sorting and merging process, such as the fixed merging order leading to unbalanced resource utilization, low efficiency of single-way merging, easy omission of products or disordered sorting during the merging process, and slow global sorting result generation speed that cannot adapt to the high concurrency scenario of e-commerce. Thus, the global merging efficiency is highly adapted to the initial task allocation granularity, fully releasing parallel computing resources and greatly shortening the merging time. This ensures that the final generated global sorting result completely includes all products to be sorted and is arranged in descending order according to the comprehensive weight value, meeting the real-time, accuracy and completeness requirements of global sorting of massive products in e-commerce scenarios.

[0057] like Figure 2 As shown, embodiments of the present invention also provide a multi-threaded product sorting task scheduling and data processing system based on load balancing, comprising: The acquisition module is used to acquire the real-time load metrics of each worker thread in the multi-thread pool and acquire the product dataset to be processed. The product dataset is decomposed into multiple tasks to be sorted according to the preset initial task granularity. The real-time load metrics include the current task queue depth and the task processing rate per unit time. The extraction module is used to extract the load metric vector of each worker thread at the current moment based on the real-time load index of each worker thread, map the load metric vector to the preset load feature space, and construct a dynamic load distribution surface in the load feature space according to the spatial distribution of each load metric vector. The partitioning module is used to perform spatial partitioning on the dynamic load distribution surface, dividing the dynamic load distribution surface into multiple geometric sub-regions; calculating the spatial volume ratio of each geometric sub-region, and using the spatial volume ratio of each geometric sub-region as the theoretical load-bearing ratio of the corresponding working thread. The fitting module is used to numerically fit the theoretical acceptance ratio with the total number of tasks to be sorted to obtain the task acceptance weight of each worker thread; and to schedule multiple tasks to be sorted to the corresponding worker threads according to the task acceptance weight. The sorting module is used for each worker thread to execute the sorting task in parallel. It performs multi-dimensional weight calculation and sorting on the product subsets assigned to each worker thread to obtain the local sorting results of each worker thread. The processing module is used to collect the local sorting results of each worker thread, determine the merging strategy based on the initial task granularity used when decomposing the task to be sorted, and merge the local sorting results in an orderly manner according to the merging strategy to obtain the global sorting result.

[0058] It should be noted that this system is a system corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0059] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0060] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0061] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for scheduling and processing multi-threaded product sorting tasks based on load balancing, characterized in that, The method includes: Get the real-time load metrics of each worker thread in the multi-thread pool, and get the product dataset to be processed. Decompose the product dataset into multiple tasks to be sorted according to the preset initial task granularity. The real-time load metrics include the current task queue depth and the task processing rate per unit time. Based on the real-time load metrics of each worker thread, the load metric vector of each worker thread at the current moment is extracted, and the load metric vector is mapped to a preset load feature space. In the load feature space, a dynamic load distribution surface is constructed according to the spatial distribution of each load metric vector. The dynamic load distribution surface is spatially partitioned into multiple geometric sub-regions; the spatial volume ratio of each geometric sub-region is calculated, and the spatial volume ratio of each geometric sub-region is used as the theoretical load-bearing ratio of the corresponding working thread. The theoretical task acceptance ratio is numerically fitted to the total number of tasks to be sorted to obtain the task acceptance weight of each worker thread; and multiple tasks to be sorted are scheduled to the corresponding worker threads according to the task acceptance weight. Each worker thread executes the sorting task in parallel, and performs multi-dimensional weight calculation and sorting on the product subset assigned to each worker thread to obtain the local sorting result of each worker thread. By collecting the local sorting results of each worker thread, a merging strategy is determined based on the initial task granularity used when decomposing the tasks to be sorted; the local sorting results are then merged in an orderly manner according to the merging strategy to obtain the global sorting result.

2. The multi-threaded product sorting task scheduling and data processing method based on load balancing according to claim 1, characterized in that, Based on the real-time load metrics of each worker thread, the load metric vector of each worker thread at the current moment is extracted. This load metric vector is then mapped to a preset load feature space. A dynamic load distribution surface is constructed in the load feature space based on the spatial distribution of each load metric vector, including: Based on the current task queue depth and task processing rate per unit time of each worker thread, the current task queue depth of each worker thread is used as the first dimension component and the task processing rate per unit time is used as the second dimension component, and the two-dimensional load metric vector of each worker thread at the current moment is obtained by combining them. The two-dimensional load metric vector of each worker thread is used as a spatial data point and mapped to a preset two-dimensional load feature space to form a set of spatial data points corresponding to each worker thread in the two-dimensional load feature space. Based on the distribution of spatial data points in the two-dimensional load feature space, a spatial surface fitting method is adopted, using the spatial data points as surface interpolation nodes. By constructing a continuous surface passing through each spatial data point, a dynamic load distribution surface is obtained. The dynamic load distribution surface is used to characterize the comprehensive load distribution state of each working thread at the current moment.

3. The method for scheduling and processing multi-threaded product sorting tasks based on load balancing according to claim 2, characterized in that, The dynamic load distribution surface is spatially partitioned into multiple geometric sub-regions. Calculate the spatial volume ratio of each geometric sub-region, and use the spatial volume ratio of each geometric sub-region as the theoretical carrying ratio of the corresponding working thread, including: In the two-dimensional load feature space, the domain range of the dynamic load distribution surface is determined, and the domain range is limited by the distribution boundary of the spatial data points corresponding to each working thread; The dynamic load distribution surface is spatially partitioned, dividing it into multiple geometric sub-regions along the coordinate axes. Calculate the spatial volume of each geometric sub-region under the dynamic load distribution surface, and calculate the ratio of the spatial volume of each geometric sub-region to the total volume of the dynamic load distribution surface to obtain the spatial volume ratio of each geometric sub-region. The spatial volume ratio of each geometric sub-region is determined as the theoretical load-bearing ratio of the corresponding worker thread. The theoretical load-bearing ratio is used to characterize the proportion of the total workload that each worker thread should undertake under load balancing conditions.

4. The multi-threaded product sorting task scheduling and data processing method based on load balancing according to claim 3, characterized in that, The task acceptance weight of each worker thread is obtained by numerically fitting the theoretical acceptance ratio with the total number of tasks to be sorted. And based on the task acceptance weight, multiple tasks to be sorted are assigned to the corresponding worker threads, including: The initial task allocation for each worker thread is obtained by multiplying the theoretical capacity of each worker thread by the total number of tasks to be sorted. The initial task allocation quantity for each worker thread is rounded down and corrected. The remaining tasks after rounding down are then allocated sequentially according to the theoretical capacity of each worker thread until all tasks to be sorted are allocated, thus obtaining the task capacity weight of each worker thread.

5. The method for scheduling and processing multi-threaded product sorting tasks based on load balancing according to claim 4, characterized in that, And based on the task acceptance weight, multiple tasks to be sorted are scheduled to their corresponding worker threads, including: Obtain the task acceptance weight of each worker thread. The task acceptance weight represents the specific number of tasks to be scheduled by each worker thread in the form of task quantity. Based on the task acceptance weight of each worker thread, extract the number of unsorted tasks that are equal to the task acceptance weight of each worker thread from multiple unsorted tasks, and obtain the subset of unscheduled tasks corresponding to each worker thread. Each subset of tasks to be scheduled is dispatched to the task queue of the corresponding worker thread, thus completing dynamic task scheduling based on task acceptance weight.

6. The multi-threaded product sorting task scheduling and data processing method based on load balancing according to claim 5, characterized in that, Each worker thread executes the sorting task in parallel, performing multi-dimensional weight calculations and sorting on the product subset assigned to each worker thread, resulting in the local sorting results for each worker thread, including: Step 5.1: Each worker thread retrieves the scheduled tasks to be sorted from its corresponding task queue, and extracts the assigned subset of products from the product dataset based on the product identifier corresponding to the task to be sorted. Step 5.2: Each worker thread performs multi-dimensional weight calculations for each product in the extracted product subset in parallel to obtain the comprehensive weight value of each product. Step 5.3: Each worker thread sorts the products in the extracted product subset in descending order according to the comprehensive weight value of each product, and obtains the local sorting result of each worker thread.

7. The multi-threaded product sorting task scheduling and data processing method based on load balancing according to claim 6, characterized in that, By collecting the local sorting results from each worker thread, a merging strategy is determined based on the initial task granularity used when decomposing the tasks to be sorted. The local sorting results are then merged in an ordered manner according to the merging strategy to obtain the global sorting result, including: By collecting the local sorting results of each worker thread, each local sorting result is an ordered sequence of goods arranged in descending order according to the comprehensive weight value; the merging order required for the merge operation is determined according to the initial task granularity used when the sorting task is performed. Based on the merging order, a multi-way merging method is used to perform parallel merging of each local sorting result. The comprehensive weight value of the currently unmerged products in each local sorting result is compared one by one, and the products with the highest comprehensive weight value are taken out and merged into the global sorting result sequence. The process iteratively compares the overall weight values ​​of unmerged items in each local sorting result. During the comparison, the item with the highest overall weight value is extracted and merged into the global sorting result sequence. This continues until all items in the local sorting results have been merged, resulting in a complete global sorting result.

8. A load-balanced multi-threaded product sorting task scheduling and data processing system, wherein the system implements the method as described in any one of claims 1 to 7, characterized in that, include: The acquisition module is used to acquire the real-time load metrics of each worker thread in the multi-thread pool and acquire the product dataset to be processed. The product dataset is decomposed into multiple tasks to be sorted according to the preset initial task granularity. The real-time load metrics include the current task queue depth and the task processing rate per unit time. The extraction module is used to extract the load metric vector of each worker thread at the current moment based on the real-time load index of each worker thread, map the load metric vector to the preset load feature space, and construct a dynamic load distribution surface in the load feature space according to the spatial distribution of each load metric vector. The partitioning module is used to perform spatial partitioning on the dynamic load distribution surface, dividing the dynamic load distribution surface into multiple geometric sub-regions; calculating the spatial volume ratio of each geometric sub-region, and using the spatial volume ratio of each geometric sub-region as the theoretical load-bearing ratio of the corresponding working thread. The fitting module is used to numerically fit the theoretical acceptance ratio with the total number of tasks to be sorted, and obtain the task acceptance weight of each worker thread. And based on the task acceptance weight, multiple tasks to be sorted are scheduled to the corresponding worker threads; The sorting module is used for each worker thread to execute the sorting task in parallel. It performs multi-dimensional weight calculation and sorting on the product subsets assigned to each worker thread to obtain the local sorting results of each worker thread. The processing module is used to collect the local sorting results of each worker thread, determine the merging strategy based on the initial task granularity used when decomposing the task to be sorted, and merge the local sorting results in an orderly manner according to the merging strategy to obtain the global sorting result.

9. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images