Dynamic computing power matching and scheduling method based on multi-dimensional labels

Abstract

The invention discloses a dynamic computing power matching and scheduling method based on a multi-dimensional label. The dynamic computing power matching and scheduling method specifically comprises the following steps: S1, task calibration and feature vector generation; s2, domain mapping and initial routing are calculated; s3, carrying out quantitative evaluation on the computing power demand; s4, monitoring the domain load and judging the blockage condition of the domain load; s5, performing priority judgment and dynamic routing decision; s6, free domain selection and task migration; according to the method, the basic efficiency of data processing is guaranteed through multi-dimensional label calibration and initial mapping of the computational domain, differential processing is carried out by monitoring the load of the computational domain in real time and dynamically judging the blockage condition in combination with the task priority, and continuous blockage of a single computational domain is avoided. Finally, the free domain with the lowest load is selected for task migration, balanced utilization of computing power resources is achieved, and the overall data processing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a dynamic computing power matching and scheduling method based on multi-dimensional labels. Background Technology

[0002] In the current field of data processing, existing data processing methods generally rely on processors to analyze data types and then add multi-dimensional tags to the data. These multi-dimensional tags cover key information such as data source, data format, and data importance. Subsequently, corresponding computing resources are allocated to the data according to the tag type, so as to achieve differentiated processing of different types of data and ensure that basic data processing needs are met.

[0003] However, when faced with a large number of data processing tasks, existing methods usually divide the massive data into several categories according to the data type, and then input the data of different categories into the computing domains corresponding to their data types for processing. However, when a large amount of data of a single type floods into a certain computing domain at the same time, it is very easy to cause data accumulation and computing power saturation in the computing domain, resulting in data processing progress stalling and the inability to complete data processing tasks in a timely manner. This has limitations and is difficult to cope with high-concurrency data processing scenarios.

[0004] The reason for this problem is that existing data processing methods have a fixed allocation of computing domains. This means that a single type of data is only injected into its corresponding specific computing domain. This not only causes the aforementioned congestion in a single computing domain, but also leaves other computing domains that are not compatible with that data type idle, resulting in a serious waste of computing resources. More importantly, because these idle computing domains differ significantly from the original computing domain in terms of hardware architecture, algorithm support, and other operating principles, even if data is migrated from the congested computing domain to these idle domains, the computation time will be much longer than required in the original computing domain. This is especially problematic because high-priority data cannot be processed in a timely manner due to computational delays, thus affecting the normal operation of subsequent business processes and presenting limitations. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic computing power matching and scheduling method based on multi-dimensional tags to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a dynamic computing power matching and scheduling method based on multi-dimensional tags, comprising the following steps:

[0007] S1. Task labeling and feature vector generation: Perform multi-dimensional labeling on the input task and generate task type labels.

[0008] S2. Computation domain mapping and initial routing: Based on the task type label, the task is assigned to the corresponding specialized computation domain for processing through a mapping function;

[0009] S3. Quantitative assessment of computing power requirements: Based on the data volume and type of the task, a computing power requirement function is constructed to quantify the computing resources required.

[0010] S4. Monitor domain load and determine its congestion status. Monitor the load of each computing domain in real time and dynamically determine whether it is in a congested state by comparing it with a preset threshold.

[0011] S5. Priority determination and dynamic routing decision: If the target domain is congested, the task priority is determined; high-priority tasks are queued in this domain, and medium and low-priority tasks trigger rerouting.

[0012] S6. Idle Domain Selection and Task Migration: For tasks that need to be migrated, select the domain with the lowest load from the set of idle domains, and perform computing resource matching and task handover.

[0013] S7. Task execution and resource allocation: The task is executed in the final determined computing domain, and the resource manager allocates matching computing resources to complete the calculation.

[0014] Preferably, in step S1, when a new computing task Ti arrives at the system, the scheduler immediately generates a multidimensional label Li for it, and the calculation formula is as follows:

[0015] Li=(Type,Priority,DataVolume,...)

[0016] Where Type is the task type; Priority is the task priority, defined as P∈{high(H), medium(M), low(L)}; and DataVolume is the amount of data to be processed by the task.

[0017] Preferably, in step S2, the specialized computing domain is divided into D1AI training domain, D2 database query domain, and D3 Web service domain.

[0018] Preferably, the calculation formula in step S2 is as follows:

[0019] Dtar get =Fma p (Typei)Dtarget is the target computational domain assigned to task Ti; F map Tyupei is a mapping function; Tyupei is the type label for task Ti.

[0020] Preferably, the formula for quantifying and evaluating computing power requirements in step S3 is as follows:

[0021] R need(Ti)=C Type ×DataVolumei

[0022] R need (Ti) represents the estimated computing power requirement for task Ti; C Type The computational intensity coefficient is related to the task type; DataVolumei is the amount of data to be processed in task Ti.

[0023] Preferably, the formula for monitoring and determining the congestion status of the domain load in step S4 is as follows:

[0024]

[0025] IsCongested(Dj) represents the congestion status of domain Dj, with 1 indicating congestion and 0 indicating normal operation; Lood(Dj) represents the real-time load rate of domain Dj; θc ongestion The preset congestion detection threshold for the system.

[0026] Preferably, the priority determination logic in step S5 is as follows:

[0027] When the target computing domain is blocked, a priority determination process is initiated. For high-priority tasks, the task continues to wait in the target domain queue; for medium and low-priority tasks, the task is triggered to be rerouted to a set of idle computing domains.

[0028] Preferably, the priority determination formula in step S5 is as follows:

[0029]

[0030] A ction Pi represents the basic operations to be performed on the task; Pi represents the priority of task Ti.

[0031] Preferably, the routing decision formula in step S5 is as follows:

[0032]

[0033] Route(Ti) is the final route target of task Ti; Dtarget is the initial target domain of the task determined in step S2; Sidle is the set of idle domains of the current system.

[0034] Preferably, the formula for selecting the free space in step S6 is as follows:

[0035]

[0036] D idle The selected, least loaded free area; S idle Let S be the set of all non-blocking domains. idle ={Dk |IsCongested(D k )=0};argmin represents the parameter corresponding to the minimum value of the function, which is the domain D in this case. k .

[0037] The technical effects and advantages of this invention are as follows:

[0038] 1. This method ensures that tasks are preferentially assigned to the most suitable specialized computing domain by using multidimensional labeling and initial mapping of the computing domain, thus guaranteeing the basic efficiency of data processing.

[0039] 2. By monitoring the load of the computing domain in real time and dynamically judging the congestion situation, and taking into account the task priority, differentiated processing is carried out. High-priority tasks are queued in the original domain to avoid cross-domain time consumption, while medium and low-priority tasks are rerouted. This ensures the timeliness of high-priority tasks and avoids continuous congestion in a single computing domain.

[0040] 3. By selecting the idle domain with the lowest load for task migration, the balanced utilization of computing resources is achieved, solving the problem of wasted computing resources in existing methods. At the same time, compatibility verification ensures the feasibility of cross-domain computing and improves the overall data processing efficiency. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0042] Figure 2 This is a schematic diagram of the task preprocessing process of the present invention;

[0043] Figure 3 This is a schematic diagram of the load determination and routing decision of the present invention;

[0044] Figure 4 This is a schematic diagram of the task migration process of the present invention;

[0045] Figure 5 This is a schematic diagram of the task execution process of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This invention provides, for example Figure 1 The method for dynamic computing power matching and scheduling based on multi-dimensional labels, as shown, includes the following steps:

[0048] S1. Task labeling and feature vector generation: Perform multi-dimensional labeling on the input task and generate task type labels.

[0049] S2. Computation domain mapping and initial routing: Based on the task type label, the task is assigned to the corresponding specialized computation domain for processing through a mapping function;

[0050] S3. Quantitative assessment of computing power requirements: Based on the data volume and type of the task, a computing power requirement function is constructed to quantify the computing resources required.

[0051] S4. Monitor domain load and determine its congestion status. Monitor the load of each computing domain in real time and dynamically determine whether it is in a congested state by comparing it with a preset threshold.

[0052] S5. Priority determination and dynamic routing decision: If the target domain is congested, the task priority is determined; high-priority tasks are queued in this domain, and medium and low-priority tasks trigger rerouting.

[0053] S6. Idle Domain Selection and Task Migration: For tasks that need to be migrated, select the domain with the lowest load from the set of idle domains, and perform computing resource matching and task handover.

[0054] S7. Task execution and resource allocation: The task is executed in the final determined computing domain, and the resource manager allocates matching computing resources to complete the calculation.

[0055] Example 1: In this example, data is input to the task labeling module: User behavior data and transaction data are aggregated through the system interface and then enter the task labeling module. The module labels each piece of data with multi-dimensional tags, covering data source, format, priority, and data volume. Then, a unique task type tag is generated based on the multi-dimensional tags. Subsequently, the data with tags is transferred to the computational domain mapping module.

[0056] The mapping module assigns initial routes: The computation domain mapping module, relying on its built-in tag-to-specialized computation domain mapping relationship library, matches data with the corresponding specialized computation domain and generates initial route instructions. The data first carries these instructions to the computational power demand quantification module to calculate the required computational power, and then carries the quantification results to the corresponding specialized computation domain.

[0057] Computational domain load assessment and load balancing: After data enters a specialized computing domain, it first goes into the task queue. The load monitoring module collects real-time operational data from the computing domain and sends it to the congestion assessment module for comparison with thresholds. Queue data from a normal computing domain is directly transferred to the domain's resource allocation module; queue data from a congested computing domain first goes to the priority assessment module.

[0058] Priority processing and migration: The priority determination module splits congested domain data, returning high-priority data to its original domain queue, while medium- and low-priority data, along with rerouting instructions and computing power results, are transferred to the idle domain filtering module. This module filters normal computing domains and sorts them by load to select target migration domains. After verifying compatibility, it generates migration instructions, and the data is transferred to the data migration module.

[0059] Migration Processing and Result Output: The data migration module transfers data to the target migration domain. After the data enters the domain queue, the migration module notifies the resource allocation module. Each computing domain resource allocation module allocates resources according to computing power requirements, and the data is then processed by the task execution module. The processing result is first verified by the result verification module, and then sent to the result output module for feedback to the corresponding business system.

[0060] like Figures 2 to 5 The method for dynamic computing power matching and scheduling based on multi-dimensional labels, as shown, includes the following steps:

[0061] S1. Task labeling and feature vector generation: Perform multi-dimensional labeling on the input task and generate task type labels.

[0062] S2. Computation domain mapping and initial routing: Based on the task type label, the task is assigned to the corresponding specialized computation domain for processing through a mapping function;

[0063] S3. Quantitative assessment of computing power requirements: Based on the data volume and type of the task, a computing power requirement function is constructed to quantify the computing resources required.

[0064] S4. Monitor domain load and determine its congestion status. Monitor the load of each computing domain in real time and dynamically determine whether it is in a congested state by comparing it with a preset threshold.

[0065] S5. Priority determination and dynamic routing decision: If the target domain is congested, the task priority is determined; high-priority tasks are queued in this domain, and medium and low-priority tasks trigger rerouting.

[0066] S6. Idle Domain Selection and Task Migration: For tasks that need to be migrated, select the domain with the lowest load from the set of idle domains, and perform computing resource matching and task handover.

[0067] S7. Task execution and resource allocation: The task is executed in the final determined computing domain, and the resource manager allocates matching computing resources to complete the calculation.

[0068] In step S1, when a new computational task Ti arrives at the system, the scheduler immediately generates a multidimensional label Li for it, and the calculation formula is as follows:

[0069] Li=(Type,Priority,DataVolume,...)

[0070] Where Type is the task type; Priority is the task priority, defined as P∈{high(H), medium(M), low(L)}; and DataVolume is the amount of data to be processed by the task.

[0071] This step is fundamental to the method and aims to accurately classify and route the input computational tasks.

[0072] In step S2, the specialized computing domain is divided into D1AI training domain, D2 database query domain, and D3 Web service domain. The system is simply divided into three domains, and the number of domains can be adjusted according to needs. Each domain specializes in handling a specific type of task.

[0073] The calculation formula in step S2 is as follows:

[0074] Dtar get =Fma p (Typei)Dtarget is the target computational domain assigned to task Ti; F map Tyupei is a mapping function; Tyupei is the type label for task Ti.

[0075] This formula defines the initial mapping relationship between tasks and computational domains.

[0076] The formula for quantifying and evaluating computing power requirements in step S3 is as follows:

[0077] R need (Ti)=C Type ×DataVolumei

[0078] R need (Ti) represents the estimated computing power requirement for task Ti; C Type The calculation intensity coefficient is related to the task type; DataVolumei is the amount of data to be processed in task Ti;

[0079] This formula is used to quantify the computing resources required to evaluate a task.

[0080] The formula for monitoring and determining the congestion status of the domain load in step S4 is as follows:

[0081]

[0082] IsCongested(Dj) represents the congestion status of domain Dj, with 1 indicating congestion and 0 indicating normal operation; Lood(Dj) represents the real-time load rate of domain Dj; θc ongestion The system's preset congestion detection threshold;

[0083] This formula is a Boolean conditional function used to determine whether the computational domain is in a blocked state.

[0084] The priority determination logic in step S5 is as follows:

[0085] When the target computing domain is blocked, a priority determination process is initiated. For high-priority tasks, the task continues to wait in the target domain queue; for medium and low-priority tasks, the task is triggered to be rerouted to a set of free computing domains.

[0086] This step is the core scheduling logic. First, it determines whether the target computing domain is blocked. If it is blocked, it further determines the task priority and executes a differentiated routing strategy based on the priority.

[0087] The priority determination formula in step S5 is as follows:

[0088]

[0089] A ction Pi represents the basic operations to be performed on the task; Pi represents the priority of task Ti.

[0090] This formula defines the initial action decision based on task priority when the target domain is blocked.

[0091] The routing decision formula in step S5 is as follows:

[0092]

[0093] Route(Ti) is the final route target of task Ti; Dtarget is the initial target domain of the task determined in step S2; Sidle is the set of idle domains in the current system;

[0094] This formula determines the final routing target of the task based on the priority judgment result.

[0095] The formula for selecting the free domain in step S6 is as follows:

[0096]

[0097] D idle The selected, least loaded free area; S idle Let S be the set of all non-blocking domains. idle ={D k |IsCongested(D k )=0};argmin represents the parameter corresponding to the minimum value of the function, which is the domain D in this case. k ;

[0098] This formula defines the strategy for selecting the target domain from the set of free domains.

[0099] Example 2: In this example, the idle computing domain and the original computing domain differ in hardware architecture and algorithm support. The original domain uses a dedicated chip adapted to specific data, while the idle domain uses a general-purpose chip. In terms of algorithm support, the original domain has targeted processing algorithms, while the idle domain has no optimized algorithms. After data migration, it is necessary to adapt to new hardware and algorithms, which increases the number of calculation steps and therefore takes longer than in the original domain.

[0100] The reason why extended processing time is not a concern for medium and low-level data is that this type of data does not affect core business, and delayed processing will not cause direct losses; moreover, its timeliness requirements are low, and slightly longer computing time will not affect user experience or subsequent business processes. Therefore, medium and low-level data can be accepted to be sent to an idle computing domain for processing.

[0101] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic computing power matching and scheduling method based on multi-dimensional labels, characterized in that, Includes the following steps: S1. Task labeling and feature vector generation: Perform multi-dimensional labeling on the input task and generate task type labels. S2. Computation domain mapping and initial routing: Based on the task type label, the task is assigned to the corresponding specialized computation domain for processing through a mapping function; S3. Quantitative assessment of computing power requirements: Based on the data volume and type of the task, a computing power requirement function is constructed to quantify the computing resources required. S4. Monitor domain load and determine its congestion status. Monitor the load of each computing domain in real time and dynamically determine whether it is in a congested state by comparing it with a preset threshold. S5. Priority determination and dynamic routing decision: If the target domain is congested, the task priority is determined; high-priority tasks are queued in this domain, and medium and low-priority tasks trigger rerouting. S6. Idle Domain Selection and Task Migration: For tasks that need to be migrated, select the domain with the lowest load from the set of idle domains, and perform computing resource matching and task handover. S7. Task execution and resource allocation: The task is executed in the final determined computing domain, and the resource manager allocates matching computing resources to complete the calculation.

2. The dynamic computing power matching and scheduling method based on multi-dimensional labels according to claim 1, characterized in that, In step S1, when a new computational task Ti arrives at the system, the scheduler immediately generates a multidimensional label Li for it, and the calculation formula is as follows: Li=(Type,Priority,DataVolume,...) Where Type is the task type; Priority is the task priority, defined as P∈{high(H), medium(M), low(L)}; and DataVolume is the amount of data to be processed by the task.

3. The dynamic computing power matching and scheduling method based on multi-dimensional tags according to claim 1, characterized in that, In step S2, the specialized computing domain is divided into D1AI training domain, D2 database query domain, and D3 Web service domain.

4. The dynamic computing power matching and scheduling method based on multi-dimensional tags according to claim 1, characterized in that, The calculation formula in step S2 is as follows: D target =F map (Typei) Dtarget is the target computation domain assigned to task Ti; F map Tyupei is a mapping function; Tyupei is the type label for task Ti.

5. The dynamic computing power matching and scheduling method based on multi-dimensional tags according to claim 1, characterized in that, The formula for quantifying and evaluating computing power requirements in step S3 is as follows: R need (You)=C Type ×DataVolumes R need (Ti) represents the estimated computing power requirement for task Ti; C Type The computational intensity coefficient is related to the task type; DataVolumei is the amount of data to be processed in task Ti.

6. The dynamic computing power matching and scheduling method based on multi-dimensional tags according to claim 1, characterized in that, The formula for monitoring and determining the congestion status of the domain load in step S4 is as follows: IsCongested(Dj) represents the congestion status of domain Dj, with 1 indicating congestion and 0 indicating normal operation; Lood(Dj) represents the real-time load rate of domain Dj; θc ongestion The preset congestion detection threshold for the system.

7. The dynamic computing power matching and scheduling method based on multi-dimensional labels according to claim 1, characterized in that, The priority determination logic in step S5 is as follows: When the target computing domain is blocked, a priority determination process is initiated. For high-priority tasks, the task continues to wait in the target domain queue; for medium and low-priority tasks, the task is triggered to be rerouted to a set of idle computing domains.

8. The dynamic computing power matching and scheduling method based on multi-dimensional labels according to claim 7, characterized in that, The priority determination formula in step S5 is as follows: A ction Pi represents the basic operations to be performed on the task; Pi represents the priority of task Ti.

9. The dynamic computing power matching and scheduling method based on multi-dimensional labels according to claim 1, characterized in that, The routing decision formula in step S5 is as follows: Route(Ti) is the final route target of task Ti; Dtarget is the initial target domain of the task determined in step S2; Sidle is the set of idle domains of the current system.

10. The dynamic computing power matching and scheduling method based on multi-dimensional labels according to claim 1, characterized in that, The formula for selecting the free domain in step S6 is as follows: D idle The selected, least loaded free area; S idle Let S be the set of all non-blocking domains. idle ={D k |IsCongested(D k )=0};argmin represents the parameter corresponding to the minimum value of the function, which is the domain D in this case. k .

Images