A heterogeneous resource scheduling and management system and method of a multi-modal AI model

Abstract

The application discloses a kind of multi-modal AI model's heterogeneous resource scheduling and management system and method, it is related to resource scheduling and management technical field, the present application includes multi-modal task splitting module, configuration module, heterogeneous resource scheduling module and heterogeneous resource management module;The multi-modal task splitting module is used to obtain several single modal subtasks;The configuration module is based on hierarchical parent rule system and matches the target heterogeneous computing power execution node set of corresponding architecture for each single modal subtask;The heterogeneous resource scheduling module is used to determine the target execution node of each single modal subtask;The heterogeneous resource management module is used to monitor the whole life cycle scheduling state of each single modal subtask.The present application realizes conflict risk analysis to heterogeneous computing power execution node from two dimensions of queue accumulation trend and resource preemption conflict, effectively avoids the risk of heterogeneous computing power node overload and heterogeneous computing power node collapse due to resource depletion in scheduling process.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling and management technology, specifically a heterogeneous resource scheduling and management system and method based on a multimodal AI model. Background Technology

[0002] As mobile AI applications become increasingly complex, evolving from single-model inference to multi-task, multi-model concurrency, issues such as scheduling conflicts between models and preemption of execution resources are beginning to emerge.

[0003] Currently, resource scheduling is achieved by binding a pool of heterogeneous computing power nodes with fixed execution tasks. However, when faced with dynamically changing mixed tasks, this method is prone to situations where high-heterogeneous computing power execution nodes are idle for a long time due to the fixed selection logic of heterogeneous computing power nodes, while low-heterogeneous computing power execution nodes are overloaded and crash. This increases the probability of scheduling conflicts and resource preemption in the heterogeneous resource scheduling system during operation, and fails to achieve real-time response and accuracy in heterogeneous resource scheduling. Summary of the Invention

[0004] The purpose of this invention is to provide a heterogeneous resource scheduling and management system and method for multimodal AI models, so as to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a heterogeneous resource scheduling and management system for multimodal AI models, the system comprising a multimodal task splitting module, a configuration module, a heterogeneous resource scheduling module, and a heterogeneous resource management module; The multimodal task splitting module generates a candidate modality set for the multimodal main task by traversing all metadata fields of the multimodal main task to be processed, and splits the multimodal main task to be processed into several single-modal subtasks according to the candidate modality set. The configuration module matches the target heterogeneous computing power execution node set of the corresponding architecture for each single-modal subtask based on a hierarchical parent rule system; The heterogeneous resource scheduling module selects the heterogeneous computing power execution nodes to be executed based on the fusion scenario tags of each target heterogeneous computing power execution node, and determines the target execution node of each single-modal subtask based on the real-time conflict risk index of each selected heterogeneous computing power execution node. The heterogeneous resource management module is used to monitor the scheduling status of each single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion.

[0006] Furthermore, the multimodal task splitting module includes a modality composition generation unit and a splitting unit; The modality generation unit traverses all metadata fields of the multimodal main task to be processed, identifies all independent raw data types, and generates a candidate modality set for the multimodal main task according to a preset raw data type mapping rule. The candidate modality set is a collection of candidate modality groups composed of data source access source identifiers and modality tags. The raw data type mapping rule refers to the mapping rule between raw data types and modality tags. The full metadata fields refer to the set of structured attributes attached to the multimodal main task to be processed, which do not contain any original business data content, but fully describe its data source, format, structure, transmission protocol, and lifecycle. Specifically, the full metadata fields include data identifier source, file magic number, MIME type, and pre-tagged type. By traversing the full metadata fields of the multimodal main task to be processed, the main task is split. This process does not require parsing the original business content, thus improving the splitting speed of the main task. The splitting unit splits the multimodal main task to be processed into several single-modal subtasks based on the candidate modality composition set of the multimodal main task to be processed, and binds the modality tags and data source access source identifiers stored in the candidate modality composition set to each single-modal subtask.

[0007] Furthermore, the configuration module includes a hierarchical parent rule system construction unit, a heterogeneous computing power execution node matching unit, a dynamic scene tag generation unit, and a heterogeneous computing power execution node filtering unit; The hierarchical parent rule system construction unit matches the modal tags corresponding to each single-modal subtask with the basic heterogeneous computing power pool, and matches the data source access source identifiers corresponding to each single-modal subtask with each execution node in the basic heterogeneous computing power pool. Based on the parent rules of modal tag-computing power pool and the child rules of data source access source identifier-static scene tag mapping-execution node, a tree-like decision chain is constructed to obtain the hierarchical parent rule system. Among them, the basic heterogeneous computing power pool is a logical resource collection that categorizes and aggregates hardware of different architectures such as CPU, GPU, NPU, and FPGA according to their modal processing capabilities. Specifically, NPU clusters specifically adapted for audio processing will be assigned to the Audio computing power pool, and GPU clusters adapted for video encoding and decoding will be assigned to the Video pool. In the computing power pool, the modal label Audio is matched with the Audio computing power pool, and the modal label Video is matched with the Video computing power pool. Both the Video and Audio computing power pools are basic heterogeneous computing power pools, serving as a scheduling front-end layer. By using modal labels, the candidate computing power range that meets the requirements of a specific single-modal subtask is quickly filtered out, thereby narrowing the global search space and improving scheduling efficiency. Execution nodes refer to physical or virtual computing units that are managed by the pooling system, have heterogeneous computing capabilities, and actually carry out workload operations. A hierarchical parent rule system is constructed through modal labels and data source access source identifiers, and confidence is introduced to further narrow the search space range of heterogeneous computing power execution nodes within the hierarchical parent rule system, reducing the scheduling computation complexity. The heterogeneous computing power execution node matching unit maps the data source access source identifier and modality label bound to each single-modal subtask to the hierarchical parent rule system, and matches all candidate heterogeneous computing power execution nodes corresponding to each single-modal subtask. Within the execution time window of each single-modal subtask, the dynamic scene tag generation unit extracts the number of requests processed per second for all candidate heterogeneous computing power execution nodes corresponding to each single-modal subtask, calculates the real-time load adaptation coefficient between each candidate heterogeneous computing power execution node and each single-modal subtask, and uses the real-time load adaptation coefficient as the dynamic scene tag for each candidate heterogeneous computing power node. The heterogeneous computing power execution node screening unit constructs a fused scene label for each candidate heterogeneous computing power execution node based on static scene labels and dynamic scene labels. It then filters each candidate heterogeneous computing power execution node according to the fused scene labels to obtain a target heterogeneous computing power execution node set for each single-modal subtask. The real-time load adaptation coefficient of the selected candidate heterogeneous computing power execution nodes can be greater than 0.8.

[0008] Furthermore, the specific method for generating the static scene tags is as follows: Based on historical single-modal subtasks, obtain all combinations of modality tags and data source access source identifiers. Randomly select a combination, specifically in the format (Audio, audio_material). Periodically retrieve the execution record library of all historical single-modal subtasks with the selected combination. Extract the scene tags bound to each historical single-modal subtask from the execution record library. Classify historical single-modal subtasks with the same scene tags into the same category. Calculate the confidence level of the static scene tags bound to each category of historical single-modal subtasks based on the scheduling success rate and average execution efficiency of the same category of historical single-modal subtasks. The specific formula for calculating the confidence level is as follows:

[0009] in, , All are weighting coefficients and , This represents the ID corresponding to each type of historical single-modal subtask, where n represents the total number of categories corresponding to historical single-modal subtasks. This represents the scheduling success rate of the i-th type of historical single-modal subtask. Specifically, let the scene label corresponding to the i-th type of historical single-modal subtask be the target scene label. The scheduling success rate = the total number of historical single-modal subtasks that successfully completed node scheduling under the target scene label / the total number of historical single-modal subtasks that initiated scheduling requests under the target scene label. If the total number of historical single-modal subtasks that initiated scheduling requests under the target scene label is 0, then the scheduling success rate is directly output as 0.5. This represents the average execution efficiency of the i-th type of historical single-modal subtasks. Average execution efficiency = (Total number of historical single-modal subtasks initiating scheduling requests under the target scene label) / (Sum of execution times of all historical single-modal subtasks under the target scene label). If the sum of execution times is 0, then the average execution efficiency is directly output as 0.5. This represents the maximum execution efficiency of the i-th type of historical single-modal subtask. Denotes the minimum execution efficiency of the i-th type of historical single-modal subtask, if Then let , Represents the confidence level of the i-th type of historical single-modal subtask; Select the scene label with the highest confidence level as the static scene label; By traversing all combinations of modal tags and data source access source identifiers, a static mapping table of data source access source identifiers and static scene tags is obtained. Specifically, the static mapping table is executed asynchronously offline in the background, completely decoupled from the scheduling link of heterogeneous resources, which helps improve the scheduling rate of heterogeneous resources. When the static mapping table is missing, an idle heterogeneous computing power execution node is randomly selected to ensure system availability.

[0010] Furthermore, the specific method for generating the dynamic scene tags is as follows: Among all the single-modal subtasks obtained from the decomposition of the multimodal main task to be processed, single-modal subtasks with selection combination forms are selected, and the execution time window of the selected single-modal subtasks is determined. The execution time window refers to a statistical window of a preset duration that is traced back from the moment the selected single-modal subtask enters the scheduling queue. For example, if the window starts at 12:00 and the preset duration is 60 seconds, then the execution time window is [11:59, 12:00]. Within the execution time window, the number of requests processed per second of all candidate heterogeneous computing power execution nodes corresponding to the selected single-modal subtasks is extracted. The ratio between the extracted number of requests processed per second and the average execution efficiency corresponding to the historical single-modal subtask set with selection combination forms is calculated to obtain the real-time load adaptation coefficient between each candidate heterogeneous computing power execution node and the selected single-modal subtask. The calculated real-time load adaptation coefficient is used as the dynamic scene label of each candidate heterogeneous computing power execution node. Based on the selected combination form, the static scene labels of the selected single-modal subtasks are determined. The determined static scene labels are combined with the dynamic scene labels to obtain the fused scene labels corresponding to each candidate heterogeneous computing power execution node.

[0011] Furthermore, the heterogeneous resource scheduling module includes a heterogeneous computing power execution node selection unit, a conflict analysis unit, and a heterogeneous resource scheduling unit; The heterogeneous computing power execution node selection unit calculates the product between the confidence level of the static scene label and the real-time load adaptation coefficient of the dynamic scene label in the fusion scene label corresponding to each target heterogeneous computing power execution node. The product of the confidence level and the real-time load adaptation coefficient is sorted in descending order of value. The target heterogeneous computing power execution nodes corresponding to the first k values ​​of the selected sequence are all selected as the heterogeneous computing power execution nodes to be executed for the corresponding single-modal subtask. The value of k can be in the range of [3,5], and the specific value range can be adaptively adjusted according to the actual situation. The heterogeneous computing power execution node is selected by the maximum value of the product, which takes into account the stability of historical scheduling experience and the real-time load status of the current heterogeneous computing power execution node, thus improving the selection accuracy. The conflict analysis unit predicts the conflict risk index of each heterogeneous computing power execution node when executing the corresponding single-modality subtask based on the historical scheduling interaction data of each node within the execution time window corresponding to the single-modality subtask. The historical scheduling interaction data refers to the set of running data generated during the interaction between the scheduling system and the heterogeneous computing power execution nodes within the preset execution time window. Specifically, the historical scheduling interaction data includes the real-time load queue length, the number of times the heterogeneous computing power execution node refuses scheduling, and the number of preemption failure events. A preemption failure event refers to an event where the critical hardware resources of the heterogeneous computing power execution node have been fully occupied, and the newly issued task cannot successfully preempt the required resources, directly triggering a scheduling failure or being forced to wait in the queue. The conflict risk index analyzes two dimensions: queue accumulation trend and resource preemption conflicters. It effectively avoids the risk of heterogeneous computing nodes being overloaded during the scheduling process, as well as the risk of heterogeneous computing nodes crashing due to resource exhaustion, thereby improving the scheduling success rate.

[0012] The heterogeneous resource scheduling unit uses the heterogeneous computing power execution node corresponding to the minimum conflict risk index as the target execution node for the corresponding single-modal subtask.

[0013] Furthermore, the specific method for predicting the conflict risk index when execution nodes with heterogeneous computing power execute corresponding single-modal subtasks is as follows: Based on the real-time load queue length in historical scheduling interaction data, the real-time queue backlog conflict risk index for each heterogeneous computing power node to be executed within the execution time window corresponding to the single-modal subtask is calculated. The specific calculation formula is as follows: ; in, This represents the execution node number corresponding to each heterogeneous computing power execution node to be executed for the corresponding single-modal subtask, and t represents the current time value. This represents the real-time load queue length of the heterogeneous computing power execution node j at time t. The real-time load queue length refers to the cumulative number of all single-modal subtasks that the heterogeneous computing power execution node is currently processing but has not yet completed in the task scheduling system. d represents the collection interval of historical scheduling interaction data, and u represents a very small constant used to avoid the denominator being equal to 0. This represents the real-time load queue length of the heterogeneous computing power execution node j at time td. This represents the maximum allowed queue length threshold for the heterogeneous computing power execution node j to be executed. , Both represent weight values ​​and min indicates the minimum value, and max indicates the maximum value. This represents the queue backlog conflict risk index of the heterogeneous computing power execution node j at time t; Based on the number of scheduling rejections and preemption failures of heterogeneous computing power execution nodes in historical scheduling interaction data, the real-time resource contention conflict risk index of each heterogeneous computing power node to be executed within the execution time window corresponding to the single-modal subtask is calculated. The specific calculation formula is as follows: ; in, This represents the total number of times that the heterogeneous computing power execution node j actively returned at time t due to insufficient resources or abnormal status within the execution time window. This represents the alarm threshold for the number of times heterogeneous computing power execution node j refuses scheduling within the execution time window. This represents the total number of preemption failure events that occur on execution node j of the heterogeneous computing power to be executed at time t within the execution time window. This represents the alarm threshold for the failure of heterogeneous computing power execution node j to preempt the execution within the execution time window. , Both represent internal weight values ​​and , This represents the resource contention conflict risk index of the heterogeneous computing power execution node j at time t. Assign weight values ​​to the resource contention conflict risk index and queue backlog conflict risk index of the heterogeneous computing power execution node j at time t, and the sum of the two assigned weight values ​​is 1. Calculate the product between the resource contention conflict risk index or the queue backlog conflict risk index and their respective assigned weight values, and sum the two calculated product values ​​to obtain the conflict risk index of the heterogeneous computing power execution node j at time t.

[0014] Furthermore, the heterogeneous resource management module includes a heterogeneous resource scheduling path determination unit and a heterogeneous resource management unit; The heterogeneous resource scheduling path determination unit generates a heterogeneous resource scheduling path for a corresponding single-modal subtask based on the target execution node of the corresponding single-modal subtask. The heterogeneous resource scheduling path for the corresponding single-modal subtask is as follows: the heterogeneous resource scheduling unit sends the scheduling request of the corresponding single-modal subtask to the target execution node. After receiving the request, the target execution node adds the corresponding single-modal subtask to its local queue and starts execution. After the execution is completed, the result is sent back to the heterogeneous resource scheduling unit. The heterogeneous resource management unit monitors the scheduling status of the corresponding single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion, based on the heterogeneous resource scheduling path of the corresponding single-modal subtask.

[0015] A method for heterogeneous resource scheduling and management of multimodal AI models, the method comprising: S10: Traverse all metadata fields of the multimodal main task to be processed, generate a candidate modality set of the multimodal main task to be processed, and split the multimodal main task to be processed according to the candidate modality set to obtain several single modality subtasks. S20: Based on a hierarchical parent rule system, match the target heterogeneous computing power execution node set corresponding to the architecture for each single-modal subtask; S30: Select the heterogeneous computing power execution node to be executed based on the fusion scenario label of each target heterogeneous computing power execution node, and determine the target execution node of each single-modal subtask based on the real-time conflict risk index of each selected heterogeneous computing power execution node. S40: Monitor the scheduling status of each single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention splits the main task by traversing all metadata fields of the multimodal main task to be processed, without needing to parse the original business content throughout the process, thus achieving real-time response. It decouples the multimodal main task into independent single-modal subtasks, with each single-modal subtask matched with a heterogeneous computing power execution node. This differs from the fixed logic of dynamically changing multi-type mixed tasks being bound to the same heterogeneous computing power execution node, avoiding resource conflicts caused by different modal computation flows flowing to the same node, and eliminating cross-modal task resource conflicts from the root.

[0017] 2. This invention constructs a hierarchical parent rule system through modal tags and data source access source identifiers, and introduces confidence scores to further narrow down the search space of heterogeneous computing power execution nodes within the hierarchical parent rule system. Based on the fusion scenario tags of heterogeneous computing power nodes within the search space, the heterogeneous computing power nodes within the search space are further filtered. Then, historical scheduling interaction data is introduced to achieve conflict risk assessment of each filtered heterogeneous computing power node. Finally, the selection of heterogeneous computing power execution nodes is achieved. Compared with the fixed matching logic of heterogeneous computing power nodes, this process not only ensures the selection efficiency and selection adaptability of heterogeneous computing power execution nodes, but also ensures the smooth execution of single-modal subtasks by the selected heterogeneous computing power execution nodes.

[0018] 3. This invention analyzes the conflict risk of heterogeneous computing power execution nodes from two dimensions: queue accumulation trend and resource preemption conflicters. It effectively avoids the risk of heterogeneous computing power nodes being overloaded during scheduling and the risk of heterogeneous computing power nodes crashing due to resource exhaustion, thereby improving the response accuracy of the heterogeneous resource scheduling system. Attached Figure Description

[0019] Figure 1This is a schematic diagram illustrating the workflow of a heterogeneous resource scheduling and management method for a multimodal AI model according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, this invention provides a technical solution for heterogeneous resource scheduling and management of multimodal AI models, the method comprising: S10: Traverse all metadata fields of the multimodal main task to be processed, generate a candidate modality set of the multimodal main task to be processed, and split the multimodal main task to be processed according to the candidate modality set to obtain several single modality subtasks. S20: Based on a hierarchical parent rule system, match the target heterogeneous computing power execution node set corresponding to the architecture for each single-modal subtask; S30: Select the heterogeneous computing power execution node to be executed based on the fusion scenario label of each target heterogeneous computing power execution node, and determine the target execution node of each single-modal subtask based on the real-time conflict risk index of each selected heterogeneous computing power execution node. S40: Monitor the scheduling status of each single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion.

[0022] A heterogeneous resource scheduling and management system for a multimodal AI model, the system including a multimodal task decomposition module, a configuration module, a heterogeneous resource scheduling module and a heterogeneous resource management module; The multimodal task splitting module generates a candidate modality set for the multimodal main task by traversing all metadata fields of the multimodal main task to be processed, and splits the multimodal main task to be processed into several single-modal subtasks based on the candidate modality set. The multimodal task decomposition module includes a modality generation unit and a decomposition unit; The modality composition generation unit traverses all metadata fields of the multimodal main task to be processed, identifies all independent raw data types, and generates a candidate modality composition set of the multimodal main task to be processed according to the preset raw data type mapping rules. The raw data type mapping rules refer to the mapping rules between raw data types and modality tags. The full metadata fields refer to the set of structured attributes attached to the multimodal main task to be processed that do not contain any business raw data content, but fully describe its data source, format, structure, transmission protocol and life cycle. The splitting unit splits the multimodal main task to be processed into several single-modal subtasks based on the candidate modality composition set of the multimodal main task to be processed, and binds the modality tags and data source access source identifiers stored in the candidate modality composition set to each single-modal subtask respectively. The configuration module matches the target heterogeneous computing power execution node set of the corresponding architecture for each single-modal subtask based on a hierarchical parent rule system; The configuration module includes a hierarchical parent rule system construction unit, a heterogeneous computing power execution node matching unit, a dynamic scene tag generation unit, and a heterogeneous computing power execution node filtering unit; The hierarchical parent rule system construction unit matches the modal tags corresponding to each single-modal subtask with the basic heterogeneous computing power pool, and matches the data source access source identifiers corresponding to each single-modal subtask with each execution node in the basic heterogeneous computing power pool. Based on the parent rules of modal tag-computing power pool, and the child rules of data source access source identifier-static scene tag mapping-execution node, a tree-like decision chain is formed to obtain the hierarchical parent rule system. Among them, the basic heterogeneous computing power pool is a logical resource collection that classifies and aggregates hardware of different architectures such as CPU, GPU, NPU, and FPGA according to their modal processing capabilities. Specifically, NPs specifically adapted for audio processing The U cluster will be assigned to the Audio computing power pool, and the GPU cluster adapted for video encoding and decoding will be assigned to the Video computing power pool. At this time, the modal label Audio is matched with the Audio computing power pool, and the modal label Video is matched with the Video computing power pool. Both the Video computing power pool and the Audio computing power pool are basic heterogeneous computing power pools. As a scheduling front-end layer, the candidate computing power range that meets the requirements of a specific single-modal subtask is quickly filtered out through modal labels, thereby narrowing the global search space and improving scheduling efficiency. The execution node refers to the physical or virtual computing unit that is managed by the pooling system, has heterogeneous computing capabilities, and actually carries the workload. The specific method for generating static scene tags is as follows: Based on historical single-modal subtasks, obtain all combinations of modality tags and data source access source identifiers. Randomly select a combination, specifically in the format (Audio, audio_material). Periodically retrieve the execution record library of all historical single-modal subtasks with the selected combination. Extract the scene tags bound to each historical single-modal subtask from the execution record library. Scene tags are directly mapped from the data source access source identifier. For example, for the data source access source identifier "audio_material", its corresponding scene tag is "uploaded from an audio material library"; for the data source access source identifier "video_short", its corresponding scene tag is "uploaded from a short video platform". Classify historical single-modal subtasks with the same scene tag into the same category. Calculate the confidence level of the static scene tags bound to each category of historical single-modal subtasks based on the scheduling success rate and average execution efficiency of the same category of historical single-modal subtasks. The specific formula for calculating the confidence level is: ; in, , All are weighting coefficients and , This represents the ID corresponding to each type of historical single-modal subtask, where n represents the total number of categories corresponding to historical single-modal subtasks. This represents the scheduling success rate of the i-th type of historical single-modal subtask. Specifically, let the scene label corresponding to the i-th type of historical single-modal subtask be the target scene label. The scheduling success rate = the total number of historical single-modal subtasks that successfully completed node scheduling under the target scene label / the total number of historical single-modal subtasks that initiated scheduling requests under the target scene label. If the total number of historical single-modal subtasks that initiated scheduling requests under the target scene label is 0, then the scheduling success rate is directly output as 0.5. This represents the average execution efficiency of the i-th type of historical single-modal subtasks. Average execution efficiency = (Total number of historical single-modal subtasks initiating scheduling requests under the target scene label) / (Sum of execution times of all historical single-modal subtasks under the target scene label). If the sum of execution times is 0, then the average execution efficiency is directly output as 0.5. This represents the maximum execution efficiency of the i-th type of historical single-modal subtask. Denotes the minimum execution efficiency of the i-th type of historical single-modal subtask, if Then let , Represents the confidence level of the i-th type of historical single-modal subtask; Select the scene label with the highest confidence level as the static scene label; Traverse all combinations of modal tags and data source access source identifiers to obtain a static mapping table of data source access source identifiers and static scene tags. Specifically, the static mapping table is a background offline asynchronous execution link that is completely decoupled from the scheduling link of heterogeneous resources, which is conducive to improving the scheduling rate of heterogeneous resources. The heterogeneous computing power execution node matching unit maps the data source access source identifier and modality label bound to each single-modal subtask to the hierarchical parent rule system, and matches all candidate heterogeneous computing power execution nodes corresponding to each single-modal subtask; Within the execution time window of each single-modal subtask, the dynamic scene tag generation unit extracts the number of requests processed per second for all candidate heterogeneous computing power execution nodes corresponding to each single-modal subtask, calculates the real-time load adaptation coefficient between each candidate heterogeneous computing power execution node and each single-modal subtask, and uses the real-time load adaptation coefficient as the dynamic scene tag for each candidate heterogeneous computing power node. The specific method for generating dynamic scene tags is as follows: Among all the single-modal subtasks obtained from the decomposition of the multimodal main task to be processed, single-modal subtasks with selection and combination forms are selected. The execution time window of the selected single-modal subtasks is determined. The execution time window refers to a statistical window of a preset duration that starts from the moment the selected single-modal subtask enters the scheduling queue. For example, if the window starts at 12:00 and the preset duration is 60 seconds, then the execution time window is [11:59, 12:00]. Within the execution time window, all candidate heterogeneous computing power execution nodes corresponding to the selected single-modal subtasks are extracted. The number of requests processed per second is calculated, and the ratio between the extracted number of requests processed per second and the average execution efficiency corresponding to the historical single-modal subtask set with selected combination forms is calculated to obtain the real-time load adaptation coefficient between each candidate heterogeneous computing power execution node and the selected single-modal subtask. The calculated real-time load adaptation coefficient is used as the dynamic scene label of each candidate heterogeneous computing power execution node. The number of requests processed per second refers to the total number of requests / tasks that the heterogeneous computing power execution node can successfully process in a unit time of 1 second. Based on the selected combination form, the static scene labels of the selected single-modal subtasks are determined. The determined static scene labels are combined with the dynamic scene labels to obtain the fused scene labels corresponding to each candidate heterogeneous computing power execution node. The heterogeneous computing power execution node screening unit constructs a fused scene label for each candidate heterogeneous computing power execution node based on static scene labels and dynamic scene labels. It then filters each candidate heterogeneous computing power execution node according to the fused scene labels to obtain a target heterogeneous computing power execution node set for each single-modal subtask. The real-time load adaptation coefficient of the selected candidate heterogeneous computing power execution nodes can be greater than 0.8. The heterogeneous resource scheduling module selects the heterogeneous computing power execution nodes to be executed based on the fusion scenario tags of each target heterogeneous computing power execution node, and determines the target execution node for each single-modal subtask based on the real-time conflict risk index of each selected heterogeneous computing power execution node. The heterogeneous resource scheduling module includes a selection unit for heterogeneous computing power execution nodes, a conflict analysis unit, and a heterogeneous resource scheduling unit. The unit for selecting heterogeneous computing power execution nodes calculates the product between the confidence level of the static scene label and the real-time load adaptation coefficient of the dynamic scene label in the fusion scene label corresponding to each target heterogeneous computing power execution node. The product between the confidence level and the real-time load adaptation coefficient is sorted in descending order of value. The target heterogeneous computing power execution nodes corresponding to the first k values ​​of the selected sequence are all used as the heterogeneous computing power execution nodes to be executed for the corresponding single-modal subtask. The value range of k can be [3,5], and the specific value range can be adaptively adjusted according to the actual situation. The conflict analysis unit predicts the conflict risk index of each heterogeneous computing power execution node when executing the corresponding single-modality subtask based on the historical scheduling interaction data of each node within the execution time window corresponding to the single-modality subtask. The specific method is as follows: Based on the real-time load queue length in historical scheduling interaction data, the real-time queue backlog conflict risk index for each heterogeneous computing power node to be executed within the execution time window corresponding to the single-modal subtask is calculated. The specific calculation formula is as follows: ; in, This represents the execution node number corresponding to each heterogeneous computing power execution node to be executed for the corresponding single-modal subtask, and t represents the current time value. represents the real-time load queue length of the heterogeneous computing power execution node j at time t. The real-time load queue length refers to the cumulative number of all single-modal subtasks that the heterogeneous computing power execution node is currently processing but has not yet completed in the task scheduling system. d represents the collection interval of historical scheduling interaction data, and u represents a minimal constant used to avoid the denominator being equal to 0. Specifically, the value of u is less than... 0.1%, This represents the real-time load queue length of the heterogeneous computing power execution node j at time td. When the value of is 0, then The value of and The values ​​are consistent. This represents the maximum allowed queue length threshold for the heterogeneous computing power execution node j to be executed. , Both represent weight values ​​and min indicates the minimum value, and max indicates the maximum value. This represents the queue backlog conflict risk index of execution node j in heterogeneous computing power at time t. This represents an exponential function with base e and e = 2.72; Based on the number of scheduling rejections and preemption failures of heterogeneous computing power execution nodes in historical scheduling interaction data, the real-time resource contention conflict risk index of each heterogeneous computing power node to be executed within the execution time window corresponding to the single-modal subtask is calculated. The specific calculation formula is as follows: ; in, This represents the total number of times that the heterogeneous computing power execution node j actively returned at time t due to insufficient resources or abnormal status within the execution time window. This represents the alarm threshold for the number of times heterogeneous computing power execution node j refuses scheduling within the execution time window. This represents the total number of preemption failure events that occur on execution node j of the heterogeneous computing power to be executed at time t within the execution time window. This represents the alarm threshold for the failure of heterogeneous computing power execution node j to preempt the execution within the execution time window. , Both represent internal weight values ​​and , This represents the resource contention conflict risk index of execution node j in the heterogeneous computing power at time t. and All are greater than 0. This represents the static resource utilization rate of the heterogeneous computing power execution node j at time t. This represents the dynamic resource utilization rate of the heterogeneous computing power execution node j at time t; Assign weight values ​​to the resource contention conflict risk index and queue backlog conflict risk index of the heterogeneous computing power execution node j at time t. Calculate the product between the resource contention conflict risk index or the queue backlog conflict risk index and their respective assigned weight values. Sum the two calculated products to obtain the conflict risk index of the heterogeneous computing power execution node j at time t. Historical scheduling interaction data refers to the set of runtime data generated during the interaction between the scheduling system and the heterogeneous computing power execution nodes within a preset execution time window. Specifically, historical scheduling interaction data includes the real-time load queue length, the number of times the heterogeneous computing power execution nodes refused scheduling, and the number of preemption failure events. Preemption failure events refer to events where the critical hardware resources of the heterogeneous computing power execution nodes have been fully occupied, and newly issued tasks cannot successfully preempt the required resources, directly triggering scheduling failures or being forced to wait in the queue. The heterogeneous resource scheduling unit will use the heterogeneous computing power execution node corresponding to the minimum conflict risk index as the target execution node for the corresponding single-modal subtask. The heterogeneous resource management module is used to monitor the scheduling status of each single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion. The heterogeneous resource management module includes a heterogeneous resource scheduling path determination unit and a heterogeneous resource management unit; The heterogeneous resource scheduling path determination unit generates a heterogeneous resource scheduling path for a corresponding single-modal subtask based on the target execution node of the corresponding single-modal subtask. The heterogeneous resource scheduling path for the corresponding single-modal subtask is as follows: the heterogeneous resource scheduling unit sends the scheduling request of the corresponding single-modal subtask to the target execution node. After receiving the request, the target execution node adds the corresponding single-modal subtask to its local queue and starts execution. After the execution is completed, the result is sent back to the heterogeneous resource scheduling unit. The heterogeneous resource management unit monitors the scheduling status of the corresponding single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion, based on the heterogeneous resource scheduling path of the corresponding single-modal subtask.

[0023] Example 1: In the audio and video multimodal content classification task, the data source access identifier, file magic number, MIME type and pre-labeled type are extracted from the audio and video data. The set of data source identifier, file magic number, MIME type and pre-labeled type is the full metadata field. The file magic number is a fixed sequence of bytes located at the beginning / specific offset of a file, used to uniquely identify the file format. For example, for a WAV waveform audio file, the first 4 bytes of RIFF are the general identifier of the RIFF container to which the WAV belongs, followed by 4 bytes of the total file length field. Skipping these 8 bytes, at offset 8 in the file, there is exactly 4 bytes of WAVE-specific identifier. Therefore, the file magic number of WAV is: the first 4 bytes of RIFF and the 4 bytes of WAVE at offset 8. The MIME type specifies how a file with a certain extension should be opened by an application. When a file with that extension is accessed, the browser will automatically use the specified application to open it. For example, the MIME type for WAV files includes audio / wa v; When uploading audio and video data to the gateway, only the feature information of the transmission protocol header and the source access identifier of the data source are used to concatenate the two types of external metadata to generate the corresponding string form of the original data type label. This label is a pre-tagged type. This process does not parse any original audio and video business content. For example, when a WAV file is uploaded from the audio material library to the gateway, the source access identifier is mapped to the audio_material semantic label. The MIME type of the WAV file in the transmission protocol header is audio / wav. The audio category identifier is extracted, and the pre-tagged type of the WAV file is generated according to the format of source semantic label + transmission protocol header MIME feature + current date as: audio_material_audio_20260623; Randomly select any two of the three categories from the full metadata fields—file magic number, MIME type, and pre-tagged type—for cross-validation. If both results point to the audio / video category, then the original data type corresponding to the audio / video multimodal content classification task is determined to include audio / video. Traverse all metadata fields to identify all independent raw data types for the audio and video multimodal content classification task; Query the preset original data type mapping table to uniquely point the identified audio data to the modal label "Audio"; and uniquely point the identified video data to the modal label "Video". Then, the candidate modal composition set for the audio and video multimodal content classification task is: [{data source access source identifier: audio_material, modal label: Audio}, {data source access source identifier: video_short, modal label: Video}]. Among them, when MP4 video is uploaded from the short video platform to the access gateway, the data source access identifier is mapped to video_short. Ultimately, the audio and video multimodal content classification task was broken down into an audio content classification subtask and a video content classification subtask.

[0024] Example 2: Let the data source access source identifier bound to the audio content classification subtask be: audio_material, and the modal tag be: Audio, then: The data source access source identifier is designated as "audio_material" and the modal label is designated as "Audio". These are mapped to the hierarchical parent rule system. The parent rule identifies the modal label "Audio" and selects a set of all execution nodes with audio processing capabilities from the global resource pool to obtain the audio basic heterogeneous computing power pool. The child rule identifies the data source access source identifier "audio_material" and obtains the mapped static scene label. Execution nodes with this static scene label are selected from the audio basic heterogeneous computing power pool. All selected execution nodes are used as candidate heterogeneous computing power execution nodes corresponding to the audio content classification subtask.

[0025] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A heterogeneous resource scheduling and management system for a multimodal AI model, characterized in that: The system includes a multimodal task splitting module, a configuration module, a heterogeneous resource scheduling module, and a heterogeneous resource management module; The multimodal task splitting module generates a candidate modality set for the multimodal main task by traversing all metadata fields of the multimodal main task to be processed, and splits the multimodal main task to be processed into several single-modal subtasks according to the candidate modality set. The configuration module matches the target heterogeneous computing power execution node set of the corresponding architecture for each single-modal subtask based on a hierarchical parent rule system; The heterogeneous resource scheduling module selects the heterogeneous computing power execution nodes to be executed based on the fusion scenario tags of each target heterogeneous computing power execution node, and determines the target execution node of each single-modal subtask based on the real-time conflict risk index of each selected heterogeneous computing power execution node. The heterogeneous resource management module is used to monitor the scheduling status of each single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion.

2. The heterogeneous resource scheduling and management system for a multimodal AI model according to claim 1, characterized in that: The multimodal task splitting module includes a modality composition generation unit and a splitting unit; The modality composition generation unit traverses all metadata fields of the multimodal main task to be processed, identifies all independent raw data types, and generates a candidate modality composition set for the multimodal main task to be processed according to the preset raw data type mapping rules. The candidate modality composition set is a set of candidate modality groups composed of data source access source identifiers and modality tags. The raw data type mapping rules refer to the mapping rules between raw data types and modality tags. The all metadata fields refer to the set of structured attributes attached to the multimodal main task to be processed that do not contain any business raw data content, but fully describe its data source, format, structure, transmission protocol and life cycle. The splitting unit splits the multimodal main task to be processed into several single-modal subtasks based on the candidate modality composition set of the multimodal main task to be processed, and binds the modality tags and data source access source identifiers stored in the candidate modality composition set to each single-modal subtask.

3. The heterogeneous resource scheduling and management system for a multimodal AI model according to claim 2, characterized in that: The configuration module includes a hierarchical parent rule system construction unit, a heterogeneous computing power execution node matching unit, a dynamic scene tag generation unit, and a heterogeneous computing power execution node filtering unit; The hierarchical parent rule system construction unit matches the modal tags corresponding to each single modal subtask with the basic heterogeneous computing power pool, and matches the data source access source identifiers corresponding to each single modal subtask with each execution node in the basic heterogeneous computing power pool. Based on the parent rules of modal tag-computing power pool and the child rules of data source access source identifier-static scene tag mapping-execution node, a tree-like decision chain is formed to obtain the hierarchical parent rule system. The heterogeneous computing power execution node matching unit maps the data source access source identifier and modality label bound to each single-modal subtask to the hierarchical parent rule system, and matches all candidate heterogeneous computing power execution nodes corresponding to each single-modal subtask. Within the execution time window of each single-modal subtask, the dynamic scene tag generation unit extracts the number of requests processed per second for all candidate heterogeneous computing power execution nodes corresponding to each single-modal subtask, calculates the real-time load adaptation coefficient between each candidate heterogeneous computing power execution node and each single-modal subtask, and uses the real-time load adaptation coefficient as the dynamic scene tag for each candidate heterogeneous computing power node. The heterogeneous computing power execution node screening unit constructs a fused scene label for each candidate heterogeneous computing power execution node based on static scene labels and dynamic scene labels. It then filters each candidate heterogeneous computing power execution node according to the fused scene labels to obtain a target heterogeneous computing power execution node set for each single-modal subtask.

4. The heterogeneous resource scheduling and management system for a multimodal AI model according to claim 3, characterized in that: The specific method for generating the static scene tags is as follows: Based on historical single-modal subtasks, obtain all combinations of modality tags and data source access source identifiers, randomly select a combination, periodically obtain the execution record library of all historical single-modal subtasks with the selected combination, extract the scene tags bound to each historical single-modal subtask in the execution record library, classify historical single-modal subtasks with the same scene tags into the same category, and calculate the confidence level of static scene tags bound to each type of historical single-modal subtask based on the scheduling success rate and average execution efficiency of the same type of historical single-modal subtasks. Select the scene label with the highest confidence level as the static scene label; Traverse all combinations of modal tags and data source access source identifiers to obtain a static mapping table of data source access source identifiers and static scene tags.

5. A heterogeneous resource scheduling and management system for a multimodal AI model according to claim 4, characterized in that: The specific method for generating the dynamic scene tags is as follows: Among all the single-modal subtasks obtained from the decomposition of the multimodal main task to be processed, single-modal subtasks with selection combination forms are selected, and the execution time window of the selected single-modal subtasks is determined. The execution time window refers to a statistical window of a preset duration that is traced back from the moment the selected single-modal subtask enters the scheduling queue. Within the execution time window, the number of requests processed per second of all candidate heterogeneous computing power execution nodes corresponding to the selected single-modal subtasks is extracted. The ratio between the extracted number of requests processed per second and the average execution efficiency corresponding to the historical single-modal subtask set with selection combination forms is calculated to obtain the real-time load adaptation coefficient between each candidate heterogeneous computing power execution node and the selected single-modal subtask. The calculated real-time load adaptation coefficient is used as the dynamic scene label of each candidate heterogeneous computing power execution node. Based on the selected combination form, the static scene labels of the selected single-modal subtasks are determined. The determined static scene labels are combined with the dynamic scene labels to obtain the fused scene labels corresponding to each candidate heterogeneous computing power execution node.

6. The heterogeneous resource scheduling and management system for a multimodal AI model according to claim 5, characterized in that: The heterogeneous resource scheduling module includes a heterogeneous computing power execution node selection unit, a conflict analysis unit, and a heterogeneous resource scheduling unit. The unit for selecting heterogeneous computing power execution nodes to be executed calculates the product between the confidence level of the static scene label and the real-time load adaptation coefficient of the dynamic scene label in the fusion scene label corresponding to each target heterogeneous computing power execution node. The unit sorts the product between the confidence level and the real-time load adaptation coefficient in descending order of the values ​​and selects the target heterogeneous computing power execution nodes corresponding to the first k values ​​of the sequence as the heterogeneous computing power execution nodes to be executed for the corresponding single-modal subtask. The conflict analysis unit predicts the conflict risk index of each heterogeneous computing power execution node when executing the corresponding single-modal subtask based on the historical scheduling interaction data of each heterogeneous computing power execution node within the execution time window corresponding to the corresponding single-modal subtask. The historical scheduling interaction data refers to the set of running data generated during the interaction between the scheduling system and the heterogeneous computing power execution node within the preset execution time window. The heterogeneous resource scheduling unit uses the heterogeneous computing power execution node corresponding to the minimum conflict risk index as the target execution node for the corresponding single-modal subtask.

7. A heterogeneous resource scheduling and management system for a multimodal AI model according to claim 6, characterized in that: The specific method for predicting the conflict risk index when execution nodes of heterogeneous computing power execute corresponding single-modal subtasks is as follows: Based on the real-time load queue length in the historical scheduling interaction data, calculate the real-time load level and real-time relative growth rate of each heterogeneous computing power node to be executed, and assign weight values ​​to the real-time load level and real-time relative growth rate to determine the real-time queue backlog conflict risk index of each heterogeneous computing power node to be executed within the execution time window corresponding to the corresponding single-modal subtask. Based on the number of times the heterogeneous computing power execution nodes refused scheduling and the number of preemption failure events in the historical scheduling interaction data, the real-time static resource utilization rate and real-time dynamic resource utilization rate of each heterogeneous computing power node to be executed are calculated. Weight values ​​are assigned to the real-time static resource utilization rate and the real-time dynamic resource utilization rate respectively to determine the real-time resource competition conflict risk index of each heterogeneous computing power node to be executed within the execution time window corresponding to the single-modal subtask. Assign weight values ​​to the resource contention conflict risk index and queue backlog conflict risk index of the heterogeneous computing power execution node j at time t. Calculate the product between the resource contention conflict risk index or the queue backlog conflict risk index and their respective assigned weight values. Sum the two calculated products to obtain the conflict risk index of the heterogeneous computing power execution node j at time t.

8. A heterogeneous resource scheduling and management system for a multimodal AI model according to claim 7, characterized in that: The heterogeneous resource management module includes a heterogeneous resource scheduling path determination unit and a heterogeneous resource management unit; The heterogeneous resource scheduling path determination unit generates a heterogeneous resource scheduling path for a corresponding single-modal subtask based on the target execution node of the corresponding single-modal subtask. The heterogeneous resource scheduling path for the corresponding single-modal subtask is as follows: the heterogeneous resource scheduling unit sends the scheduling request of the corresponding single-modal subtask to the target execution node. After receiving the request, the target execution node adds the corresponding single-modal subtask to its local queue and starts execution. After the execution is completed, the result is sent back to the heterogeneous resource scheduling unit. The heterogeneous resource management unit monitors the scheduling status of the corresponding single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion, based on the heterogeneous resource scheduling path of the corresponding single-modal subtask.

9. A method for heterogeneous resource scheduling and management of a multimodal AI model applied to a heterogeneous resource scheduling and management system for a multimodal AI model according to any one of claims 1-8, characterized in that: The method includes: S10: Traverse all metadata fields of the multimodal main task to be processed, generate a candidate modality set of the multimodal main task to be processed, and split the multimodal main task to be processed according to the candidate modality set to obtain several single modality subtasks. S20: Based on a hierarchical parent rule system, match the target heterogeneous computing power execution node set corresponding to the architecture for each single-modal subtask; S30: Select the heterogeneous computing power execution node to be executed based on the fusion scenario label of each target heterogeneous computing power execution node, and determine the target execution node of each single-modal subtask based on the real-time conflict risk index of each selected heterogeneous computing power execution node. S40: Monitor the scheduling status of each single-modal subtask throughout its entire lifecycle, from issuance and queuing to execution completion.

Images