Dynamic scheduling method and system of heterogeneous computing resources of multi-modal end-to-side model

Abstract

The application discloses a heterogeneous computing resource dynamic scheduling method and system of a multi-modal end-side model, relates to the technical field of heterogeneous computing resource management, and comprises the following steps: obtaining an operator sequence output by a multi-modal end-side model, and identifying data exchange points among the operator sequence based on execution characteristics; identifying execution pause positions based on the data exchange points, obtaining an execution pause set, interval dividing the operator sequence, extracting an intermediate execution state of each interval connection in a preset visual task after the interval dividing, and obtaining a visual task structure diagram; traversing the visual task structure diagram, identifying a cut-in interval position, obtaining a cut-in sequence, embedding an execution section of a preset language task, obtaining an interleaved execution sequence, and coordinately controlling the tasks, so that the problem that the execution section of the language task cannot be timely inserted into the visual task due to the continuity and non-interruptibility of GPU operator execution, resulting in insufficient utilization of heterogeneous computing resources and task coordination lag is solved.

Description

Technical Field

[0001] This invention relates to the field of heterogeneous computing resource management technology, and more specifically, to a method and system for dynamic scheduling of heterogeneous computing resources in a multimodal edge model. Background Technology

[0002] In existing technologies, with the rapid development of edge-side multimodal AI applications, visual and language tasks often need to be executed in parallel on the same device. These tasks include image recognition, object detection, scene understanding, and various functions such as natural language processing, speech recognition, and text generation, requiring the simultaneous processing of high-dimensional feature data and complex model calculations. However, edge devices are typically constrained by limited computing resources, limited storage bandwidth, and a complex variety of heterogeneous computing units. This makes it easy to encounter computing bottlenecks, memory congestion, and data transmission delays when running multiple computationally intensive tasks simultaneously. Furthermore, different types of processing units (such as CPUs, GPUs, NPUs, or FPGAs) differ in computing power, storage access patterns, and energy efficiency, increasing the complexity of task scheduling and resource management.

[0003] However, in existing edge-side multimodal model scheduling, GPU tasks are typically scheduled in "batches" or "layers" as the smallest scheduling unit, with execution times generally between 10 and 50 milliseconds. This means that operators in each batch or layer must be completed continuously during execution, unlike traditional real-time tasks which can be flexibly segmented with interrupt granularity. This continuity leads to a situation where, when vision and language tasks need to be interleaved, the language task cannot be inserted in time during the execution of the current GPU operator and must wait for the entire batch or layer to complete before starting, resulting in significant time gaps and execution delays. In multimodal edge devices, this scheduling mode not only results in low GPU resource utilization but also leads to insufficient coordination between vision and language tasks, increasing overall task response latency and impacting real-time performance and user experience. To address these issues, this invention proposes a solution. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a method and system for dynamic scheduling of heterogeneous computing resources for multimodal edge models. By identifying possible execution pauses between visual task operators and dynamically scheduling and interleaving visual and language tasks, the method addresses the problem that when visual and language tasks need to be interleaved, the execution segment of the language task cannot be inserted in time due to the continuity and uninterruptibility of GPU operator execution, resulting in insufficient utilization of heterogeneous computing resources and delayed coordination between tasks.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for dynamic scheduling of heterogeneous computing resources for a multimodal edge model includes the following steps: acquiring the operator sequence output by the multimodal edge model, and identifying data exchange points between operator sequences based on their execution characteristics; identifying execution pause locations based on the data exchange points between operator sequences to obtain an execution pause set; dividing the operator sequence into intervals based on the execution pause set, and extracting the intermediate execution states of each interval connection point in a preset visual task to obtain a visual task structure diagram; traversing the visual task structure diagram and identifying cut-in interval locations to obtain a cut-in sequence; embedding the execution segment of a preset language task into the cut-in sequence to obtain an interleaved execution sequence, and coordinating and controlling the visual task and the language task based on the interleaved execution sequence.

[0007] In a preferred embodiment, the step of acquiring the operator sequence output by the multimodal endpoint model and identifying the data exchange points between operator sequences based on the execution characteristics of the operator sequences specifically involves: acquiring the execution pipeline of the multimodal endpoint model, extracting the operator sequence and constructing an operator execution trajectory matrix; identifying data exchange feature points in the operator execution trajectory matrix, and performing topological analysis on the data exchange feature points using a graph convolutional network to obtain data exchange regions; and expanding the data points using a breadth-first search algorithm based on the data exchange regions to obtain data exchange points.

[0008] In a preferred embodiment, the step of expanding data points based on the data exchange area using a breadth-first search algorithm to obtain data exchange points specifically involves: constructing an operator graph based on the execution pipeline of the multimodal end-side model; determining a preset central node from the data exchange area as the expansion starting point, and using breadth-first search to expand the expansion starting point hop by hop until a preset number of hops is reached or a preset resource constraint condition is triggered, thereby obtaining the data exchange point.

[0009] In a preferred embodiment, the step of identifying execution pause locations based on data exchange points between operator sequences to obtain an execution pause set specifically involves: obtaining the operator execution context corresponding to the data exchange point and analyzing resource waiting events in the operator execution context; comparing the duration of the resource waiting events with a preset time window to obtain an execution delay interval; mapping the execution delay interval to the operator sequence to obtain the execution pause intervals before and after the data exchange point; and merging overlapping execution pause intervals to obtain an execution pause set.

[0010] In a preferred embodiment, the step of segmenting the operator sequence based on the execution pause set and extracting the intermediate execution states of each segmented interval connection in the preset visual task to obtain a visual task structure diagram specifically involves: segmenting the operator sequence into several operator intervals based on the execution pause set; extracting the original operator execution sequence table based on the execution pipeline of the multimodal edge model; for each operator interval, performing the forward computation of each operator sequentially according to the original operator execution sequence table; extracting the intermediate execution states of each operator interval connection in the preset visual task based on the computation results to obtain an intermediate feature map; analyzing the semantic content of the intermediate feature map, extracting visual task features, and constructing a visual task structure diagram based on the visual task features.

[0011] In a preferred embodiment, the step of traversing the visual task structure graph and identifying the locations of intrusive intervals to obtain an intrusive sequence specifically involves: traversing the visual task structure graph using a depth-first traversal algorithm and constructing a data flow graph; calculating the data input / output degree of each node in the data flow graph, where the data input / output degree includes the number of direct predecessor nodes and the number of direct successor nodes; based on the data input / output degree, selecting nodes whose number of direct predecessor nodes and direct successor nodes are both lower than a preset first threshold to obtain a candidate node set; extracting the data transmission path of each candidate node in the data flow graph and calculating the data packet size and transmission delay in the data transmission path; performing concurrent execution evaluation on the data transmission path based on the data packet size and transmission delay, and identifying intrusive intervals based on the evaluation results; and sorting the intrusive intervals according to a preset topological order of the data flow graph to obtain an intrusive sequence.

[0012] In a preferred embodiment, the step of embedding the execution segment of a preset language task into the cuttable sequence to obtain an interleaved execution sequence specifically involves: extracting the interval subgraph corresponding to the cuttable sequence in the data flow graph, and generating several interval execution paths in the interval subgraph using a preset random walk algorithm; obtaining resource access records for several interval execution paths, and calculating the resource access overlap frequency between interval execution paths based on the resource access records; obtaining candidate embedding positions based on the resource access overlap frequency, and embedding the execution segment of the preset language task based on the candidate embedding positions to obtain an interleaved execution sequence.

[0013] In a preferred embodiment, the step of obtaining candidate embedding positions based on resource access overlap frequency and embedding execution segments of a preset language task based on the candidate embedding positions to obtain an interleaved execution sequence specifically involves: identifying resource contention regions in the cut-in sequence based on resource access overlap frequency to obtain a set of prohibited embedding intervals; removing the prohibited embedding intervals from the cut-in sequence and rearranging the remaining sequence according to the original order of the cut-in sequence to obtain candidate embedding positions; segmenting the candidate embedding positions using a preset path discretization method and dividing the preset language task atomic units into execution segments matching the size of the candidate embedding positions; and inserting execution segments into the segmented embedding positions to obtain an interleaved execution sequence.

[0014] In a preferred embodiment, the coordinated control of visual and language tasks based on the interleaved execution sequence specifically involves: parsing the visual and language task execution segments in the interleaved execution sequence and analyzing their dependencies; establishing task segment execution priorities based on the dependencies and allocating heterogeneous computing resources to the visual and language task execution segments based on these priorities; monitoring execution progress in real time, detecting resource allocation conflicts, and dynamically adjusting the execution order based on resource allocation conflicts.

[0015] The technical effects and advantages of the heterogeneous computing resource dynamic scheduling method and system of the multimodal edge model of this invention are as follows:

[0016] This invention acquires the operator sequence output by a multimodal model and identifies data exchange points based on the execution characteristics of the operator sequence, effectively analyzing the dependencies and potential execution pauses between operators. Furthermore, it segments the operator sequence into intervals based on the execution pause set and extracts the intermediate execution states of each interval connection in the visual task, thereby constructing a complete visual task structure diagram. By traversing the visual task structure diagram, it identifies intrusive intervals to obtain intrusive sequences, providing precise locations for language task insertion. It embeds the execution segments of a preset language task into the intrusive sequences, generating interleaved execution sequences, and coordinates and controls the visual and language tasks based on these interleaved execution sequences, achieving reasonable resource allocation and optimized execution order between tasks. Through the above steps, the problem of language tasks not being able to be inserted in a timely manner due to the fact that GPU tasks are scheduled in "batch" or "layer" as the smallest scheduling unit and cannot be interrupted in traditional edge scheduling is solved. This effectively improves the utilization of heterogeneous computing resources, reduces scheduling lag between tasks, realizes efficient interleaved execution of vision and language tasks, reduces latency, improves the throughput and real-time performance of edge multimodal tasks, and significantly enhances the execution efficiency and responsiveness of edge multimodal artificial intelligence applications. Attached Figure Description

[0017] Figure 1This is a flowchart illustrating the heterogeneous computing resource dynamic scheduling method of the multimodal edge model of the present invention.

[0018] Figure 2 This is a schematic diagram of the heterogeneous computing resource dynamic scheduling system of the multimodal end-side model of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1, Figure 1 The present invention provides a method for dynamic scheduling of heterogeneous computing resources in a multimodal edge-side model, comprising the following steps:

[0021] S1, obtain the operator sequence output by the multimodal end-side model, and identify the data exchange points between operator sequences based on the execution characteristics of the operator sequences;

[0022] In this example, the operator sequence output by the multimodal end-side model is obtained, and the data exchange points between the operator sequences are identified based on the execution characteristics of the operator sequences. Specifically:

[0023] Obtain the execution pipeline of the multimodal edge model, extract the operator sequence, and construct the operator execution trajectory matrix;

[0024] Identify data exchange feature points in the operator execution trajectory matrix, and use graph convolutional networks to perform topological analysis on the data exchange feature points to obtain the data exchange region;

[0025] Based on the data exchange area, a breadth-first search algorithm is used to expand the data points and obtain the data exchange points.

[0026] It's important to note that in real-world scenarios, multimodal edge models internally record the execution process of an inference cycle from input to output. Each processing step is represented as an operator, such as convolution, feature concatenation, matrix multiplication, and attention modules. To obtain the execution pipeline, lightweight execution recording hooks can be added to each computational node of the model. These hooks record information such as timestamps, input and output tensor sizes, operator type, and accelerator type (e.g., NPU or DSP) when an operator starts and ends execution. For example, a convolution operator might record "started at 8.3ms, ended at 8.9ms, output features are 32×32×64". After the entire inference process is completed, all the recorded operators are concatenated in the order they occurred to obtain the execution pipeline, which describes the actual order in which the model called operators during an inference cycle. An example of the execution pipeline's content is: Conv→ReLU→Conv→Add→Attention→Linear→Softmax. This complete sequence is the operator sequence. To further analyze the behavioral characteristics between operators, the execution status of operators needs to be structured. Therefore, the attributes of each operator during execution (such as start time, end time, running device, memory usage, input / output size, etc.) are arranged by row, and these attributes are arranged by column, thus constructing the operator execution trajectory matrix. For example, the 5th row of the matrix corresponds to the 5th operator (such as Attention), and the columns are filled with "start time 15.2ms, execution duration 2.8ms, running device is NPU, input size 768×64, output size 768×64, peak memory usage 8MB".

[0027] Furthermore, in the operator execution trajectory matrix, the execution results of some operators are transferred from one computing unit to another, such as switching from NPU to CPU or from GPU to NPU. These locations are usually accompanied by significant data movement behaviors, such as increased memory bandwidth, increased execution pauses, and sudden changes in output feature size. Therefore, identifying data exchange feature points in the matrix can be done by observing whether there is a switching of running devices, a sudden increase in feature size, or an abnormally long execution interval between rows (of a certain operator) and the next row (of the next operator). For example, the operator corresponding to row 12 of the matrix is ​​executed on the NPU with an output size of 64×64×128, while the operator in row 13 runs on the CPU, and there is a 3ms gap between rows 12 and 13, which is significantly longer than the average connection interval of 0.2ms in the pipeline. This can be marked as a data exchange feature point. To identify larger data exchange regions from these feature points, the connections between operators are constructed into a directed graph. Using the feature points as initial nodes, a graph convolutional network (GCN) performs topological analysis on the local neighborhood structure of each node. For example, GCN aggregates more obvious patterns based on the attributes of surrounding nodes (input size, device type, latency), thereby identifying a set of adjacent operators exhibiting the same exchange behavior. This set constitutes the data exchange region, indicating that not just one operator is involved in data movement, but rather multiple operators collectively form a cross-device data processing segment. For instance, the 3–6 operators before and after Attention need to travel between different accelerators, and this area is identified as a data exchange region.

[0028] Secondly, once a data exchange area is identified, it's necessary to determine which specific locations within it best represent the actual data migration nodes. This is the process of expanding the data exchange points. First, the execution pipeline of the multimodal edge model is mapped to an operator graph, where each node represents an operator, and edges represent execution dependencies between operators. For example, a dependency edge between Conv and ReLU indicates that ReLU depends on the output of Conv. Then, a central node is selected within the previously identified data exchange area. For instance, the 15th operator in the area, if it exhibits the highest latency or the most significant device switching characteristic, can serve as the starting point. A breadth-first search is performed starting from this node, visiting the directly related operators before and after it from the first hop, such as the 14th and 16th operators. If they also exhibit certain data migration characteristics (e.g., different devices or higher latency), the search continues to expand to the second hop, for example, visiting the 13th and 17th operators. The expansion process continues until a preset number of hops is reached (e.g., two hops) or resource conditions are detected as unsuitable for further expansion (e.g., when hopping to the 18th operator, the running device and feature size are no longer relevant to the region, then expansion stops). The final set of marked nodes is the data exchange point. For example, if the final BFS expansion results in nodes 14, 15, 16, and 17, all of which exhibit clear processing unit switching characteristics, then these four locations are identified as the actual data exchange points throughout the inference process.

[0029] Finally, it should be noted that the preset resource constraints refer to a set of restrictions set in advance when performing breadth-first expansion of the data exchange area to avoid excessive expansion or entering resource-scarce regions. These constraints limit the range and depth of the expansion path, ensuring that the final data exchange point is representative without falling into regions with excessive computational resource pressure that cannot accommodate other tasks. Specifically, these resource constraints typically include three categories: computational load constraints, GPU memory or intermediate feature cache occupancy constraints, and execution latency constraints. For example, in the execution pipeline of an edge device, if the GPU load of a certain operator link remains above 85% for a long time, it indicates that the region is approaching a performance bottleneck. At this time, when performing breadth-first search, once these nodes with high loads are explored, resource constraints will be triggered, and the expansion process will be stopped, thus avoiding misidentifying these nodes as regions that can be parallelized or cut into. Another example is that some operators generate large intermediate feature maps in video memory after execution. For example, a 224×224×64 feature map may occupy tens of megabytes of space. If BFS finds that continuing to expand will cause video memory usage to exceed the pre-set limit (such as exceeding 70% of available video memory), resource constraints will also be triggered and the expansion will be terminated early. In addition, there may be latency conditions. For example, if the BFS expands to a certain node and the cumulative execution time of the path before that node has exceeded 20 milliseconds, while the maximum allowed waiting time for the language task to insert is only 15 milliseconds, then BFS will also stop at this node to avoid expanding to deep paths that are meaningless to the insertion task. By setting these constraints, we can ensure that the data exchange points obtained by BFS extension are close to the sensitive locations of task switching in actual execution, but do not enter resource-overloaded areas, thus providing a reliable basis for subsequent identification of cut-in intervals and insertion of language task execution segments.

[0030] In this example, based on the data exchange area, a breadth-first search algorithm is used to expand the data points and obtain the data exchange points, specifically:

[0031] Constructing operator graphs for execution pipelines based on multimodal end-side models;

[0032] The data exchange point is obtained by determining a preset central node in the data exchange area as the starting point for expansion, and then using breadth-first search to expand the starting point hop by hop until a preset number of hops is reached or a preset resource constraint condition is triggered.

[0033] It should be noted that in practical applications, the execution pipeline already provides the actual execution order, dependencies, and operating device information of all operators during a single inference process. To transform this information into an operator graph suitable for structural analysis, each operator needs to be treated as a node in the graph according to the data flow dependencies between operators. Operators that are adjacent in the execution order or have feature transitive relationships are connected by directed edges. For example, if Relu1 is executed immediately after Conv1 in the execution pipeline, and the input of Relu1 comes from Conv1, then an edge is created from Conv1 to Relu1. If a branching structure exists subsequently, such as the output of Conv2 being used by both MaxPool and Add, then Conv2 needs to be connected to both operators respectively, forming a multi-branching structure. Furthermore, when constructing the operator graph, some key attributes of the operators are included as additional information for the nodes. These include the execution time (e.g., 3.8ms to 4.2ms), input and output feature sizes (e.g., 32×32×64), and the type of computational unit used (e.g., Conv1 executes on the NPU while the Attention module executes on the CPU in a given inference). These attributes are used in subsequent algorithms to determine the location of data exchange. The resulting operator graph is no longer a simple sequential list, but a directed graph structure that reflects the actual execution dependencies between operators, data flow paths, and feature map transmission directions. For example, it can be a chain structure from Conv→ReLU→Conv→Add→Attention→Linear→Softmax, or it can contain complex topologies with multiple branches and multiple inputs.

[0034] Furthermore, after identifying a data exchange area, the most representative operator within that area needs to be selected as the starting point for expansion. This node typically exhibits significant cross-device switching characteristics or sudden latency increases. For example, if the 22nd operator in the area executes on the CPU, while the adjacent operator runs on the NPU, and there is a 2.5ms gap between them, which is more than ten times the pipeline average, then the 22nd operator is suitable as the central node. A breadth-first search is then performed starting from this central node. The first hop visits the directly connected nodes before and after it, such as the 21st and 23rd operators. If the 21st operator runs on the NPU and its output tensor size suddenly increases, or the 23rd operator executes on the CPU with increased latency, it indicates that these adjacent nodes also exhibit data migration behavior. They should be added to the expansion set, and the second hop should continue, for example, to the 20th and 24th operators. When expanding to a certain hop, if the node's attributes no longer meet the resource switching characteristics—for example, the 24th operator is still executing on the CPU but the execution interval returns to normal, and the input / output size does not change significantly—it indicates that the expansion has reached the data exchange boundary, and the expansion can be stopped. Alternatively, when the expansion reaches a preset number of hops, such as a maximum of two hops, BFS will also terminate. The series of nodes ultimately captured by BFS—such as the 21st, 22nd, 23rd, and in some cases the 20th or 24th operator—are data exchange points. They represent the specific locations where memory relocation, cross-device data transfer, or waiting events occur during actual execution. For example, in edge devices, the back-and-forth data switching between the NPU and CPU is often the main cause of performance degradation. These data exchange points mark the precise location of bottlenecks, allowing subsequent pause identification, interval partitioning, and task interleaving scheduling to accurately pinpoint which operators to insert language task segments between, thereby improving overall execution efficiency.

[0035] S2, based on the data exchange points between operator sequences, identify the execution pause locations and obtain the execution pause set;

[0036] In this example, execution pause locations are identified based on data exchange points between operator sequences, resulting in an execution pause set, specifically:

[0037] Obtain the operator execution context corresponding to the data exchange point, and analyze the resource wait events in the operator execution context;

[0038] The execution delay interval is obtained by comparing the duration of the resource waiting event with the preset time window;

[0039] By mapping the execution delay interval to the operator sequence, the execution pause interval before and after the data exchange point is obtained;

[0040] Merge overlapping execution pause intervals to obtain the execution pause set.

[0041] It should be noted that if operators 18, 19, and 20 in the operator sequence are identified as critical locations where cross-device data migration may occur, the execution context of these operators needs to be retrieved from the execution log or runtime monitoring module. This includes the operator's start and end times, the size of the input and output tensors, the execution device type, peak memory usage, whether data migration occurred, the execution idle time before and after the operator, and the resource usage of parallel tasks. For example, when operator 19 is triggered by the NPU, its context records "started at 25.4ms, ended at 26.1ms, device is NPU, input feature size is 128×64, and there was a 1.8ms wait before the call." A resource wait event refers to an operator that should have immediately obtained computing or memory resources before planned execution, but due to resources being occupied by other tasks, data still being migrated, or waiting for the previous operator to complete transmission on another device, the operator must wait for a certain period of time. This waiting time does not generate effective computation and only prolongs the overall execution time. For example, the 18th operator waits for the NPU to complete a DMA transfer, and the 20th operator waits for the CPU to release the cache area.

[0042] After acquiring resource wait events, it's necessary to determine whether these waits constitute a performance bottleneck. This requires comparing the duration of the resource wait events with a preset time window, such as a 1ms window, to distinguish between natural scheduling jitter and actual performance latency. Suppose the 18th operator waits for 1.4ms, the 19th operator waits for 1.8ms, while the 20th operator waits for only 0.3ms. The wait times of the first two operators significantly exceed the preset time window, thus being considered execution latency and marked as an execution latency interval. An execution latency interval refers to the actual execution delay caused by insufficient resources, device switching, or data transmission. It's not the operator's computation time, but rather the period during which the operator is forced to pause in the queue. This period is crucial in task scheduling because it represents an underutilized accelerator. If language tasks or other subtasks are to be inserted later, these latency intervals become usable time windows, preventing interference with the main execution flow of the vision task.

[0043] Furthermore, after identifying the execution delay intervals, these delay periods need to be marked in the operator sequence to pinpoint where the visual task naturally generates gaps in time. For example, the delay interval for the 18th operator extends from 24.0ms to 25.4ms in the inference timeline, while the delay interval for the 19th operator extends from 25.4ms to 27.2ms. By mapping these time periods back to the operator's index position in the sequence, we obtain the execution pause interval corresponding to that operator. An execution pause interval is a pause caused by data exchange or resource contention. It is usually located before or after the data exchange point, so the vicinity of the data exchange point can often map multiple pause intervals. For example, in this case, the pause intervals for the 18th and 19th operators are almost consecutive, meaning that during this execution segment, the device experienced a continuous gap in time due to cross-device data transfer.

[0044] Finally, after mapping, multiple execution pause intervals are often obtained, many of which may overlap or be adjacent. Therefore, they need to be integrated to simplify the subsequent scheduling logic. For example, the pause interval for the 18th operator is 24.0ms-25.4ms, the pause interval for the 19th operator is 25.4ms-27.2ms, and although the 20th operator does not have a significant delay, there is a 0.2ms instantaneous gap before its execution, forming a 27.2ms-27.4ms interval. These three intervals are continuous on the timeline, so they can be merged into a larger execution pause interval of 24.0ms-27.4ms. After this processing, the final set of execution pauses is a set of non-overlapping pause time periods with clear boundaries. Each pause segment represents a continuous interval during model execution where resource waiting or data movement occurs.

[0045] S3, based on the execution pause set, the operator sequence is divided into intervals, and the connection point of each interval after division is extracted to the intermediate execution state of the preset visual task, so as to obtain the visual task structure diagram.

[0046] In this example, the operator sequence is segmented into intervals based on the execution pause set, and the connection points of each segmented interval are extracted to represent the intermediate execution states of the preset visual task, resulting in a visual task structure diagram, as follows:

[0047] Based on the execution pause set, the operator sequence is divided into several operator intervals;

[0048] Based on the multimodal end-side model, the execution pipeline extracts the original operator execution sequence table;

[0049] For each operator interval, the forward computation of each operator is executed sequentially according to the original operator execution order list;

[0050] Based on the calculation results, the intermediate execution state of each operator interval connection point in the preset visual task is extracted to obtain the intermediate feature map;

[0051] Analyze the semantic content of the intermediate feature map, extract visual task features, and construct a visual task structure map based on the visual task features.

[0052] It should be noted that the execution pause set obtained in the previous step, such as the inference timeline 24.0ms-27.4ms, 41.2ms-42.0ms, and 55.5ms-57.8ms, were identified as three consecutive pause intervals. These pause intervals, when corresponding to the operator sequence, will fall between certain operators, such as between operators 17 and 18, 26 and 27, and 34 and 35. To facilitate subsequent task insertion and structural analysis, these pause intervals can be used as dividing points to segment the complete operator sequence into multiple sub-intervals based on the location of the pauses. For example, if the original operator sequence contains 50 operators, and the pause points fall near operators 17, 26, and 34, then it can be divided into four segments: operators 1-17 as the first segment, operators 18-26 as the second segment, operators 27-34 as the third segment, and operators 35-50 as the fourth segment. This segmentation method uses pause boundaries to automatically divide continuously executed operators into natural computational stages, making the execution within each operator continuous, while there are obvious execution gaps between segments. This also means that subsequent language tasks can prioritize inserting near these boundaries.

[0053] Secondly, the execution pipeline of the multimodal edge model records the actual execution order of all operators in a single inference, such as Conv1→Relu1→Conv2→Add→Attention1→Linear1→Softmax. These records typically come from the execution hooks or the model framework's runtime logs. To ensure that the segmented operator intervals can be re-executed according to the model's original semantics, these original orders need to be extracted to form a complete operator execution order table. This order table is actually a list structure, arranging the name or identifier of each operator from the first convolutional layer to the last output layer in a strict forward dependency order. It reflects the complete topological call order of the model structure, rather than the temporal misalignment caused by scheduling during execution. For example, Conv3 must precede Add in the order table because it provides the input for Add, and Add must precede Attention because Attention reads the result of Add. The significance of the order table is that it ensures that when performing separate forward inference on operator intervals later, the model's true computational dependencies are still followed, and the forward flow is not disrupted by the segmentation of the intervals.

[0054] Furthermore, after obtaining each operator interval, forward inference needs to be performed on the operators within the interval to generate the interval output state. However, this must be done sequentially according to the original operator execution order list to ensure consistent computational dependencies. For example, the second interval contains operators 18-26, and these operators are arranged in the original order list as "Conv5→BN5→Relu5→Attention2→Linear3→Add→Relu6→Conv6". Therefore, when executing the interval, the operators must be executed one by one in this order, even if the actual execution time of some operators is not continuous. The process of performing forward computation can be illustrated as follows: the input feature map size is 64×64×128, which becomes 64×64×256 after Conv5, then normalized by BN5, then activated by Relu5, then Attention2 will reallocate the channel weights, Linear3 will perform tensor mapping, and Add will add the results of the two branches, and so on until the last operator in the interval.

[0055] Furthermore, after completing the forward inference of a certain operator interval, the end of the interval usually corresponds to a natural boundary in the execution pipeline. Therefore, the internal state of the model at this boundary is the intermediate execution state of that interval. To extract this state, the output tensor of the last operator in the interval can be recorded. For example, in intervals 18-26, the output of Conv6 at the end is a feature tensor with a shape of 32×32×128, which represents the model's abstract representation of the visual input (such as an image) at that position. Saving these tensors at the end of the intervals forms the intermediate feature map of that interval. The intermediate feature map is the semantic expression of the input content by the model at a specific computation stage. For example, in the earlier interval feature maps, the model tends to represent low-level visual information such as edges and textures, while in the interval feature maps closer to Attention, it begins to present high-level features such as semantically sensitive regions and object structural relationships. The significance of the intermediate feature map is that it is an important basis for determining which stages in the model execution process are suitable for inserting language tasks, because the semantic information richness varies at different stages, and the selection of insertion points needs to ensure that it does not disrupt the logic of the visual task.

[0056] Finally, to construct a visual task structure graph, semantic analysis needs to be performed on each intermediate feature map. This can be achieved through features visualization, channel response analysis, or clustering methods to determine whether the feature map focuses on edges, textures, object regions, or global relationships. For example, in the feature map of interval 1, the response is mainly concentrated in high-frequency texture regions, such as clothing textures and object edges; in the feature map of interval 2, the activation after the Attention layer is significantly enhanced in the target subject region, indicating that the model focuses more on structural information at this stage; and in the feature map of interval 3, the overall channel response tends to be sparse, representing the model entering the task discrimination stage. Based on these semantic attributes, interval nodes can be abstracted into categories such as "texture feature nodes," "structural feature nodes," and "semantic discrimination nodes," and a visual task structure graph can be constructed based on the feature dependencies between intervals (e.g., structural features depend on texture features, and high-level semantics depend on structural features). This structure graph represents the semantic flow path during the execution of the visual task, reflecting how the model gradually moves from low-level information to high-level semantics, which helps in subsequent identification of inaccessible intervals for task interleaving.

[0057] S4, Traverse the visual task structure map and identify the cut-in interval positions to obtain the cut-in sequence;

[0058] In this example, the visual task structure graph is traversed and the positions of the intrusive intervals are identified to obtain the intrusive sequence, specifically:

[0059] A depth-first traversal algorithm is used to traverse the visual task structure graph and construct a data flow graph;

[0060] Calculate the data input / output degree of each node in the data flow graph, where the data input / output degree includes the number of direct predecessor nodes and the number of direct successor nodes;

[0061] Based on the data input-output degree, nodes whose number of direct predecessor nodes and direct successor nodes are both lower than a preset first threshold are selected to obtain a candidate node set.

[0062] Extract the data transmission path of each candidate node in the data flow graph, and calculate the data packet size and transmission delay in the data transmission path;

[0063] The concurrent execution evaluation of the data transmission path is performed based on the data packet size and transmission delay, and the cut-in interval is identified based on the evaluation results;

[0064] The cut-in intervals are sorted according to the preset topological order of the data flow graph to obtain the cut-in sequence.

[0065] In this example, the data packet size and transmission delay in the data transmission path are calculated as follows:

[0066] The formula for calculating the size of data packets in the data transmission path is as follows:

[0067]

[0068] The formula for calculating the transmission delay in the data transmission path is as follows:

[0069]

[0070] in, This represents the total size of the data packets transmitted along data transmission path p. Let n be the amount of data output by node n in the data transmission path p. Let p be the set of nodes along the data transmission path. The transmission delay of data transmission path p, The computation time for node n itself. Let p be the set of directed edges on the data transmission path. This represents the transmission delay required for the output from node n to node m.

[0071] It's important to note that the visual task structure graph consists of multiple nodes representing visual semantic stages, such as "texture extraction nodes," "region attention nodes," "object feature nodes," and "global relationship nodes." Edges between nodes indicate the direction of semantic information flow. To more accurately depict the data dependencies between these nodes, a depth-first traversal is performed on the structure graph starting from the input direction. For example, starting from the texture extraction node, the graph sequentially visits the structural nodes it leads to, then continues into semantic nodes, until all reachable paths have been traversed. During this traversal, each flow of information between nodes is recorded. For instance, if information flows from node A to node B, a directed edge from A to B is added to the data flow graph. Finally, all access paths are organized to obtain a more clearly structured data flow graph. The significance of this data flow graph lies in the fact that it not only preserves the semantic hierarchy of the visual task structure graph but also accurately reflects the actual data transfer paths between intermediate feature maps. It transforms the complex semantic structure into an analytical model that facilitates the assessment of dependency strength, data scale variations, and execution feasibility, providing a foundation for subsequent identification of potential entry points.

[0072] After obtaining the data flow graph, it is necessary to determine the position and structural characteristics of each node in the data flow. This can be achieved by calculating the input-output degree of the node, that is, recording how many direct predecessor nodes and how many direct successor nodes the node has. For example, a texture feature node may have only one predecessor node (from a specific convolutional interval) and one successor node (entering the structural feature node), with both its input and output degrees being 1; while an object feature node may receive input from both region interest nodes and edge enhancement nodes, thus having 2 predecessor nodes; and a global relation node may output results to multiple discriminant nodes, thus having 3 successor nodes. By traversing each directed edge in the data flow graph, the number of predecessors and successors of each node can be counted and used as the data input-output degree of that node. These degrees reflect the structural position of the node: nodes with low input-output degrees are usually located in positions with weaker dependencies in the data flow and have greater insertion flexibility; nodes with high input-output degrees are often on the critical path and are more difficult to interrupt or cut into.

[0073] Secondly, to identify suitable locations for insertion into language task execution segments, nodes are filtered based on their input-output degrees. For example, a threshold of 2 means that only nodes with both the number of predecessor and successor nodes less than 2 are selected as candidate nodes. This is because such nodes have fewer path dependencies, reducing the risk of modification or pause. For instance, a texture feature node with a predecessor degree of 1 and a successor degree of 1 passes the screening; while an object integration node with a predecessor degree of 3 indicates numerous dependencies, and intervention might disrupt upstream and downstream semantics, thus it is not considered a candidate node. Assuming there are 10 nodes in the structure graph, after screening, four nodes may remain that meet the criteria, such as nodes 2, 4, 7, and 9. These nodes form a candidate node set, which is a crucial target for subsequent path analysis and concurrent execution evaluation.

[0074] Furthermore, for each candidate node, its data transmission path in the data flow graph needs to be analyzed. For example, starting from node 4, tracing backwards along the graph to nodes 5 and 6, and then to node 8, until the data flow no longer extends downwards, constitutes a data transmission path for node 4. Within each path, the size of the data packet can be estimated by combining the tensor sizes recorded in the model execution pipeline. For example, the feature map of node 4 might be 32×32×128, the feature map of node 5 expands to 64×64×128, and the feature map of node 6 shrinks to 16×16×256. These feature map sizes represent the size of the data packets transmitted along the path. Simultaneously, by referencing the timestamps in the execution pipeline, the actual latency between nodes along the path can be estimated. For example, the interval from node 4 output to node 5 input is 0.7ms, from node 5 to node 6 is 0.4ms, and from node 6 to node 8 is 1.1ms. These intervals constitute the transmission latency.

[0075] Furthermore, packet size and transmission latency are key indicators for measuring the feasibility of concurrent execution: larger packet sizes and longer transmission latency indicate that the visual task requires higher computational resources and memory bandwidth near that node, making it unsuitable for inserting language tasks. Conversely, if a node's path contains small packets and short latency, it indicates low dependency strength, and inserting a language task will not cause significant interference. For example, the packet sizes of node 4 are all within a medium range, with a maximum of only 32×32×128, and the longest path latency is only 1.1ms, so it can be assessed as feasible for concurrent execution. However, node 7 has a packet size of 224×224×64, and a transmission latency exceeding 1.8ms, so the assessment result is that it is not suitable for insertion. Through such evaluation, nodes with a high probability of concurrent execution can be selected from the candidate node set, and the path locations of these nodes can be marked as insertion intervals.

[0076] Finally, after determining the cut-in intervals, they need to be sorted according to the topological order of the data flow graph to ensure a clear sequential relationship for subsequent language task insertions. The topological order follows the natural flow from upstream to downstream, such as from texture feature nodes to structural feature nodes and then to semantic nodes. If the cut-in intervals correspond to nodes 2, 4, and 9 respectively, and the topological positional relationship in the graph is: node 2 in the earliest stage, node 4 in the middle stage, and node 9 in the later stage, then the sorting result is [node 2 → node 4 → node 9]. This sorting forms a cut-in sequence, allowing the language task to follow the natural semantic progression of the visual task during execution, preventing order confusion, and also facilitating segment-by-segment processing by subsequent insertion algorithms.

[0077] S5 embeds the execution segment of the preset language task into the cut-in sequence to obtain the interleaved execution sequence, and coordinates and controls the visual task and the language task based on the interleaved execution sequence.

[0078] In this example, execution segments of a predefined language task are embedded within the intrusive sequence to obtain an interleaved execution sequence, specifically:

[0079] Extract the cut-in sequence into the interval subgraph corresponding to the data flow graph, and use a preset random walk algorithm to generate several interval execution paths in the interval subgraph;

[0080] Obtain resource access records for several interval execution paths, and calculate the frequency of resource access overlap between interval execution paths based on the resource access records;

[0081] Candidate embedding positions are obtained based on the frequency of resource access overlap, and the execution segments of the preset language task are embedded based on the candidate embedding positions to obtain an interleaved execution sequence.

[0082] It's important to note that once the cut-in sequence is obtained, the specific topological intervals containing these cut-in nodes need to be identified on the data flow graph to further analyze their execution structure. Therefore, corresponding continuous subgraphs are extracted from the data flow graph based on the upstream and downstream dependencies of the cut-in nodes. For example, if the cut-in sequence is nodes 2, 4, and 9, then the entire connected portion from node 2 through nodes 3, 4, and 6, to either node 8 or 9, is extracted from the data flow graph, forming an interval subgraph to describe the actual topological structure of the cut-in region during visual task execution. Next, using a pre-defined random walk algorithm, starting from these cut-in nodes, successor nodes are randomly selected within the interval subgraph to continue moving forward. This process is repeated within a certain step limit, generating multiple different interval execution paths. For example, starting from node 2, the path might be "2→3→4→6→8" or "2→4→6→9". Each path represents a possible execution route within the interval. The significance of interval execution paths lies in their ability to reveal the diversity of execution within the intrusive region. This is because visual tasks may exhibit different execution orders in actual operation due to factors such as data dependencies, module parallelism, and scheduling strategies. By collecting multiple paths obtained from random walks, we can more comprehensively estimate the resource conflicts and execution impacts that may result from inserting language tasks into the interval.

[0083] Furthermore, after generating multiple interval execution paths, it is necessary to analyze the operator execution records involved in each path to extract the hardware resource access information during these executions, such as memory bandwidth usage, cache read / write counts, NPU core scheduling frequency, and DMA channel usage. For example, path A (2→3→4→6→8) may trigger three high-bandwidth accesses at node 4, while path B (2→4→6→9) may experience two acceleration core contentions at node 6. All of these are recorded as resource access records. Subsequently, the resource accesses of different paths are compared. For example, it is compared whether resource accesses at node 4 all occur in the same time segment, and whether consecutive read / write conflicts occur at node 6 in different paths. Through these comparisons, the frequency of resource access overlap between two paths can be calculated. If a resource is accessed frequently and simultaneously in both paths, such as having high-frequency cache read / write events in multiple paths, its overlap frequency is high; if a resource occurs in different paths with a large time difference, its overlap frequency is low. The higher the frequency of resource access overlap, the more easily the execution of that interval is interfered with by the insertion of language tasks, and the embedding position needs to be carefully selected.

[0084] Finally, by calculating the frequency of resource access overlap between execution paths in different intervals, we can identify node positions with low overlap frequencies. For example, the resource access of node 3 in paths A, B, and C has almost no overlap, and node 8 also shows as a low-conflict point in most paths. These nodes are selected as candidate embedding positions because inserting the execution segment of the language task at these positions is least likely to cause interference. Subsequently, suitable nodes are selected from these candidate positions according to the interval topology order. For example, the "text precoding segment" of the language task is inserted after node 3, and the "feature fusion segment" is inserted before node 8. These inserted segments are then integrated according to the original execution order of the visual task to form the final interleaved execution sequence. For example, the original visual execution path is "2→3→4→6→8→9", which may become "2→3→language segment A→4→6→language segment B→8→9" after embedding. The final interleaved execution sequence is the specific arrangement in which the visual and language tasks can be executed concurrently on the edge, which can maximize the balance between performance and resource consumption, reduce execution interference, and improve overall task efficiency.

[0085] In this example, candidate embedding positions are obtained based on the frequency of resource access overlap, and the execution segments of the preset language task are embedded based on the candidate embedding positions to obtain an interleaved execution sequence, specifically:

[0086] Based on the frequency of overlapping resource accesses, resource contention regions that can be entered into the sequence are identified, and a set of prohibited embedding intervals is obtained.

[0087] Remove the forbidden embedding interval set from the cutable sequence, and rearrange the remaining sequence according to the original order of the cutable sequence to obtain the candidate embedding positions;

[0088] A preset path discretization method is used to segment the candidate embedding positions, and the preset language task atomic units are divided into execution segments that match the size of the candidate embedding positions.

[0089] The execution segment is inserted at the embedding position after segmentation to obtain an interleaved execution sequence.

[0090] It's important to note that after obtaining the resource access overlap frequency for each node or interval in the cutaway sequence, further analysis of the distribution of these frequencies across the time axis and execution structure is needed to identify which nodes belong to resource contention regions. A resource contention region refers to a region where multiple operators frequently access the same type of hardware resource simultaneously in different execution paths. For example, multiple nodes may exhibit high-intensity memory bandwidth usage or acceleration core contention in different paths. This means that inserting a language task near these nodes could potentially cause additional queuing, blocking, and execution jitter. For instance, in the cutaway sequence of nodes 3, 5, and 8, node 5 exhibits three consecutive high-frequency cache read / write operations on multiple execution paths, while nodes 3 and 8 show lighter access on most paths. Therefore, the interval containing node 5 is marked as a resource contention region. After identifying the resource contention region, it is used as a prohibited embedding interval set. This means that all nodes or regions with intense resource contention cannot be selected when embedding language tasks in the future, to avoid conflicts between the vision task and the language task regarding critical resources.

[0091] Secondly, after obtaining the set of prohibited embedding intervals, these regions need to be completely removed from the original cutaway sequence. For example, if the cutaway sequence is [3,5,7,8], and the prohibited embedding interval set obtained through resource contention analysis is [5,7], then the remaining nodes after removal are [3,8]. Next, based on the original execution order of the cutaway sequence in the visual task, these remaining nodes need to be reordered according to the original topological order to ensure that the candidate embedding positions conform to the execution logic of the visual task itself. For example, [3,8] obtained after removal originally passed through node 3 first and then node 8 during execution, so it can be used as a candidate embedding position without adjustment. If there are many remaining regions, for example, if the original sequence is [3,5,8,12,15], and after removal we get [3,8,15], rearranging them still maintains this order, thus finally obtaining the candidate position list for the language insertion task.

[0092] Furthermore, to ensure the reasonable embedding of language task segments into candidate positions, these positions need to be discretized. This involves further dividing a candidate node or interval into multiple independently usable insertion segments. For example, the execution interval corresponding to candidate node 8 might include the main operator execution stage and the post-processing stage, which have different durations. Therefore, the interval can be divided into two segments using path discretization, corresponding to "8 - front segment" and "8 - back segment" respectively. Simultaneously, the pre-defined language task often contains multiple atomic units, such as the three minimum execution modules: "text precoding," "tag embedding," and "attention calculation." These atomic units need to be divided or merged into execution segments matching the available execution window size of the candidate interval segments. For instance, "precoding + embedding" can be merged into a larger segment corresponding to node 3, while "attention calculation" can be separated into a smaller segment corresponding to node 8. This ensures that the embedding of the language task on the edge does not exceed resource capacity or disrupt the execution rhythm of the visual task.

[0093] Finally, after matching the candidate position segments with the language task execution segments, the next step is to truly embed these language task execution segments into the execution sequence of the visual task, forming the final interleaved execution sequence. For example, the original execution order of the visual task is "2→3→4→6→8→9", while the candidate embedding positions, after discretization, include "3-after segment" and "8-before segment". The language task execution segments are divided into "precoding segments" and "attention segments". Finally, the "precoding segment" is inserted after "3-after segment", and the "attention segment" is inserted before "8-before segment", so the resulting interleaved execution sequence may become "2→3→language segment A→4→6→8→language segment B→9". This interleaved arrangement allows the language operator to be executed during the natural gaps in the visual operator, thereby minimizing resource conflicts, improving the overall throughput efficiency on the edge, and maintaining the timeliness of multimodal task linkage.

[0094] In this example, visual and language tasks are coordinated and controlled based on interleaved execution sequences, specifically as follows:

[0095] Analyze the visual task execution segments and language task execution segments in the interleaved execution sequence, and the dependencies between the visual task execution segments and language task execution segments;

[0096] The execution priority of task segments is established based on the dependency relationship, and heterogeneous computing resources are allocated to the visual task execution segment and the language task execution segment based on the execution priority of task segments;

[0097] Monitor execution progress in real time, detect resource allocation conflicts, and dynamically adjust the execution order based on resource allocation conflicts.

[0098] It's important to note that after the interleaved execution sequence is generated, the visual task execution segments and language task execution segments need to be parsed. The purpose is to identify the sequence order, input-output dependencies, and resource usage relationships between these two types of task segments. For example, a typical interleaved sequence might be "V2 →V3 →LA →V4 →V6 →LB →V8 →V9", where "V3" represents the third segment of the visual task and "LA" represents the pre-coding segment of the language task. During parsing, the data input sources required by each task segment are checked segment by segment. For example, does "LA" require visual feature maps, and does "V4" depend on the text embeddings generated by "LA"? The resource dependencies of the task segments are also analyzed. For instance, some visual segments need to complete memory preparation in advance, while some language segments need to wait for the NPU cores to be idle before performing attention operations. For example, the parsing result might show a dependency chain like this: V3 → LA (because the language segment needs intermediate visual features), LA → V4 (because the visual segment needs the text features output by the language segment), V6 → LB (because the attention module of the language segment needs the output of V6 as input). Analysis can determine how the two tasks should be connected, ensuring that the language segment does not start prematurely without visual input, and that the visual segment does not stagnate meaninglessly while waiting for the language processing results.

[0099] Furthermore, after clarifying the dependencies between each task segment, it is necessary to establish execution priorities for all task segments to ensure that execution strictly follows the dependency chain where sequential order is required, and maximizes hardware utilization where parallelism is allowed. For example, based on the parsed dependency chain, if "V3→LA→V4", then V3 must have the highest priority, followed by LA, and finally V4. If there is no dependency between V6 and LA, they can be placed at the same priority level and run in parallel if resources permit. Then, hardware resources are allocated to the task segments according to priority. For example, the high-priority V3 is allocated a GPU acceleration core, while a portion of NPU cores and cache bandwidth are reserved for LA to ensure that LA can start immediately after V3 finishes execution. If it is found that LB and V8 can be partially executed in parallel, V8 will be allocated a more powerful computing core, while LB will be allocated to a low-power core to avoid resource conflicts. For example, in an actual execution arrangement, V3 is allocated three NPU cores and two DMA channels, LA obtains one NPU core and some shared memory, and V4 waits until LA completes its output before allocating remaining resources for execution. Priority-driven resource allocation enables reasonable scheduling of multimodal tasks on the edge, preventing mutual blocking.

[0100] Finally, once the task officially enters the execution phase, the execution progress needs to be monitored in real time, including the current running status of each task, the resources it consumes, and the estimated completion time. Any resource allocation conflicts must be adjusted promptly. For example, if the vision segment V6 is using a large amount of memory bandwidth, causing significant queuing delays when the language segment's load balancer (LB) attempts scheduling, the monitoring module will detect this resource contention and classify it as a conflict event based on a preset threshold. In this case, the execution order will be dynamically adjusted according to the severity of the conflict. For example, the LB might be postponed until after V6's peak execution period, or certain steps of the LB might be divided and run earlier on idle computing cores to eliminate bandwidth contention. If monitoring data shows that a vision task completes ahead of schedule, for example, V3 is expected to take 15 milliseconds but actually only takes 10 milliseconds, the LA can be started earlier, and the start time of subsequent segments can be adjusted to shorten the overall latency. Conversely, if a segment's execution slows down, for example, if the LA's latency increases due to a decrease in cache hit rate, the subsequent vision segments can temporarily increase their parallel execution ratio to avoid the overall pace being slowed down by the language segment. Through this real-time monitoring and dynamic adjustment mechanism, the interleaved execution sequence can maintain a stable and efficient operating state on the edge, making the collaboration between vision and language tasks smoother and more reliable.

[0101] Example 2, Figure 2 The present invention provides a heterogeneous computing resource dynamic scheduling system for a multimodal edge model, comprising a sequence acquisition module, a pause identification module, an interval segmentation module, a segmentation identification module, and a task coordination module.

[0102] The sequence acquisition module is used to acquire the operator sequence output by the multimodal end-side model and identify the data exchange points between operator sequences based on the execution characteristics of the operator sequences.

[0103] The pause identification module is used to identify execution pause locations based on data exchange points between operator sequences, and to obtain an execution pause set.

[0104] The interval segmentation module is used to segment the operator sequence into intervals based on the execution pause set, and extract the connection point of each segmented interval in the intermediate execution state of the preset visual task to obtain the visual task structure diagram.

[0105] The cut-in recognition module is used to traverse the visual task structure diagram and identify the cut-in interval positions to obtain the cut-in sequence;

[0106] The task coordination module is used to embed the execution segments of preset language tasks into the intrusive sequence to obtain an interleaved execution sequence, and to coordinate and control the visual task and the language task based on the interleaved execution sequence.

[0107] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0108] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0109] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0112] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dynamic scheduling of heterogeneous computing resources in a multimodal edge-side model, characterized in that, Includes the following steps: Obtain the operator sequence output by the multimodal end-side model, and identify the data exchange points between operator sequences based on the execution characteristics of the operator sequences; The execution pause locations are identified based on the data exchange points between operator sequences, resulting in an execution pause set. The operator sequence is segmented into intervals based on the execution pause set, and the connection points of each segmented interval are extracted to represent the intermediate execution states of the preset visual task, resulting in a visual task structure diagram, as follows: Based on the execution pause set, the operator sequence is divided into several operator intervals; based on the execution pipeline of the multimodal edge model, the original operator execution order table is extracted; for each operator interval, the forward computation of each operator is executed sequentially according to the original operator execution order table; based on the computation results, the intermediate execution states of the connection points of each operator interval in the preset visual task are extracted to obtain intermediate feature maps; the semantic content of the intermediate feature maps is analyzed, visual task features are extracted, and a visual task structure map is constructed based on the visual task features; The visual task structure diagram is traversed and intrusive intervals are identified to obtain an intrusive sequence, specifically: A depth-first traversal algorithm is used to traverse the visual task structure graph and construct a data flow graph. The data input / output degree of each node in the data flow graph is calculated, including the number of direct predecessor nodes and the number of direct successor nodes. Based on the data input / output degree, nodes with both the number of direct predecessor nodes and the number of direct successor nodes below a preset first threshold are selected to obtain a candidate node set. The data transmission path of each candidate node in the data flow graph is extracted, and the data packet size and transmission delay in the data transmission path are calculated. The data transmission path is evaluated concurrently based on the data packet size and transmission delay, and the cut-in intervals are identified according to the evaluation results. The cut-in intervals are sorted according to a preset topological order of the data flow graph to obtain a cut-in sequence. The execution segments of the pre-defined language task are embedded in the cut-in sequence to obtain the interleaved execution sequence, and the visual task and the language task are coordinated and controlled based on the interleaved execution sequence.

2. The method for dynamic scheduling of heterogeneous computing resources in a multimodal edge model according to claim 1, characterized in that, The process of acquiring the operator sequence output by the multimodal end-side model and identifying the data exchange points between operator sequences based on the execution characteristics of the operator sequences specifically involves: Obtain the execution pipeline of the multimodal edge model, extract the operator sequence, and construct the operator execution trajectory matrix; Identify data exchange feature points in the operator execution trajectory matrix, and use graph convolutional networks to perform topological analysis on the data exchange feature points to obtain the data exchange region; Based on the data exchange area, a breadth-first search algorithm is used to expand the data points and obtain the data exchange points.

3. The method for dynamic scheduling of heterogeneous computing resources in a multimodal edge model according to claim 2, characterized in that, The process of expanding data points based on the data exchange area using a breadth-first search algorithm to obtain data exchange points is as follows: Constructing operator graphs for execution pipelines based on multimodal end-side models; The data exchange point is obtained by determining a preset central node in the data exchange area as the starting point for expansion, and then using breadth-first search to expand the starting point hop by hop until a preset number of hops is reached or a preset resource constraint condition is triggered.

4. The method for dynamic scheduling of heterogeneous computing resources in a multimodal edge model according to claim 3, characterized in that, The method of identifying execution pause locations based on data exchange points between operator sequences to obtain an execution pause set is as follows: Obtain the operator execution context corresponding to the data exchange point, and analyze the resource wait events in the operator execution context; The execution delay interval is obtained by comparing the duration of the resource waiting event with the preset time window; By mapping the execution delay interval to the operator sequence, the execution pause interval before and after the data exchange point is obtained; Merge overlapping execution pause intervals to obtain the execution pause set.

5. The method for dynamic scheduling of heterogeneous computing resources in a multimodal edge model according to claim 4, characterized in that, The process of embedding execution segments of a preset language task into the intrusive sequence to obtain an interleaved execution sequence is as follows: Extract the cut-in sequence into the interval subgraph corresponding to the data flow graph, and use a preset random walk algorithm to generate several interval execution paths in the interval subgraph; Obtain resource access records for several interval execution paths, and calculate the frequency of resource access overlap between interval execution paths based on the resource access records; Candidate embedding positions are obtained based on the frequency of resource access overlap, and the execution segments of the preset language task are embedded based on the candidate embedding positions to obtain an interleaved execution sequence.

6. The method for dynamic scheduling of heterogeneous computing resources in a multimodal edge model according to claim 5, characterized in that, The process of obtaining candidate embedding positions based on resource access overlap frequency and embedding execution segments of a preset language task based on these candidate embedding positions to obtain an interleaved execution sequence is as follows: Based on the frequency of overlapping resource accesses, resource contention regions that can be entered into the sequence are identified, and a set of prohibited embedding intervals is obtained. Remove the forbidden embedding interval set from the cutable sequence, and rearrange the remaining sequence according to the original order of the cutable sequence to obtain the candidate embedding positions; A preset path discretization method is used to segment the candidate embedding positions, and the preset language task atomic units are divided into execution segments that match the size of the candidate embedding positions. The execution segment is inserted at the embedding position after segmentation to obtain an interleaved execution sequence.

7. The method for dynamic scheduling of heterogeneous computing resources in a multimodal edge model according to claim 6, characterized in that, The coordinated control of visual and language tasks based on interleaved execution sequences is specifically as follows: This paper analyzes the visual task execution segments and language task execution segments in an interleaved execution sequence and examines their dependencies. The execution priority of task segments is established based on the dependency relationship, and heterogeneous computing resources are allocated to the visual task execution segment and the language task execution segment based on the execution priority of task segments; Monitor execution progress in real time, detect resource allocation conflicts, and dynamically adjust the execution order based on resource allocation conflicts.

8. A heterogeneous computing resource dynamic scheduling system for a multimodal edge model, applied to the heterogeneous computing resource dynamic scheduling method for a multimodal edge model as described in any one of claims 1-7, characterized in that, It includes a sequence acquisition module, a pause recognition module, a range segmentation module, a cut-in recognition module, and a task coordination module. The sequence acquisition module is used to acquire the operator sequence output by the multimodal end-side model and identify the data exchange points between operator sequences based on the execution characteristics of the operator sequences. The pause identification module is used to identify execution pause locations based on data exchange points between operator sequences, and to obtain an execution pause set. The interval segmentation module is used to segment the operator sequence into intervals based on the execution pause set, and extract the intermediate execution states of each segmented interval connection point in the preset visual task to obtain the visual task structure diagram, specifically: Based on the execution pause set, the operator sequence is divided into several operator intervals; based on the execution pipeline of the multimodal edge model, the original operator execution order table is extracted; for each operator interval, the forward computation of each operator is executed sequentially according to the original operator execution order table; based on the computation results, the intermediate execution states of the connection points of each operator interval in the preset visual task are extracted to obtain intermediate feature maps; the semantic content of the intermediate feature maps is analyzed, visual task features are extracted, and a visual task structure map is constructed based on the visual task features; The cut-in recognition module is used to traverse the visual task structure diagram and identify cut-in intervals to obtain a cut-in sequence, specifically: A depth-first traversal algorithm is used to traverse the visual task structure graph and construct a data flow graph. The data input / output degree of each node in the data flow graph is calculated, including the number of direct predecessor nodes and the number of direct successor nodes. Based on the data input / output degree, nodes with both the number of direct predecessor nodes and the number of direct successor nodes below a preset first threshold are selected to obtain a candidate node set. The data transmission path of each candidate node in the data flow graph is extracted, and the data packet size and transmission delay in the data transmission path are calculated. The data transmission path is evaluated concurrently based on the data packet size and transmission delay, and the cut-in intervals are identified according to the evaluation results. The cut-in intervals are sorted according to a preset topological order of the data flow graph to obtain a cut-in sequence. The task coordination module is used to embed the execution segments of preset language tasks into the intrusive sequence to obtain an interleaved execution sequence, and to coordinate and control the visual task and the language task based on the interleaved execution sequence.

Images