A large language model operator revenue boundary fingerprint scheduling multiplexing method, system, electronic device and readable storage medium for RVV

Abstract

The application discloses a large language model operator revenue boundary fingerprint scheduling multiplexing method and system for RVV, electronic equipment and readable storage medium. The method comprises the following steps: according to a target hardware configuration and a probe scheduling template set, probe imaging is performed on each working load in a working load set to generate a revenue boundary fingerprint of the working load; the fingerprint distance and migration risk between the working loads are calculated according to the revenue boundary fingerprint; a scheduling multiplexing relationship is formed according to the fingerprint distance, migration risk, execution stage, operator type, quantization format and main bottleneck type; a coverage group is formed according to the scheduling multiplexing relationship, and a representative working load is determined in the coverage group; a complete scheduling search is performed on the representative working load, the representative scheduling scheme is migrated to the target working load in the corresponding coverage group, and a migration scheduling scheme is further formed. The application can determine the multiplexing relationship according to the revenue boundary state and migration risk of the working load on the RVV hardware.

Description

Technical Field

[0001] This invention relates to the field of scheduling and search technology for large language model quantized linear operators, and in particular to a method, system, electronic device, and readable storage medium for scheduling and reusing large language model operator revenue boundary fingerprints for RVV. Background Technology

[0002] In the inference process of low-bit quantization large language models, matrix-vector multiplication, matrix-matrix multiplication, and related linear operators are repeatedly executed. In the deployment of edge RISC-V Vector Extension (RVV) hardware, these operators usually determine the main execution time of the pre-filling stage and the generation stage. Therefore, their scheduling scheme directly affects the execution cost of the quantization model on the target hardware.

[0003] RVV employs a variable vector length mechanism and controls the vector execution mode through vector element width, vector register group multiples, tail processing strategies, and vector configuration instructions. The same quantized linear operator can correspond to different data packing, loop rearrangement, vectorization dimension, block size, expansion factor, number of threads, and tail processing schemes under different matrix dimensions, quantization group sizes, memory access layouts, and execution stages, significantly increasing the overall scheduling search space.

[0004] Performing a full schedule search for every workload requires repeatedly enumerating candidate scheduling actions and performing measurements, simulations, and correctness verification. As the number of model layers, execution stages, context length, batch size, and quantization format changes, and the number of workloads increases, the number of full searches, measurement costs, and scheduling scheme management costs also rise.

[0005] Existing methods can merge workloads with similar matrix shapes. However, for RVV quantization linear operators, workloads with similar matrix shapes may still have different payoff boundaries. For example, two workloads may have similar N-dimensionality, but due to differences in quantization group size, scaling factor access method, tail element distribution, and vector configuration frequency, their register pressure, cache working set pressure, and vector configuration overhead may differ significantly. In this case, directly reusing the scheduling scheme according to matrix shape may result in the migrated block boundaries, tail processing, and register usage not meeting the target hardware constraints.

[0006] Therefore, a reuse determination method is needed that can be computed before the full scheduling search, so that it can determine the reuse relationship based on the benefit boundary state and migration risk of the workload on the target RVV hardware, and provide a definite exception handling path when migration fails. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, electronic device, and readable storage medium for scheduling reuse of large language model operators with revenue boundary fingerprints for RVV. The technical problem to be solved is: when there are multiple workloads and multiple candidate scheduling schemes for RVV quantized linear operators, how to determine which workloads can safely reuse the same representative scheduling scheme before the complete scheduling search, and avoid erroneous reuse when migration fails.

[0008] The preferred technical solutions among the many technical solutions provided by this invention can produce a variety of technical effects, which are described in detail below.

[0009] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a method for reusing operator benefit boundary fingerprint scheduling in large language models oriented towards RVV, the method comprising: Based on the target hardware configuration and probe scheduling template set, probe profiles are generated for each workload in the workload set formed by inference execution records, and the revenue boundary fingerprint of the workload is generated. The main bottleneck type of the workload is determined based on the revenue boundary fingerprint. The fingerprint distance and migration risk between workloads are calculated based on the revenue boundary fingerprint. A scheduling reuse relationship is formed based on the fingerprint distance, the migration risk, and the workload attributes. A coverage group is formed based on the scheduling reuse relationship, and a representative workload is determined within the coverage group. The workload attributes include execution stage, operator type, quantization format, and main bottleneck type. A complete scheduling search is performed on the representative workload to obtain a representative scheduling scheme. The representative scheduling scheme is then migrated to the target workload within the corresponding coverage group. After inheriting stable fields, recalculating repairable fields, fixing legality, and verifying correctness, a migration scheduling scheme is formed.

[0010] In some embodiments, forming a workload set based on inference execution records includes: Operator calls with the same execution stage, operator type, matrix dimension, and quantization format in the inference execution record are grouped into the same workload. The call weight of each workload is determined based on the number of calls and the average time per call, resulting in the workload set. The workload set includes the same workload and workloads with different execution stages, operator types, matrix dimensions, and quantization formats in the inference execution record. The inference execution record includes the actual quantized linear operator calls triggered during the inference process of the large language model, as well as the execution stage, operator type, matrix dimension, quantization format, number of calls, and average time per call for each call.

[0011] In some embodiments, generating the revenue boundary fingerprint of the workload includes: Based on the target hardware configuration, probe profiling is performed on each workload in the workload set under the low-overhead vector probe template, the computationally intensive probe template, and the memory access streaming probe template, respectively, to obtain multiple probe boundary vectors for each workload. Multiple probe boundary vectors of workloads under different probe templates are aggregated to obtain the benefit boundary fingerprint corresponding to the workload; the benefit boundary fingerprint includes the memory access boundary component, register pressure component, cache working set pressure component, tail element overhead component, vector configuration overhead component, and thread bandwidth contention component.

[0012] In some embodiments, the probe boundary vector includes calculation of memory access boundaries, register pressure, cache working set pressure, tail element overhead, vector configuration overhead, and thread bandwidth contention. The computation memory access boundary is determined based on the arithmetic operation volume of the workload, the number of bytes accessed, and the peak arithmetic throughput and available bandwidth of the target hardware configuration. The register pressure is determined based on the vector register usage of the workload under the corresponding probe template and the number of available vector registers in the target hardware configuration; The cache working set pressure is determined based on the number of cache working set bytes corresponding to the matrix block of the workload under the corresponding probe template and the L1 cache capacity of the target hardware configuration; The tail element overhead is determined based on the ratio of the number of elements that cannot be processed by a single vector instruction under the corresponding probe template in the matrix dimension-related loop of the workload and the number of remaining elements that are divisible by the block size to the number of elements that can be processed by a single vector instruction under the corresponding probe template. The vector configuration overhead is determined based on the number of vector configuration instructions and the number of vector execution fragments under the corresponding probe template for the workload. The thread bandwidth contention is determined based on the total number of memory access bytes of all threads within the probe run window under the corresponding probe template and the probe run window duration.

[0013] In some embodiments, before calculating the fingerprint distance and migration risk between the workloads, the method further includes: for any two workloads, determining whether their execution stage, operator type, quantization format and main bottleneck type are consistent; if they are consistent, calculating the fingerprint distance and migration risk between them. After cropping and normalizing the revenue boundary fingerprints of the two workloads, the fingerprint distance between them is calculated. The migration risk between the two workloads is calculated based on the differences in register pressure component, cache working set pressure component, tail element overhead component, vector configuration overhead component, and thread bandwidth contention component corresponding to the revenue boundary fingerprint. When the execution stage, operator type, quantization format and main bottleneck type of the source workload and the target workload are consistent, and the fingerprint distance between them does not exceed the fingerprint distance threshold and the migration risk between them does not exceed the migration risk threshold, it is determined that the representative scheduling scheme of the source workload is allowed to migrate to the target workload, thus forming a directed scheduling reuse relationship from the source workload to the target workload.

[0014] In some embodiments, forming an overlay group based on the scheduling multiplexing relationship, and identifying representative workloads within the overlay group, includes: A reuse neighborhood is formed for each workload based on the scheduling reuse relationship. The reuse neighborhood includes the source workload and the target workload to which the representative scheduling scheme of the source workload is allowed to migrate. The coverage gain is determined based on the call weight of the workload within the reuse neighborhood, the coverage radius penalty is determined based on the maximum fingerprint distance within the reuse neighborhood, the migration risk penalty is determined based on the weighted sum of migration risks within the reuse neighborhood, and the coverage score of the reuse neighborhood for each workload is determined by the coverage gain, coverage radius penalty and migration risk penalty. The reuse neighborhood with the highest coverage score is selected to form a coverage group, and within each coverage group, the workload with the lowest fingerprint distance to other members and the lowest overall cost of migration risk is selected as the representative workload.

[0015] In some embodiments, the complete scheduling search includes searching the scheduling action fields of candidate scheduling schemes. The scheduling action fields include data packaging method, circular rearrangement method, matrix partitioning method, vectorization dimension, vector element width, vector register group multiple, expansion factor, fusion method, number of threads, and tail processing method. When migrating the representative scheduling scheme to the target workload, the data packaging method, circular rearrangement method, vectorization dimension, vector element width, vector register group multiple, and fusion method in the representative scheduling scheme are inherited to form an initial migration scheduling scheme. Based on the matrix dimension, quantization format, and target hardware resource constraints of the target workload, the matrix block boundary, block size, expansion factor, number of threads, and tail processing method are recalculated.

[0016] In some embodiments, the method further includes: performing an exception point re-search on the target workload that fails to perform legitimacy repair or correctness verification, and forming a final set of scheduling schemes based on the representative scheduling scheme, the migration scheduling scheme, and the exception point re-search results.

[0017] According to another aspect of the present invention, a large language model operator benefit boundary fingerprint scheduling reuse system for RVV is also provided, comprising: The revenue boundary fingerprint generation module is used to perform probe profiling on each workload in the workload set formed by inference execution records according to the target hardware configuration and probe scheduling template set, generate the revenue boundary fingerprint of the workload, and determine the main bottleneck type of the workload based on the revenue boundary fingerprint. The reuse coverage generation module is used to calculate the fingerprint distance and migration risk between workloads based on the revenue boundary fingerprint, form a scheduling reuse relationship based on the fingerprint distance, the migration risk and workload attributes, form a coverage group based on the scheduling reuse relationship, and determine representative workloads within the coverage group; wherein, the workload attributes include execution stage, operator type, quantization format and main bottleneck type; The scheduling scheme generation module is used to perform a complete scheduling search on the representative workload to obtain a representative scheduling scheme, migrate the representative scheduling scheme to the target workload within the corresponding coverage group, and form a migration scheduling scheme after inheriting stable fields, recalculating repairable fields, performing legality repair, and verifying correctness; it is also used to perform an exception point re-search on the target workload that fails to perform legality repair or correctness verification, and form a final scheduling scheme set based on the representative scheduling scheme, the migration scheduling scheme, and the re-search results of exception points.

[0018] According to another aspect of the present invention, an electronic device is also provided, including a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the above-described method for scheduling and reusing the revenue boundary fingerprint of a large language model operator for RVV.

[0019] According to another aspect of the present invention, a non-transitory computer-readable storage medium is also provided, wherein the non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-described method for scheduling and reusing the revenue boundary fingerprint of a large language model operator for RVV.

[0020] Implementing one of the above-described technical solutions of the present invention has the following advantages or beneficial effects: First, without performing a complete scheduling search, the present invention constructs a benefit boundary fingerprint of the workload through probe profiling, thereby transforming the scheduling reuse determination from experience or shape matching into a quantitative decision based on the boundary state of the target RVV hardware resources.

[0021] Second, by introducing a joint constraint mechanism of fingerprint distance and migration risk, the present invention performs dual-condition screening of scheduling reuse relationships, effectively eliminating workload combinations with large differences in resource pressure and high probability of migration failure while ensuring boundary similarity.

[0022] Third, this invention transforms the original complete scheduling search per workload into a representative-driven local search by dividing the coverage group and selecting representative workloads. Combined with the legality repair, correctness verification and exception point re-search mechanism, this invention ensures that each workload in the final scheduling scheme set has a clear source path and meets the verifiability constraint. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart of a method for scheduling and reusing the revenue boundary fingerprint of a large language model for RVV according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a large language model operator revenue boundary fingerprint scheduling and reuse system for RVV according to an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments, illustrating various exemplary embodiments that may be used to implement the present invention. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of the present invention disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of the present invention.

[0025] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," etc., indicate the orientation or positional relationship based on the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0026] To illustrate the technical solution described in this invention, specific embodiments are described below, showing only the parts related to the embodiments of this invention.

[0027] Example 1: like Figure 1 As shown in this embodiment, a method for scheduling and reusing the operator revenue boundary fingerprint for large language models oriented to RVV is provided. The method includes: S100. Based on the target hardware configuration and probe scheduling template set, perform probe profiling on each workload in the workload set formed by inference execution records, generate the revenue boundary fingerprint of the workload, and determine its main bottleneck type based on the revenue boundary fingerprint of the workload.

[0028] In some embodiments, the inference execution record includes the actual quantized linear operator calls triggered during the inference process of the large language model, as well as the execution stage, operator type, matrix dimension, quantization format, number of calls, and average time per call for each call.

[0029] Inference execution records are derived from model graph execution logs, offline analysis logs, and deployment tool records; a complete scheduling search result is not required.

[0030] In some embodiments, a workload set is formed based on inference execution records, including: Operator calls with consistent execution stages, operator types, matrix dimensions, and quantization formats in the inference execution record are grouped into the same workload. The call weight of each workload is determined based on the number of calls and the average time per call, resulting in a workload set. It is understandable that the workload set includes both the same workload and different workloads.

[0031] This step groups operator calls with consistent execution phases, operator types, matrix dimensions, and quantization formats into a single workload. Specifically, the execution phase distinguishes between the pre-filling and generation phases; the operator type differentiates between matrix-vector multiplication, matrix-matrix multiplication, and the fixed computational form within quantized linear operators; the matrix dimension describes the input / output scale; and the quantization format describes the weighted quantization bit width, quantization group size, and scaling factor organization.

[0032] In some implementations, during the merging process, the calls to quantized linear operators are first filtered according to the model layer identifier, execution stage, and operator type. Then, the M-dimensionality, N-dimensionality, K-dimensionality, and quantization format of each call are read. Here, the M-dimensionality represents the output row dimension, the N-dimensionality represents the output column dimension, and the K-dimensionality represents the reduction dimension.

[0033] For calls that occur multiple times within the same layer and have the same dimension and quantization format, this method merges them into a single workload; for calls with different dimensions, quantization group sizes, or scaling factor organization methods, the system retains them as separate workloads. This process ensures that subsequent scheduling and reuse checks will not incorrectly classify calls with different data layouts as the same object.

[0034] Each workload in the workload set includes at least the execution phase, operator type, M-dimension, N-dimension, K-dimension, quantization format, number of calls, average execution time per call, and call weight. The number of calls represents the number of times the workload appears in the inference record; the average execution time per call represents the average execution time of the workload during the recording phase.

[0035] The quantization format fields mentioned above include at least the weight quantization bit width, quantization group size, scaling factor data type, and scaling factor arrangement. For the Q4_0 format, the weight quantization bit width is 4 bits, the quantization group size is given by the model weight format, and the scaling factor and quantization weight blocks are stored in a fixed order. This field directly affects the number of bytes accessed, the tail quantization group boundary, and the scaling factor index; therefore, it must be used as a workload merging condition.

[0036] In some implementations, the first Call weights for each workload Based on the number of calls and average time per run Confirmed. Specifically, ; in, Used to indicate the first The contribution of each workload to the overall inference execution cost. This represents the workload index. The technical meaning of this formula is to combine call frequency and single-instance cost into a normalized weight, so that subsequent representatives will prioritize covering workloads that contribute more to execution cost.

[0037] In some embodiments, generating a revenue boundary fingerprint for a workload includes: Based on the target hardware configuration, probe profiling is performed on each workload in the workload set under low-overhead vector probe templates, computationally intensive probe templates, and memory access streaming probe templates, resulting in multiple probe boundary vectors for each workload. These probe boundary vectors include computational memory access boundaries, register pressure, cache working set pressure, tail element overhead, vector configuration overhead, and thread bandwidth contention. Multiple probe boundary vectors for the same workload are aggregated to obtain the revenue boundary fingerprint corresponding to the workload.

[0038] The target hardware configurations mentioned above include RVV vector register length, number of available vector registers, L1 cache capacity, available bandwidth, peak arithmetic throughput, vector configuration overhead, and thread synchronization overhead. The probe scheduling template set includes low-overhead vector probe templates, computationally intensive probe templates, and memory-access-streaming probe templates.

[0039] The aforementioned low-overhead vector probe template is used to reduce vector configuration changes and observe the impact of vector configuration overhead on scheduling benefits during the mini-batch generation stage; the computationally intensive probe template is used to increase data reuse and expansion, and observe register pressure and computational throughput boundaries; the memory access streaming probe template is used to enhance continuous memory access characteristics and observe cache working set pressure and thread bandwidth contention. All three probe templates are used for boundary profiling and are not intended as the final deployment scheduling scheme.

[0040] Furthermore, the low-overhead vector probe template fixes the vector element width and vector register set multiples, and ensures that the same vector configuration covers consecutive execution segments, exposing the number of vector configuration instructions and the number of tail segments. The computationally intensive probe template fixes smaller cache blocks and increases the K-dimensional expansion, exposing the register pressure caused by the number of accumulators and intermediate vectors. The memory access streaming probe template fixes the consecutive weight access order and reduces data reuse, exposing the pressure on cache capacity and bandwidth from weight blocks, scaling factor blocks, and output blocks.

[0041] During probe profiling, this method applies three probe templates sequentially to each workload. Each probe run records the probe template number, target hardware configuration number, workload number, number of probe execution segments, number of vector configuration instructions, register usage, number of bytes in the tile working set, number of tail elements, total memory access bytes per thread, and probe run window duration. These records do not include the final scheduling action; they are only used to calculate the probe boundary vector.

[0042] The probe profile uses the same matrix dimensions, quantization format, weight layout, and scaling factor layout as the target workload. The probe template only changes the fixed scheduling pattern required to observe the boundary states, without altering the data semantics of the workload itself. This results in probe boundary vectors that reflect the resource pressure of the target workload under real RVV execution conditions, while avoiding the search cost of fully enumerating scheduling actions.

[0043] In some embodiments, after probe profiling is completed, a validity check is performed on each probe result. The validity check includes ensuring that the probe input dimension is consistent with the workload dimension, the quantization format is consistent with the workload quantization format, the probe running window is not empty, the number of vector configurations is non-negative, the number of registers used does not exceed the total number of hardware registers, and the number of bytes in the tile working set is greater than zero. Probe results that pass the validity check proceed to boundary component calculation, while probe results that fail the check are marked as unusable for fingerprint aggregation.

[0044] For each workload, this method generates probe boundary vectors under three probe scheduling templates. The probe boundary vectors are represented as follows: ; Where i represents the workload index and r represents the probe template index. This represents the memory access boundary for the i-th working negative under the r-th probe template. This represents the register pressure of the i-th working sample under the r-th probe template. This indicates the pressure of the cache working set under the r-th probe template for the i-th working sample. This represents the tail element cost of the i-th working negative under the r-th probe template. This represents the vector configuration cost of the i-th working negative under the r-th probe template. This indicates that the i-th job is responsible for the thread bandwidth contention under the r-th probe template.

[0045] In some embodiments, the memory access boundary is determined based on the arithmetic operations of the workload, the number of bytes accessed, and the peak arithmetic throughput and available bandwidth of the target hardware configuration.

[0046] For ease of understanding, the compute memory access boundary is used to describe the position of the workload's arithmetic strength relative to the target hardware's compute bandwidth.

[0047] In a specific example, the arithmetic operation quantity of the i-th workload is represented as: ; in, , , These represent the M-dimensional (output row dimension), N-dimensional (output column dimension), and K-dimensional (reduction dimension) of the i-th workload, respectively.

[0048] The number of bytes of data access for the i-th workload is represented as: ; in, Indicates the number of bytes of active data. Indicates the weight quantization bit width. Indicates the size of the quantization group. Indicates the number of bytes of the scaling factor. Indicates the number of bytes of output data.

[0049] Furthermore, the computational memory access boundary of the i-th workload under the r-th probe template is represented as: ; Where Peak represents the peak arithmetic throughput of the target hardware configuration, and BW (bandwidth) represents the available bandwidth of the target hardware configuration.

[0050] It should be noted that when the computation and memory access boundary is close to the equilibrium position, it means that the corresponding workload is near the benefit boundary between computation-constrained and memory-constrained workloads.

[0051] In some embodiments, register pressure is determined based on the amount of vector registers used by the workload under the corresponding probe template and the number of available vector registers in the target hardware configuration.

[0052] For ease of understanding, register pressure is used to describe the extent to which vector registers are used under a probe template relative to the number of available vector registers.

[0053] The register pressure of the i-th workload under the r-th probe template is expressed as: ; in, This indicates the number of vector registers used by the i-th workload under the r-th probe template (the number of accumulator vectors, intermediate vectors, and address vectors that need to be retained simultaneously), and Ravail indicates the number of vector registers in the target hardware configuration that can be used for this operator.

[0054] It should be noted that the higher the pressure on this register, the more likely the scheduling scheme will run out of registers during migration due to changes in the expansion factor, the multiple of the vector register group, and the number of intermediate accumulated variables.

[0055] In some embodiments, the cache working set pressure is determined based on the number of cache working set bytes corresponding to the matrix blocks of the workload under the corresponding probe template and the L1 cache capacity of the target hardware configuration.

[0056] To put it simply, cache working set pressure is used to describe the extent to which the number of bytes in the working set generated by matrix partitioning is used relative to the capacity of the L1 cache.

[0057] The cache working set pressure of the i-th workload under the r-th probe template is expressed as: ; in, L1 (Level 1 cache) represents the number of bytes of the activation block, weight block, scaling factor block, and output block that the i-th workload needs to reside simultaneously in the next tile of the r-th probe template. L1 represents the L1 cache capacity of the target hardware configuration.

[0058] It should be noted that the higher the pressure on the cache working set, the more likely it is that the block size in the representative scheduling scheme will exceed the cache capacity constraint when it is migrated to the target workload.

[0059] In some embodiments, the tail element overhead is determined based on the ratio of the number of elements in the workload's matrix dimension-related loop that cannot be processed by a single vector instruction under the corresponding probe template and the number of remaining elements that are divisible by the block size to the number of elements that can be processed by a single vector instruction under the corresponding probe template.

[0060] For ease of understanding, the tail element overhead describes the proportion of the remaining number of elements relative to the number of elements that can be processed by a single vector instruction. This remaining number of elements is determined based on the case where the target matrix dimension-related loop cannot be divided by the number of elements that can be processed by a single vector instruction and the block size.

[0061] In some implementations, it is set This represents the number of elements that can be processed by a single vector command under the r-th probe template. The matrix dimension representing the i-th workload , , Cannot be in related loops The number of remaining elements when the block size is divisible by the block size.

[0062] Then, the tail element cost of the i-th workload under the r-th probe template is: .

[0063] To put it simply, the higher the overhead of the tail element, the more the mask tail processing and segmented tail processing need to be regenerated during migration.

[0064] In some embodiments, the vector configuration overhead is determined based on the number of vector configuration instructions and the number of vector execution fragments under the corresponding probe template for the workload.

[0065] It should be noted that vector configuration overhead describes the proportion of overhead of vector configuration operations such as vector length setting instructions (vsetvli) relative to vector execution fragments.

[0066] The vector configuration cost of the i-th workload under the r-th probe template is represented as: ; in, This represents the number of vector configuration commands for the i-th workload under the r-th probe template. This represents the number of times the i-th workload performs a vector execution segment under the r-th probe template.

[0067] To put it simply, the higher the vector configuration overhead, the more sensitive the workload is to vector element width, vector register set multiples, and vector configuration merging strategies.

[0068] In some embodiments, thread bandwidth contention is determined based on the total number of memory access bytes for all threads within the probe run window under the corresponding probe template and the duration of the probe run window.

[0069] It should be noted that thread bandwidth contention is used to describe the degree of competition between the total memory access requirements and the available bandwidth of the target hardware during multi-threaded execution.

[0070] The thread bandwidth contention for the i-th workload under the r-th probe template is represented as follows: ; in, This represents the total number of memory accesses (bytes) for all threads within the probe execution window of the i-th workload under the r-th probe template. This represents the duration of the probe run window for the i-th workload under the r-th probe template.

[0071] To put it simply, the higher the bandwidth contention for a given thread, the more likely it is to cause bandwidth contention when the number of threads and the thread partitioning strategy in the scheduling scheme are migrated to the target workload.

[0072] Furthermore, workloads (including the same workload) will yield multiple probe boundary vectors under different probe templates. To form a stable benefit boundary fingerprint, this step aggregates the results from multiple probes. The memory access boundary component is calculated by taking the probe result closest to the boundary between computationally constrained and memory-constrained states, which represents the benefit boundary position of the workload. Register pressure, cache working set pressure, tail element overhead, vector configuration overhead, and thread bandwidth contention are calculated by taking the maximum pressure value among the probe results, which is used to preserve the risky states most likely to trigger failure during migration.

[0073] In some implementations, the aggregated revenue boundary fingerprint is represented as: .in, Pick The smallest probe result among the three probe templates, i.e. =min( ), r=1, 2, 3, corresponding to the low-overhead vector probe template, computationally intensive probe template, and memory-accessing streaming probe template, respectively; similarly, Take the maximum value from the corresponding probe results.

[0074] After the above fusion, the components corresponding to the revenue boundary fingerprint are obtained: memory access boundary component, register pressure component, cache working set pressure component, tail element overhead component, vector configuration overhead component, and thread bandwidth contention component.

[0075] It should be noted that each component comes from the probe profiling results, not the complete scheduling search results. Therefore, the revenue boundary fingerprint can be calculated before the complete scheduling search.

[0076] After obtaining the revenue boundary fingerprint, this step determines the primary bottleneck type based on the relationship between each component and a preset threshold. For example, if the memory access boundary reaches the corresponding preset threshold, then the memory access boundary is determined as the primary bottleneck type. If multiple components reach the threshold simultaneously, a unique primary bottleneck type is determined according to the priority order of memory access constraints and computational constraints: register pressure, cache working set pressure, thread bandwidth contention, tail element overhead, and vector configuration overhead. It should be noted that the primary bottleneck type is used for subsequent reuse relationship determination to avoid workloads with different dominant limiting factors being incorrectly assigned to the same coverage group.

[0077] In some implementations, the aforementioned preset thresholds are derived from the target hardware configuration file and the deployment tool's scheduling policy file. The register pressure threshold is determined by the number of available vector registers and the number of reserved registers; the cache working set pressure threshold is determined by the L1 cache capacity and cache retention ratio; the thread bandwidth contention threshold is determined by the target hardware's available bandwidth and thread synchronization overhead; and the tail element overhead threshold and vector configuration overhead threshold are given by the deployment tool's configuration of tail processing cost and vsetvli (vector length setting instruction) cost. These thresholds are used as input parameters in the calculation and are not obtained through a full search in this step.

[0078] It should be noted that this step only forms a workload set based on the inference execution records; it does not require performing a full schedule search on any workload, nor does it depend on the performance results of existing scheduling schemes. Therefore, the output of this step can be obtained before the full schedule search.

[0079] S200: Calculate the fingerprint distance and migration risk between workloads based on the revenue boundary fingerprint. Establish scheduling reuse relationships based on the fingerprint distance, migration risk, and workload attributes. Form coverage groups based on these relationships, and identify representative workloads within each coverage group. Workload attributes include execution stage, operator type, quantization format, and main bottleneck type.

[0080] In some embodiments, for any two workloads, this step first determines whether their execution stage, operator type, quantization format, and main bottleneck type are consistent. Different execution stages may lead to different call patterns; different operator types may lead to different loop structures; different quantization formats may lead to different data packaging and scaling factor access methods; different main bottleneck types indicate that the two workloads have different limiting factors. Therefore, only workloads with consistent attributes above will proceed to fingerprint distance and migration risk calculation.

[0081] Furthermore, the fingerprint distance between the two workloads is calculated after cropping and normalizing the revenue boundary fingerprints.

[0082] It should be noted that fingerprint distance is used to measure whether the benefit boundaries of two workloads on the target hardware are similar.

[0083] In some implementations, the components of the workload's revenue boundary fingerprint are first clipped and normalized, and then the fingerprint distances between them are calculated using the following formula: ; in, Let be the fingerprint distance between the i-th workload and the j-th workload. The above aggregated revenue boundary fingerprint representation; m represents the fingerprint component index, i.e., m=1,2,...,6, where 1 represents the fused computation memory access boundary component, 2 represents the fused register pressure component, 3 represents the fused cache working set pressure component, 4 represents the fused tail element overhead component, 5 represents the fused vector configuration overhead component, and 6 represents the fused thread bandwidth contention component. denoted by , where represents the distance weight of the m-th component, and 'clip' represents the clipping normalization function.

[0084] To put it simply, the technical meaning of this formula (fingerprint distance) is to map boundary states of different dimensions to the same comparison scale.

[0085] Furthermore, the migration risk between the two workloads is calculated based on the differences in register pressure component, cache working set pressure component, tail element overhead component, vector configuration overhead component, and thread bandwidth contention component.

[0086] The migration risk described above measures the likelihood of validity repair failures and correctness verification failures when a representative scheduling scheme migrates from one workload to another.

[0087] In some implementations, the migration risk between the i-th workload and the j-th workload Represented as: .

[0088] in, , , , and These represent the weights of the corresponding register pressure component, cache working set pressure component, tail element overhead component, vector configuration overhead component, and thread bandwidth contention component, respectively. The technical implication of this formula is to synthesize resource pressure differences that could lead to migration repair failures into a risk quantity.

[0089] When two workloads i and j satisfy the property consistency and When this happens, it indicates that the scheduling scheme can be migrated from workload i to workload j. Wherein, ζ represents the fingerprint distance threshold, and ζ represents the migration risk threshold. This determination simultaneously constrains the similarity of the benefit boundary and the risk of migration failure, avoiding the incorrect reuse of scheduling schemes simply because of similar shapes.

[0090] The above fingerprint distance threshold The migration risk threshold ζ is derived from the scheduling reuse policy in the target hardware configuration file. The similarity of the boundary states of the benefit control is controlled, and ζ controls the probability of repair failure after migration. For RVV cores with limited register resources, the register pressure weight in ζ can be higher than the cache pressure weight; for RVV cores with smaller cache capacity, the cache working set pressure weight in ζ can be higher than the register pressure weight. The weight difference is given by the target hardware configuration file to ensure that reuse determination is consistent with hardware resource bottlenecks.

[0091] In some implementations, a directed multiplexing decision matrix A can be formed. The representative scheduling scheme for the i-th workload allows migration to the j-th workload; This indicates that direct migration is not allowed. The matrix is ​​directed because the risks of migrating from i to j and from j to i are determined by the resource pressure differences between the source and target workloads, respectively.

[0092] Based on the directed reuse decision matrix, a reuse neighborhood N(i) can be formed for the i-th workload. The sum of call weights of workloads within the reuse neighborhood represents the execution cost contribution that can be covered when the i-th workload is used as a representative; the maximum fingerprint distance within the reuse neighborhood represents the coverage radius; and the weighted sum of migration risks within the reuse neighborhood represents the migration failure risk. Therefore, the coverage score can be composed of coverage benefits, coverage radius penalties, and migration risk penalties.

[0093] In some embodiments, forming coverage groups based on scheduling multiplexing relationships, and identifying representative workloads within the coverage groups, includes: Each workload is assigned a reuse neighborhood based on the scheduling reuse relationship. The reuse neighborhood includes the source workload and the target workload to which the representative scheduling scheme of the source workload is allowed to migrate. The coverage gain is determined based on the call weight of the workload within the reuse neighborhood, the coverage radius penalty is determined based on the maximum fingerprint distance within the reuse neighborhood, the migration risk penalty is determined based on the weighted sum of migration risks within the reuse neighborhood, and the coverage score of the reuse neighborhood for each workload is determined by the coverage gain, coverage radius penalty, and migration risk penalty. Prioritize the selection of reused neighborhoods with high coverage scores to form coverage groups, and within each coverage group, select workloads with low fingerprint distance to other members and low overall migration risk as representative workloads.

[0094] In some implementations, the coverage score of the reuse neighborhood of the i-th workload Represented as: ; in, This represents the call weight of the j-th workload. This represents the fingerprint distance from the i-th workload to the j-th workload. Let μ represent the migration risk from workload i to workload j, μ represent the coverage radius penalty coefficient, and η represent the migration risk penalty coefficient.

[0095] The higher the coverage score, the more execution cost contribution can be covered when the i-th workload is used as a representative, and the coverage radius and migration risk are at a lower level.

[0096] When forming coverage groups, reuse neighborhoods with high coverage scores are prioritized, and workloads that have already been covered are removed from subsequent candidates. For each coverage group, the representative workload is selected as the workload with the lowest combined cost of distance to fingerprints of other members within the group and migration risk. This selection of representative workloads can both cover higher call weights and reduce the risk of repair failures when migrating to members within the group.

[0097] In each coverage group C kInside, representing the workload r k The overall migration cost can be determined based on the total migration cost, which is expressed as: ; in, This indicates the workload of the candidate representatives. This indicates the workload of the members of the coverage group. Indicates the call weight of the member's workload. (Select) The lowest candidate representative is r k This minimizes the distance and risk-weighted total amount of representative workload to group members.

[0098] S300 performs a complete scheduling search on the representative workload to obtain a representative scheduling scheme. The representative scheduling scheme is then migrated to the target workload within the corresponding coverage group. After inheriting stable fields, recalculating repairable fields, fixing legality, and verifying correctness, a migration scheduling scheme is formed.

[0099] In some embodiments, a complete scheduling search includes searching the scheduling action fields of candidate scheduling schemes. The scheduling action fields include data packaging method, circular rearrangement method, matrix partitioning method, vectorization dimension, vector element width, vector register group multiple, expansion factor, fusion method, number of threads, and tail processing method.

[0100] The data packaging method determines the memory organization of quantization weights and scaling factors; the loop rearrangement method determines the traversal order of M, N, and K dimensions; the matrix partitioning method determines the size of the cache working set; the vectorization dimension, vector element width, and vector register set multiple determine the RVV vector execution form; the expansion factor affects register usage; the number of threads affects bandwidth contention; and the tail processing method determines the correct handling of non-divisible dimensions.

[0101] Candidate scheduling schemes are generated in a fixed order. First, data packing methods and loop rearrangement methods are enumerated; then, matrix partitioning methods and vectorization dimensions are enumerated; next, vector element widths, vector register group multiples, and expansion factors are enumerated; finally, the number of threads and tail processing methods are enumerated. This order ensures that the data layout and loop structure are determined first, followed by resource consumption-related fields, facilitating the early removal of candidate schemes that do not meet hardware constraints during the search process.

[0102] After candidate scheduling schemes are generated, the block boundaries are first calculated based on the M, N, and K dimensions representing the workload. Then, the addresses of weight blocks, scaling factor blocks, and output blocks are calculated according to the quantization format. Subsequently, the required number of vector registers, the number of bytes in the tile working set, the number of vector configurations, and the total memory access requirements of threads are calculated. The results of these calculations serve as input for validity checks and performance evaluation.

[0103] Before candidate scheduling schemes enter performance evaluation, they need to pass a series of validity checks. These checks include: dimension boundary checks to ensure that the blocks cover all matrix elements without exceeding boundaries; register checks to ensure that the accumulator, intermediate variables, and vector register sets do not exceed the number of available registers; cache working set checks to ensure that the activation, weight, scaling factor, and output working set of the current tile do not exceed the preset cache capacity ratio; vector configuration checks to ensure that the vector element width, vector register set multiple, and vector length settings comply with RVV constraints; and thread bandwidth checks to ensure that the total memory access requirements of multiple threads do not exceed the preset bandwidth contention threshold.

[0104] After passing the legality check, the candidate scheduling schemes proceed to performance evaluation. Performance evaluation is conducted through a fixed method among target hardware testing, a local RVV test program, and a simulated execution program; in this embodiment, target hardware testing is used.

[0105] The complete scheduling search is constrained by passing correctness verification and aims to minimize execution latency. In some deployment environments, the comprehensive cost function can also combine execution latency, register overflow penalty, cache out-of-bounds penalty, vector configuration overhead penalty, and thread bandwidth contention penalty into an evaluation value.

[0106] In some implementations, the evaluation value of the candidate scheduling scheme S is expressed as: ; in, This indicates the execution latency of the candidate scheduling scheme on the target hardware. Indicates register over-limit penalty, This indicates the penalty for exceeding the cache working set limit. Indicates vector configuration overhead penalty. This represents a penalty for thread bandwidth contention. , , and This indicates the corresponding penalty weight. Candidate scheduling schemes must first pass the correctness verification; after verification, they are then assigned a penalty weight. Sort from lowest to highest.

[0107] The full scheduled search also performs phased pruning. The first phase prunes candidate solutions that cannot cover the target matrix based on dimensionality and quantization group boundaries; the second phase prunes resource-overloaded candidate solutions based on register pressure and cache working set pressure; the third phase prunes candidate solutions with significantly high execution overhead based on vector configuration overhead and thread bandwidth contention; and the fourth phase performs correctness verification and performance evaluation on the remaining candidate solutions. Through phased pruning, the full search representing the workload still retains complete search semantics but reduces the number of invalid candidate solutions.

[0108] Once the representative scheduling scheme is determined, the action fields and resource status of the representative scheduling scheme are recorded. Resource status includes register usage, tile working set bytes, vector configuration counts, total thread memory access bytes, tail processing type, and correctness verification summary. These resource statuses serve as an initial reference for migration remediation, used to determine what type of resource pressure change the target workload has experienced relative to the representative workload.

[0109] For each representative workload, the complete schedule search outputs a representative scheduling scheme. This scheme includes scheduling action fields, validity check results, correctness verification results, and performance evaluation results. The representative scheduling scheme does not only serve the representative workload but also serves as a migration source for other workloads within the same coverage group.

[0110] During migration, the data packaging method, circular rearrangement method, vectorization dimension, vector element width, and fusion method are inherited as stable fields; the matrix block boundary, block size, expansion factor, number of threads, and tail processing method are treated as repairable fields and regenerated according to the matrix dimension and quantization format of the target workload.

[0111] Stable fields can be inherited because workloads within the same coverage group maintain consistency in execution phase, operator type, quantization format, and main bottleneck type, and their data layout, main loop direction, and vectorization dimension typically exhibit strong stability. Repairable fields require recalculation because the matrix dimension, tail remainder, cache working set, and thread memory access requirements of the target workload may differ from those of the representative workload.

[0112] It should be noted that stable fields represent the default inherited fields when the migration scheme is generated, and do not mean that all stable fields remain absolutely unchanged during the validity repair phase. Data packaging method, circular rearrangement method, vectorization dimension, and fusion method remain unchanged during the migration process; vector element width and vector register group multiples are inherited during the initial migration, representing the scheduling scheme, and are updated according to the vector configuration repair rules when vector configuration validity checks fail. This update only occurs during the validity repair phase, and the repaired fields are recorded as the repair result of the migration scheduling scheme.

[0113] Migration field generation is performed in a fixed order. The first step is to copy stable fields to form an initial migration scheduling plan. The second step is to recalculate the output row dimension (Tile_M), output column dimension (Tile_N), and reduction dimension (Tile_K) of the blocks based on the M, N, and K dimensions of the target workload. The third step is to recalculate the quantization group boundaries and scaling factor indexes based on the quantization format of the target workload. The fourth step is to recalculate the unrolling factor, number of threads, and tail processing method based on the target hardware configuration. The fifth step combines the above fields into a migration plan to be repaired, and then proceeds to the legality repair process.

[0114] In some embodiments, the legality repair may select to perform one or more of the following repairs based on the triggering conditions: The dimensional boundary repair is performed by recalculating the start and end boundaries of the blocks based on the matrix dimensions of the target workload. To put it simply, dimensional boundary repair is used to recalculate the start and end boundaries of Tile_M, Tile_N, and Tile_K based on the M, N, and K dimensions of the target workload, ensuring that the migrated scheduling scheme covers all elements of the target matrix and does not access out-of-bounds addresses.

[0115] When the matrix dimension of the target workload is not divisible by the current vector length, the corresponding block size, or the quantization group size, or when the tail element crosses the quantization group boundary, masked tail processing or segmented tail processing is generated for tail repair. For ease of understanding, tail repair is used to handle cases where the target dimension is not divisible by the vector length, block size, or quantization group size. When non-divisible tails exist, masked tail processing and segmented tail processing are generated, and the scaling factor and quantization group index are corrected simultaneously.

[0116] When the cached working set exceeds a preset cache capacity ratio, cache working set repair involves reducing the size of one or more of the following chunk dimensions that caused the cached working set to exceed the limit: the output row dimension (Tile_M), the output column dimension (Tile_N), or the reduced dimension (Tile_K). For clarity, cache working set repair addresses situations where the migrated tile working set exceeds the cache capacity threshold. When the working set corresponding to activation, weight, scaling factor, and output exceeds the preset cache ratio, the chunk size in Tile_N and Tile_K corresponding to the exceeding working set is reduced until the cached working set meets the constraints.

[0117] When register usage exceeds the number of available registers, register usage is corrected by reducing the spread factor, adjusting the vector register set multiple, or splitting computation fragments. For clarity, register usage correction addresses situations where vector register usage exceeds the available number after migration. When the number of accumulators, intermediate vectors, and vector register set multiples cause register pressure to exceed a threshold, the spread factor is reduced, the vector register set multiple is adjusted, and computation fragments are split if the limit is still exceeded after adjustment.

[0118] When vector configuration overhead exceeds a threshold, vector configuration repair is performed by merging adjacent vector configurations and adjusting the vector element width or vector register set multiple, without altering the data semantics and vectorization dimension. For clarity, vector configuration repair addresses situations where vector configuration overhead is excessive after migration. When the number of vsetvli executions relative to the number of vector execution fragments exceeds a threshold, adjacent vector configurations are merged, and the vector element width, vector register set multiple, and vectorization dimension are adjusted to reduce redundant configuration overhead.

[0119] Thread bandwidth competition repair that reduces the number of threads and adjusts the thread block range when the thread bandwidth competition exceeds a threshold. For ease of understanding, the thread bandwidth competition repair is used to handle the situation where the multi-thread memory access competition is too high after migration. When the total memory access demand of the threads exceeds the available bandwidth threshold, the number of threads is reduced, the thread block is adjusted, and part of the parallelism is transferred to a dimension with higher cache reuse.

[0120] The above six types of legality repairs are executed in a fixed order: first, the dimension boundary repair is executed, then the tail processing repair, followed by the cache working set repair, register usage repair, vector configuration repair, and thread bandwidth competition repair. This order ensures that the address boundary is correct first, the tail coverage is correct subsequently, the cache and register resources are then constrained, and the vector configuration and thread strategy converge finally. After each type of repair is completed, the system recalculates the corresponding boundary components and passes the repaired fields to the next type of repair.

[0121] When a certain type of repair changes the fields on which subsequent repairs depend, only the affected subsequent fields are recalculated. For example, after the cache working set repair reduces Tile_K, the register usage, the number of vector configuration times, and the thread memory access bytes are recalculated; after the register usage repair reduces the unroll factor, the number of vector execution segments and the vector configuration overhead are recalculated. This local recalculation avoids inconsistent fields in the migration scheme during the repair process.

[0122] Furthermore, the migration scheme to be repaired needs to generate an address mapping before entering the legality repair. The address mapping includes the activation matrix address, quantization weight address, scale factor address, and output matrix address. Each address mapping is based on the matrix dimension and quantization format of the target workload, rather than沿用 the address boundary representing the workload. This processing ensures that the block boundary representing the scheduling scheme will not be wrongly applied to the target workload after migration.

[0123] After the legality repair is completed, correctness verification is performed. When the quantized linear operator requires bit-exact output, the migration scheduling scheme passes the verification if the output is element-wise consistent with the reference implementation output; when the quantized linear operator allows numerical errors, it passes the verification according to the preset absolute error threshold and relative error threshold. The verification samples cover the aligned dimension, unaligned tail dimension, and quantization group boundary dimension to prevent tail processing and scale factor index errors.

[0124] Correctness verification uses input tensors, quantization weights, scaling factors, and output layouts consistent with the target workload. Aligned dimension samples are used to check the main loop; unaligned tail dimension samples are used to check masked tail processing and segmented tail processing; quantization group boundary dimension samples are used to check scaling factor indexing and quantization group cross-boundary access. Only after all three samples pass is the migration scheduling scheme written into the final scheduling scheme set.

[0125] In some embodiments, the method further includes: performing a re-examination of exception points for target workloads that fail to resolve legitimacy issues or fail to verify correctness, and forming a final set of scheduling schemes based on representative scheduling schemes, migration scheduling schemes, and re-examination results of exception points.

[0126] When legitimacy repair or correctness verification fails, the target workload no longer uses the representative scheduling scheme. Instead, it is treated as an exception point and a full scheduling search is performed. The scheduling scheme obtained from the re-search at the exception point is directly added to the final scheduling scheme set, and the source of failure for the corresponding reuse relationship is recorded to prevent the same representative scheme from being incorrectly migrated to the target workload again.

[0127] Exception records should include at least the target workload ID, representative workload ID, failure stage, failure reason, triggered remediation items, and the re-searched scheduling scheme ID. Failure stages include dimensional boundary repair, tail processing repair, cached working set repair, register usage repair, vector configuration repair, thread bandwidth contention repair, and correctness verification. Failure reasons are used for subsequent scheduling strategy analysis and are not included in the migration scheme output for the current target workload.

[0128] Each item in the final scheduling scheme set includes at least the workload number, scheme source, scheduling action field, stability field source, recalculation result of repairable fields, legality repair status, correctness verification status, and final scheme number. Scheme source includes three categories: representative search, migration reuse, and exception point re-search. Using these fields, the deployment tool can directly read the final scheduling scheme for each workload and trace which technical path the scheme originated from.

[0129] Ultimately, representative workloads are assigned to the representative scheduling scheme, successfully migrated workloads are assigned to the migration scheduling scheme, and exception workloads are assigned to the re-searched scheduling scheme, thus forming the final set of scheduling schemes. This final set of scheduling schemes can be read by the compiler backend, offline deployment tools, and model inference building processes to select the appropriate scheduling scheme for quantized linear operators.

[0130] In summary, this embodiment, without performing a complete scheduling search, constructs a benefit boundary fingerprint of the workload through probe profiling, thereby transforming the scheduling reuse determination from experience or shape matching into a quantitative decision based on the boundary state of the target RVV hardware resources. By introducing a joint constraint mechanism of fingerprint distance and migration risk, scheduling reuse relationships are screened under two conditions, effectively eliminating workload combinations with large differences in resource pressure and high probability of migration failure while ensuring boundary similarity. Through coverage group division and representative workload selection mechanisms, the original complete scheduling search per workload is transformed into a representative-driven local search. Combined with legality repair, correctness verification, and exception point re-search mechanisms, each workload in the final scheduling scheme set has a clear source path and meets verifiability constraints.

[0131] Example 2: This embodiment illustrates how the revenue boundary fingerprint drives representation search, migration reuse, and exception point re-search. The example data is used to illustrate the input-output chain and is not intended to limit the scope of this invention. Matrix-vector multiplication (MV) represents the generation stage operator type in the "Type" field of Table 1.

[0132] Table 1. Examples of simplified workload representations The six workloads in Table 1 all originate from the quantization matrix-vector multiplication operator in the generation phase, and all use the Q4_0 quantization format. W1, W2, W3, and W6 have the same K dimension but different N dimensions, while W4 and W5 have a larger K dimension, thus their cache working set pressure differs from that of group W1. Call weights are used for subsequent coverage benefit calculations.

[0133] Table 2. Summary of Fingerprint Reuse Detection and Subsequent Processing Table 2 illustrates that reuse determination does not solely depend on matrix shape. W1 and W2 have low fingerprint distance and migration risk, forming the same coverage group; although W1 and W3 share the same main bottleneck, their high migration risk prevents them from entering W1's coverage group; W2 and W4 have different main bottlenecks, and both have high fingerprint distance and migration risk, therefore reuse is not permitted.

[0134] Table 3 Source Table of Final Solution In this embodiment, W1, W2, and W6 are all MV operators in the generation stage, all with the quantization format Q4_0, and their profit boundary fingerprints are all dominated by vector configuration overhead. The fingerprint distance and migration risk between W1 and W2 are both below the threshold, therefore W2 can enter W1's coverage group. The fingerprint distance and migration risk between W1 and W6 also meet the initial reuse conditions, so W6 can be used as a migration target to enter the coverage group. However, after migration, the correctness verification fails, so W6 is written to the exception point and the full scheduling search is re-executed.

[0135] Although W3 shares the same execution stage, operator type, and main bottleneck type as W1, its migration risk exceeds the threshold due to differences in tail element overhead and cached working set, thus preventing it from entering W1's coverage group. W4 and W5 both exhibit cached working set pressure dominance, and their fingerprint distance and migration risk are both below the threshold, thus forming another coverage group, with W4 serving as the representative workload. Ultimately, W1 and W4 use a representative scheduling scheme, W2 and W5 use a migration scheduling scheme, W3 uses a single-point full search scheme, and W6 uses an exception-point re-search scheme.

[0136] This embodiment forms a complete input-output closed loop: Table 1 provides a set of workloads, Table 2 forms a reuse determination based on the revenue boundary fingerprint, and Table 3 outputs the source of the final solution based on representative search, migration verification, and exception point re-search. Each workload has a unique, deterministic, and verifiable source of scheduling solutions.

[0137] Example 3: like Figure 2 As shown, this embodiment also provides a large language model operator benefit boundary fingerprint scheduling and reuse system for RVV, including: The revenue boundary fingerprint generation module is used to perform probe profiling on each workload in the workload set formed by inference execution records based on the target hardware configuration and probe scheduling template set, generate the revenue boundary fingerprint of the workload, and determine its main bottleneck type based on the revenue boundary fingerprint of the workload.

[0138] Furthermore, the revenue boundary fingerprint generation module includes an execution record parsing unit, a workload merging unit, a probe profiling unit, and a fingerprint aggregation unit. The execution record parsing unit is used to read inference execution records; the workload merging unit is used to form a workload set and call weights; the probe profiling unit is used to generate probe boundary vectors based on the target hardware configuration and probe scheduling template set; and the fingerprint aggregation unit is used to form the revenue boundary fingerprint and determine the main bottleneck type.

[0139] The reuse coverage generation module calculates the fingerprint distance and migration risk between workloads based on the revenue boundary fingerprint. It then forms scheduling reuse relationships based on the fingerprint distance, migration risk, and workload attributes. These relationships form coverage groups, within which representative workloads are identified. Workload attributes include execution stage, operator type, quantization format, and main bottleneck type. Furthermore, the reuse coverage generation module includes a fingerprint distance calculation unit, a migration risk calculation unit, a reuse relationship generation unit, and a representative selection unit. The fingerprint distance calculation unit is used to compare the benefit boundary fingerprints of workloads; the migration risk calculation unit is used to calculate resource pressure differences; the reuse relationship generation unit is used to generate directed reuse relationships; and the representative selection unit is used to form coverage groups and select representative workloads.

[0140] The scheduling scheme generation module is used to perform a complete scheduling search on the representative workload to obtain a representative scheduling scheme. It then migrates the representative scheduling scheme to the target workload within the corresponding coverage group. After inheriting stable fields, recalculating repairable fields, performing legality repair, and verifying correctness, a migration scheduling scheme is formed. The module is also used to perform a re-search for exception points for target workloads that fail to perform legality repair or correctness verification. Based on the representative scheduling scheme, the migration scheduling scheme, and the re-search results for exception points, a final set of scheduling schemes is formed.

[0141] Furthermore, the scheduling scheme generation module includes a representative scheduling search unit, a migration field generation unit, a validity repair unit, a correctness verification unit, and an exception handling unit. The representative scheduling search unit performs a complete scheduling search on the representative workload; the migration field generation unit inherits stable fields and recalculates repairable fields; the validity repair unit performs dimension boundary, tail processing, cache working set, register usage, vector configuration, and thread bandwidth contention repair; the correctness verification unit verifies the migration scheme output; and the exception handling unit re-executes a complete scheduling search on workloads that fail to be repaired or verified.

[0142] Example 4: This embodiment also provides an electronic device, including: One or more processors; One or more memories are used to store one or more computer programs, and one or more processors are used to execute the one or more computer programs stored in the memories, so that the one or more processors perform the features / steps of a large language model operator revenue boundary fingerprint scheduling reuse method for RVV as described in Embodiment 1.

[0143] Furthermore, at the hardware level, the device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other necessary hardware. One or more embodiments of this application can be implemented in software, for example, by the processor reading the corresponding computer program from the non-volatile memory into memory and then running it.

[0144] Of course, in addition to the software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software. That is to say, the execution subject of the above processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0145] Example 5: Those skilled in the art will understand that all or part of the features / steps of the above-described method embodiments can be implemented by methods, data processing systems, or computer programs. These features may be implemented without hardware, entirely in software, or in a combination of hardware and software. The aforementioned computer program may be stored in a non-transitory computer-readable storage medium. When the computer program is executed (e.g., by a processor), it performs the steps of the above-described embodiment of a method for scheduling and reusing large language model operators for RVV.

[0146] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the processes of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0147] The above description is merely a preferred embodiment of the present invention. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A method for reusing operator revenue boundary fingerprint scheduling in a large language model oriented towards RVV, characterized in that, The method includes: Based on the target hardware configuration and probe scheduling template set, probe profiles are generated for each workload in the workload set formed by inference execution records, and the revenue boundary fingerprint of the workload is generated. The main bottleneck type of the workload is determined based on the revenue boundary fingerprint. The fingerprint distance and migration risk between workloads are calculated based on the revenue boundary fingerprint. A scheduling reuse relationship is formed based on the fingerprint distance, the migration risk, and the workload attributes. A coverage group is formed based on the scheduling reuse relationship, and a representative workload is determined within the coverage group. The workload attributes include execution stage, operator type, quantization format, and main bottleneck type. A complete scheduling search is performed on the representative workload to obtain a representative scheduling scheme. The representative scheduling scheme is then migrated to the target workload within the corresponding coverage group. After inheriting stable fields, recalculating repairable fields, fixing legality, and verifying correctness, a migration scheduling scheme is formed. It also includes: performing a re-examination of exception points for the target workloads that fail to achieve legitimacy or correctness verification, and forming a final set of scheduling schemes based on the representative scheduling scheme, the migration scheduling scheme, and the re-examination of exception points.

2. The method for scheduling and reusing operator revenue boundary fingerprints for large language models oriented towards RVV as described in claim 1, characterized in that, The process of forming a workload set based on inference execution records includes: Operator calls with the same execution stage, operator type, matrix dimension, and quantization format in the inference execution record are grouped into the same workload. The call weight of each workload is determined based on the number of calls and the average time per call, resulting in the workload set. The workload set includes the same workload and workloads with different execution stages, operator types, matrix dimensions, and quantization formats in the inference execution record. The inference execution record includes the actual quantized linear operator calls triggered during the inference process of the large language model, as well as the execution stage, operator type, matrix dimension, quantization format, number of calls, and average time per call for each call.

3. The method for reusing operator revenue boundary fingerprint scheduling for large language models oriented to RVV according to claim 1, characterized in that, The generated workload's revenue boundary fingerprint includes: Based on the target hardware configuration, probe profiling is performed on each workload in the workload set under the low-overhead vector probe template, the computationally intensive probe template, and the memory access streaming probe template, respectively, to obtain multiple probe boundary vectors for each workload. Multiple probe boundary vectors of workloads under different probe templates are aggregated to obtain the benefit boundary fingerprint corresponding to the workload; the benefit boundary fingerprint includes the memory access boundary component, register pressure component, cache working set pressure component, tail element overhead component, vector configuration overhead component, and thread bandwidth contention component.

4. The method for scheduling and reusing operator revenue boundary fingerprints for large language models oriented to RVV according to claim 3, characterized in that, The probe boundary vector includes calculation of memory access boundaries, register pressure, cache working set pressure, tail element overhead, vector configuration overhead, and thread bandwidth contention. The computation memory access boundary is determined based on the arithmetic operation volume of the workload, the number of bytes accessed, and the peak arithmetic throughput and available bandwidth of the target hardware configuration. The register pressure is determined based on the vector register usage of the workload under the corresponding probe template and the number of available vector registers in the target hardware configuration; The cache working set pressure is determined based on the number of cache working set bytes corresponding to the matrix block of the workload under the corresponding probe template and the L1 cache capacity of the target hardware configuration; The tail element overhead is determined based on the ratio of the number of elements that cannot be processed by a single vector instruction under the corresponding probe template in the matrix dimension-related loop of the workload and the number of remaining elements that are divisible by the block size to the number of elements that can be processed by a single vector instruction under the corresponding probe template. The vector configuration overhead is determined based on the number of vector configuration instructions and the number of vector execution fragments under the corresponding probe template for the workload. The thread bandwidth contention is determined based on the total number of memory access bytes of all threads within the probe run window under the corresponding probe template and the probe run window duration.

5. The method for scheduling and reusing operator revenue boundary fingerprints for large language models oriented to RVV according to claim 1, characterized in that, Before calculating the fingerprint distance and migration risk between the workloads, the method further includes: for any two workloads, determining whether their execution stage, operator type, quantization format and main bottleneck type are consistent; if they are consistent, calculating the fingerprint distance and migration risk between them. After cropping and normalizing the revenue boundary fingerprints of the two workloads, the fingerprint distance between them is calculated. The migration risk between the two workloads is calculated based on the differences in register pressure component, cache working set pressure component, tail element overhead component, vector configuration overhead component, and thread bandwidth contention component corresponding to the revenue boundary fingerprint. When the execution stage, operator type, quantization format and main bottleneck type of the source workload and the target workload are consistent, and the fingerprint distance between them does not exceed the fingerprint distance threshold and the migration risk between them does not exceed the migration risk threshold, it is determined that the representative scheduling scheme of the source workload is allowed to migrate to the target workload, thus forming a directed scheduling reuse relationship from the source workload to the target workload.

6. The method for scheduling and reusing operator revenue boundary fingerprints for large language models oriented to RVV according to claim 1, characterized in that, Coverage groups are formed based on the scheduling and multiplexing relationships, and representative workloads are identified within the coverage groups, including: A reuse neighborhood is formed for each workload based on the scheduling reuse relationship. The reuse neighborhood includes the source workload and the target workload to which the representative scheduling scheme of the source workload is allowed to migrate. The coverage gain is determined based on the call weight of the workload within the reuse neighborhood, the coverage radius penalty is determined based on the maximum fingerprint distance within the reuse neighborhood, the migration risk penalty is determined based on the weighted sum of migration risks within the reuse neighborhood, and the coverage score of the reuse neighborhood for each workload is determined by the coverage gain, coverage radius penalty and migration risk penalty. The reuse neighborhood with the highest coverage score is selected to form a coverage group, and within each coverage group, the workload with the lowest fingerprint distance to other members and the lowest overall cost of migration risk is selected as the representative workload.

7. The method for reusing operator revenue boundary fingerprint scheduling for large language models oriented to RVV according to claim 1, characterized in that, The complete scheduling search includes searching the scheduling action fields of candidate scheduling schemes. The scheduling action fields include data packaging method, circular rearrangement method, matrix partitioning method, vectorization dimension, vector element width, vector register group multiple, expansion factor, fusion method, number of threads, and tail processing method. When migrating the representative scheduling scheme to the target workload, the data packaging method, circular rearrangement method, vectorization dimension, vector element width, vector register group multiple, and fusion method in the representative scheduling scheme are inherited to form an initial migration scheduling scheme. Based on the matrix dimension, quantization format, and target hardware resource constraints of the target workload, the matrix block boundary, block size, expansion factor, number of threads, and tail processing method are recalculated.

8. A large language model operator revenue boundary fingerprint scheduling and reuse system for RVV, characterized in that, include: The revenue boundary fingerprint generation module is used to perform probe profiling on each workload in the workload set formed by inference execution records according to the target hardware configuration and probe scheduling template set, generate the revenue boundary fingerprint of the workload, and determine the main bottleneck type of the workload based on the revenue boundary fingerprint. The reuse coverage generation module is used to calculate the fingerprint distance and migration risk between workloads based on the revenue boundary fingerprint, form a scheduling reuse relationship based on the fingerprint distance, the migration risk and workload attributes, form a coverage group based on the scheduling reuse relationship, and determine representative workloads within the coverage group; wherein, the workload attributes include execution stage, operator type, quantization format and main bottleneck type; The scheduling scheme generation module is used to perform a complete scheduling search on the representative workload to obtain a representative scheduling scheme, migrate the representative scheduling scheme to the target workload within the corresponding coverage group, and form a migration scheduling scheme after inheriting stable fields, recalculating repairable fields, performing legality repair, and verifying correctness; it is also used to perform an exception point re-search on the target workload that fails to perform legality repair or correctness verification, and form a final scheduling scheme set based on the representative scheduling scheme, the migration scheduling scheme, and the re-search results of exception points.

9. An electronic device comprising a processor and a memory, characterized in that, The memory stores a computer program, which, when executed by the processor, implements the method for scheduling and reusing the revenue boundary fingerprint of a large language model operator for RVV as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the method for scheduling and reusing the revenue boundary fingerprint of a large language model operator for RVV as described in any one of claims 1 to 7.

Images