Artificial intelligence model training resource adaptive distribution system

Abstract

The invention belongs to the technical field of artificial intelligence, and discloses an artificial intelligence model training resource adaptive distribution system. The method comprises the following steps: acquiring and calculating graph structure data and hardware resource state data in real time, and calculating a data reuse rate and generating a candidate operator fusion scheme by constructing an operator execution time sequence constraint matrix and identifying an operator cluster of data locality characteristics; a resource competition hotspot prediction mechanism is introduced, memory bandwidth occupation fluctuation characteristics are analyzed, a resource conflict probability is calculated for a fusion scheme, and a dynamic balance optimization model of fusion income and resource conflicts is constructed. An optimal operator fusion decision sequence and a resource allocation strategy are generated through iterative solution, and accurate dynamic adjustment of computing resources in the training process is achieved. The training efficiency and the resource utilization rate are improved, the energy consumption is reduced, and the system stability is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and more particularly, to an artificial intelligence model training resource adaptive allocation system. BACKGROUND

[0002] Operator fusion is a core technology for deep learning compilation optimization, which reduces the overhead of intermediate result storage and data transfer by merging multiple consecutive computation operators into a single optimized unit. Mainstream deep learning frameworks such as TensorFlow, PyTorch and MXNet have integrated basic operator fusion functions, which can identify and merge common patterns such as convolution-activation, matrix multiplication-addition, etc. For example, inference optimization tools such as NVIDIA's TensorRT and Intel's OpenVINO can automatically identify fusible operator combinations and generate optimized code. However, these fusion strategies are mainly designed for inference scenarios, and most of them use static rule matching methods, lacking adaptability to the characteristics of the training process.

[0003] The traditional framework uses a static or simple heuristic rule-based operator fusion mechanism, which cannot perceive the dynamic change characteristics of the underlying hardware resources, resulting in frequent resource allocation conflicts in complex training environments. For example, in a multi-tenant GPU cluster environment, when the automatically fused operator group is allocated to a computing node whose memory bandwidth is close to saturation, the fusion decision originally intended to improve computational density instead triggers severe memory access competition, causing significant performance degradation. In particular, in the training process of large-scale Transformer models, operator fusion of attention mechanisms and feedforward network layers can improve theoretical computational efficiency, but due to the lack of a collaborative perception mechanism with memory hierarchy and PCIe bandwidth usage, the actual performance improvement is often low, and even negative optimization occurs when resource competition is intense. The existing technology also has the limitation of fixed operator fusion granularity and inability to identify dynamic data locality patterns. When the training batch size or model parallelism changes, the pre-determined fusion strategy cannot adapt to the new data flow characteristics, resulting in a sharp decline in cache hit rate. The existing system lacks the ability to model the complex interaction between operator fusion decisions and resource allocation decisions, and cannot dynamically predict and adjust during the training process, resulting in uneven resource allocation and waste of computational capacity in different training stages (such as warm-up period, stable period and convergence period).

[0004] In view of this, the present application proposes an artificial intelligence model training resource adaptive allocation system to solve the above problems. SUMMARY

[0005] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purpose, the present application provides the following technical scheme: an artificial intelligence model training resource adaptive allocation system, comprising:

[0006] a data acquisition module configured to acquire computation graph structure data of a to-be-trained model and real-time hardware resource state data;

[0007] a constraint matrix construction module configured to construct an operator execution timing constraint matrix according to data dependency relationships between operators in the computation graph structure data;

[0008] a data reuse rate calculation module configured to perform multi-granularity decomposition on the computation graph structure data, identify an operator cluster having a data locality feature, and calculate a data reuse rate within each operator cluster;

[0009] a fusion scheme generation module configured to generate a candidate operator fusion scheme set based on the data reuse rate and the operator execution timing constraint matrix;

[0010] a competition hotspot prediction module configured to predict a resource competition hotspot area in a future time window by analyzing memory bandwidth occupancy fluctuation characteristics in the real-time hardware resource state data;

[0011] a conflict probability calculation module configured to calculate a resource conflict probability for each scheme in the candidate operator fusion scheme set according to a spatiotemporal distribution feature of the resource competition hotspot area;

[0012] an optimization model construction module configured to construct a collaborative optimization model of operator fusion and resource allocation based on the resource conflict probability, the collaborative optimization model including a dynamic balance mechanism of fusion benefits and resource conflicts;

[0013] a strategy generation module configured to generate an operator fusion decision sequence and a corresponding resource allocation strategy by iteratively solving the collaborative optimization model;

[0014] a resource configuration adjustment module configured to dynamically adjust a computing resource configuration in a model training process according to the operator fusion decision sequence and the resource allocation strategy, to realize conflict-free collaboration of operator fusion and resource allocation.

[0015] Preferably, the multi-granularity decomposition on the computation graph structure data, the identification of the operator cluster having the data locality feature, and the calculation of the data reuse rate within each operator cluster include:

[0016] the computation graph structure data is represented as a directed acyclic graph, where a node represents an operator and an edge represents a data flow;

[0017] the directed acyclic graph is subjected to a depth-first traversal, and a memory address mapping relationship between an input tensor and an output tensor of each operator is recorded;

[0018] a memory overlap degree of a shared tensor between adjacent operators is calculated, and the memory overlap degree is determined by a ratio of a shared tensor size to a total tensor size;

[0019] adjacent operators with memory overlap greater than a preset overlap threshold are classified as initial operator pairs;

[0020] initial operator pairs with transitive data locality are expanded into operator clusters by a transitive closure algorithm;

[0021] For each operator cluster, data access records of input and output tensors of operators in the cluster are extracted;

[0022] The proportion of tensors with repeated references in the data access records is analyzed, and the ratio of the reuse times of the tensors in the cluster to the total tensor access times is calculated;

[0023] Based on the ratio, combined with the execution frequency of the operators in the operator cluster, the weighted data reuse rate of each operator cluster is calculated; the weighted data reuse rate is normalized to obtain the data reuse rate of the corresponding operator cluster.

[0024] Preferably, based on the data reuse rate and the operator execution timing constraint matrix, a candidate operator fusion scheme set is generated, including:

[0025] According to the operator execution timing constraint matrix, a parallel execution relationship graph between operators is constructed;

[0026] In the parallel execution relationship graph, identify operator combinations without data dependency conflicts;

[0027] For each operator combination, calculate the theoretical calculation density improvement ratio after fusion;

[0028] The weighted sum of the theoretical calculation density improvement ratio and the data reuse rate is taken as the fusion benefit index;

[0029] Select operator combinations with fusion benefit index greater than a preset benefit threshold to form a candidate operator fusion scheme set.

[0030] Preferably, by analyzing the memory bandwidth occupancy rate fluctuation characteristics in the hardware resource real-time state data, the resource competition hotspot area in the future time window is predicted, including:

[0031] The memory bandwidth occupancy rate is decomposed into time series, and the trend component, periodic component and random component are extracted;

[0032] Based on the frequency spectrum analysis result of the periodic component, the periodic mode of resource occupation is identified;

[0033] According to the slope change of the trend component, the mutation point of resource occupation is detected;

[0034] The periodic patterns before and after the mutation point are clustered and analyzed to obtain a typical resource competition pattern;

[0035] Based on the similarity of the typical resource competition pattern and the current resource state, the resource competition hotspot area is predicted.

[0036] Preferably, the resource conflict probability is calculated for each scheme in the candidate operator fusion scheme set according to the spatiotemporal distribution characteristics of the resource competition hotspot area, including:

[0037] The resource competition hotspot area is mapped to a time-resource two-dimensional space to construct a hotspot distribution matrix;

[0038] For each candidate operator fusion scheme, the resource occupation spatiotemporal trajectory during execution is estimated;

[0039] The overlapping area of the resource occupation spatiotemporal trajectory and the hotspot distribution matrix is calculated;

[0040] According to the ratio of the overlapping area to the total area of the resource occupation spatiotemporal trajectory, the initial conflict probability is determined;

[0041] The initial conflict probability is Bayes corrected based on historical execution data to obtain the resource conflict probability.

[0042] Preferably, a collaborative optimization model of operator fusion and resource allocation is constructed based on the resource conflict probability, including:

[0043] An optimization function with the goal of maximizing training throughput is constructed, and the optimization function includes an operator fusion benefit term and a resource conflict penalty term;

[0044] The resource conflict probability is introduced as a weight factor into the resource conflict penalty term;

[0045] The granularity constraint condition of operator fusion is set to limit the upper limit of the number of operators fused at a time;

[0046] The capacity constraint condition of resource allocation is added to ensure that the allocated resources do not exceed the physical limit of hardware;

[0047] The optimization function and the constraint condition are combined to form a collaborative optimization model.

[0048] Preferably, the operator fusion decision sequence and the corresponding resource allocation strategy are generated by iteratively solving the collaborative optimization model, including:

[0049] The operator fusion state vector and the resource allocation vector are initialized;

[0050] In each iteration, the gradient information of the collaborative optimization model under the current state is calculated;

[0051] Based on the gradient information, the operator fusion state is updated using an adaptive learning rate;

[0052] According to the updated operator fusion state, the resource demand distribution is recalculated;

[0053] The resource allocation subproblem is solved using the Lagrange multiplier method to obtain the resource allocation scheme for the current iteration;

[0054] Determine whether the change in the objective function value between two consecutive iterations is less than the convergence threshold. If so, output the operator fusion decision sequence and resource allocation strategy.

[0055] Preferably, the initial pair of operators with transitive data locality is expanded into a family of operators using a transitive closure algorithm, including:

[0056] Construct an adjacency matrix for the initial operator pairs, where the matrix elements indicate whether there are shared operators between the operator pairs;

[0057] Perform matrix exponentiation on the adjacency matrix to calculate the k-step reachability matrix;

[0058] When the reachability matrix no longer changes in k steps, the transitive closure matrix is ​​obtained.

[0059] Based on the connected components of the transitive closure matrix, operator clusters are partitioned;

[0060] For each operator cluster, the ratio of the total internal data transmission volume to the total external data transmission volume is calculated and used as an index of cluster aggregation degree.

[0061] Preferably, based on the similarity between typical resource competition patterns and the current resource status, resource competition hotspots are predicted, including:

[0062] Extract multi-dimensional feature vectors of the current resource status, including the occupancy rate of various resources, the rate of change of occupancy rate, and the occupancy distribution entropy;

[0063] Calculate the Euclidean distance between the multidimensional feature vector and the feature center of each typical resource competition pattern;

[0064] Use the Gaussian kernel function to convert the Euclidean distance into a similarity score;

[0065] Select the typical resource competition pattern with the highest similarity score as the matching pattern;

[0066] Based on the historical evolution of matching patterns, we can infer the distribution of resource competition hotspots within future time windows.

[0067] Preferably, dynamically adjusting the allocation of computational resources during model training includes:

[0068] Analyze the operator fusion decision sequence to extract the timing and parameters of the operator fusion operation;

[0069] Based on the resource allocation strategy, the real-time availability of hardware resources is mapped to the operator requirements, and the mapped operator fusion operation is obtained.

[0070] In the model training loop, the mapped operator fusion operation is applied to merge the corresponding computational resource allocation;

[0071] Real-time monitoring of resource utilization fluctuations during training, and detection of potential conflicts based on resource utilization fluctuations;

[0072] Based on the monitoring results of potential conflicts, update the parameters of resource configuration to ensure the continuity of operator execution and overall training efficiency.

[0073] The technical effects and advantages of the adaptive allocation system for training resources of artificial intelligence models in this invention are as follows:

[0074] This invention shortens the model iteration cycle, enabling researchers to explore more complex model architectures or handle larger datasets under the same hardware conditions. The stability of the training process is improved, effectively eliminating common performance fluctuations and unpredictable resource contention issues. In multi-tenant shared computing environments, the dynamic adaptability of this invention ensures efficient collaboration among training tasks, improving the throughput and quality of service of the computing cluster. For large-scale distributed training scenarios, the collaborative optimization mechanism effectively alleviates communication bottlenecks and improves the scalability of model parallelism and data parallelism strategies. The adaptive nature of this invention allows it to adapt to various hardware platforms, from edge devices to supercomputing centers. This improvement in training efficiency directly accelerates the AI ​​innovation cycle, enabling researchers to more quickly validate hypotheses and iterate model designs. Attached Figure Description

[0075] Figure 1 This is a schematic diagram of the adaptive allocation system for training resources of the artificial intelligence model according to the present invention. Detailed Implementation

[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] This application provides an adaptive allocation system for artificial intelligence model training resources. The execution entities of the system include, but are not limited to, AI training servers, distributed computing clusters, edge computing devices, high-performance computing centers, etc., which can be regarded as general computing nodes of this application. The system controller includes, but is not limited to, at least one of the following: a training scheduler, a resource manager, and an adaptive optimization controller.

[0078] Please see Figure 1In this embodiment of the invention, the artificial intelligence model training resource adaptive allocation system includes: a data acquisition module, a constraint matrix construction module, a data reuse rate calculation module, a fusion scheme generation module, a competition hotspot prediction module, a conflict probability calculation module, an optimization model construction module, a strategy generation module, and a resource configuration adjustment module; the modules are connected to each other via wired and / or wireless means to realize data transmission between modules.

[0079] The data acquisition module is used to acquire the computational graph structure data and real-time hardware resource status data of the model to be trained. The computational graph structure data includes key information such as the type of operators in the model, computational complexity, data dependencies, and memory access patterns, which is extracted in real time by the model parsing engine. The real-time hardware resource status data includes metrics such as processor utilization, memory bandwidth usage, cache hit rate, and energy consumption levels, which are continuously collected by the hardware monitoring module. This data provides a foundation for formulating resource allocation strategies, ensuring the targeted and effective optimization.

[0080] The constraint matrix construction module is used to construct an operator execution timing constraint matrix based on the data dependencies between operators in the computation graph structure data. This matrix quantifies the execution order constraints and timing dependency strength between operators, reflecting the parallel and sequential execution relationships in the computation process. The operator execution timing constraint matrix is ​​derived by analyzing the data flow and control dependencies in the computation graph, providing a decision-making basis for subsequent operator fusion optimization. High constraint values ​​in the matrix represent strong dependencies, and the corresponding operator sequences must be executed in a specific order to ensure the correctness of the computation results.

[0081] The data reuse rate calculation module performs multi-granularity decomposition on computation graph structure data, identifies operator clusters with data locality characteristics, and calculates the data reuse rate within each operator cluster. Multi-granularity decomposition is an adaptive partitioning technique that identifies operator combinations with high internal data sharing based on operator characteristics and data access patterns. By analyzing the memory overlap and data access records between adjacent operators, the system identifies operator clusters with strong data locality and calculates the data reuse rate within each cluster, providing optimization targets for operator fusion and improving cache utilization efficiency.

[0082] The fusion scheme generation module generates a set of candidate operator fusion schemes based on data reuse rate and operator execution timing constraint matrices. This set includes various possible operator combination strategies, each with different performance characteristics and resource requirements. The system evaluates the computational density improvement and data reuse benefits of operator combinations, identifying those with high fusion gains and using them as candidate schemes. This provides options for subsequent optimization, balancing computational efficiency and resource consumption.

[0083] The resource contention hotspot prediction module analyzes the fluctuation characteristics of memory bandwidth utilization in real-time hardware resource status data to predict resource contention hotspots within future time windows. Resource contention hotspots refer to the spatiotemporal locations where resource contention may occur during training, directly impacting training performance and stability. The system performs time-series analysis on memory bandwidth utilization, extracts trend and periodic features, identifies typical resource contention patterns, and predicts future hotspot distributions based on the current state, providing forward-looking guidance for resource allocation and avoiding performance bottlenecks.

[0084] The conflict probability calculation module calculates the resource conflict probability for each scheme in the candidate operator fusion scheme set based on the spatiotemporal distribution characteristics of resource contention hotspots. The resource conflict probability quantifies the likelihood of a fusion scheme encountering resource contention with a predicted hotspot during execution, serving as a crucial indicator for evaluating scheme feasibility. The calculation process considers the spatial overlap between the fusion scheme's resource demand trajectory and the hotspots, generating an accurate conflict risk assessment to help the system select the most suitable fusion strategy for the current hardware state.

[0085] The optimization model building module is used to construct a collaborative optimization model for operator fusion and resource allocation based on resource conflict probabilities. This collaborative optimization model includes a dynamic balance mechanism between fusion benefits and resource conflict. The model uses maximizing training throughput as the objective function, while simultaneously considering the performance improvements brought by operator fusion and the potential risks of resource conflict, achieving a dynamic balance between the two. The model design includes constraints on the granularity of operator fusion and the capacity limitations of resource allocation, ensuring that the generated optimization scheme can improve performance while maintaining system stability.

[0086] The strategy generation module generates operator fusion decision sequences and corresponding resource allocation strategies by iteratively solving a collaborative optimization model. The solution process employs an adaptive optimization algorithm, starting from an initial state and gradually adjusting the operator fusion state and resource allocation scheme until convergence to the optimal solution. Each iteration calculates the gradient information of the objective function based on the current state and uses the Lagrange multiplier method to handle resource constraints, ultimately outputting a complete decision sequence and allocation strategy to provide precise guidance for actual execution.

[0087] Based on the operator fusion decision sequence and resource allocation strategy, the system dynamically adjusts the computational resource configuration during model training to achieve conflict-free collaboration between operator fusion and resource allocation. Dynamic adjustment is the execution phase of the system, translating the optimization plan into practical operations. The system analyzes the decision sequence, maps the fusion operations to hardware resources, and monitors resource utilization in real time during training, adjusting configuration parameters as needed to ensure high training efficiency and stability, achieving the goal of optimal resource utilization.

[0088] In this embodiment of the invention, the detailed implementation steps for constructing the operator execution timing constraint matrix include:

[0089] The computation graph structure data is parsed into an operator dependency graph, where nodes represent operators and edges represent data dependencies. The operator dependency graph is an abstract representation of the computation graph, intuitively showing the execution constraints between operators. The parsing process employs graphical analysis techniques to identify node attributes and edge connections in the computation graph, generating a dependency graph with a clear topological structure. The dependency graph retains the key dependency information of the original computation graph while simplifying non-critical paths, facilitating subsequent analysis and optimization.

[0090] The edges in the operator dependency graph are analyzed to extract the input and output dependencies for each operator. Input-output dependencies are fundamental to determining the execution order of operators and directly impact the design of the scheduling strategy. The analysis process employs a graph traversal algorithm, scanning each operator node to identify its predecessor and successor nodes and recording the corresponding dependency types and strengths. This step generates a complete list of operator dependencies, providing a data foundation for subsequent time-series constraint calculations.

[0091] The topological sorting of the operator dependency graph is calculated to determine the execution order constraints between operators. Topological sorting is a linear sorting method that guarantees dependency satisfaction, ensuring that each operator only begins execution after all its input dependent operators have been executed. The calculation process uses the Kahn algorithm, starting from the source node with no input dependencies, and progressively removing processed nodes and their corresponding edges to generate one or more valid execution sequences. These sequences reflect the sequential relationships between operators and are an important basis for constructing the temporal constraint matrix.

[0092] Based on execution order constraints, an operator execution timing constraint matrix is ​​constructed, where matrix elements represent the dependency strength and timing delay between operators. The timing constraint matrix is ​​a two-dimensional representation that intuitively displays the dependencies between all operator pairs. The construction process calculates the timing constraint strength between each pair of operators based on the topological sorting result and dependency type, considering the necessity and urgency of data dependencies. Matrix elements are calculated using the following formula:

[0093]

[0094] Among them, TM ij D represents the temporal constraint strength between operators i and j. ij Let C be the data dependency between operators i and j, Dmax be the maximum data dependency, and C be the maximum data dependency. ij To control the degree of dependency, Cmax is the maximum degree of control dependency, and β is the balance factor between data dependency and control dependency, with a value range of [0,1]. This formula comprehensively considers the influence of data flow and control flow, generating a more accurate representation of time-series constraints.

[0095] The timing constraint matrix for operator execution is normalized to ensure that the matrix values ​​conform to a preset timing range. Normalization eliminates scale differences between different types of operators, making the constraint values ​​comparable and consistent. The process employs a Min-Max normalization method, mapping matrix elements to the [0,1] interval, where 0 represents no dependency and 1 represents the strongest constraint. The normalized matrix is ​​easier for subsequent optimization algorithms to process, ensuring the rationality and effectiveness of operator scheduling.

[0096] In this embodiment of the invention, the detailed implementation steps for performing multi-granularity decomposition on computation graph structure data, identifying operator clusters with data locality characteristics, and calculating the data reuse rate within each operator cluster include:

[0097] Representing computational graph structure data as a directed acyclic graph (DAG) allows us to use nodes to represent operators and edges to represent data flows. DAGs are a standard method for representing computational models, clearly expressing the data transfer relationships between operators. The representation process preserves the complete topological structure and data dependencies of the original computational graph, while adding auxiliary information such as operator attributes and edge weights to facilitate subsequent locality analysis. Edge weights in the graph typically represent the amount of data transferred, directly reflecting the strength of data interaction between operators.

[0098] A depth-first search (DFS) traversal is performed on the directed acyclic graph (DAG), recording the memory address mapping between the input and output tensors of each operator. This memory address mapping is fundamental for identifying data reuse opportunities and directly reflects the distribution of tensors in physical storage. The traversal process employs an improved DFS algorithm, visiting each node in the graph and recording the memory address range and size information of the input and output tensors processed. This step generates a complete memory access map, providing a basis for identifying shared data regions.

[0099] The memory overlap of shared tensors between adjacent operators is calculated. Memory overlap is determined by the ratio of the size of the shared tensor to the total size of the tensors. Memory overlap is a direct indicator for evaluating data locality between operators, reflecting potential data reuse opportunities. The calculation formula is:

[0100]

[0101] Among them, MO ij S represents the memory overlap between operators i and j. ij S is the size of the tensor memory shared by operators i and j. i and S j These represent the total tensor memory size processed by operators i and j, respectively. This metric ranges from [0,1], with larger values ​​indicating higher overlap and stronger data locality. This calculation considers the data sharing characteristics between operators, providing a quantitative basis for selecting initial operator pairs.

[0102] Adjacent operators with a memory overlap greater than a preset overlap threshold are grouped into initial operator pairs. These initial operator pairs are the basic units for building operator clusters, representing combinations of operators with significant data-sharing characteristics. The selection process sets an appropriate overlap threshold (typically 0.3-0.5) to ensure that the selected operator pairs have sufficient data locality. These initial operator pairs will be expanded into complete operator clusters in subsequent steps, achieving broader data reuse optimization.

[0103] The transitive closure algorithm expands initial operator pairs with transitive data locality into operator clusters. Transitive closure is an effective method for discovering indirect relationships, identifying groups of operators with transitive data sharing. The algorithm constructs an adjacency matrix of the initial operator pairs, calculates the transitive closure through matrix exponentiation, and identifies connected components as operator clusters. This step expands local pairwise relationships into global group relationships, forming operator clusters with high internal data sharing, providing a foundation for data reuse rate calculations.

[0104] For each operator cluster, data access records for the input and output tensors of the operators within the cluster are extracted. These data access records are crucial information for evaluating the effectiveness of actual data reuse, containing the time, frequency, and method of tensor access. The extraction process analyzes the operator's execution trajectory, recording the number of times and patterns each tensor is read and written, forming a complete spatiotemporal sequence of access. These records detail the data flow path within the operator cluster, providing a precise basis for reusability calculations.

[0105] The proportion of tensors repeatedly referenced in the data access records is analyzed, and the ratio of tensor reuse counts within a cluster to total tensor access counts is calculated. This ratio directly reflects the degree of data reuse within the operator cluster and is a core indicator for evaluating the potential for local optimization. The calculation process involves counting the total number of times each tensor is accessed and the number of times it is repeatedly accessed by different operators, and the ratio of these two values ​​yields the initial reuse rate. This initial reuse rate reflects the basic data sharing characteristics of the operator cluster and needs further weighting based on execution frequency.

[0106] Based on the ratio and the execution frequency of operators within the operator cluster, the weighted data reuse rate of each operator cluster is calculated. The weighted data reuse rate is then normalized to obtain the data reuse rate of the corresponding operator cluster. The weighting process considers the differences in the impact of operator execution frequency on data reuse, making the reuse rate more accurately reflect the data access patterns during actual training. The calculation formula is:

[0107]

[0108] Among them, f i Let r be the execution frequency of operator i. i DR represents the original data reuse rate of operator i. cLet be the weighted data reuse rate of operator cluster c. Normalization maps the weighted reuse rate to the [0,1] interval, which facilitates comparison and sorting among different operator clusters and provides an important basis for subsequent operator fusion decisions.

[0109] In this embodiment of the invention, the detailed implementation steps for generating a set of candidate operator fusion schemes based on data reuse rate and operator execution timing constraint matrix include:

[0110] Based on the operator execution timing constraint matrix, a parallelizable execution relationship graph is constructed among operators. This graph is a special graph structure that represents which operators can be executed concurrently without violating dependency constraints. The construction process is based on the timing constraint matrix; when the matrix element value is less than a preset threshold, operators i and j are considered to be able to execute in parallel. This graph provides information on the degrees of freedom for operator scheduling, reveals the potential parallel optimization space, and provides a basis for feasibility analysis of operator fusion.

[0111] In the parallelizable relational graph, operator combinations without data dependency conflicts are identified. The absence of data dependency conflicts is a necessary condition for operator fusion, ensuring the correctness of the fused computational result. The identification process employs subgraph matching technology, searching the parallelizable relational graph for operator groups that satisfy specific patterns, excluding combinations with conflicts such as write-after-read, read-after-write, and write-after-write. This step filters out candidate operator groups that are safe for fusion, providing a legitimate candidate set for subsequent benefit evaluation.

[0112] For each operator combination, calculate the theoretical increase in computational density after fusion. This increase in computational density is one of the main benefits of operator fusion, reflecting the degree of optimization in the ratio of computation to data transmission. The calculation formula is as follows:

[0113] Where Cfused represents the computational cost after fusion, Tfused represents the data transfer cost after fusion, Ci represents the computational cost of a single operator, Ti represents the data transfer cost of a single operator, and CDI represents the percentage increase in computational density after fusion. A positive value indicates that fusion has led to an increase in computational density, and a larger value indicates a more significant increase. This calculation takes into account the impact of fusion on the balance between computation and memory access, providing a quantitative assessment of the benefits of fusion.

[0114] The weighted sum of the theoretical computational density improvement ratio and data reuse rate is used as the fusion benefit index. This fusion benefit index comprehensively considers both computational efficiency improvement and data locality optimization, thus fully evaluating the potential value of the fusion solution. The calculation formula is:

[0115] FB i =α×CDI i +(1-α)×DR i ;

[0116] Among them, CDIi DR represents the computational density improvement ratio of scheme i. i Let α represent the data reuse rate of the operator cluster involved in scheme i, and let α be a balancing factor with a value range of [0,1], which is dynamically adjusted according to the characteristics of the training task. This fusion index unifies the optimization effects of different dimensions into a single score, facilitating comparison and selection among schemes.

[0117] Operator combinations with fusion benefit metrics exceeding a preset benefit threshold are selected to form a candidate operator fusion scheme set. The preset benefit threshold is the criterion for selecting valuable fusion schemes, ensuring that the selected schemes can bring significant performance improvements. The threshold setting is usually based on historical optimization experience and can be dynamically adjusted according to different models and hardware platforms. This step generates the final candidate fusion scheme set, which includes multiple operator combination schemes with high potential benefits, providing selection space for subsequent optimization decisions.

[0118] In this embodiment of the invention, the detailed implementation steps for predicting resource contention hotspots within a future time window by analyzing the memory bandwidth utilization fluctuation characteristics in real-time hardware resource status data include:

[0119] Time series decomposition of memory bandwidth utilization is performed to extract trend, periodic, and random components. Time series decomposition is a fundamental step in understanding bandwidth utilization patterns, enabling the identification of variation patterns across different time scales. The decomposition employs the STL (Seasonal and Trend decomposition using Loess) method, splitting the original time series into a trend component reflecting long-term trends, a periodic component reflecting periodic changes, and a residual component reflecting random fluctuations. This decomposition provides a multi-layered perspective on bandwidth utilization, facilitating subsequent pattern recognition and prediction.

[0120] Based on the spectral analysis results of periodic components, periodic patterns of resource occupancy are identified. These periodic patterns reflect the regular changes in system resource usage and are an important basis for predicting future occupancy. The analysis process uses Fast Fourier Transform (FFT) technology to calculate the spectral density of the periodic components and identify the main frequency components and their intensities. The main frequencies usually correspond to regular behaviors such as batch processing cycles and model layer switching during training. These identified periodic patterns provide a temporal framework for subsequent predictions.

[0121] Based on the slope changes of the trend component, abrupt changes in resource usage are detected. These abrupt changes signify significant changes in the system's resource usage pattern, typically corresponding to events such as training phase transitions or concurrent task interference. The detection employs the CUSUM (cumulative sum) algorithm or the Bayesian change point detection method to identify moments in the trend component where the slope change exceeds a statistical threshold. These abrupt changes segment the time series into intervals with different characteristics, enabling the prediction model to adapt to resource usage features at different stages.

[0122] Cluster analysis of periodic patterns before and after the mutation point yields typical resource competition patterns. These typical competition patterns represent resource usage scenarios, summarizing the resource competition characteristics of the system under different states. Cluster analysis employs K-means or hierarchical clustering algorithms to group periodic patterns with similar spectral characteristics and intensity distributions into the same class. Each cluster center represents a typical resource competition pattern, encompassing specific temporal distribution and intensity characteristics, providing a template reference for hotspot prediction.

[0123] Based on the similarity between typical resource competition patterns and the current resource state, resource competition hotspots are predicted. The prediction process first extracts the feature vector of the current resource state, then calculates the similarity with each typical pattern, selecting the best-matching pattern as the basis for prediction. According to the historical evolution trajectory of the matching patterns, the possible states of resource use within future time windows are inferred, and spatiotemporal regions where resource occupancy may exceed a threshold are identified and marked as predicted resource competition hotspots. This prediction result provides forward-looking guidance for subsequent conflict assessment and optimization decisions.

[0124] In this embodiment of the invention, the detailed implementation steps for calculating the resource conflict probability for each scheme in the candidate operator fusion scheme set based on the spatiotemporal distribution characteristics of resource competition hotspots include:

[0125] Resource competition hotspots are mapped to a two-dimensional time-resource space to construct a hotspot distribution matrix. This matrix provides a spatial representation of resource competition, intuitively displaying the degree of competition for various resources at different points in time. The mapping process transforms predicted hotspots into a two-dimensional matrix representation of time and resource type, with matrix element values ​​representing the competition intensity at that spatiotemporal point. This representation method unifies the time and resource dimensions, facilitating subsequent comparison and analysis with operator execution trajectories.

[0126] For each candidate operator fusion scheme, its spatiotemporal resource consumption trajectory during execution is estimated. This trajectory describes the changes in resource requirements during the fusion scheme's execution and serves as fundamental information for assessing potential conflicts. The estimation process, based on operator characteristics and historical execution data, combined with the computational characteristics of the fused scheme, generates three-dimensional trajectory data containing execution time, resource type, and resource consumption. This trajectory reflects the resource demand pattern of the fusion scheme, providing precise input for conflict analysis.

[0127] The overlap area between the spatiotemporal trajectory of resource occupancy and the hotspot distribution matrix is ​​calculated. The overlap area directly reflects the degree of conflict between the fusion scheme and the predicted hotspots, and is a key indicator for assessing conflict risk. The calculation employs a spatial integration method, integrating the trajectory and hotspot matrix over the shared spatiotemporal coverage area to obtain the total overlap. The calculation considers the product effect of occupancy intensity and hotspot intensity; the higher the intensity of the overlap area, the greater the potential conflict risk.

[0128] The initial conflict probability is determined based on the ratio of the overlapping area to the total area of ​​the resource occupation spatiotemporal trajectory. This initial conflict probability is a conflict risk estimate based on pure spatial analysis, reflecting the degree of overlap between the resource requirements of the fusion scheme and system hotspots. The value ranges from [0,1], with a higher value indicating a higher probability of conflict. This preliminary estimate considers spatial overlap factors but has not yet been adjusted based on historical execution experience.

[0129] The initial conflict probability (ICP) is corrected using Bayesian methods based on historical execution data to obtain the resource conflict probability. Bayesian correction is a method that combines prior knowledge to optimize probability estimation, improving prediction accuracy based on historical observations. The correction process applies Bayes' theorem:

[0130]

[0131] Where P(C|O) is the conditional probability of an actual conflict occurring after an overlap is observed in historical data, P(O) is the probability of an overlap being observed in historical data, and CP is the resource conflict probability. This correction takes into account historical execution experience, making the conflict probability estimate more accurate and reliable, and providing a precise risk assessment for subsequent optimization decisions.

[0132] In this embodiment of the invention, the detailed implementation steps for constructing a collaborative optimization model for operator fusion and resource allocation based on resource conflict probability include:

[0133] An optimization function is constructed with the goal of maximizing training throughput. This function includes an operator fusion benefit term and a resource conflict penalty term. The optimization function is the core guiding principle of the decision-making process, balancing the two key objectives of performance improvement and system stability. The construction process first defines the mathematical expression of training throughput, then introduces the two influencing factors of fusion benefit and conflict penalty to form a complete objective function. The fusion benefit term represents the performance improvement brought about by fusion, and the conflict penalty term represents the performance loss caused by potential conflicts; together, they determine the final system performance.

[0134] Resource conflict probability is incorporated as a weighting factor into the resource conflict penalty term. Conflict probability is a key indicator for assessing the risk of fusion schemes and directly affects the strength of the penalty term. The induction process multiplies the conflict probability by the severity of the conflict consequences, quantifying the expected loss of potential conflicts. This design results in stronger penalties for schemes with high conflict probabilities, guiding the optimization process to avoid high-risk areas and ensuring the stability and reliability of the system.

[0135] Set granular constraints for operator fusion to limit the upper limit of the number of operators in a single fusion. Granular constraints are a safety mechanism to prevent over-fusion, ensuring the practicality and manageability of the optimization results. The constraints specify the maximum number of operators in a single fusion group, usually determined based on hardware characteristics and compiler capabilities. This constraint prevents problems such as increased compilation complexity and decreased scheduling flexibility caused by excessively large fusion groups, ensuring the practical feasibility of the optimization results.

[0136] Add capacity constraints to resource allocation to ensure that allocated resources do not exceed hardware physical limitations. Capacity constraints are a fundamental principle of resource allocation, preventing system crashes or performance degradation caused by over-allocation. Constraints include multiple dimensions such as the number of processor cores, memory capacity, and bandwidth limits, ensuring that at any given time, the total resource requirements of all tasks do not exceed the system's available resources. This constraint guarantees the feasibility of the optimization scheme in a real-world environment, avoiding situations where the theoretically optimal solution is impractical.

[0137] By combining the optimization function and constraints, a collaborative optimization model is formed. The collaborative optimization model is a complete mathematical description, including the objective function and various constraints, formally expressing all elements of the optimization problem. The model takes the following form:

[0138] Maximize: F(X,Y)=∑(FB) i ×X i )-∑(CP j ×S j ×Y j );

[0139] Constraints:

[0140] 1.∑X i≤Nmax (fusion granularity constraint);

[0141] 2.∑(R k ×Y k )≤Cmax(resource capacity constraint);

[0142] 3. Other logic and dependency constraints;

[0143] Where F(X,Y) is the objective function value, and X represents the fusion decision vector. i Indicates whether to select option i, Y represents the resource allocation vector, and FB i For the benefits of solution i integration, CP j Let S be the conflict probability of resource j. j Y represents the conflict severity of resource j. j R represents the allocation amount of resource j; k Y represents the unit demand for resource type k. k Let Nmax be the allocation amount or proportion of resource type k in the resource allocation vector Y, Nmax be the maximum number of fusion operators, and Cmax be the upper limit of resource capacity. This model comprehensively considers the balance between performance and stability, providing a complete framework for subsequent optimization solutions.

[0144] In this embodiment of the invention, the detailed implementation steps for generating operator fusion decision sequences and corresponding resource allocation strategies by iteratively solving the collaborative optimization model include:

[0145] Initialize the operator fusion state vector and resource allocation vector. The initial state is the starting point for optimization iterations, and a reasonable initial value can accelerate the convergence process. The initialization process usually adopts a heuristic strategy, assigning conservative initial values ​​to the fusion state vector (e.g., setting all values ​​to 0 to indicate no fusion) and uniform initial values ​​to the resource allocation vector (e.g., allocating resources proportionally to the operator's computational load). This initial state ensures the safety and rationality of the optimization starting point, providing a foundation for subsequent iterations.

[0146] In each iteration, the gradient information of the collaborative optimization model in the current state is calculated. The gradient information indicates the optimization direction and serves as a guide for iterative updates. The calculation process uses the partial derivatives of the objective function with respect to the fusion state and resource allocation vectors to obtain the optimization direction and suggested step size for the current state. For discrete variables such as fusion decisions, approximate gradients or differencing methods are used to evaluate the direction of change; for continuous variables such as resource allocation ratios, standard gradient calculation methods are used to obtain the accurate gradient. This step provides a theoretical basis for state updates, ensuring that iterations proceed in the direction of improving the objective function value.

[0147] Based on gradient information, the operator fusion state is updated using an adaptive learning rate. The update process is the core step in the optimization iteration, directly changing the system's decision state. The update employs gradient ascent (for maximization problems), adjusting the fusion state according to the gradient direction and the learning rate. An adaptive learning rate strategy is used, initially larger to accelerate convergence and then reduced to improve accuracy as the system approaches the optimal solution. For discrete fusion decisions, binarization is typically performed after the update, converting continuous values ​​into 0 / 1 decision variables. This step enables the gradual optimization of the fusion scheme, approximating the optimal decision combination.

[0148] Based on the updated operator fusion state, the resource demand distribution is recalculated. This distribution reflects the system's new demand patterns for various resources after the fusion decision change and serves as input for solving the resource allocation subproblem. The calculation process analyzes the resource demand characteristics of each fusion group based on the latest fusion state, including multiple dimensions such as computational requirements, memory requirements, and bandwidth requirements. This updated demand distribution reflects the resource preferences of the current fusion scheme, providing a precise basis for resource allocation optimization.

[0149] The resource allocation subproblem is solved using the Lagrange multiplier method to obtain the resource allocation scheme for the current iteration. The resource allocation subproblem is an optimization problem that seeks the optimal resource allocation strategy under the condition of a fixed fusion decision. The solution process involves constructing a Lagrange function, transforming resource constraints into penalty terms, solving an unconstrained optimization problem, and finally verifying the optimality and feasibility of the results using KKT conditions. The Lagrange multiplier method can efficiently handle constrained optimization problems, providing accurate solutions for complex resource allocation scenarios. This step generates the optimal resource allocation scheme matching the current fusion state, completing the full update for a single iteration.

[0150] The system determines whether the change in the objective function value between two consecutive iterations is less than a convergence threshold. If so, it outputs the operator fusion decision sequence and resource allocation strategy. Convergence is the condition for terminating the iteration, ensuring that the optimization process stops when sufficient accuracy is achieved. The criterion is typically the relative rate of change of the objective function. When the rate of change in multiple consecutive iterations is less than a preset threshold (e.g., 0.1%), the optimization is considered to have converged to a local or global optimum. After convergence, the system outputs the final operator fusion decision sequence, containing the fusion decision for each pair of fusionable operators, and the corresponding resource allocation strategy, including the resource allocation ratio for each operator or fusion group. These outputs directly guide subsequent training processes, optimizing resource utilization and improving training efficiency.

[0151] In this embodiment of the invention, the detailed implementation steps for expanding an initial pair of operators with transitive data locality into a family of operators using the transitive closure algorithm include:

[0152] Construct an adjacency matrix for the initial operator pairs, where each element indicates whether the operator pairs share an operator. The adjacency matrix is ​​a standard method for representing node relationships in graph theory, visually illustrating the connections between operator pairs. The construction process analyzes the initial set of operator pairs; if two operator pairs share an operator, a 1 is marked at the corresponding matrix position, otherwise a 0 is marked. This matrix provides a complete view of the relationships between operator pairs and is the fundamental data structure for computing transitive closures.

[0153] Matrix exponentiation is performed on the adjacency matrix to calculate the k-step reachability matrix. The k-step reachability matrix indicates whether it is possible to reach another node through k or fewer intermediate nodes, reflecting a broader range of indirect relationships. The calculation uses matrix multiplication and logical OR operations, recursively defined as follows:

[0154] M 1 = M(original adjacency matrix);

[0155] M k =M (k-1) or (M) (k-1) ×M);

[0156] Here, "or" represents the logical OR operation, and "×" represents Boolean matrix multiplication. This calculation reveals the transitive connections between different operator pairs, providing a theoretical basis for identifying complete operator families.

[0157] When the reachability matrix no longer changes after k steps, the transitive closure matrix is ​​obtained. The transitive closure matrix represents whether there is a direct or indirect connection path between any two nodes, and is the mathematical expression of a complete relational network. The criterion is that the matrix is ​​exactly the same in two consecutive iterations, i.e., M... k =M (k+1) At this point, the matrix already contains all possible connection relationships. The calculation of the transitive closure matrix usually requires no more than n iterations (where n is the number of nodes), ensuring the finite convergence of the algorithm.

[0158] Based on the connected components of the transitive closure matrix, operator clusters are partitioned. Connected components are the largest mutually reachable subsets of nodes in the graph, corresponding to operator groups with transitive data locality. The partitioning process employs depth-first search or breadth-first search algorithms to identify independent connected components in the transitive closure matrix, with each connected component corresponding to an operator cluster. This step expands the initial pairwise relationships into complete group relationships, forming a set of operator clusters with high internal data sharing.

[0159] For each operator cluster, the ratio of total internal data transfer to total external data transfer is calculated as an intra-cluster aggregation index. This index reflects the strength of data locality within the operator cluster and is an important reference for evaluating cluster quality. The calculation process involves separately counting the data transfer volume between operators within the cluster and the data transfer volume between the cluster and external operators, and then calculating the ratio. A higher ratio indicates that intra-cluster data exchange is dominant, and the locality characteristic is more significant. This index can be used to further screen and optimize operator clusters, ensuring that the final operator clusters have significant data locality and reusability value.

[0160] In this embodiment of the invention, the detailed implementation steps for predicting resource competition hotspots based on the similarity between typical resource competition patterns and the current resource status include:

[0161] A multi-dimensional feature vector of the current resource status is extracted, including the occupancy rate of various resources, the rate of change of occupancy rate, and the occupancy distribution entropy. This multi-dimensional feature vector is a comprehensive numerical representation of the resource status, capturing both the static and dynamic characteristics of the system. The extraction process calculates features across multiple dimensions from hardware monitoring data, including occupancy rate reflecting the current load level, the rate of change reflecting the trend, and entropy values ​​reflecting the uniformity of distribution. These features collectively constitute a complete feature space describing the current resource status, providing input for subsequent similarity calculations.

[0162] Calculate the Euclidean distance between the multidimensional feature vector and the feature centers of each typical resource competition pattern. Euclidean distance is a standard method for measuring feature vector similarity, directly reflecting the degree of difference between the current state and the typical pattern. The calculation formula is the standard multidimensional Euclidean distance calculation. The distance calculation considers the differences in all feature dimensions, comprehensively assesses the overall similarity between states, and provides a quantitative basis for subsequent matching.

[0163] The Euclidean distance is converted into a similarity score using a Gaussian kernel function. The similarity score is a standardized similarity measure that facilitates comparison and ranking between different patterns. The conversion uses a Gaussian kernel function:

[0164]

[0165] Where D(x,y) is the Euclidean distance between feature vectors x and y, S(x,y) is the similarity score between feature vectors x and y, and σ is the bandwidth parameter of the kernel function, which controls the rate at which the similarity decays with distance. This transformation maps the Euclidean distance to the interval [0,1], with values ​​closer to 1 indicating higher similarity, providing a standardized scoring metric for pattern matching.

[0166] The most typical resource competition pattern with the highest similarity score is selected as the matching pattern. The matching pattern is a historical pattern most similar to the current state, used as a reference template for predicting future states. The selection process simply compares the similarity scores of all typical patterns and selects the one with the highest score as the matching result. When multiple patterns have similar scores, weighted voting or ensemble methods can be used to improve the robustness and accuracy of the matching.

[0167] Based on the historical evolution trajectory of matching patterns, the distribution of resource competition hotspots within future time windows is inferred. The prediction process is based on the past development patterns of matching patterns to infer the possible future evolution paths of the current state. Specific methods include time series extrapolation, application of pattern evolution rules, or statistical prediction models, generating resource usage predictions for future time windows based on historical data of matching patterns. The prediction results are then filtered through thresholds to identify spatiotemporal areas where resource occupancy may exceed safe levels, marking them as expected resource competition hotspots, providing early warning information and risk alerts for subsequent optimization decisions.

[0168] In this embodiment of the invention, the detailed implementation steps for dynamically adjusting the configuration of computing resources during model training include:

[0169] The decision sequence for operator fusion is parsed to extract the timing and parameters of the operator fusion operations. Decision sequence parsing is the first step in the execution phase, converting the output of the optimization model into specific operation instructions. The parsing process identifies each fusion decision in the decision sequence, extracts the relevant operator combinations, fusion order, and specific parameters, forming a structured operation description. These descriptions contain complete execution information, such as which operators need to be fused, when to perform the fusion, and what fusion strategy to use, providing clear guidance for subsequent mapping and execution.

[0170] Based on the resource allocation strategy, operator requirements are mapped to the real-time availability of hardware resources, resulting in the mapped operator fusion operation. Resource mapping is the process of converting logical resource requirements into physical resource allocation, ensuring the executability of operations in the actual environment. The mapping process checks the real-time status of current hardware resources, matches and adjusts the resource requirements in the operator fusion operation with available resources, and generates operation instructions containing specific execution details. When the resource status does not match expectations, the system will make appropriate adjustments to ensure that the operation can still be executed efficiently in a changing environment.

[0171] In the model training loop, the mapped operator fusion operation is applied to merge the corresponding computational resource allocations. This application process is the actual execution phase of the optimization strategy, directly impacting training performance and efficiency. During execution, the system activates the operator fusion operation according to a specified timing sequence, combining relevant operators into optimization units and allocating corresponding computational resources. This process involves multiple aspects, including scheduler configuration, memory management, and computational resource allocation, ensuring that the fusion operation can be seamlessly integrated into the training process to achieve the expected performance improvement.

[0172] The system monitors resource utilization fluctuations in real time during training and detects potential conflicts based on these fluctuations. Monitoring is the sensing link in closed-loop control, providing real-time feedback on the system status. The monitoring process continuously collects real-time data on key resource indicators, such as processor utilization, memory bandwidth usage, and cache hit rate, and calculates the fluctuation characteristics of these indicators. The system uses statistical methods to analyze fluctuation patterns, identify abnormal changes and potential resource conflicts, such as sudden drops in utilization or abnormally increased fluctuations, which may signal improper resource allocation or operator fusion conflicts.

[0173] Based on monitoring results of potential conflicts, resource configuration parameters are updated to ensure the continuity of operator execution and overall training efficiency. Parameter updating is the execution link of closed-loop control, adjusting system behavior according to monitoring feedback. The update process analyzes detected potential conflicts, determines their severity and scope of impact, and then formulates corresponding adjustment strategies. Adjustments may include reallocating resource ratios, temporarily reducing the priority of certain fusion operations, and dynamically adjusting execution timing, with the aim of eliminating or mitigating the impact of conflicts and maintaining the stability and efficiency of the training process. This dynamic adjustment mechanism enables the system to adapt to environmental changes and prediction errors, ensuring good performance and stability under various conditions.

[0174] This invention acquires real-time computational graph structure data and hardware resource status data of the model to be trained, constructs an accurate operator execution timing constraint matrix and operator cluster data reuse rate, and generates optimal operator fusion decision and resource allocation strategies by combining resource contention prediction and conflict probability assessment models. Through dynamic monitoring and parameter adjustment, it achieves efficient utilization of computational resources. It is highly adaptable, capable of optimizing training parameters in real time based on model structure characteristics and hardware status, significantly improving model training efficiency and reducing resource waste.

[0175] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0176] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0177] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An artificial intelligence model training resource self-adaptive allocation system, characterized in that, The method comprises the following steps: a data acquisition module is configured to acquire computational graph structure data of a to-be-trained model and real-time hardware resource state data; a constraint matrix construction module is configured to construct an operator execution timing constraint matrix according to a data dependency relationship between operators in the computational graph structure data; a data reuse rate calculation module is configured to perform multi-granularity decomposition on the computational graph structure data, identify an operator cluster having a data locality feature, and calculate a data reuse rate within each operator cluster; a fusion scheme generation module is configured to generate a candidate operator fusion scheme set based on the data reuse rate and the operator execution timing constraint matrix; a competition hotspot prediction module is configured to predict a resource competition hotspot area in a future time window by analyzing a memory bandwidth occupancy rate fluctuation feature in the real-time hardware resource state data; a conflict probability calculation module is configured to calculate a resource conflict probability for each scheme in the candidate operator fusion scheme set according to a space-time distribution feature of the resource competition hotspot area; an optimization model construction module is configured to construct a collaborative optimization model of operator fusion and resource allocation based on the resource conflict probability, wherein the collaborative optimization model comprises a dynamic balance mechanism of fusion benefits and resource conflicts; a strategy generation module is configured to generate an operator fusion decision sequence and a corresponding resource allocation strategy by iteratively solving the collaborative optimization model; a resource configuration adjustment module is configured to dynamically adjust a computing resource configuration in a model training process according to the operator fusion decision sequence and the resource allocation strategy. 2.The artificial intelligence model training resource self-adaptive allocation system of claim 1, wherein, The multi-granularity decomposition of the computational graph structure data, the identification of the operator cluster having the data locality feature, and the calculation of the data reuse rate within each operator cluster comprise the following steps: the computational graph structure data is represented as a directed acyclic graph, wherein a node represents an operator and an edge represents a data flow; a depth-first traversal is performed on the directed acyclic graph, and a memory address mapping relationship between an input tensor and an output tensor of each operator is recorded; a memory overlap degree of a shared tensor between adjacent operators is calculated, and the memory overlap degree is determined by a ratio of a shared tensor size to a total tensor size; adjacent operators with a memory overlap degree greater than a preset overlap threshold are grouped into an initial operator pair; the initial operator pair having a transitive data locality is expanded into the operator cluster by a transitive closure algorithm; for each operator cluster, data access records of input tensors and output tensors of operators within the cluster are extracted; a proportion of repeatedly referenced tensors in the data access records is analyzed, and a ratio of a reuse number of a cluster tensor to a total tensor access number is calculated; based on the ratio, a weighted data reuse rate of each operator cluster is calculated in combination with an execution frequency of operators within the operator cluster; and the weighted data reuse rate is normalized to obtain a data reuse rate of the corresponding operator cluster. 3.The artificial intelligence model training resource self-adaptive allocation system of claim 1, wherein, The generation of the candidate operator fusion scheme set based on the data reuse rate and the operator execution timing constraint matrix comprises the following steps: a parallel execution relationship graph between operators is constructed according to the operator execution timing constraint matrix; in the parallel execution relationship graph, an operator combination without a data dependency conflict is identified; for each operator combination, a theoretical calculation density improvement ratio after fusion is calculated; and The weighted sum of the theoretical calculation density promotion ratio and the data reuse rate is taken as a fusion benefit index; An operator combination with the fusion benefit index greater than a preset benefit threshold is selected to form the candidate operator fusion scheme set. 4.The artificial intelligence model training resource self-adaptive allocation system of claim 1, wherein, The method for predicting the resource competition hotspot region in the future time window by analyzing the memory bandwidth occupancy fluctuation characteristics in the hardware resource real-time state data comprises: Time series decomposition is performed on the memory bandwidth occupancy to extract a trend component, a periodic component and a random component; Based on the frequency spectrum analysis result of the periodic component, a periodic mode of resource occupancy is identified; Based on the slope change of the trend component, a mutation point of resource occupancy is detected; The periodic modes before and after the mutation point are subjected to cluster analysis to obtain a typical resource competition mode; Based on the similarity between the typical resource competition mode and the current resource state, the resource competition hotspot region is predicted. 5.The artificial intelligence model training resource self-adaptive allocation system of claim 1, wherein, The method for calculating a resource conflict probability for each scheme in the candidate operator fusion scheme set according to the spatiotemporal distribution characteristics of the resource competition hotspot region comprises: The resource competition hotspot region is mapped to a time-resource two-dimensional space to construct a hotspot distribution matrix; For each candidate operator fusion scheme, a resource occupancy spatiotemporal trajectory during execution is estimated; An overlapping area of the resource occupancy spatiotemporal trajectory and the hotspot distribution matrix is calculated; Based on the ratio of the overlapping area to the total area of the resource occupancy spatiotemporal trajectory, an initial conflict probability is determined; The initial conflict probability is subjected to Bayesian correction based on historical execution data to obtain the resource conflict probability. 6.The artificial intelligence model training resource self-adaptive allocation system of claim 1, wherein, The method for constructing a collaborative optimization model of operator fusion and resource allocation based on the resource conflict probability comprises: An optimization function with the maximum training throughput as the target is constructed, and the optimization function comprises an operator fusion benefit item and a resource conflict penalty item; The resource conflict probability is introduced as a weight factor into the resource conflict penalty item; A granularity constraint condition of operator fusion is set to limit the upper limit of the number of operators fused at a time; A capacity constraint condition of resource allocation is added to ensure that the allocated resources do not exceed the hardware physical limit; The optimization function and the constraint condition are combined to form the collaborative optimization model. 7.The artificial intelligence model training resource self-adaptive allocation system of claim 1, wherein, The method for generating an operator fusion decision sequence and a corresponding resource allocation strategy by iteratively solving the collaborative optimization model comprises: An operator fusion state vector and a resource allocation vector are initialized; In each iteration, gradient information of the collaborative optimization model in the current state is calculated; Based on the gradient information, the operator fusion state is updated using an adaptive learning rate; Based on the updated operator fusion state, the resource demand distribution is recalculated; A resource allocation sub-problem is solved by the Lagrange multiplier method to obtain a resource allocation scheme of the current iteration; Whether the change of the objective function value of adjacent two iterations is less than a convergence threshold is judged, and if yes, the operator fusion decision sequence and the resource allocation strategy are output. 8.The system of claim 2, wherein, The method for expanding the initial operator pair with transitive data locality into the operator cluster by a transitive closure algorithm comprises: An adjacency matrix of the initial operator pair is constructed, and a matrix element represents whether there is a shared operator between the operator pairs. performing matrix power operation on the adjacency matrix to obtain a k-step reachability matrix; obtaining a transitive closure matrix when the k-step reachability matrix no longer changes; dividing the operator cluster based on connected components of the transitive closure matrix; calculating a ratio of an internal data transmission total amount and an external data transmission total amount as an intra-cluster aggregation degree index for each of the operator clusters. 9.The artificial intelligence model training resource self-adaptive allocation system of claim 4, wherein, The prediction of the resource competition hotspot region based on the similarity between the typical resource competition mode and the current resource state comprises: extracting a multi-dimensional feature vector of the current resource state, including the occupancy rate, the occupancy rate change speed and the occupancy distribution entropy of each type of resource; calculating the Euclidean distance between the multi-dimensional feature vector and the feature center of each typical resource competition mode; using a Gaussian kernel function to convert the Euclidean distance into a similarity score; selecting the typical resource competition mode with the highest similarity score as the matching mode; inferring the resource competition hotspot region distribution in the future time window according to the historical evolution trajectory of the matching mode. 10.The artificial intelligence model training resource self-adaptive allocation system according to claim 1, characterized in that, The dynamic adjustment of the computing resource configuration in the model training process comprises: analyzing the operator fusion decision sequence to extract the timing and parameters of the operator fusion operation; mapping the operator demand to the real-time availability of hardware resources according to the resource allocation strategy to obtain the mapped operator fusion operation; applying the mapped operator fusion operation in the model training loop to combine the corresponding computing resource allocation; real-time monitoring of resource utilization fluctuation in the training process and detection of potential conflicts based on resource utilization fluctuation; updating the parameters of the resource configuration based on the monitoring results of the potential conflicts.

Images