Multi-agent collaborative large model training fault tolerance and efficient deployment system

Abstract

The invention discloses a multi-agent collaborative large model training fault tolerance and efficient deployment system, and the system comprises a training operation data collection and preprocessing module which is used for collecting and preprocessing training operation data in a large model training process; the continuous situation field construction module is used for generating continuous situation field representation; the disturbance source identification module is used for generating a disturbance region set; the multi-agent cooperation module is used for executing disturbance propagation prediction; the training track candidate generation module is used for calculating a training energy dissipation evaluation result; the collaborative decision-making and training guiding module is used for executing training track guiding operation; the stable attraction structure recognition module is used for recognizing a stable attraction structure and controlling a training iteration process; and the model deployment module is used for executing model deployment. Through continuous situation field modeling and multi-agent collaborative decision making, forward guidance control is carried out on the large model training and deployment process, and the method has the advantages of being high in training stability, high in fault-tolerant capability and high in overall efficiency.

Description

Technical Field

[0001] This invention relates to the field of large model training and deployment, and in particular to a fault-tolerant and efficient deployment system for multi-agent collaborative large model training. Background Technology

[0002] In the process of training large distributed models, data parallelism, model parallelism, or hybrid parallelism are usually adopted to distribute the computing tasks to multiple training nodes to complete the training collaboratively. Existing technologies generally maintain training consistency through parameter synchronization, gradient aggregation, and fixed communication topology. After training, the model is deployed to multiple inference nodes, and external inference services are provided through resource scheduling and service routing. In this type of technical solution, the training status is usually monitored in the form of discrete indicators or local logs, and anomaly handling mainly relies on node restart, task rollback, or resynchronization.

[0003] However, as the model size and the number of training nodes continue to increase, node anomalies, communication fluctuations, and gradient oscillations during the training process exhibit stronger spatiotemporal coupling characteristics. Existing technologies struggle to provide a unified and continuous global characterization of the training state and cannot predict the propagation trend of disturbances during the training process. They can only passively handle anomalies after they occur, which can easily lead to frequent interruptions in the training process and repeated adjustments to the training structure, resulting in decreased training efficiency and insufficient stability. At the same time, existing deployment methods lack unified modeling of the training process, making it difficult to form a consistent state control mechanism in the training and deployment phases. Summary of the Invention

[0004] One objective of this invention is to propose a multi-agent collaborative large model training fault-tolerant and efficient deployment system. This invention uses continuous situation field modeling and multi-agent collaborative decision-making to provide forward guidance and control for the large model training and deployment process, which has the advantages of high training stability, strong fault tolerance and high overall efficiency.

[0005] A multi-agent collaborative large model training fault-tolerant and efficient deployment system according to an embodiment of the present invention includes: The training data acquisition and preprocessing module is used to collect training data from each training node during the training of a large model, and to preprocess the data to generate a standardized training data set. The continuous situation field construction module is used to construct the continuous situation field for training large models and generate a continuous situation field representation in the continuous situation field coordinate system. The disturbance source identification module is used to identify disturbance sources in the continuous situation field representation and generate a set of disturbance regions; The multi-agent collaboration module is used to configure a set of multi-agents and perform disturbance propagation prediction to generate a set of high-risk situation areas; The training trajectory candidate generation module is used to generate a set of training trajectory guidance candidates and calculate the training energy dissipation evaluation results; The collaborative decision-making and training guidance module is used to select low-energy-dissipation guidance strategies based on training energy dissipation evaluation results and execute training trajectory guidance operations. A stable attraction structure identification module is used to identify stable attraction structures in the updated continuous situation field representation and control the training iteration process; The model deployment module is used to perform model deployment after the large model has been trained.

[0006] Optionally, modules can be integrated using the following methods: Collect training data from each training node during the training of a large model, perform preprocessing on the training data, and generate a standardized training data set. Construct a continuous situation field for large model training, mapping the parameter space dimension, communication space dimension, and time dimension to a unified continuous situation field coordinate system, and generating a continuous situation field representation based on a standardized training operation data set in the continuous situation field coordinate system; Identify disturbance sources in the continuous situation field representation and generate a set of corresponding disturbance regions in the continuous situation field coordinate system based on the disturbance sources; Configure a multi-agent set, each agent collects situation field information in its own observation domain, and performs disturbance propagation prediction based on the disturbance region set to generate a high-risk situation region set; A candidate set of training trajectories is generated for high-risk situation areas, and training energy dissipation evaluation results are generated based on gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost. Based on the training energy dissipation evaluation results, a collaborative decision is made to select a low-energy dissipation guidance strategy from the training trajectory guidance candidate set, and the training trajectory guidance operation is performed on the large model training process according to the low-energy dissipation guidance strategy. After performing the training trajectory guidance operation, the continuous situation field representation is updated, and stable attraction structures are identified in the updated continuous situation field representation. When the large model training situation reverts to a stable attraction structure, the large model training iteration is continuously executed. After the large model has been trained, the model is deployed.

[0007] Optionally, the training data includes computational load data, gradient statistics, parameter update data, communication latency data, and communication bandwidth usage data, and the preprocessing includes time alignment, outlier removal, and normalization.

[0008] Optionally, the generation of the continuous situation field representation includes the following specific steps: Obtain a standardized training run dataset and the parallel training structure and communication topology used in the training of large models; Based on the parallel training structure, a parameter space index is established for parameter partitioning; based on the communication topology, a communication space index is established for the communication link; and based on the global reference time axis, a time index is established for the sampling time, forming a set of index triples. Establish a mapping relationship between the set of index triples and continuous coordinates, map the parameter space index to continuous coordinates in the parameter space, map the communication space index to continuous coordinates in the communication space, and map the time index to continuous coordinates in the time dimension, forming a continuous situation field coordinate system composed of parameter space dimension, communication space dimension and time dimension; Construct a multi-channel state structure in a continuous state field coordinate system; For each index triplet, the corresponding standardized training data is extracted from the standardized training data set and written into the multi-channel state structure corresponding to the mapped coordinate position of the index triplet to form a discrete situation sampling point set. The discrete situation sampling point set is processed to be continuous. Continuous neighborhood ranges are set in the parameter space dimension, communication space dimension, and time dimension, respectively. Based on the neighborhood range, the spatially and temporally adjacent discrete situation sampling points are weighted and fused to generate a continuous situation field representation that changes continuously in the three-dimensional coordinate system.

[0009] Optionally, the generation of the set of perturbation regions includes the following specific steps: In the continuous situation field representation, multi-channel state data of each discrete situation sampling point is obtained, and the multi-channel state data of each discrete situation sampling point is arranged in time sequence according to the time dimension to form a situation evolution sequence. For each discrete situation sampling point in the situation evolution sequence, statistical calculations are performed on the state changes of each channel to extract the change amplitude between adjacent time windows and generate a set of change features. A weighted aggregation process is performed on the set of changing features to generate a disturbance intensity assessment result, and the disturbance intensity assessment result is compared with a preset disturbance judgment threshold. When the disturbance intensity assessment result is greater than the preset disturbance judgment threshold, the corresponding discrete situation sampling point is marked as the disturbance source, and the parameter space coordinates, communication space coordinates and time dimension coordinates of the disturbance source in the continuous situation field coordinate system are recorded. For each disturbance source, the disturbance extension range is set along the parameter space dimension, communication space dimension and time dimension in the continuous situation field coordinate system, and discrete situation sampling points adjacent to the disturbance source are selected within the disturbance extension range to generate a set of disturbance regions corresponding to the disturbance source. When multiple disturbance sources exist and multiple disturbance region sets overlap, the overlapping disturbance region sets are merged to generate a joint disturbance region set, and the joint disturbance region set replaces the corresponding disturbance region set.

[0010] Optionally, the generation of the high-risk situation area set includes the following specific steps: Configure a multi-agent set, assign a unique agent identifier to each agent, and divide the observation domain for each agent in the continuous situation field coordinate system; Each agent locates the corresponding disturbance source within its observation domain and extracts a set of disturbance propagation features based on the set of disturbance regions corresponding to the disturbance source. The set of disturbance propagation features includes parameter space expansion span value, communication space expansion span value, time dimension expansion span value, parameter space propagation direction value, and communication space propagation direction value. Perturbation propagation prediction is performed based on the perturbation propagation feature set, and perturbation propagation prediction results are generated; The degree of propagation impact is calculated in the continuous situation field coordinate system based on the disturbance propagation prediction results. The situation sampling points corresponding to the coordinates of the propagation impact level that exceed the preset high-risk judgment threshold are marked as high-risk situation points, and the high-risk situation points are formed into a set of high-risk situation regions.

[0011] Optionally, the generation of the training energy dissipation evaluation result includes the following specific steps: Obtain the set of high-risk situation areas and the continuous situation field representation, extract the regional coordinate information of the high-risk situation areas in the continuous situation field coordinate system, and generate the coordinate set of high-risk situation areas. Based on the coordinate set of high-risk situation areas, candidate sets for parameter update path adjustment, parallel training structure reconstruction, and communication topology adjustment are generated respectively. The candidate sets for adjusting parameter update paths, reconstructing parallel training structures, and adjusting communication topology are combined to generate a candidate set for guiding training trajectories. For each training trajectory guidance candidate in the training trajectory guidance candidate set, calculate the gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost respectively. The weighted summation of gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost generates a training energy dissipation evaluation result.

[0012] Optionally, the execution of the training trajectory guidance operation includes the following specific steps: Based on the training energy dissipation evaluation results, for each training trajectory guidance candidate in the training trajectory guidance candidate set, the corresponding training energy dissipation value is determined, and a training energy dissipation value set is generated. For each agent in the multi-agent set, based on the set of high-risk situation areas in the observation domain corresponding to the agent, the risk impact of each training trajectory guidance candidate on the high-risk situation area in the observation domain is determined, and the set of local risk impact values ​​corresponding to the agent is generated. For each agent in the multi-agent set, based on the set of training energy dissipation values ​​and the set of local risk impact values ​​corresponding to that agent, a weighted summation is performed on the corresponding training energy dissipation values ​​and local risk impact values ​​to determine the agent decision-making value corresponding to each training trajectory guidance candidate, and the set of agent decision-making value corresponding to that agent is generated. Based on the set of agent decision generation values ​​corresponding to each agent, for the same training trajectory guidance candidate, the agent decision generation values ​​of each agent are aggregated to determine the global collaborative decision generation value corresponding to the training trajectory guidance candidate, and the training trajectory guidance candidate with the smallest global collaborative decision generation value is selected as the collaborative decision result. The collaborative decision-making results are used as a low-energy dissipation guidance strategy, which includes parameter update path adjustment, parallel training structure reconstruction, and communication topology adjustment. Training trajectory guidance operations are performed during the training process of large models based on a low-energy dissipation guidance strategy.

[0013] Optionally, the continuous execution of large model training iterations includes the following specific steps: After completing the training trajectory guidance operation, a description of the current training state is generated based on the standardized training operation data set; The continuous situation field coordinate system is updated based on the training state description results, and an updated continuous situation field representation is generated. In the updated continuous situation field representation, the set of training state description results corresponding to the consecutive preset number of rounds is selected, and the candidate stable attraction structure center value is determined based on the set of training state description results. Based on the degree of offset between the current training state description result and the center value of the candidate stable attraction structure, the training convergence offset value is determined, and based on the change range of the training state description result set within a consecutive preset number of rounds, the training fluctuation intensity value is determined, and a set of stability judgment indicators is generated. The stability determination index set is judged. When the training convergence offset value and the training fluctuation intensity value simultaneously meet the threshold condition, the stable attraction structure is determined to be established, and the center value of the candidate stable attraction structure is confirmed as the center value of the stable attraction structure. When a stable attraction structure is established, the current training trajectory guidance operation is maintained during subsequent training, and large model training iterations are continuously executed until the stable attraction structure determination result changes.

[0014] Optionally, the execution model deployment includes the following specific steps: After the model training is complete, the model is deployed to multiple inference nodes; Resource assessment and service routing are performed based on the runtime status data of each inference node; During model deployment and operation, the running status of inference nodes is monitored. When an abnormal inference node is detected, the abnormal node is restarted, requests are suspended, or it is removed. Resources and service routing are reconfigured to maintain the operation of the model inference service.

[0015] The beneficial effects of this invention are: This invention reconstructs the large-scale model training process from the traditional discrete index monitoring method to a continuous situational field representation based on parameter space, communication space, and time dimensions. This achieves unified modeling and global characterization of the distributed training state, enabling the continuous expression of the operating state, parameter evolution state, and communication state of each training node in the same coordinate system. Based on the continuous situational field, this invention can identify disturbance sources and generate a set of disturbance regions in a timely manner during training. Then, through a multi-agent collaborative mechanism, it can predict the propagation trend of disturbances in advance and locate high-risk situational areas in advance, thereby avoiding the problem of passively dealing with anomalies only after they occur. Compared with the fault-tolerant methods of existing technologies that rely on node restarts or training rollbacks, this invention guides the training trajectory, allowing the training process to continue even when disturbances exist, reducing the frequency of training interruptions and improving the overall training stability and resource utilization efficiency.

[0016] Furthermore, this invention introduces a training energy dissipation evaluation mechanism, unifying gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost into a quantitative evaluation framework. Based on multi-agent collaborative decision-making, it selects a low-energy-dissipation training trajectory guidance strategy, controlling the adjustment of parameter update paths, parallel training structures, and communication topologies. This ensures training effectiveness while suppressing unnecessary structural oscillations and communication overhead. Simultaneously, this invention uses a stable attracting structure as the criterion for determining training stability, allowing the training structure to dynamically adjust within a certain range. It can continue iterating as long as the training state returns to a stable attracting structure, fundamentally improving robustness in complex training environments. After model training is complete, this invention naturally extends the aforementioned situational awareness and control concepts to the model deployment stage. Through operational status monitoring, resource evaluation, and routing adjustments, it achieves stable operation of the inference service, ensuring consistent control logic and state management between the training and deployment stages. This comprehensively improves the reliability, continuity, and engineering feasibility of large-scale model training and deployment systems. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a block diagram of the overall structure of a multi-agent collaborative large model training fault-tolerant and efficient deployment system proposed in this invention; Figure 2 This is a schematic diagram illustrating the generation of a high-risk situation region set in a multi-agent collaborative large model training fault-tolerant and efficient deployment system proposed in this invention. Figure 3 This is a schematic diagram illustrating the execution of training trajectory guidance operations in a multi-agent collaborative large model training fault-tolerant and efficient deployment system proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figures 1-3 A multi-agent collaborative large model training fault-tolerant and efficient deployment system, comprising: The training data acquisition and preprocessing module is used to collect training data from each training node during the training of a large model, and to preprocess the data to generate a standardized training data set. The continuous situation field construction module is used to construct the continuous situation field for training large models and generate a continuous situation field representation in the continuous situation field coordinate system. The disturbance source identification module is used to identify disturbance sources in the continuous situation field representation and generate a set of disturbance regions; The multi-agent collaboration module is used to configure a set of multi-agents and perform disturbance propagation prediction to generate a set of high-risk situation areas; The training trajectory candidate generation module is used to generate a set of training trajectory guidance candidates and calculate the training energy dissipation evaluation results; The collaborative decision-making and training guidance module is used to select low-energy-dissipation guidance strategies based on training energy dissipation evaluation results and execute training trajectory guidance operations. A stable attraction structure identification module is used to identify stable attraction structures in the updated continuous situation field representation and control the training iteration process; The model deployment module is used to perform model deployment after the large model has been trained.

[0020] In this embodiment, the modules are connected through the following method: Collect training data from each training node during the training of a large model, perform preprocessing on the training data, and generate a standardized training data set. Construct a continuous situation field for large model training, mapping the parameter space dimension, communication space dimension, and time dimension to a unified continuous situation field coordinate system, and generating a continuous situation field representation based on a standardized training operation data set in the continuous situation field coordinate system; Identify disturbance sources in the continuous situation field representation and generate a set of corresponding disturbance regions in the continuous situation field coordinate system based on the disturbance sources; Configure a multi-agent set, each agent collects situation field information in its own observation domain, and performs disturbance propagation prediction based on the disturbance region set to generate a high-risk situation region set; A candidate set of training trajectories is generated for high-risk situation areas, and training energy dissipation evaluation results are generated based on gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost. Based on the training energy dissipation evaluation results, a collaborative decision is made to select a low-energy dissipation guidance strategy from the training trajectory guidance candidate set, and the training trajectory guidance operation is performed on the large model training process according to the low-energy dissipation guidance strategy. After performing the training trajectory guidance operation, the continuous situation field representation is updated, and stable attraction structures are identified in the updated continuous situation field representation. When the large model training situation reverts to a stable attraction structure, the large model training iteration is continuously executed. After the large model has been trained, the model is deployed.

[0021] In this embodiment, the training operation data includes computational load data, gradient statistics data, parameter update data, communication latency data, and communication bandwidth usage data. The preprocessing includes time alignment, outlier removal, and normalization. The gradient statistics data refers to the data formed by statistically summarizing the gradient values ​​calculated by each training node during the backpropagation phase in the large model training process. It is used to characterize the intensity and stability of the changes in model parameters at the current training step size. The parameter update data refers to the parameter change data generated when each training node performs update operations on the model parameters after the gradient calculation is completed. It is used to characterize the actual adjustment result of the model parameters from the previous training state to the current training state.

[0022] In this embodiment, the generation of the continuous situation field representation includes the following specific steps: Obtain a standardized training run dataset and the parallel training structure and communication topology used in the training of large models; In the distributed large model training process, the parallel training structure and communication topology are used to determine the parallel execution relationship of model parameters or computational tasks among training nodes and the communication connection relationship formed between training nodes to complete the interaction of training data. Based on the parallel training structure, a parameter space index is established for parameter partitioning; based on the communication topology, a communication space index is established for the communication link; and based on the global reference time axis, a time index is established for the sampling time, forming a set of index triples to uniquely identify the position of each training run data in the continuous situation field coordinate system. Establish a mapping relationship between the set of index triples and continuous coordinates, map the parameter space index to continuous coordinates in the parameter space, map the communication space index to continuous coordinates in the communication space, and map the time index to continuous coordinates in the time dimension, forming a continuous situation field coordinate system composed of the parameter space dimension, the communication space dimension, and the time dimension, so that each piece of training data in the standardized training data set has a unique coordinate position in the continuous situation field coordinate system; The continuous situation field coordinate system is a continuous coordinate system constructed with the parameter space dimension, communication space dimension and time dimension as coordinate axes. It is used to represent the continuous distribution relationship of the training state of the large model with the parameter position, communication position and time in a unified space. A multi-channel state structure is constructed in a continuous situation field coordinate system. The multi-channel state structure includes a calculation load channel, a gradient statistics channel, a parameter update channel, a communication delay channel, and a communication bandwidth occupancy channel. For each index triplet, the corresponding standardized training data is extracted from the standardized training data set and written into the multi-channel state structure corresponding to the mapped coordinate position of the index triplet to form a discrete situation sampling point set. A continuous processing method is applied to the discrete situation sampling point set. Continuous neighborhood ranges are set in the parameter space dimension, communication space dimension, and time dimension, respectively. Based on the neighborhood range, spatially and temporally adjacent discrete situation sampling points are weighted and fused to generate a continuous situation field representation that changes continuously in the three-dimensional coordinate system. The continuous situation field representation refers to the overall training state representation formed by uniformly organizing and continuously processing the multi-channel state data at each coordinate position in the coordinate system composed of the parameter space dimension, communication space dimension, and time dimension. It is used to characterize the continuous evolution relationship of the large model training state in the global scope.

[0023] In this embodiment, the generation of the set of disturbance regions includes the following specific steps: In the continuous situation field representation, multi-channel state data of each discrete situation sampling point is obtained, and the multi-channel state data of each discrete situation sampling point is arranged in time sequence according to the time dimension to form a situation evolution sequence. For each discrete situation sampling point in the situation evolution sequence, statistical calculations are performed on the state changes of each channel to extract the change amplitude between adjacent time windows and generate a set of change features. A weighted aggregation process is performed on the set of changing features to generate a disturbance intensity assessment result, and the disturbance intensity assessment result is compared with a preset disturbance judgment threshold. The generation of the preset disturbance judgment threshold specifically includes: after the continuous situation field is constructed, the continuous situation field representation corresponding to the stable training phase is used as the benchmark situation sample set. In the benchmark situation sample set, the disturbance intensity evaluation results corresponding to each discrete situation sampling point are statistically analyzed to obtain the historical distribution range of the disturbance intensity evaluation results. Based on the historical distribution range, the 95th percentile value of the disturbance intensity evaluation results is selected as the preset disturbance judgment threshold so that when the disturbance intensity evaluation result of the discrete situation sampling point exceeds the threshold, it is possible to distinguish between normal training fluctuations and abnormal disturbance states. When the disturbance intensity assessment result is greater than the preset disturbance judgment threshold, the corresponding discrete situation sampling point is marked as the disturbance source, and the parameter space coordinates, communication space coordinates and time dimension coordinates of the disturbance source in the continuous situation field coordinate system are recorded. For each disturbance source, the disturbance extension range is set along the parameter space dimension, communication space dimension and time dimension in the continuous situation field coordinate system, and discrete situation sampling points adjacent to the disturbance source are selected within the disturbance extension range to generate a set of disturbance regions corresponding to the disturbance source. When generating the set of disturbance regions corresponding to the disturbance source, the parameter space coordinates, communication space coordinates, and time dimension coordinates of the disturbance source in the continuous situation field coordinate system are read. The parameter space disturbance radius value is determined with the parameter space coordinate as the center, the communication space disturbance radius value is determined with the communication space coordinate as the center, and the time disturbance window length is determined with the time dimension coordinate as the center. The parameter space disturbance radius value, communication space disturbance radius value, and time disturbance window length are used as disturbance expansion range parameters. All discrete situation sampling points are traversed in the continuous situation field coordinate system, and the distance relationship between each discrete situation sampling point and the corresponding coordinates of the disturbance source in the parameter space dimension, communication space dimension, and time dimension is calculated. Discrete situation sampling points that fall within the parameter space disturbance radius value range in the parameter space dimension, the communication space disturbance radius value range in the communication space dimension, and the time disturbance window length range in the time dimension are selected, and discrete situation sampling points that meet the above three-dimensional constraints are marked as affected situation sampling points. The affected situation sampling points are associated with the corresponding disturbance source to form the set of disturbance regions corresponding to the disturbance source. When multiple disturbance sources exist and multiple disturbance region sets overlap, the overlapping disturbance region sets are merged to generate a joint disturbance region set, and the joint disturbance region set replaces the corresponding disturbance region set.

[0024] In this embodiment, the generation of the high-risk situation area set includes the following specific steps: Configure a multi-agent set, assign a unique agent identifier to each agent, and divide the observation domain for each agent in the continuous situation field coordinate system; During large-scale model training, the number of agents is determined based on the number of training nodes and the scale of the parallel training structure. An agent instance set is generated, and a unique agent identifier is assigned to each agent. The global range of the parameter space dimension, communication space dimension, and time dimension is read in the continuous situation field coordinate system. Based on the number of agents, the global range is partitioned to generate the parameter space dimension range, communication space dimension range, and time dimension range corresponding to each agent. The parameter space dimension range, communication space dimension range, and time dimension range corresponding to each agent are combined to form the observation domain of that agent. The observation domain is bound and stored with the corresponding agent identifier, resulting in a multi-agent set configuration result containing the mapping relationship between agent identifiers and corresponding observation domains. The multi-agent ensemble is a set of control entities configured during the training and deployment of large models, consisting of multiple agents with independent observation capabilities and collaborative decision-making capabilities. Each agent is assigned a corresponding continuous situation field observation domain, which is used to obtain training situation information from the continuous situation field representation and participate in disturbance propagation prediction and training trajectory guidance decision-making. The observation domain is the state perception range divided for each agent in the continuous situation field coordinate system. This state perception range is jointly defined by the parameter space dimension range, the communication space dimension range, and the time dimension range, and is used to determine the spatial and temporal boundaries of the agent's ability to collect training situation information from the continuous situation field representation. Each agent locates the corresponding disturbance source within its observation domain and extracts a set of disturbance propagation features based on the set of disturbance regions corresponding to the disturbance source. The set of disturbance propagation features includes parameter space expansion span value, communication space expansion span value, time dimension expansion span value, parameter space propagation direction value, and communication space propagation direction value. The generation of the disturbance propagation feature set specifically includes: each agent, based on its corresponding observation domain, filters disturbance sources in the continuous situation field coordinate system whose parameter space coordinates, communication space coordinates, and time dimension coordinates all fall within the observation domain, generating a disturbance source subset corresponding to that agent; and for each disturbance source in the disturbance source subset, reads its disturbance region set, extracts parameter space coordinates, communication space coordinates, and time dimension coordinates for discrete situation sampling points within the disturbance region set, respectively, to obtain parameter space coordinate sequences, communication space coordinate sequences, and time coordinate sequences; each agent calculates the minimum and maximum coordinate values ​​for the parameter space coordinate sequences and generates a parameter space expansion span value, and calculates the minimum and maximum coordinate values ​​for the communication space coordinate sequences. The maximum coordinate value is calculated and a communication space expansion span value is generated. The minimum and maximum time values ​​are calculated for the time coordinate sequence and a time dimension expansion span value is generated. Each agent selects the discrete situation sampling points with a preset ratio that are farthest from the disturbance source in the continuous situation field coordinate system from the disturbance region set, forming a far-end sampling point set. The propagation direction value of the parameter space and the propagation direction value of the communication space are determined by the displacement direction of the center position of the far-end sampling point set in the parameter space coordinate and the communication space coordinate relative to the corresponding coordinate of the disturbance source. Each agent combines the parameter space expansion span value, the communication space expansion span value, the time dimension expansion span value, the parameter space propagation direction value, and the communication space propagation direction value to form a disturbance propagation feature set. Perturbation propagation prediction is performed based on the perturbation propagation feature set, and perturbation propagation prediction results are generated; The disturbance propagation prediction result is obtained by each agent based on the disturbance propagation feature set, after performing parameter space reachability determination, communication space reachability determination and time reachability determination on the situation sampling points, and then filtering out the situation sampling points that are determined to be affected by disturbance propagation. The generation of the disturbance propagation prediction results specifically includes: each agent reading the disturbance propagation feature set of the corresponding disturbance source within its observation domain, generating a propagation feature reading result; obtaining the parameter space coordinates, communication space coordinates, and time dimension coordinates of the disturbance source in the continuous situation field coordinate system, and combining the three as the corresponding coordinate point of the disturbance source to generate the propagation start point coordinate value; each agent using the propagation start point coordinate value as a reference, determining the unidirectional propagation direction of the parameter space dimension based on the parameter space propagation direction value, and using the parameter space expansion span value as the unidirectional extrapolation distance to generate the parameter space prediction endpoint coordinate value, and then using the propagation start point coordinate value and the parameter space prediction endpoint coordinate value together to determine the parameter space propagation constraint; each agent using the propagation start point coordinate value as a reference, determining the unidirectional propagation direction of the communication space dimension based on the communication space propagation direction value, and using the communication space expansion span value as the unidirectional extrapolation distance to generate the parameter space prediction endpoint coordinate value. One-way extrapolation distance generates the predicted endpoint coordinates of the communication space. The propagation start point coordinates and the predicted endpoint coordinates of the communication space are used together to determine the communication space propagation constraints. Each agent uses the propagation start point coordinates as the starting time point and the time dimension extension span value as the prediction duration to generate the predicted endpoint coordinates of the time. The propagation start point coordinates and the predicted endpoint coordinates of the time are used together to determine the time dimension propagation constraints. Each agent reads the parameter space coordinates, communication space coordinates, and time dimension coordinates corresponding to the situation sampling points in the continuous situation field one by one in its observation domain, and performs parameter space reachability determination, communication space reachability determination, and time reachability determination in sequence. Situation sampling points that meet all three determinations are written into the disturbance propagation candidate sampling point set to generate the disturbance propagation candidate sampling point set. Each agent outputs the disturbance propagation candidate sampling point set as the disturbance propagation prediction result. The parameter space reachability determination refers to the process of determining whether the coordinates of the situation sampling point in the parameter space dimension are located on a unidirectional propagation path defined by the propagation starting point coordinate value along the propagation direction value of the parameter space and the extension span value of the parameter space. The communication space reachability determination refers to the process of determining whether the coordinates of the situation sampling point in the communication space dimension are located on a unidirectional propagation path defined by the propagation starting point coordinate value along the propagation direction value of the communication space and the extension span value of the communication space. The time reachability determination refers to the process of determining whether the coordinates of the situation sampling point in the time dimension are located within a propagation time period defined by the propagation starting point coordinate value as the starting time point and the extension span value of the time dimension. This invention extracts the directional and span characteristics of disturbance propagation based on the actual distribution of the disturbance region set in a continuous situation field, and then uses the disturbance propagation characteristics to perform propagation reachability determination in the continuous situation field, thereby achieving forward prediction of the disturbance's impact range. The degree of propagation impact is calculated in the continuous situation field coordinate system based on the disturbance propagation prediction results. The generation of the propagation impact level specifically includes: each agent reading the disturbance propagation prediction result, which is a set of situational sampling points, and obtaining the parameter space coordinates, communication space coordinates, and time dimension coordinates corresponding to each situational sampling point; reading the coordinate points corresponding to the disturbance source, and calculating the coordinate differences of each situational sampling point relative to the corresponding coordinate points of the disturbance source in the parameter space dimension, communication space dimension, and time dimension, respectively, to obtain the parameter space displacement value, communication space displacement value, and time displacement value; taking the absolute values ​​of the parameter space displacement value, communication space displacement value, and time displacement value, respectively, to obtain the parameter space displacement amplitude value, communication space displacement amplitude value, and time displacement amplitude value; using the parameter space expansion span value, communication space expansion span value, and time dimension expansion span value as normalization scales, respectively, normalizing the corresponding displacement amplitude values ​​to obtain the parameter space normalized displacement value, communication space normalized displacement value, and time normalized displacement value; and performing weighted summation to obtain the situational sampling point impact score, which is used as the propagation impact level; The situation sampling points corresponding to the coordinates of the propagation impact level that exceed the preset high-risk judgment threshold are marked as high-risk situation points, and the high-risk situation points are formed into a set of high-risk situation regions.

[0025] In this embodiment, the generation of the training energy dissipation evaluation result includes the following specific steps: Obtain the set of high-risk situation areas and the continuous situation field representation, extract the regional coordinate information of the high-risk situation areas in the continuous situation field coordinate system, and generate the coordinate set of high-risk situation areas. Based on the coordinate set of high-risk situation areas, candidate sets for parameter update path adjustment, parallel training structure reconstruction, and communication topology adjustment are generated respectively. Using the parameter space coordinates, communication space coordinates, and time dimension coordinates of each high-risk situation area in the continuous situation field coordinate system as constraints, a set of parameter update stages affected by high-risk situation areas is determined in the parameter space dimension. Within this set, a restricted rearrangement is performed on the update order and update grouping of parameter update data. This restricted rearrangement avoids the parameter space coordinate intervals corresponding to high-risk situation areas, generating multiple parameter update path adjustment results. These results are then aggregated to form a parameter update path adjustment candidate set. At the parallel training structure level, based on the distribution of high-risk situation areas in the time dimension coordinates, the affected parallel training time window is determined, and within this window... Within the training node parallel group partitioning method and the synchronization boundary between parallel groups, a restricted adjustment is performed. The restricted adjustment is constrained by reducing the parallel synchronization intensity within the coverage area of ​​the high-risk situation area. Multiple parallel training structure reconstruction results are generated, and the parallel training structure reconstruction results are aggregated to form a parallel training structure reconstruction candidate set. At the communication structure level, based on the distribution of the high-risk situation area in the communication space coordinates, the set of affected communication paths is determined. Within the communication path set, a restricted reconstruction is performed on the communication adjacency relationship and synchronization path between training nodes. The restricted reconstruction is constrained by reducing the synchronization interaction on the communication path covered by the high-risk situation area. Multiple communication topology adjustment results are generated, and the communication topology adjustment results are aggregated to form a communication topology adjustment candidate set. The parameter update path adjustment candidate set, the parallel training structure reconstruction candidate set, and the communication topology adjustment candidate set are combined to generate a training trajectory guidance candidate set. Each training trajectory guidance candidate in the training trajectory guidance candidate set consists of a parameter update path adjustment candidate, a parallel training structure reconstruction candidate, and a communication topology adjustment candidate. For each training trajectory guidance candidate in the training trajectory guidance candidate set, the gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost are calculated. Specifically, taking a single training trajectory guidance candidate as the object, the set of training nodes that need to perform gradient recalculation under the action of the training trajectory guidance candidate is determined, and the change magnitude of the computational load data and gradient statistics data corresponding to each training node in the training node set is multiplied item by item and summed to generate the gradient recalculation cost. Then, the set of communication synchronization paths corresponding to the training trajectory guidance candidate is determined, and the communication latency data and communication bandwidth usage data corresponding to each communication path in the communication synchronization path set are multiplied item by item and summed to generate the communication synchronization cost. The set of parallel training structure changes corresponding to the training trajectory guidance candidate is determined, and for the set of training nodes in the set of parallel training structure changes that have undergone parallel group changes, the change magnitude of the parameter update data corresponding to each training node is determined, and the change magnitude of the parameter update data is weighted and accumulated with the corresponding group division change amount to generate the training structure reconstruction cost. When generating the group partition change amount, the parallel training structure corresponding to the training trajectory guidance candidate is taken as the object. The parallel group partition result of the training nodes before the training trajectory guidance candidate takes effect is determined, a baseline parallel group set is generated, and the training node identifiers in each parallel group are recorded to form a baseline group member set. The parallel group partition result of the training nodes after the training trajectory guidance candidate takes effect is determined, an adjusted parallel group set is generated, and the training node identifiers in each parallel group are recorded to form an adjusted group member set. Then, a node-by-node comparison is performed on the baseline parallel group set and the adjusted parallel group set to determine the set of training nodes that have undergone parallel group changes during the parallel group partitioning process. The number of training nodes in the set of training nodes that have undergone parallel group changes is taken as the group partition change amount corresponding to the training trajectory guidance candidate, and normalization is performed. The weighted summation of gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost generates a training energy dissipation evaluation result. This invention is based on a set of high-risk situational regions and constructs training trajectory guidance candidates in advance to avoid unstable regions, thereby maintaining training continuity and structural stability without interrupting the training process.

[0026] In this embodiment, the execution of the training trajectory guidance operation includes the following specific steps: Based on the training energy dissipation evaluation results, for each training trajectory guidance candidate in the training trajectory guidance candidate set, the corresponding training energy dissipation value is determined, and a training energy dissipation value set is generated. For each agent in the multi-agent set, based on the set of high-risk situation areas in the observation domain corresponding to the agent, the risk impact of each training trajectory guidance candidate on the high-risk situation area in the observation domain is determined, and the set of local risk impact values ​​corresponding to the agent is generated. The generation of the risk impact specifically includes: performing discretization sampling on the parameter space dimension, communication space dimension, and time dimension in the continuous situation field coordinate system to obtain a set of situation sampling points in the observation domain; counting the number of situation sampling points in the observation domain that overlap with high-risk situation areas guided by the training trajectory; and dividing the number of overlapping situation sampling points by the total number of situation sampling points in the observation domain to obtain a normalized overlap ratio value, which is used as the risk impact. For each agent in the multi-agent set, based on the set of training energy dissipation values ​​and the set of local risk impact values ​​corresponding to that agent, a weighted summation is performed on the corresponding training energy dissipation values ​​and local risk impact values ​​to determine the agent decision-making value corresponding to each training trajectory guidance candidate, and the set of agent decision-making value corresponding to that agent is generated. Based on the set of agent decision values ​​corresponding to each agent, for the same training trajectory guidance candidate, the agent decision values ​​of each agent are aggregated to determine the global collaborative decision value corresponding to the training trajectory guidance candidate, and the training trajectory guidance candidate with the smallest global collaborative decision value is selected as the collaborative decision result. The aggregation process uses a weighted sum. The collaborative decision-making results are used as a low-energy dissipation guidance strategy, which includes parameter update path adjustment, parallel training structure reconstruction, and communication topology adjustment. The training trajectory guidance operation is performed on the large model training process based on the low energy dissipation guidance strategy. The training trajectory guidance operation includes updating the update order and update group of parameters and data, updating the parallel group division method and synchronization boundary of training nodes, and updating the communication adjacency relationship and synchronization path between training nodes.

[0027] In this embodiment, the continuous execution of large model training iterations includes the following specific steps: After completing the training trajectory guidance operation, a current training state description result is generated based on the standardized training operation data set. The training state description result consists of a normalized combination of computational load data, gradient statistics data, parameter update data, communication latency data, and communication bandwidth usage data at the same time. The continuous situation field coordinate system is updated based on the training state description results, and an updated continuous situation field representation is generated. In the updated continuous situation field representation, the set of training state description results corresponding to the consecutive preset number of rounds is selected, and the candidate stable attraction structure center value is determined based on the set of training state description results. The candidate stable attraction structure center value is obtained by calculating the mean of the training state description result set within a consecutive preset number of rounds on each corresponding data dimension, and then combining the mean values. Based on the degree of offset between the current training state description result and the center value of the candidate stable attraction structure, the training convergence offset value is determined, and based on the change range of the training state description result set within a consecutive preset number of rounds, the training fluctuation intensity value is determined, and a set of stability judgment indicators is generated. The stability determination index set is judged. When the training convergence offset value and the training fluctuation intensity value simultaneously meet the threshold condition, the stable attraction structure is determined to be established, and the center value of the candidate stable attraction structure is confirmed as the center value of the stable attraction structure. The threshold condition is that the training convergence offset value is not greater than the preset stable attraction structure neighborhood threshold, and the training fluctuation intensity value is not greater than the preset stable fluctuation threshold. When a stable attraction structure is established, the current training trajectory guidance operation is maintained during subsequent training, and large model training iterations are continuously executed until the stable attraction structure determination result changes.

[0028] In this embodiment, the deployment of the execution model includes the following specific steps: After the model training is complete, the model is deployed to multiple inference nodes; Resource assessment and service routing are performed based on the runtime status data of each inference node; During model deployment and operation, the running status of inference nodes is monitored. When an abnormal inference node is detected, the abnormal node is restarted, requests are suspended, or it is removed. Resources and service routing are reconfigured to maintain the operation of the model inference service.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a large-scale distributed training and inference service scenario within an artificial intelligence computing center. This computing center deploys multiple heterogeneous training servers, with training nodes interconnected via a high-speed network to support long-term continuous training of language models with extremely large parameter scales. In this scenario, the training process is lengthy, involves a large number of training nodes, and has complex communication links. Inevitably, problems such as node load fluctuations, communication latency jitter, and unstable gradient changes occur during training. Existing training methods often require interrupting the training process and resuming through rollback or node restart when encountering these problems, resulting in disruption of training continuity and difficulty in guaranteeing training efficiency. This is precisely the core problem that this invention aims to solve.

[0030] In this computing center, the system of this invention is deployed at the training scheduling and control layer, running in parallel with the existing training framework. After training begins, the system continuously collects training operation data such as computational load, gradient statistics, parameter updates, and communication status of each training node, and performs unified preprocessing on this data to construct a continuous situational field representation reflecting the overall training state. During training, when some training nodes experience increased load or abnormal communication status, the system can identify the corresponding disturbance source in the continuous situational field and further generate a set of disturbance regions. Subsequently, through a multi-agent collaborative mechanism, the system predicts the disturbance propagation trend in different observation domains, locating potential high-risk situational regions in advance. Based on these high-risk situational regions, the system dynamically generates multiple training trajectory guidance candidates and comprehensively evaluates different candidate schemes in terms of computational cost, communication cost, and structural adjustment cost, selecting the guidance strategy with the least impact on training. This guides and adjusts the parameter update order, parallel training structure, and communication path, thereby preventing the training process from directly entering an unstable state.

[0031] In actual operation, the training tasks of this computing center, running continuously for extended periods, did not frequently trigger training interruptions. Through continuous updates of the state field and identification of stable attraction structures, the system can determine whether the training state has returned to a stable region after adjustments to the training structure, and continuously advance training iterations in a stable state. Furthermore, after model training is completed and the inference service phase begins, the system continues to monitor the operational status of inference nodes. When some inference nodes experience abnormal loads or service latency changes, the system can promptly adjust resource allocation and service routing to ensure continuous operation of the inference service. Relevant operational logs and training process records show that, under the same time period and computing power environment, this invention can effectively reduce the number of unplanned interruptions during training, reduce the additional overhead caused by frequent reconstruction of the training structure, and improve the stability and continuity of the overall training and deployment process, thus verifying the feasibility and engineering value of this invention in practical application environments.

[0032] To verify the performance of the present invention in practice, it was compared with traditional methods.

[0033] Table 1. Overall Performance Comparison of Distributed Large Model Training and Deployment

[0034] As shown in Table 1, in terms of total training time, the traditional method requires 312 hours to complete one training session, while the method of this invention requires 274 hours, which is a reduction of about ten percentage points. Combined with the data on the number of training interruptions, it can be found that the traditional method experienced 17 interruptions during training, while the method of this invention only experienced 4. Since each interruption triggers training pause, state recovery, or structure adjustment, these operations themselves do not produce effective training results. Therefore, the significant reduction in the number of interruptions is the direct reason for the reduction in total training time.

[0035] Further comparison of average interruption recovery time shows that traditional methods require an average of 28 minutes to recover from a single interruption, while the method of this invention only takes 9 minutes. This indicates that when a local anomaly occurs, traditional methods usually need to wait for the node state to recover or perform a large-scale synchronization operation. However, this invention, through a training trajectory guidance mechanism, only adjusts the affected parameter update path, parallel structure, or communication path, making the recovery process more localized and thus shortening the recovery time.

[0036] Data on the percentage of effective training runtime further validates this point. The traditional method has an effective runtime percentage of 84.6%, meaning a significant amount of time is spent on interrupt handling and invalid waiting. In contrast, the method of this invention has an effective runtime percentage of 96.1%, indicating that the vast majority of time is devoted to actual training computation. This difference directly reflects the effectiveness of this invention in reducing unnecessary pauses.

[0037] At the communication and structural levels, the average parameter synchronization latency decreased from 41.8 milliseconds to 29.3 milliseconds, while the number of training structure reconstructions decreased from 21 to 7. This indicates that traditional methods often deal with unstable states by frequently reconstructing parallel structures and performing global synchronization, leading to increased communication overhead and structural oscillations. In contrast, this invention identifies high-risk situation areas in advance and adjusts the communication topology and parallel structure only within the necessary range, avoiding frequent and large-scale structural changes, thereby reducing synchronization latency and the number of reconstructions.

[0038] During the inference phase, the traditional method exhibited 11 anomalies, while the present invention only exhibited 3. This indicates that the stable control strategy developed during the training phase of the present invention makes the model more adaptable to load changes and node fluctuations after deployment, reducing the triggering of anomalies during the inference phase.

[0039] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multi-agent collaborative large-scale model training fault-tolerant and efficient deployment system, characterized in that, include: The training data acquisition and preprocessing module is used to collect training data from each training node during the training of a large model, and to preprocess the data to generate a standardized training data set. The continuous situation field construction module is used to construct the continuous situation field for training large models and generate a continuous situation field representation in the continuous situation field coordinate system. The disturbance source identification module is used to identify disturbance sources in the continuous situation field representation and generate a set of disturbance regions; The multi-agent collaboration module is used to configure a set of multi-agents and perform disturbance propagation prediction to generate a set of high-risk situation areas; The training trajectory candidate generation module is used to generate a set of training trajectory guidance candidates and calculate the training energy dissipation evaluation results; The collaborative decision-making and training guidance module is used to select low-energy-dissipation guidance strategies based on training energy dissipation evaluation results and execute training trajectory guidance operations. A stable attraction structure identification module is used to identify stable attraction structures in the updated continuous situation field representation and control the training iteration process; The model deployment module is used to perform model deployment after the large model has been trained.

2. The multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 1, characterized in that, The modules are connected in the following way: Collect training data from each training node during the training of a large model, perform preprocessing on the training data, and generate a standardized training data set. Construct a continuous situation field for large model training, mapping the parameter space dimension, communication space dimension, and time dimension to a unified continuous situation field coordinate system, and generating a continuous situation field representation based on a standardized training operation data set in the continuous situation field coordinate system; Identify disturbance sources in the continuous situation field representation and generate a set of corresponding disturbance regions in the continuous situation field coordinate system based on the disturbance sources; Configure a multi-agent set, each agent collects situation field information in its own observation domain, and performs disturbance propagation prediction based on the disturbance region set to generate a high-risk situation region set; A candidate set of training trajectories is generated for high-risk situation areas, and training energy dissipation evaluation results are generated based on gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost. Based on the training energy dissipation evaluation results, a collaborative decision is made to select a low-energy dissipation guidance strategy from the training trajectory guidance candidate set, and the training trajectory guidance operation is performed on the large model training process according to the low-energy dissipation guidance strategy. After performing the training trajectory guidance operation, the continuous situation field representation is updated, and stable attraction structures are identified in the updated continuous situation field representation. When the large model training situation reverts to a stable attraction structure, the large model training iteration is continuously executed. After the large model has been trained, the model is deployed.

3. The multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 2, characterized in that, The training data includes computational load data, gradient statistics, parameter update data, communication latency data, and communication bandwidth usage data. The preprocessing includes time alignment, outlier removal, and normalization.

4. The multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 2, characterized in that, The generation of the continuous situation field representation includes the following specific steps: Obtain a standardized training run dataset and the parallel training structure and communication topology used in the training of large models; Based on the parallel training structure, a parameter space index is established for parameter partitioning; based on the communication topology, a communication space index is established for the communication link; and based on the global reference time axis, a time index is established for the sampling time, forming a set of index triples. Establish a mapping relationship between the set of index triples and continuous coordinates, map the parameter space index to continuous coordinates in the parameter space, map the communication space index to continuous coordinates in the communication space, and map the time index to continuous coordinates in the time dimension, forming a continuous situation field coordinate system composed of parameter space dimension, communication space dimension and time dimension; Construct a multi-channel state structure in a continuous state field coordinate system; For each index triplet, the corresponding standardized training data is extracted from the standardized training data set and written into the multi-channel state structure corresponding to the mapped coordinate position of the index triplet to form a discrete situation sampling point set. The discrete situation sampling point set is processed to be continuous. Continuous neighborhood ranges are set in the parameter space dimension, communication space dimension, and time dimension, respectively. Based on the neighborhood range, the spatially and temporally adjacent discrete situation sampling points are weighted and fused to generate a continuous situation field representation that changes continuously in the three-dimensional coordinate system.

5. The multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 2, characterized in that, The generation of the set of disturbance regions includes the following specific steps: In the continuous situation field representation, multi-channel state data of each discrete situation sampling point is obtained, and the multi-channel state data of each discrete situation sampling point is arranged in time sequence according to the time dimension to form a situation evolution sequence. For each discrete situation sampling point in the situation evolution sequence, statistical calculations are performed on the state changes of each channel to extract the change amplitude between adjacent time windows and generate a set of change features. A weighted aggregation process is performed on the set of changing features to generate a disturbance intensity assessment result, and the disturbance intensity assessment result is compared with a preset disturbance judgment threshold. When the disturbance intensity assessment result is greater than the preset disturbance judgment threshold, the corresponding discrete situation sampling point is marked as the disturbance source, and the parameter space coordinates, communication space coordinates and time dimension coordinates of the disturbance source in the continuous situation field coordinate system are recorded. For each disturbance source, the disturbance extension range is set along the parameter space dimension, communication space dimension and time dimension in the continuous situation field coordinate system, and discrete situation sampling points adjacent to the disturbance source are selected within the disturbance extension range to generate a set of disturbance regions corresponding to the disturbance source. When multiple disturbance sources exist and multiple disturbance region sets overlap, the overlapping disturbance region sets are merged to generate a joint disturbance region set, and the joint disturbance region set replaces the corresponding disturbance region set.

6. The multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 2, characterized in that, The generation of the set of high-risk situation areas includes the following specific steps: Configure a multi-agent set, assign a unique agent identifier to each agent, and divide the observation domain for each agent in the continuous situation field coordinate system; Each agent locates the corresponding disturbance source within its observation domain and extracts a set of disturbance propagation features based on the set of disturbance regions corresponding to the disturbance source. The set of disturbance propagation features includes parameter space expansion span value, communication space expansion span value, time dimension expansion span value, parameter space propagation direction value, and communication space propagation direction value. Perturbation propagation prediction is performed based on the perturbation propagation feature set, and perturbation propagation prediction results are generated; The degree of propagation impact is calculated in the continuous situation field coordinate system based on the disturbance propagation prediction results. The situation sampling points corresponding to the coordinates of the propagation impact level that exceed the preset high-risk judgment threshold are marked as high-risk situation points, and the high-risk situation points are formed into a set of high-risk situation regions.

7. The multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 2, characterized in that, The generation of the training energy dissipation evaluation results includes the following specific steps: Obtain the set of high-risk situation areas and the continuous situation field representation, extract the regional coordinate information of the high-risk situation areas in the continuous situation field coordinate system, and generate the coordinate set of high-risk situation areas. Based on the coordinate set of high-risk situation areas, candidate sets for parameter update path adjustment, parallel training structure reconstruction, and communication topology adjustment are generated respectively. The candidate sets for adjusting parameter update paths, reconstructing parallel training structures, and adjusting communication topology are combined to generate a candidate set for guiding training trajectories. For each training trajectory guidance candidate in the training trajectory guidance candidate set, calculate the gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost respectively. The weighted summation of gradient recalculation cost, communication synchronization cost, and training structure reconstruction cost generates a training energy dissipation evaluation result.

8. The multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 2, characterized in that, The execution of the training trajectory guidance operation includes the following specific steps: Based on the training energy dissipation evaluation results, for each training trajectory guidance candidate in the training trajectory guidance candidate set, the corresponding training energy dissipation value is determined, and a training energy dissipation value set is generated. For each agent in the multi-agent set, based on the set of high-risk situation areas in the observation domain corresponding to the agent, the risk impact of each training trajectory guidance candidate on the high-risk situation area in the observation domain is determined, and the set of local risk impact values ​​corresponding to the agent is generated. For each agent in the multi-agent set, based on the set of training energy dissipation values ​​and the set of local risk impact values ​​corresponding to that agent, a weighted summation is performed on the corresponding training energy dissipation values ​​and local risk impact values ​​to determine the agent decision-making value corresponding to each training trajectory guidance candidate, and the set of agent decision-making value corresponding to that agent is generated. Based on the set of agent decision generation values ​​corresponding to each agent, for the same training trajectory guidance candidate, the agent decision generation values ​​of each agent are aggregated to determine the global collaborative decision generation value corresponding to the training trajectory guidance candidate, and the training trajectory guidance candidate with the smallest global collaborative decision generation value is selected as the collaborative decision result. The collaborative decision-making results are used as a low-energy dissipation guidance strategy, which includes parameter update path adjustment, parallel training structure reconstruction, and communication topology adjustment. Training trajectory guidance operations are performed during the training process of large models based on a low-energy dissipation guidance strategy.

9. The multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 2, characterized in that, The continuous execution of large model training iterations includes the following specific steps: After completing the training trajectory guidance operation, a description of the current training state is generated based on the standardized training operation data set; The continuous situation field coordinate system is updated based on the training state description results, and an updated continuous situation field representation is generated. In the updated continuous situation field representation, the set of training state description results corresponding to the consecutive preset number of rounds is selected, and the candidate stable attraction structure center value is determined based on the set of training state description results. Based on the degree of offset between the current training state description result and the center value of the candidate stable attraction structure, the training convergence offset value is determined, and based on the change range of the training state description result set within a consecutive preset number of rounds, the training fluctuation intensity value is determined, and a set of stability judgment indicators is generated. The stability determination index set is judged. When the training convergence offset value and the training fluctuation intensity value simultaneously meet the threshold condition, the stable attraction structure is determined to be established, and the center value of the candidate stable attraction structure is confirmed as the center value of the stable attraction structure. When a stable attraction structure is established, the current training trajectory guidance operation is maintained during subsequent training, and large model training iterations are continuously executed until the stable attraction structure determination result changes.

10. A multi-agent collaborative large model training fault-tolerant and efficient deployment system according to claim 2, characterized in that, The deployment of the execution model includes the following specific steps: After the model training is complete, the model is deployed to multiple inference nodes; Resource assessment and service routing are performed based on the runtime status data of each inference node; During model deployment and operation, the running status of inference nodes is monitored. When an abnormal inference node is detected, the abnormal node is restarted, requests are suspended, or it is removed. Resources and service routing are reconfigured to maintain the operation of the model inference service.

Images