Large model training system reliability analysis method, device, equipment and medium
By analyzing the training process of large language models layer by layer and injecting faults, the problem of traditional methods being unable to characterize the destructive power of soft errors is solved, thereby improving the stability of the model and the quality of text generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JILIN UNIVERSITY
- Filing Date
- 2026-05-08
- Publication Date
- 2026-07-31
AI Technical Summary
In the training process of existing large language models, traditional reliability analysis methods are unable to accurately characterize the destructive power of soft errors, resulting in insufficient model stability and text generation quality.
By analyzing the training process of a large language model layer by layer, the target objects for fault injection are determined, fault templates are generated based on preset fault types, fault perturbation is performed using an injector, and the model results after perturbation are compared with those without perturbation to optimize the model training process.
It improves the accuracy of reliability analysis in the training process of large language models, enhances the stability of the models and the quality of text generation, and strengthens the model's ability to resist interference in complex scenarios.
Smart Images
Figure CN122490100A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, equipment and medium for reliability analysis of a large model training system. Background Technology
[0002] Large language models, as important pre-trained models for natural language processing, have been widely applied in fields such as intelligent question answering, text generation, and code generation due to their strong language understanding and generation capabilities. Currently, reliability analysis methods for large language model training primarily rely on statistical features such as gradients, parameter update amounts, and activation value ranges during training. By processing surrogate information, they generate sensitivity rankings or distributions for different layers and operators within the model. However, both the analytical basis and the results are built upon surrogate features. The conclusions mainly reflect the sensitivity differences of operators within the model and are insufficient to directly characterize the specific impact of soft errors on the training process. These methods typically perform static or phased evaluations based on a certain type of local features, making it difficult to depict the propagation and cumulative effects of soft errors during training, as well as the differences in impact between different training stages.
[0003] As can be seen from the above, how to improve the accuracy of reliability analysis during the training process of large language models, overcome the shortcomings of traditional methods in representing the actual destructive power of soft errors, and improve the stability of the model and the quality of text generation are problems to be solved in this field. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for reliability analysis of large model training systems, which can improve the accuracy of reliability analysis during the training of large language models, overcome the shortcomings of traditional methods in characterizing the actual destructive power of soft errors, and improve the stability of the model and the quality of text generation. The specific solution is as follows: Firstly, this application discloses a reliability analysis method for a large model training system, including: The training process of the large language model used to generate text is parsed layer by layer to obtain parsed data, and the target object for fault injection is determined based on the parsed data. Based on the preset fault type, the target object, the number of fault occurrence steps, the duration interval and the disturbance method set by the user are parameterized to form a fault template to be called. According to the fault template, corresponding fault injection information is generated for each fault injection. The injector is initialized based on the fault injection information. The initialized injector is used to apply fault perturbation to the target data of the target operator in the current execution process. Then, the target operator after the fault perturbation is restored to its original state, and the training process of the large language model is continued to obtain the training result of the perturbated model. Obtain the benchmark model training results generated during the training process of a large language model without applied fault perturbations; The training results of the perturbated model and the training results of the benchmark model are compared and analyzed to obtain the comparison analysis results. Based on the comparative analysis results, the training process of the large language model used to generate text is optimized to obtain an optimized large language model, and text is generated using the optimized large language model.
[0005] Optionally, the training process of the large language model used to generate text is parsed layer by layer to obtain parsed data, including: The training process of the large language model used to generate text is analyzed layer by layer to obtain the hierarchical position of the target network layer involved in the training process, the execution direction and category of the target operator, and the data association relationship between intermediate tensors.
[0006] Optionally, the parameterized description of the target object, the user-defined fault occurrence steps, the duration interval, and the disturbance method based on a preset fault type includes: Based on user requirements, select any one variable from all variables to determine the independent variable; among them, all variables include the fault occurrence stage, target operator type, target network layer, fault injection direction, and fault perturbation method; Other variables besides the independent variables are used as control variables; Based on the preset fault type, the independent variable, and the control variable, the target object, the number of fault occurrence steps, the duration interval, and the disturbance mode set by the user are parameterized and described.
[0007] Optionally, the step of applying fault perturbation to the target data of the target operator during the current execution process using the initialized injector includes: Determine whether the currently executed operator during the training of a large language model is the target operator; If the operator currently being executed during the training of a large language model is the target operator, then the initialized injector is used to apply fault perturbation to the target data of the target operator during the current execution process.
[0008] Optionally, if the operator currently being executed during the training of the large language model is the target operator, then using the initialized injector, a fault perturbation is applied to the target data of the target operator during the current execution process, including: If the operator currently being executed during the training of a large language model is the target operator, then the hook function is registered for the target operator using the initialized injector; Based on the hook function, the target data of the target operator during the execution of the calculation is obtained, and the random values in the target data are bit-flipped to complete the fault disturbance.
[0009] Optionally, the comparative analysis of the training results of the perturbed model and the training results of the benchmark model includes: Based on the loss values in the training results of the perturbated model and the loss values in the training results of the benchmark model, a perturbation dataset and a benchmark dataset are formed. A comparative analysis was conducted on the perturbation dataset and the benchmark dataset.
[0010] Optionally, the comparative analysis of the perturbation dataset and the benchmark dataset includes: Calculate the relative error between the perturbation dataset and the benchmark dataset; Comparative analysis results are generated based on the relative error; the comparative analysis results include a characterization of the degree of influence of fault injection on the training convergence process of the large language model.
[0011] Secondly, this application discloses a reliability analysis device for a large model training system, comprising: The parsing module is used to parse the training process of the large language model used to generate text layer by layer to obtain parsed data, and to determine the target object for fault injection based on the parsed data. The fault injection information generation module is used to parameterize the target object, the number of fault occurrence steps set by the user, the duration interval and the disturbance mode based on the preset fault type, so as to form a fault template to be called, and generate corresponding fault injection information for each fault injection according to the fault template. The fault perturbation module is used to initialize the injector based on the fault injection information, apply fault perturbation to the target data of the target operator in the current execution process using the initialized injector, restore the original state of the target operator after the fault perturbation is applied, and continue to execute the training process of the large language model to obtain the perturbation model training result. The training result acquisition module is used to acquire the benchmark model training results generated during the training process of a large language model without applying fault perturbations. The comparative analysis module is used to compare and analyze the training results of the perturbated model and the training results of the benchmark model to obtain the comparative analysis results. An optimization module is used to optimize the training process of the large language model used to generate text based on the results of comparative analysis, so as to obtain an optimized large language model, and then use the optimized large language model to generate text.
[0012] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned reliability analysis method for large model training systems.
[0013] Fourthly, this application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed large model training system reliability analysis method.
[0014] As can be seen, this application provides a reliability analysis method for a large-scale model training system, including layer-by-layer analysis of the training process of a large language model used to generate text, obtaining the analyzed data, determining the target object for fault injection based on the analyzed data, realizing a refined decomposition of the large language model training process, accurately locating the core links in model training where fault injection can be performed, and ensuring that fault injection can accurately act on key nodes of the model; based on preset fault types, parameterizing the target object, the number of fault occurrence steps set by the user, the duration interval, and the perturbation method to form a fault template to be called, generating corresponding fault injection information for each fault injection based on the fault template, improving the standardization and consistency of fault injection, avoiding parameter chaos in each injection, reducing the operational complexity of fault injection, ensuring that the conditions for each fault injection are reproducible and traceable, and avoiding excessive or insufficient fault injection; initializing the injector based on the fault injection information, using the initialized injector to apply fault perturbation to the target data of the target operator in the current execution process, and then applying the fault perturbation to the target operator. The process involves restoring the target operator to its original state and continuing the training of the large language model to obtain the perturbed model training results. Without disrupting the original model structure or affecting the normal training process, a fault perturbation is performed on the target operator. The original state of the perturbed target operator is then restored to ensure that subsequent normal model training is not affected. The training results of the large language model without fault perturbation are obtained to ensure clear reference for comparative analysis, avoiding meaningless analysis and inability to determine the impact of faults due to the lack of a benchmark. A comparative analysis is performed between the perturbed model training results and the benchmark model training results to accurately pinpoint the specific impact of fault injection on the large language model training and clarify the specific problems caused by fault injection. Based on the comparative analysis results, the training process of the large language model used for text generation is optimized to obtain an optimized large language model. This optimized large language model is then used to generate text, improving model stability and text generation quality, enhancing the model's anti-interference ability in complex scenarios, and ensuring more accurate and reliable generated text. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0016] Figure 1 This is a flowchart of a reliability analysis method for a large model training system disclosed in this application; Figure 2This is a flowchart of the training workload analysis for a large language model disclosed in this application; Figure 3 This application discloses a flowchart of a fault injection execution control training process based on hook functions. Figure 4 This is a flowchart of a medical question-and-answer text generation method based on a large language model disclosed in this application; Figure 5 This is a structural diagram of a fault injection framework based on a mapping-simulation-analysis workflow disclosed in this application; Figure 6 This is a schematic diagram of the structure of a reliability analysis device for a large model training system disclosed in this application; Figure 7 This application provides a structural diagram of an electronic device. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] Large language models, as important pre-trained models for natural language processing, have been widely applied in fields such as intelligent question answering, text generation, and code generation due to their strong language understanding and generation capabilities. Currently, reliability analysis methods for large language model training primarily rely on statistical features such as gradients, parameter update amounts, and activation value ranges during training. By processing surrogate information, they generate sensitivity rankings or distributions for different layers and operators within the model. However, both the analytical basis and the results are built upon surrogate features. The conclusions mainly reflect the sensitivity differences of operators within the model and are insufficient to directly characterize the specific impact of soft errors on the training process. These methods typically perform static or phased evaluations based on a certain type of local features, making it difficult to depict the propagation and cumulative effects of soft errors during training, as well as the differences in impact between different training stages.
[0019] Large language models typically consist of word embedding layers, multi-layer Transformer (neural network) structures, and an output layer. The word embedding layer converts the input text into a numerical representation that the model can process. The multi-layer Transformer structure, through repeated stacking, forms the main body of the model, used for layer-by-layer modeling and feature extraction of the input sequence. The output layer maps the hidden states to generate corresponding prediction results. As the main computational component in the model training process, the Transformer structure involves large-scale numerical operations during forward propagation, backward propagation, and parameter updates, resulting in a computationally intensive and time-consuming training process. Hardware failures not only have a relatively higher probability of occurrence, but the numerical anomalies they introduce may also propagate and gradually amplify during subsequent training, thus affecting the model's training stability and final convergence results. Therefore, improving the accuracy of reliability analysis during the training of large language models, overcoming the shortcomings of traditional methods in representing the actual destructive power of soft errors, and improving model stability and text generation quality are problems that need to be solved in this field.
[0020] See Figure 1 As shown in the figure, this invention discloses a reliability analysis method for a large model training system, which may specifically include: Step S11: The training process of the large language model used to generate text is parsed layer by layer to obtain parsed data, and the target object for fault injection is determined based on the parsed data.
[0021] In this embodiment, the training process of the LLM (Large Language Model) used to generate text is analyzed layer by layer to obtain the hierarchical position of the target network layer involved in the training process, the execution direction and category of the target operator, and the data association relationship between intermediate tensors.
[0022] Due to the large computational scale, numerous execution stages, and diverse types of operators involved in the training process of large language models, different training stages, layers, operators, and sensitivities to numerical perturbations vary. Therefore, it is necessary to first analyze the training workload of large language models and identify target objects suitable for fault injection. Firstly, this application analyzes the training process of large language models layer by layer to obtain the hierarchical positions of the target network layers involved in the training process, the execution directions and categories of target operators, and the data relationships between intermediate tensors. Subsequently, based on the above analysis results, the abstract computational behavior during training is mapped to specific objects that can be injected with faults. This allows the injection targets to be precisely located at specific training steps, layers, operators, and tensor positions, thus providing an accurate object basis for subsequent fault injection.
[0023] In this step, the training workload analysis process for the large language model is as follows: Figure 2 As shown, the training workload refers to the ultra-large-scale pre-training process of a large language model based on the Transformer architecture. This workload requires the continuous execution of long-term, high-density activation value calculations and parameter updates using a large-scale computing cluster until the model reaches convergence. The workload of this invention not only includes the computational topology defined by the model architecture but also encompasses the long-cycle convergence process consisting of multiple iterations. Its technical characteristics lie in the enormous scale of the computational graph, the high degree of repetition in operator types (such as the attention mechanism and feedforward network in the Transformer block), and the high sensitivity of the entire training process to fluctuations in numerical accuracy.
[0024] Step S12: Based on the preset fault type, the target object, the number of fault occurrence steps set by the user, the duration interval, and the disturbance mode are parameterized to form a fault template to be called. According to the fault template, corresponding fault injection information is generated for each fault injection.
[0025] During the training of large language models, transient hardware failures can affect training computation in different ways, such as single bursts, continuous occurrences, or intermittent occurrences. Different failure modes may exhibit different behaviors in terms of timing, frequency, and scope of impact. Therefore, after mapping the error injection objects, it is necessary to further construct the failure modes and generate specific error injection information. First, based on preset failure types, the number of occurrence steps, duration interval, target location, and perturbation method of the failure are parameterized to form a callable failure template. Then, according to the established failure template, corresponding failure injection information is generated for each error injection, clearly representing when, where, and in what form the failure is injected. This application transforms abstract failure modes into structured failure injection information, achieving standardization and repeatable control of the subsequent error injection execution process, thereby providing fundamental support for failure impact analysis during training.
[0026] In this embodiment, according to user requirements, any one variable is selected from all variables and determined as the independent variable; among them, all variables include the fault occurrence stage, target operator type, target network layer, fault injection direction, and fault disturbance method; other variables besides the independent variable are used as control variables; based on the preset fault type, the independent variable, and the control variable, the target object, the number of fault occurrence steps set by the user, the duration interval, and the disturbance method are parameterized and described.
[0027] Step S13: Initialize the injector based on the fault injection information. Using the initialized injector, apply fault perturbation to the target data of the target operator in the current execution process. Then restore the original state of the target operator after applying the fault perturbation and continue to execute the training process of the large language model to obtain the perturbation model training result.
[0028] In this embodiment, the injector is initialized based on the fault injection information, and it is determined whether the operator currently being executed during the training of the large language model is the target operator. If the operator currently being executed during the training of the large language model is the target operator, the initialized injector is used to apply a fault perturbation to the target data of the target operator during the current execution process, and then the target operator after the fault perturbation is applied is restored to its original state.
[0029] Specifically, if the operator currently being executed during the training of the large language model is the target operator, then the initialized injector is used to apply fault perturbation to the target data of the target operator during the current execution process. This includes: if the operator currently being executed during the training of the large language model is the target operator, then the initialized injector is used to register a hook function for the target operator; based on the hook function, the target data of the target operator during the execution calculation process is obtained, and the random values in the target data are bit-flipped to complete the fault perturbation.
[0030] Step S14: Obtain the benchmark model training results generated during the training process of the large language model without applying fault perturbations.
[0031] Since the above steps have already completed the target object identification and fault injection information generation, this step introduces an error injection triggering mechanism during the model training execution phase to achieve online fault injection and establish a training process of a large language model without fault perturbation as a control baseline. The fault injection execution control training process based on the hook function is as follows: Figure 3As shown, firstly, the objects in the Injector are initialized using fault injection information. It is then determined whether the currently executed operator during the training of the large language model is the target operator. If the currently executed operator is the target operator, a hook function is registered for the target operator using the initialized Injector. A fault perturbation is applied to the target data during the current execution of the target operator. Then, the original state of the target operator after the fault perturbation is restored, and the training process of the large language model continues to obtain the perturbated model training result. Simultaneously, normal training without fault injection is performed under the same training configuration to obtain the baseline model training result generated by the training process of the large language model without fault perturbation. Through this method, a correspondence between training with and without fault perturbation can be established, providing an experimental basis for subsequent fault impact analysis.
[0032] Step S15: Compare and analyze the training results of the perturbated model and the training results of the benchmark model to obtain the comparison and analysis results.
[0033] In this embodiment, a perturbation dataset and a benchmark dataset are formed based on the loss value in the training results of the perturbed model and the loss value in the training results of the benchmark model; the perturbation dataset and the benchmark dataset are compared and analyzed to obtain the comparison analysis results.
[0034] The specific analysis process includes: calculating the relative error between the perturbation dataset and the benchmark dataset; generating comparative analysis results based on the relative error; and the comparative analysis results include a characterization of the degree of influence of fault injection on the training convergence process of the large language model.
[0035] Since both the training processes with and without fault perturbation have been completed, it is necessary to collect relevant data generated during training and evaluate the impact of fault injection based on the collected data, thereby obtaining comparative analysis results of the large language model training process. First, the loss values of the training set at each iteration step during the training process of the large language model with fault perturbation are recorded. Simultaneously, the loss values on the validation set are collected at different stages of training. At the same time, the corresponding benchmark model training data are acquired, forming a perturbation dataset and a benchmark dataset that can be used for comparative analysis. Further, the relative error is calculated based on the perturbation dataset and the benchmark dataset to generate comparative analysis results.
[0036] This application employs a controlled variable approach, using the fault occurrence stage, target operator type, fault mode, and fault injection direction as research dimensions. While keeping other conditions constant, it analyzes the differences in training loss evolution and validation loss shift caused by changes in a single variable. This allows for the study of the impact of different training stages, different operators, different error modes, and forward or backward propagation error injection methods on the model training process. Through this analysis, the sensitive stages, sensitive objects, and typical impact patterns of faults during training can be identified, leading to a comparative analysis of the training process of large language models and providing a basis for subsequent fault-tolerance strategy design.
[0037] Step S16: Optimize the training process of the large language model used to generate text based on the comparative analysis results to obtain the optimized large language model, and use the optimized large language model to generate text.
[0038] This invention addresses the problem of transient hardware failures during large language model training potentially introducing numerical biases and further corrupting silent data. It applies a fault injection framework based on a mapping-simulation-analysis workflow. This framework focuses on the characteristics of the large language model training workload, analyzing the workload layer by layer and establishing a mapping relationship from specific target operators in specific network layers at specific training stages to specific fault injection objects. This transforms the operator-level execution logic during training into target objects that can be precisely injected with faults. Furthermore, by parameterizing the fault occurrence steps, duration intervals, target locations, and perturbation methods, structured fault injection information is formed, enabling standardized construction and repeated invocation of fault patterns. To ensure the fault injection process aligns with the actual training execution process, this invention further implements online fault injection of target operators based on hook functions during training, and establishes a training process of a large language model without fault perturbation as a baseline. Meanwhile, by collecting the training loss of each iteration step and the validation loss of different stages in the training results of the perturbated model and the training results of the benchmark model, and calculating the relative error, and combining the control variable method to analyze the differences in training behavior under different training stages, different layers, different operator types, different fault modes and different error injection directions, we can obtain the sensitive stages, layers and operator objects that are easily affected by faults in the training process of large language models, and provide a basis for the design of subsequent fault tolerance strategies.
[0039] There are no restrictions on the specific type of text; it will be determined based on the specific scenario. For example, if the text is a medical question-and-answer text, it will be used to provide the largest model for intelligent question answering.
[0040] See Figure 4 As shown in the figure, this invention discloses a method for generating medical question-and-answer text based on a large language model, which may specifically include: Step S21: The training process of the large language model used to generate medical question-and-answer text is parsed layer by layer to obtain parsed data, and the target object for fault injection is determined based on the parsed data.
[0041] Step S22: Based on the preset fault type, the target object, the number of fault occurrence steps set by the user, the duration interval, and the disturbance mode are parameterized to form a fault template to be called. According to the fault template, corresponding fault injection information is generated for each fault injection.
[0042] Step S23: Initialize the injector based on the fault injection information. Using the initialized injector, apply fault perturbation to the target data of the target operator in the current execution process. Then restore the original state of the target operator after applying the fault perturbation and continue to execute the training process of the large language model to obtain the perturbation model training result.
[0043] Step S24: Obtain the benchmark model training results generated during the training process of the large language model without applying fault perturbations.
[0044] Step S25: Compare and analyze the training results of the perturbated model and the training results of the benchmark model to obtain the comparison and analysis results.
[0045] Step S26: Based on the comparative analysis results, optimize the training process of the large language model used to generate medical question and answer text to obtain an optimized large language model, and use the optimized large language model to generate medical question and answer text.
[0046] The structure of the fault injection framework based on the mapping-simulation-analysis workflow is as follows: Figure 5 As shown, the reliability analysis process of this invention is mainly divided into the following five stages: 1. Training workload analysis and error injection object mapping; 2. Fault mode construction and error injection information generation; 3. Fault injection execution based on hook function and comparative training; 4. Fault injection data collection and impact assessment.
[0047] This application utilizes the real training environment of a large language model, injecting online faults into the target operator during specific training phases using a hook function, and immediately restoring the original state after computation. By directly quantifying and comparing the loss changes and validation index differences between the perturbed model training results and the benchmark model training results, it overcomes the shortcomings of traditional surrogate analysis methods in characterizing the actual destructive power of soft errors, thus improving the accuracy of reliability assessment. Furthermore, this invention fully considers the different fault modes exhibited by transient hardware faults in real computing power clusters, supporting the construction of various fault templates such as single bursts, continuous occurrences, or intermittent occurrences. This multi-granularity fault simulation capability enables the evaluation process to more comprehensively and accurately characterize the actual impact of transient hardware faults on the training trajectory and final convergence results of the large language model, further enhancing the versatility and practical value of this analysis method.
[0048] By combining fine-grained workload parsing with parameterized fault template generation, precise fault injection control is achieved for different levels, operators, and computation directions. By avoiding deep modifications to the underlying training code, the invasiveness and implementation overhead of fault simulation are reduced, thus providing an efficient and reliable experimental basis for the design of subsequent fault tolerance strategies for large language models.
[0049] This application provides a reliability analysis method for a large-scale model training system. The method includes: layer-by-layer analysis of the training process of a large language model used to generate text, obtaining analyzed data; determining the target object for fault injection based on the analyzed data, thus achieving a refined decomposition of the large language model training process and accurately locating the core links in model training where fault injection can be performed, ensuring that fault injection can accurately act on key nodes of the model; based on preset fault types, parameterizing the target object, user-defined fault occurrence steps, duration interval, and perturbation method to form a fault template to be invoked; generating corresponding fault injection information for each fault injection based on the fault template, improving the standardization and consistency of fault injection, avoiding parameter chaos in each injection, reducing the operational complexity of fault injection, ensuring that the conditions for each fault injection are reproducible and traceable, and avoiding over- or under-injection of faults; initializing the injector based on the fault injection information, applying fault perturbation to the target data of the target operator in the current execution process using the initialized injector, and then applying the fault perturbation to the target operator. The process involves restoring the original state and continuing the training of the large language model to obtain the perturbed model training results. Without disrupting the original model structure or affecting the normal training process, fault perturbation is applied to the target operator. The original state of the perturbed target operator is then restored to ensure that subsequent normal model training is not affected. The training results of the benchmark model generated from the training process of the large language model without fault perturbation are obtained to ensure clear reference for comparative analysis and avoid meaningless analysis or inability to determine the impact of faults due to the lack of a benchmark. A comparative analysis is performed between the perturbed model training results and the benchmark model training results to accurately pinpoint the specific impact of fault injection on the large language model training and clarify the specific problems caused by fault injection. Based on the comparative analysis results, the training process of the large language model used for text generation is optimized to obtain an optimized large language model. This optimized large language model is then used to generate text, improving model stability and text generation quality, enhancing the model's anti-interference ability in complex scenarios, and ensuring more accurate and reliable generated text.
[0050] See Figure 6 As shown in the figure, an embodiment of the present invention discloses a reliability analysis device for a large model training system, which may specifically include: The parsing module 11 is used to parse the training process of the large language model used to generate text layer by layer to obtain parsed data, and to determine the target object for fault injection based on the parsed data. The fault injection information generation module 12 is used to parameterize the target object, the number of fault occurrence steps set by the user, the duration interval and the disturbance mode based on the preset fault type, so as to form a fault template to be called, and generate corresponding fault injection information for each fault injection according to the fault template. The fault perturbation module 13 is used to initialize the injector based on the fault injection information, apply fault perturbation to the target data of the target operator in the current execution process using the initialized injector, restore the original state of the target operator after the fault perturbation is applied, and continue to execute the training process of the large language model to obtain the perturbation model training result. Training result acquisition module 14 is used to acquire the benchmark model training results generated during the training process of a large language model without applied fault perturbation; The comparison analysis module 15 is used to compare and analyze the training results of the perturbed model and the training results of the benchmark model to obtain the comparison analysis results. The optimization module 16 is used to optimize the training process of the large language model used to generate text based on the comparative analysis results, so as to obtain an optimized large language model, and use the optimized large language model to generate text.
[0051] In some specific embodiments, the parsing module 11 may specifically include: The layer-by-layer parsing module parses the training process of the large language model used to generate text layer by layer to obtain the hierarchical position of the target network layer involved in the training process, the execution direction and category of the target operator, and the data relationship between intermediate tensors.
[0052] In some specific embodiments, the fault injection information generation module 12 may specifically include: The independent variable determination module is used to select any one variable from all variables and determine it as the independent variable according to user needs; among them, all variables include the fault occurrence stage, target operator type, target network layer, fault injection direction, and fault disturbance method; The control variable determination module is used to determine other variables besides the independent variable as control variables; The parameterized description module is used to parameterize the target object, the number of fault occurrence steps set by the user, the duration interval, and the disturbance mode based on the preset fault type, the independent variable, and the control variable.
[0053] In some specific embodiments, the fault disturbance module 13 may specifically include: The judgment module is used to determine whether the currently executed operator is the target operator during the training of the large language model; The fault perturbation application module is used to apply fault perturbations to the target data of the target operator during the current execution process by using the initialized injector if the operator currently being executed during the training of the large language model is the target operator.
[0054] In some specific embodiments, the fault disturbance module 13 may specifically include: The hook function registration module is used to register hook functions for the target operator if the currently executed operator is the target operator during the training of the large language model; The bit-flipping module is used to obtain the target data of the target operator during the execution of the calculation process based on the hook function, and to flip the bits of the random values in the target data to complete the fault disturbance.
[0055] In some specific embodiments, the comparison analysis module 15 may specifically include: The dataset generation module is used to generate perturbed datasets and benchmark datasets based on the loss values in the training results of the perturbed model and the loss values in the training results of the benchmark model. The dataset comparison and analysis module is used to compare and analyze perturbation datasets and benchmark datasets.
[0056] In some specific embodiments, the comparison analysis module 15 may specifically include: The relative error calculation module is used to calculate the relative error between the perturbation dataset and the benchmark dataset; The comparative analysis result generation module is used to generate comparative analysis results based on the relative error; the comparative analysis results include the degree of influence of fault injection on the training convergence process of the large language model.
[0057] Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the reliability analysis method for a large model training system executed by the electronic device disclosed in any of the foregoing embodiments.
[0058] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0059] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.
[0060] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform operations and processing on the data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the large model training system reliability analysis method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the large model training system reliability analysis device from external devices, as well as data collected by its own input / output interface 25.
[0061] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0062] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the steps of the large model training system reliability analysis method disclosed in any of the foregoing embodiments.
[0063] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0064] The reliability analysis method, apparatus, device, and storage medium for a large model training system provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A reliability analysis method for a large model training system, characterized in that, include: The training process of the large language model used to generate text is parsed layer by layer to obtain parsed data, and the target object for fault injection is determined based on the parsed data. Based on the preset fault type, the target object, the number of fault occurrence steps, the duration interval and the disturbance method set by the user are parameterized to form a fault template to be called. According to the fault template, corresponding fault injection information is generated for each fault injection. The injector is initialized based on the fault injection information. The initialized injector is used to apply fault perturbation to the target data of the target operator in the current execution process. Then, the target operator after the fault perturbation is restored to its original state, and the training process of the large language model is continued to obtain the training result of the perturbated model. Obtain the benchmark model training results generated during the training process of a large language model without applied fault perturbations; The training results of the perturbated model and the training results of the benchmark model are compared and analyzed to obtain the comparison analysis results. Based on the comparative analysis results, the training process of the large language model used to generate text is optimized to obtain an optimized large language model, and text is generated using the optimized large language model.
2. The reliability analysis method for a large model training system according to claim 1, characterized in that, The training process of the large language model used to generate text is analyzed layer by layer to obtain the parsed data, including: The training process of the large language model used to generate text is analyzed layer by layer to obtain the hierarchical position of the target network layer involved in the training process, the execution direction and category of the target operator, and the data association relationship between intermediate tensors.
3. The reliability analysis method for a large model training system according to claim 1, characterized in that, The parameterized description of the target object, the user-defined fault occurrence steps, the duration interval, and the disturbance method based on the preset fault type includes: Based on user requirements, select any one variable from all variables to determine the independent variable; among them, all variables include the fault occurrence stage, target operator type, target network layer, fault injection direction, and fault perturbation method; Other variables besides the independent variables are used as control variables; Based on the preset fault type, the independent variable, and the control variable, the target object, the number of fault occurrence steps, the duration interval, and the disturbance mode set by the user are parameterized and described.
4. The reliability analysis method for a large model training system according to claim 1, characterized in that, The step of applying fault perturbation to the target data of the target operator during the current execution process using the initialized injector includes: Determine whether the currently executed operator during the training of a large language model is the target operator; If the operator currently being executed during the training of a large language model is the target operator, then the initialized injector is used to apply fault perturbation to the target data of the target operator during the current execution process.
5. The reliability analysis method for a large model training system according to claim 4, characterized in that, If the operator currently being executed during the training of the large language model is the target operator, then using the initialized injector, a fault perturbation is applied to the target data of the target operator during the current execution process, including: If the operator currently being executed during the training of a large language model is the target operator, then the hook function is registered for the target operator using the initialized injector; Based on the hook function, the target data of the target operator during the execution of the calculation process is obtained, and the random values in the target data are bit-flipped to complete the fault disturbance.
6. The reliability analysis method for a large model training system according to any one of claims 1 to 5, characterized in that, The comparative analysis of the training results of the perturbated model and the training results of the baseline model includes: Based on the loss values in the training results of the perturbated model and the loss values in the training results of the benchmark model, a perturbation dataset and a benchmark dataset are formed. A comparative analysis was conducted on the perturbation dataset and the benchmark dataset.
7. The reliability analysis method for a large model training system according to claim 6, characterized in that, The comparative analysis of the perturbation dataset and the benchmark dataset includes: Calculate the relative error between the perturbation dataset and the benchmark dataset; Comparative analysis results are generated based on the relative error; the comparative analysis results include a characterization of the degree of influence of fault injection on the training convergence process of the large language model.
8. A reliability analysis device for a large model training system, characterized in that, include: The parsing module is used to parse the training process of the large language model used to generate text layer by layer to obtain parsed data, and to determine the target object for fault injection based on the parsed data. The fault injection information generation module is used to parameterize the target object, the number of fault occurrence steps set by the user, the duration interval and the disturbance mode based on the preset fault type, so as to form a fault template to be called, and generate corresponding fault injection information for each fault injection according to the fault template. The fault perturbation module is used to initialize the injector based on the fault injection information, apply fault perturbation to the target data of the target operator in the current execution process using the initialized injector, restore the original state of the target operator after the fault perturbation is applied, and continue to execute the training process of the large language model to obtain the perturbation model training result. The training result acquisition module is used to acquire the benchmark model training results generated during the training process of a large language model without applied fault perturbations. The comparative analysis module is used to compare and analyze the training results of the perturbated model and the training results of the benchmark model to obtain the comparative analysis results. An optimization module is used to optimize the training process of the large language model used to generate text based on the results of comparative analysis, so as to obtain an optimized large language model, and then use the optimized large language model to generate text.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the reliability analysis method for a large model training system as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the reliability analysis method for large model training systems as described in any one of claims 1 to 7.