A method and device for breakpoint retraining of intelligent computing cloud platforms based on heterogeneous computing power

Abstract

This invention provides a method and apparatus for resuming training after breakpoints on an intelligent computing cloud platform based on heterogeneous computing power. It relates to the fields of intelligent computing centers, smart computing centers, computing infrastructure, and smart computing cloud technologies. The method includes: Step S1, acquiring breakpoint data of the model training task executed on the source chip, and normalizing the breakpoint data into a chip-independent intermediate representation; Step S2, acquiring the capability data of the target chip; Step S3, determining a target parallel partitioning scheme based on the chip-independent intermediate representation and capability data; Step S4, repartitioning the chip-independent intermediate representation according to the target parallel partitioning scheme and distributing it to the target chip; Step S5, adapting the computational precision and operators required to continue executing the model training task on the target chip to continue the training task. The target parallel partitioning scheme of this invention can adapt to the target chip, improving the success rate of resuming training after breakpoints. By adapting the computational precision and operators, the stable operation of the training task can be guaranteed.

Description

Technical Field

[0001] This invention relates to the fields of intelligent computing centers, smart computing centers, computing infrastructure, and smart cloud technologies, specifically to a method and apparatus for resuming training at breakpoints on an intelligent computing cloud platform based on heterogeneous computing power. Background Technology

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "smart computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models) by utilizing large-scale heterogeneous computing resources, including general-purpose and intelligent computing power. Intelligent computing centers encompass facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0004] "Intelligent computing center" includes, but is not limited to, "intelligent computing center".

[0005] "Intelligent computing center" or artificial intelligence computing center is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting artificial intelligence computing architecture.

[0006] "Computing power" is the core of "intelligent computing center" and "smart computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to execute a certain computing requirement. It is the computing power to achieve the target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] In intelligent computing cloud platforms, model training typically requires periodically saving checkpoints. When the intelligent computing cloud platform needs to replace different types of chips due to reasons such as cost, supply chain security, or computing power diversification, training can continue on the target chip based on the saved checkpoints. However, the current checkpoint is incompatible with the physical partitioning method of the source chip and the memory capacity, interconnect topology, and parallel strategy of the target chip, making it impossible to directly continue training the model on a target chip of a different type than the source chip.

[0008] It is evident that since the emergence of intelligent computing centers, the problem of cross-vendor chip replacement and continued training has been a pressing issue that needs to be addressed in this field. This is because the strong coupling between model training checkpoints and the chip type and parallel partitioning strategy of the source chip makes it impossible to directly perform training. Summary of the Invention

[0009] This invention provides a method and apparatus for resuming training at breakpoints on an intelligent computing cloud platform based on heterogeneous computing power, in order to solve the problem in the prior art that cross-vendor chip replacement training cannot be directly performed due to the strong coupling between model training checkpoints and the chip type and parallel splitting strategy of the source chip.

[0010] To solve the above problems, the present invention is implemented as follows: In a first aspect, the present invention provides a method for resuming training from breakpoints on an intelligent computing cloud platform based on heterogeneous computing power, comprising: step S1, acquiring breakpoint data of the intelligent computing cloud platform executing a model training task on a source chip in heterogeneous computing power, and normalizing the breakpoint data into a chip-independent intermediate representation, wherein the chip-independent intermediate representation includes at least the topological metadata of the training task, the weight data of the logical tensor, the first and second moments of the optimizer corresponding to the logical tensor, and the state of the random number generator during the training process; Step S2: Obtain the capability data of the target chip in the heterogeneous computing power that is of a different chip type than the source chip; Step S3: Based on the chip-independent intermediate representation and the capability data, determine the target parallel partitioning scheme for the chip-independent intermediate representation; Step S4: Re-segment the chip-independent intermediate representation according to the target parallel segmentation scheme and distribute and load it onto the target chip; Step S5: Based on the computational precision and operators of the source chip and the target chip respectively, adapt the computational precision and operators required to continue the training task of the model on the target chip, so as to continue the training task of the model on the target chip.

[0011] In some embodiments, step S4 includes: Step S4.1: Determine the tensor parallelism, pipeline parallelism, and data parallelism of the target parallel partitioning scheme, as well as the first partitioning method for each parallelism. Step S4.2: According to the tensor parallelism, pipeline parallelism, and data parallelism, as well as the first partitioning method, partition the weight data of the logic tensor, the first moment and the second moment of the optimizer to obtain the partitioned data. Step S4.3: Load the fragmented data onto the target chip and restore the state of the random number generator.

[0012] In some embodiments, step S5 includes: Step S5.1: Obtain the logical value precision of the chip-independent intermediate representation and the actual execution precision of the target chip; Step S5.2: In response to the inconsistency between the logical numerical precision and the actual execution precision, obtain the loss value obtained by the model through forward propagation on the target chip; Step S5.3: During the process of the model performing the training task, the loss value is subjected to loss scaling processing to adapt to the computational accuracy required to continue performing the training task of the model on the target chip.

[0013] In some embodiments, step S5.3 includes: Step S5.3.1: Obtain the loss scaling factor for each training step; Step S5.3.2: Before the backpropagation of the model, the loss value is multiplied based on the loss scaling factor to obtain the loss value after the calculation accuracy is adapted. Step S5.3.3: Before updating the model parameters, perform a division operation on the gradient values ​​obtained by backpropagation based on the loss scaling factor to adapt to the computational accuracy required to continue the training task of the model on the target chip.

[0014] In some embodiments, step S5.3.1 includes at least one of the following steps: Step S5.3.1.1: For the t-th training step, in response to the gradient overflow of the gradient value of the t-th training step, reduce the loss scaling factor of the t-th training step to obtain the loss scaling factor of the (t+1)-th training step, and discard the gradient value of the t-th training step. Step S5.3.1.2: In response to the absence of gradient overflow in the gradient values ​​of K consecutive training steps, the loss scaling factor of the t-th training step is amplified to obtain the loss scaling factor of the (t+1)-th training step; wherein, the t-th training step is the last training step in the K consecutive training steps. Step S5.3.1.3: In response to the absence of gradient overflow in the gradient value of the t-th training step and the fact that the t-th training step has not reached the last training step in the K consecutive training steps, the loss scaling factor of the t-th training step is used as the loss scaling factor of the (t+1)-th training step.

[0015] In some embodiments, step S5 further includes: Step S5.1': Determine the first operator used by the model to perform the training task on the source chip; Step S5.2': Obtain the set of operators supported by the target chip; Step S5.3': In response to the first operator not existing in the operator set, determine the second operator corresponding to the first operator to adapt the operator required to continue the training task of the model on the target chip.

[0016] In some embodiments, step S5.3' includes: Step S5.3.1': Obtain the operator replacement table; Step S5.3.2': Based on the first operator, query the operator replacement table to obtain an operator combination with the same calculation logic as the first operator, and use it as the second operator; or, Step S5.3.3': Based on the first operator, query the operator replacement table to obtain an operator with a similar value to the first operator, and use it as the second operator.

[0017] In some embodiments, step S3 includes: Step S3.1: Obtain the memory capacity of the target chip from the capability data; Step S3.2: Determine the memory requirements of the chip-independent intermediate representation; Step S3.3: Under the constraint of the video memory capacity, based on the video memory requirements, determine multiple candidate parallel partitioning schemes; Step S3.4: Based on the topology metadata, determine the first communication group for each of the multiple candidate parallel splitting schemes; Step S3.5: Based on the first communication packet, determine the target parallel segmentation scheme from the plurality of candidate parallel segmentation schemes.

[0018] In some embodiments, step S3.4 includes: Step S3.4.1: Obtain the second partitioning method of tensor parallelism, pipeline parallelism and data parallelism corresponding to the topology metadata; Step S3.4.2: Based on the second segmentation method, the parallelism of the multiple candidate parallel segmentation schemes is dimensionally aligned to obtain the first communication group.

[0019] In some embodiments, step S3.5 includes: Step S3.5.1: Determine the second communication group that needs to be used for high-frequency communication from the first communication group; Step S3.5.2: Determine the bandwidth domain corresponding to the second communication group, and determine the candidate parallel segmentation scheme corresponding to the second communication group with the largest bandwidth in the bandwidth domain, as the target parallel segmentation scheme.

[0020] Secondly, the present invention also provides a breakpoint training device for an intelligent computing cloud platform based on heterogeneous computing power, comprising: a processing module, used to acquire breakpoint data of the intelligent computing cloud platform executing model training tasks on the source chip in heterogeneous computing power, and normalize the breakpoint data into a chip-independent intermediate representation, wherein the chip-independent intermediate representation includes at least the topological metadata of the training task, the weight data of the logical tensor, the first and second moments of the optimizer corresponding to the logical tensor, and the state of the random number generator during the training process; The acquisition module is used to acquire the capability data of a target chip in the heterogeneous computing power that is of a different chip type than the source chip; The determination module is used to determine a target parallel partitioning scheme for the chip-independent intermediate representation based on the chip-independent intermediate representation and the capability data; The segmentation module is used to re-segment the chip-independent intermediate representation according to the target parallel segmentation scheme and distribute and load it onto the target chip; The training continuation module is used to adapt the computational precision and operators required to continue the training task of the model on the target chip based on the respective computational precision and operators of the source chip and the target chip, so as to continue the training task of the model on the target chip.

[0021] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps in the breakpoint resume training method of the intelligent computing cloud platform based on heterogeneous computing power as described in the first aspect above.

[0022] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps in the breakpoint continuation training method of the intelligent computing cloud platform based on heterogeneous computing power as described in the first aspect above.

[0023] Fifthly, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the breakpoint continuation training method of the intelligent computing cloud platform based on heterogeneous computing power as described in the first aspect above.

[0024] In this invention, step S1 involves acquiring breakpoint data of the intelligent computing cloud platform in the heterogeneous computing power executing model training tasks on the source chip, and normalizing the breakpoint data into a chip-independent intermediate representation; step S2 involves acquiring the capability data of a target chip in the heterogeneous computing power that has a different chip type than the source chip; step S3 involves determining a target parallel partitioning scheme for the chip-independent intermediate representation based on the chip-independent intermediate representation and capability data; step S4 involves repartitioning the chip-independent intermediate representation according to the target parallel partitioning scheme and distributing it to the target chip; and step S5 involves adapting the computational precision and operators required to continue executing the model training task on the target chip based on the respective computational precision and operators of the source chip and the target chip, so that the model training task can continue to be executed on the target chip. Thus, by acquiring the chip-independent intermediate representation and solving the target parallel partitioning scheme for partitioning the chip-independent intermediate representation based on the capability data of the target chip, the target parallel partitioning scheme can be adapted to the capability data of target chips of different chip types, avoiding partitioning errors and improving the success rate of resuming training after breakpoints. By adapting the computational precision and operators required to continue training the model on the target chip, the hardware differences between the source and target chips due to their different chip types can be resolved. This ensures that when the model is trained across different types of chips using heterogeneous computing power, the training task can run stably on the target chip. By scaling the computational precision of the source chip and replacing its operators, the computational precision and operators of the source chip can be adapted to target chips of different chip types. This improves the universality and applicability of the computational precision and operator adaptation, thereby reducing the cost of adapting between different chip types. Attached Figure Description

[0025] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of a breakpoint retraining method for an intelligent computing cloud platform based on heterogeneous computing power, provided by the present invention. Figure 2 This is a structural diagram of the breakpoint resume training system of the intelligent computing cloud platform based on heterogeneous computing power provided by the present invention; Figure 3 This is a flowchart of the breakpoint continuation training method for the intelligent computing cloud platform based on heterogeneous computing power provided by the present invention; Figure 4 This is a flowchart of the parallel strategy remapping process provided by the present invention. Figure 5This is a structural diagram of a breakpoint resuming training device for an intelligent computing cloud platform based on heterogeneous computing power provided by the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in this invention. Detailed Implementation

[0027] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] The “computing power” mentioned in this invention refers to: the ability of computer equipment or computing / data center to process information; the ability of computer hardware and software to work together to perform a certain computing requirement; the computing power to achieve the target result output by processing information data; and a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, mainly providing services to society through computing power infrastructure.

[0029] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0030] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capability of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capability.

[0031] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), while the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB). Disaster recovery ratio is an important indicator of security and reliability.

[0032] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0033] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0034] The “computing power” mentioned in this invention includes: general computing power, intelligent computing power, and supercomputing power.

[0035] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0036] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0037] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0038] The "intelligent computing center" described in this invention refers to a facility that, through the use of large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), primarily provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0039] The "intelligent computing cloud platform" mentioned in this invention, abbreviated as "intelligent computing cloud", refers to a cloud computing platform that integrates hardware and software resources based on an intelligent computing center.

[0040] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0041] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0042] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0043] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0044] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0045] The "model" mentioned in this invention includes, but is not limited to, "large language model" and "multimodal large model".

[0046] The "large language model" mentioned in this invention refers to a large-scale language model (LLM), which is a language model with a large number of parameters. It is designed to understand and generate human language, and is trained with a large amount of text data. It can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0047] The “Multimodal Large Models” mentioned in this invention refer to models that combine multimodal information such as text, images, videos, and audio for training, including but not limited to multimodal large language models.

[0048] The "heterogeneous computing power" of this invention refers to a computing power supply model that integrates various computing resources such as chips (CPUs or GPUs) through an intelligent computing cloud platform, and relies on virtualization, containerization and intelligent scheduling technologies to achieve collaborative scheduling and efficient utilization of diverse computing power.

[0049] The “computing power running task” mentioned in this invention refers to a specific workload or job executed on computing power resources that requires a certain amount of computing power support, usually involving complex data processing, numerical calculation, model training or simulation scenarios.

[0050] It is important to emphasize that the breakpoint training method based on heterogeneous computing power in the intelligent computing cloud platform of this invention can realize the migration of model training tasks from one chip to another chip of a different chip type for continued training. This is far more difficult than the existing technology of migrating training tasks to a chip of the same chip type for continued training.

[0051] In this invention, different chip types can refer to different chip manufacturers.

[0052] In other words, this invention performs breakpoint resume training between chips of different types manufactured by different chip manufacturers, while existing technologies perform breakpoint resume training between chips of the same type manufactured by the same company. The following is a detailed technical analysis of why the difficulty of breakpoint resume training in this invention is far greater than that in existing technologies: I. The Native Heterogeneous Barrier of Chip Instruction Sets and Hardware Architecture Chips from different manufacturers often have incompatible instruction sets, employing completely independent instruction set architectures. Their underlying instructions are mutually unrecognizable, causing operators in directly loaded breakpoint data to fail to execute, thus preventing the training task from continuing. Furthermore, the computing power of different types of chips is unequal; when the precision saved on the source chip is directly transferred to the target chip, underflow and result drift are highly likely to occur. Traditional breakpoint-based training methods do not consider the incompatibility of operators and precision across different chip types, severely impacting the success rate of continuing model training tasks on different types of chips.

[0053] II. Consistency Challenges in Full Migration of Breakpoint Data For chips from the same vendor, breakpoint training can directly utilize the existing parallel partitioning scheme. However, in cross-vendor heterogeneous computing scenarios, the parallel partitioning scheme of the source chip is completely unsuitable for the computing power topology of the target chip. Moreover, when resuming training across different vendor chips, state misalignment can occur due to differences in computational accuracy. Direct loading will cause the training convergence trajectory to deviate completely, making breakpoint training impossible.

[0054] In summary, existing breakpoint training solutions are difficult to apply in heterogeneous computing scenarios. The industry urgently needs a breakpoint training solution that can load training tasks onto target chips of different types than the source chip and ensure successful execution of the training tasks on the target chip. The breakpoint training method based on heterogeneous computing power in the intelligent computing cloud platform of this invention solves this pressing problem in the field of computing power.

[0055] Please see Figure 1 , Figure 1 This is a flowchart of a breakpoint continuation training method for an intelligent computing cloud platform based on heterogeneous computing power, provided by the present invention. Figure 1 As shown, the method includes: Step S1: Obtain the breakpoint data of the intelligent computing cloud platform in the heterogeneous computing power executing the model training task on the source chip, and normalize the breakpoint data into a chip-independent intermediate representation.

[0056] It should be noted that the training task of the model in this invention can be one of the computing power running tasks.

[0057] It should be noted that when a model is trained on an intelligent computing cloud platform, it relies on chips to provide computing power to the platform. The chip where the model is currently being trained is called the source chip. The source chip is one of the chips in the heterogeneous computing power.

[0058] During training, the model can periodically save checkpoints to preserve breakpoint data. This allows the model to retrieve breakpoint data based on the saved checkpoints when the chip needs to be replaced for model training.

[0059] In some embodiments, if a chip needs to be replaced at the current moment, the checkpoint closest to the current moment can be determined based on the current moment and the checkpoint's retention period. Based on the retained checkpoint, breakpoint data including model weights, optimizer states, and other data can be determined as breakpoint data for the intelligent computing cloud platform to execute model training tasks on the source chip in heterogeneous computing power.

[0060] It should be noted that when saving breakpoint data, its saved format is strongly coupled with the physical parallel partitioning scheme used during saving and the numerical precision format of the chip. In order to decouple the breakpoint data from the chip, the breakpoint data can be normalized to normalize the breakpoint data into a chip-independent intermediate representation.

[0061] Optionally, the chip-independent intermediate representation includes at least the topological metadata of the training task, the weight data of the logical tensor, the first and second moments of the optimizer corresponding to the logical tensor, and the state of the random number generator during the training process.

[0062] Optionally, the chip-independent intermediate representation can be defined as: (1) in, Global shape description for the i-th logical tensor , Let be the logical precision of the i-th logical tensor. The total number of logical tensors. The first moment of the optimizer for the i-th logical tensor Let be the second-order moment of the optimizer for the i-th logical tensor. The state of the random number generator during the training process. This is topology metadata. This is a chip-independent intermediate representation.

[0063] The topology metadata includes the parallel partitioning scheme for the training task, namely tensor parallelism, pipeline parallelism, data parallelism, and their respective partitioning methods.

[0064] In some embodiments, after periodically saving the breakpoint data, the breakpoint data can be normalized to obtain a chip-independent intermediate representation.

[0065] Alternatively, when saving breakpoint data, the breakpoint data can be directly normalized to periodically save the obtained chip-independent intermediate representation.

[0066] The beneficial technical effect of step S1 is that by normalizing the breakpoint data into a chip-independent intermediate representation, the breakpoint data and the chip can be decoupled during the model training process. This is beneficial for continuing to train the model directly based on the chip-independent intermediate representation after changing to a different chip type.

[0067] Step S2: Obtain the capability data of the target chip, which has a different chip type from the source chip in the heterogeneous computing power.

[0068] Among them, the target chip, which is of a different type than the source chip, can mean that the target chip and the source chip can be chips from different manufacturers in heterogeneous computing power.

[0069] In other words, the target chip and the source chip are manufactured by different companies, and their corresponding chip types are also different.

[0070] For example, heterogeneous computing power consists of chip 1, chip 2, and chip 3, each corresponding to a different chip type. If the intelligent computing cloud platform performs a model training task on chip 1, then chip 1 is the source chip.

[0071] If the intelligent computing cloud platform needs to be replaced with chip 3 for model training, then chip 3 is the target chip.

[0072] In some embodiments, the target chip's capability data may include the chip's memory capacity, the numerical precision formats supported by the chip, interconnect topology bandwidth, and the set of operators supported by the chip. The numerical precision formats supported by the chip may be the chip's actual execution precision.

[0073] In some embodiments, the capability data of the target chip can be defined as: (2) in, This represents the memory capacity of the target chip. The numerical precision format supported by the target chip. Let be the interconnect bandwidth function between any two chips. The set of operators supported by the target chip. This is capability data.

[0074] Optionally, capability data of the target chip can be obtained from the development information of the target chip manufacturer.

[0075] Optionally, the target chip can also be tested to obtain capability data of the target chip.

[0076] In this invention, the source chip and the target chip can be a chip cluster.

[0077] The beneficial technical effect of step S2 is that by acquiring the capability data of the target chip, the capability data can be used as a constraint to automatically determine the segmentation scheme for continuing to execute the training task of the model on the target chip, thereby improving the automation of model training.

[0078] Step S3: Based on chip-independent intermediate representation and capability data, determine the target parallel partitioning scheme for chip-independent intermediate representation.

[0079] Understandably, since the source chip and the target chip have different hardware and computing power, the segmentation scheme of the source chip cannot be directly applied to the target chip. Therefore, it is necessary to re-segment the chip-independent intermediate representation so that the segmented data can adapt to the hardware differences and achieve stable and efficient operation of the training task.

[0080] In some embodiments, the hardware constraints of the target chip can be determined based on capability data. Under the hardware constraints, multiple candidate parallel partitioning schemes can be obtained by enumerating parallel partitioning schemes. Thus, the target parallel partitioning scheme with chip-independent intermediate representation can be determined from the multiple candidate parallel partitioning schemes.

[0081] Optionally, the hardware constraint of the target chip can be the memory capacity of the target chip. In other words, the candidate parallel partitioning scheme can completely partition the chip-independent intermediate representation and meet the memory capacity constraint of the target chip.

[0082] Understandably, the memory capacity constraint is used to limit the memory usage of each candidate parallel partitioning scheme. Multiple parallel partitioning schemes can be enumerated first, and their corresponding memory usage determined. Parallel partitioning schemes with memory usage less than or equal to the memory capacity can then be considered as candidate parallel partitioning schemes.

[0083] Among them, the candidate parallel partitioning scheme refers to the scheme for partitioning the weight data of the logic tensor in the chip-independent intermediate representation, as well as the first and second moments of the optimizer corresponding to the logic tensor.

[0084] In some embodiments, a candidate parallel partitioning scheme can be represented as (TP', PP', DP'), where TP stands for Tensor Parallelism, PP stands for Pipeline Parallelism, and DP stands for Data Parallelism. TP can be understood as tensor parallel grouping; PP can be understood as pipeline parallel grouping; and DP can be understood as data parallel grouping.

[0085] In other words, TP' refers to the tensor parallel group in the candidate parallel partitioning scheme; PP' refers to the pipeline parallel group in the candidate parallel partitioning scheme; and DP' refers to the data parallel group in the candidate parallel partitioning scheme.

[0086] In other words, the candidate parallel partitioning scheme has multiple parallel groups, and the target parallel partitioning scheme of chip-independent intermediate representation can be determined from the candidate parallel partitioning schemes based on the communication frequency of each parallel group.

[0087] In some embodiments, parallel packets of high-frequency communication can be determined and mapped to different interconnect bandwidths to obtain parallel packets assigned to high interconnect bandwidths, thereby using the candidate parallel segmentation scheme corresponding to the packet as the target parallel segmentation scheme.

[0088] Optionally, high interconnect bandwidth can be defined as interconnect bandwidth exceeding a set threshold.

[0089] As an example, the process of determining a target parallel partitioning scheme can be divided into two stages of solution, where the first and second stages handle different content.

[0090] For example, the first stage deals with hard constraints, namely memory capacity constraints, to ensure that the memory usage of the candidate parallel partitioning schemes obtained by enumeration is less than or equal to the memory capacity; the second stage deals with soft objectives, to make the candidate parallel partitioning schemes corresponding to the parallel groups with high interconnect bandwidth the target parallel partitioning schemes.

[0091] In some embodiments, the objective parallel partitioning scheme is the only solution for solving the problem in two stages; that is, there is only one objective parallel partitioning scheme.

[0092] The beneficial technical effect of step S3 is that after obtaining the capability data of the target chip, the solution process of the target parallel partitioning scheme can be automatically executed based on the capability data of the target chip. That is, by generating multiple candidate parallel partitioning schemes corresponding to chip-independent intermediate representations, a target parallel partitioning scheme is determined from the candidate parallel partitioning schemes. This allows the target parallel partitioning scheme to be directly adapted to the target chip, which is of a different chip type than the source chip. Moreover, there is no need to manually rewrite the parallel logic for the target chip, which reduces the adaptation cycle and labor costs. This is beneficial for reducing the adaptation cost of model training and maximizing the utilization of computing power in the process of adapting heterogeneous computing power across chips.

[0093] Step S4: Re-segment the chip-independent intermediate representation according to the target parallel segmentation scheme and distribute and load it to the target chip.

[0094] In some embodiments, the target parallel partitioning scheme may indicate the tensor parallelism, pipeline parallelism, and data parallelism of tensor parallel groups, pipeline parallel groups, and data parallel groups, as well as a first partitioning method for each parallelism.

[0095] In other words, the chip-independent intermediate representation can be segmented according to tensor parallelism, pipeline parallelism, and data parallelism using the first segmentation method to obtain the segmented data.

[0096] Furthermore, the fragmented data can be loaded onto the target chip so that the target chip can store the data required to perform the model training task.

[0097] In some embodiments, when segmenting the chip-independent intermediate representation, the loading location of each data segment on the target chip, i.e., the video memory space, can be determined, so that the segmented data segments can be loaded into the corresponding video memory space.

[0098] In some embodiments, after being loaded into the target chip, the fragmented data after loading can be compared to see if the chip-independent intermediate representation is consistent, so as to ensure that there is no data deviation.

[0099] Optionally, it can also be verified whether the sharded data is consistent with the target parallel sharding scheme to avoid sharding errors.

[0100] The beneficial technical effect of step S4 is that by re-segmenting the chip-independent intermediate representation according to the target parallel segmentation scheme and distributing it to the target chip, the segmented data can be adapted to the target chip, achieving cross-chip compatibility of heterogeneous computing power and further improving the success rate of migrating training tasks to the target chip.

[0101] Step S5: Based on the computational precision and operators of the source chip and the target chip respectively, adapt the computational precision and operators required to continue the model training task on the target chip so as to continue the model training task on the target chip.

[0102] It should be noted that after the chip-independent intermediate representation is re-segmented and distributed to the target chip according to the target parallel segmentation scheme, the model has not yet started to continue the training task on the target chip. At this time, the data recovery and initialization before training are completed.

[0103] In some embodiments, after loading the chip-independent intermediate representation onto the target chip, the model calculates the model loss value on the target chip using the loaded weights, i.e., performs forward propagation of the model, to formally begin continuing the model training task on the target chip.

[0104] It should be noted that the model training task continues to be executed on the target chip by performing forward propagation, back propagation, and parameter updates.

[0105] During the forward propagation process, the model's loss value is obtained, and the accuracy of this loss value is the same as the computational accuracy of the source chip.

[0106] Since the computational precision of the source chip and the target chip may differ, it may be necessary to adapt the computational precision required to continue the model training task on the target chip, and then continue the model training task on the target chip after the computational precision has been adapted.

[0107] In other words, after obtaining the loss value through forward propagation, the loss value can be scaled to make its accuracy the same as that of the target chip. This allows backpropagation to be performed to obtain a gradient with the same accuracy as the target chip. Before updating the parameters, the gradient can be scaled to restore the computational accuracy, and the parameters can be updated based on this gradient value.

[0108] Optionally, the loss value and gradient can be scaled by setting a loss scaling factor.

[0109] It should be noted that when training a model on a chip, operators are needed to perform the model training task. The operators may be the same or different between different chips.

[0110] In other words, we can first determine whether the source chip and the target chip support the same operators. If they are different, we can replace the operators on the source chip with equivalent or approximate operators supported by the target chip to adapt the operators required to continue the training task of the model on the target chip.

[0111] In some embodiments, if the computational precision and operators of the source chip are the same as those of the target chip, the model training task can be directly performed on the target chip.

[0112] The beneficial technical effect of step S5 is that by adapting the computational precision and operators required to continue the model training task on the target chip, model training can be performed on the target chip with the computational precision of chip-independent intermediate representations. Furthermore, even if the target chip and the source chip are of different types, model training can be performed on the target chip using operators equivalent to those on the source chip. This solves the problem of training failure after chip replacement caused by the strong coupling between chip-independent intermediate representations and the chip itself. It also facilitates chip replacement without needing to switch to a chip with the same computational precision and operators as the source chip, i.e., without needing to switch to a chip of the same type as the source chip, thus improving chip selectivity. By scaling the computational precision and replacing the operators, it is beneficial to adapt to different chip types, improving the applicability of the intelligent computing cloud platform for breakpoint resume training based on heterogeneous computing power, and further reducing the cost of chip adaptation.

[0113] In this invention, step S1 involves acquiring breakpoint data of the intelligent computing cloud platform in the heterogeneous computing power executing model training tasks on the source chip, and normalizing the breakpoint data into a chip-independent intermediate representation; step S2 involves acquiring the capability data of the target chip in the heterogeneous computing power, which has a different chip type than the source chip; step S3 involves determining the target parallel partitioning scheme for the chip-independent intermediate representation based on the chip-independent intermediate representation and capability data; step S4 involves repartitioning the chip-independent intermediate representation according to the target parallel partitioning scheme and distributing it to the target chip; and step S5 involves adapting the computational precision and operators required to continue executing the model training task on the target chip based on the respective computational precision and operators of the source chip and the target chip, so that the model training task can continue to be executed on the target chip. Therefore, by acquiring the chip-independent intermediate representation and solving the target parallel partitioning scheme based on the capability data of the target chip, the target parallel partitioning scheme can be adapted to the capability data of target chips of different chip types, avoiding partitioning errors and improving the success rate of resuming training after breakpoints. By adapting the computational precision and operators required to continue training the model on the target chip, the hardware differences between the source and target chips due to their different chip types can be resolved. This ensures that when the model is trained across different types of chips using heterogeneous computing power, the training task can run stably on the target chip. By scaling the computational precision of the source chip and replacing its operators, the computational precision and operators of the source chip can be adapted to target chips of different chip types. This improves the universality and applicability of the computational precision and operator adaptation, thereby reducing the cost of adapting between different chip types.

[0114] In some embodiments, step S4 includes: Step S4.1: Determine the tensor parallelism, pipeline parallelism, and data parallelism of the target parallel partitioning scheme, as well as the first partitioning method for each parallelism.

[0115] In some embodiments, after determining the target parallel partitioning scheme, the tensor parallelism, pipeline parallelism, and data parallelism included in the target parallel partitioning scheme, as well as the first partitioning method for each parallelism, can be obtained. Based on the first partitioning method, the chip-independent intermediate representation can be partitioned into tensor parallel groups, pipeline parallel groups, and data parallel groups corresponding to tensor parallelism, pipeline parallelism, and data parallelism.

[0116] The beneficial technical effect of step S4.1 is that by clarifying the tensor parallelism, pipeline parallelism, and data parallelism of the target parallel segmentation scheme, as well as the first segmentation method of each parallelism, the chip-independent intermediate representation can be accurately segmented, which is conducive to accurately loading multiple parallel groups onto the target chip, so that the target chip can be fully loaded with the chip-independent intermediate representation, and further improve the success rate of model training.

[0117] Step S4.2: According to tensor parallelism, pipeline parallelism, data parallelism, and the first partitioning method, partition the weight data of the logic tensor, the first moment and the second moment of the optimizer to obtain the partitioned data.

[0118] In some embodiments, a first partitioning method can be used to partition the weight data of the logic tensor, the first moment and the second moment of the optimizer into tensor parallelism, pipeline parallelism and data parallelism respectively, to obtain the tensor parallel grouping, pipeline parallel grouping and data parallel grouping corresponding to the weight data of the logic tensor, the first moment and the second moment of the optimizer.

[0119] In other words, the fragmented data after splitting includes tensor parallel grouping, pipeline parallel grouping, and data parallel grouping.

[0120] For example, when splitting tensor parallelism, the tensor can be split along the feature dimension of the weight data of the logical tensor according to the tensor parallelism to obtain tensor parallel groups.

[0121] For pipeline parallelism partitioning, the optimizer's first and second moments can be divided into pipeline stages according to pipeline parallelism, with each stage corresponding to a pipeline parallel group.

[0122] For data parallelism partitioning, the training data can be partitioned into non-overlapping datasets according to the degree of data parallelism, with each dataset corresponding to a data parallelism group.

[0123] In some embodiments, tensor parallel groups, pipeline parallel groups, and data parallel groups together constitute a chip-independent intermediate representation, and there is no redundant division among tensor parallel groups, pipeline parallel groups, and data parallel groups.

[0124] The beneficial technical effect of step S4.2 is that by dividing the weight data of the logic tensor, the first moment and the second moment of the optimizer, the divided data can be adapted to the topological parallel logic of the target chip, eliminating the hardware dependence of the data on the chip type, which is conducive to realizing the breakpoint continuation training of heterogeneous computing power across chips.

[0125] Step S4.3: Load the fragmented data onto the target chip and restore the state of the random number generator.

[0126] In some embodiments, when segmenting the chip-independent intermediate representation, the loading location of each data segment on the target chip, i.e., the video memory space, can be determined, so that the segmented data segments can be loaded into the corresponding video memory space.

[0127] In some embodiments, after being loaded into the target chip, the fragmented data after loading can be compared to see if the chip-independent intermediate representation is consistent, so as to ensure that there is no data deviation.

[0128] Optionally, it can also be verified whether the sharded data is consistent with the target parallel sharding scheme to avoid sharding errors.

[0129] It should be noted that after loading the fragmented data onto the target chip, the state of the random number generator can be restored. The state of the random number generator is used to fix the random number sequence, ensuring that the random number sequence on the target chip is completely consistent with the breakpoint data, and that there is no deviation in the random results.

[0130] In other words, restoring the state of the random number generator ensures that the training process can be accurately reproduced, enabling seamless training continuation across different chip scenarios.

[0131] The beneficial technical effect of step S4.3 is that by loading the fragmented data onto the target chip, the target chip can have the data required to perform the model training task. By restoring the state of the random number generator, the sequence of data from the breakpoint can be resumed during model training, without problems such as sample duplication, skipping, or rollback due to state loss.

[0132] In some embodiments, step S5 includes: Step S5.1: Obtain the logic value precision of the chip-independent intermediate representation and the actual execution precision of the target chip.

[0133] In some embodiments, before adapting the calculation precision of the source chip and the target chip, the logical numerical precision of the chip-independent intermediate representation and the actual execution precision of the target chip can be obtained, and the consistency between the logical numerical precision and the actual execution precision can be compared.

[0134] Optionally, if the precision of the logical value is consistent with the actual execution precision, there is no need to adapt the actual execution precision. The back propagation and parameter update of the model can be performed directly on the target chip according to the precision of the logical value of the chip-independent intermediate representation.

[0135] In some embodiments, the precision of the logic values ​​in the chip-independent intermediate representation is related to the logic tensor. That is, the precision of the logic tensors included in the chip-independent intermediate representation can be determined as the precision of the logic values.

[0136] In some embodiments, the actual execution accuracy of the target chip is related to the target chip manufacturer; that is, the actual execution accuracy of the target chip can be obtained from the development information of the target chip manufacturer.

[0137] The beneficial technical effect of step S5.1 is that by obtaining the logical numerical precision of the chip-independent intermediate representation and the actual execution precision of the target chip, it is possible to determine whether the calculation precision of the source chip and the target chip needs to be adapted before performing calculation precision adaptation, thereby effectively preventing the abuse of computing power resources.

[0138] Step S5.2: In response to the inconsistency between the logical numerical precision and the actual execution precision, obtain the loss value of the model obtained through forward propagation on the target chip.

[0139] In some embodiments, the adaptation of logical numerical precision to actual execution precision can occur after the model's forward propagation. That is, forward propagation on the target chip refers to training using training data, which takes place within the model itself and is independent of the target chip's actual execution precision.

[0140] After the model undergoes forward propagation on the target chip, the loss value is obtained. The operators on the target chip then need to process the data based on this loss value. Because the precision of the logical values ​​differs from the actual execution precision, numerical overflow may occur.

[0141] In other words, the loss value needs to be scaled so that the accuracy of the loss value is the same as the actual execution accuracy.

[0142] The beneficial technical effect of step S5.2 is that when the precision of the logical numerical value is inconsistent with the actual execution precision, by obtaining the loss value obtained by the model through forward propagation on the target chip, the loss value can be processed to make the loss value match the actual execution precision of the target chip, thereby avoiding numerical overflow and achieving precision consistency.

[0143] Step S5.3: During the model training task, the loss value is scaled to adapt to the computational accuracy required to continue the model training task on the target chip.

[0144] In some embodiments, after obtaining the loss value obtained by the model through forward propagation, the model training task, namely backpropagation and parameter update, can continue to be performed.

[0145] Before backpropagation, the loss value can be scaled up or down to obtain a loss value that is the same as the actual execution accuracy of the target chip. Backpropagation is then performed based on this loss value to obtain the gradient value.

[0146] Furthermore, the model's parameters need to be updated based on the gradient values. These updates are performed by the optimizer updating the model's weights according to the gradient values. The precision of the gradient values ​​is the same as the actual execution precision of the target chip, but differs from the logical numerical precision of the source chip.

[0147] In other words, before updating the parameters, the gradient values ​​need to be scaled to obtain gradient values ​​with the same logical numerical precision as the source chip, and the optimizer uses these gradient values ​​to update the model's weight values.

[0148] In other words, during the model training process, the computational precision needs to be adjusted multiple times to ensure that the numerical computational precision during model training matches both the model's computational precision and the target chip's computational precision.

[0149] The beneficial technical effect of step S5.3 is that by performing loss scaling on the loss value, the computational accuracy of the model can be adapted to the target chip when the model continues to perform training tasks on the target chip. This avoids numerical overflow caused by the difference in computational accuracy of different chip types, thereby causing information loss and ensuring the stability and reliability of training on the target chip.

[0150] In some embodiments, step S5.3 includes: Step S5.3.1: Obtain the loss scaling factor for each training step.

[0151] It should be noted that the model can continue training on the target chip in multiple training steps. For each training step, the computational precision needs to be adjusted.

[0152] In other words, by obtaining the loss scaling factor for each training step, the computational accuracy can be adapted using the loss scaling factor.

[0153] The loss scaling factor for each training step can be the same or different.

[0154] Optionally, an initial loss scaling factor can be set for the first training step, and the initial loss scaling factor can be updated during the execution of the training step, so that different training steps can use different loss scaling factors to adapt the computational accuracy.

[0155] The beneficial technical effect of step S5.3.1 is that by obtaining the loss scaling factor for each training step, different loss scaling factors can be used to adapt the computational accuracy for different training steps, which improves the accuracy of computational accuracy adaptation and helps to improve the stability of model training.

[0156] Step S5.3.2: Before backpropagation of the model, multiply the loss values ​​based on the loss scaling factor to obtain the loss value after the calculation accuracy is adapted.

[0157] In some embodiments, by multiplying the loss value based on the loss scaling factor before backpropagation of the model, the loss value can be amplified or reduced to obtain a loss value with the same execution accuracy as the target chip, that is, a loss value after accuracy adaptation.

[0158] Furthermore, the model can be backpropagated based on the loss value after the computational accuracy is adapted, thereby obtaining the gradient value. The computational accuracy of this gradient value is the same as the actual execution accuracy of the target chip.

[0159] The beneficial technical effect of step S5.3.2 is that before backpropagation of the model, the loss value can be multiplied based on the loss scaling factor to adapt the calculation accuracy of the loss value so that the loss value is adapted to the actual execution accuracy of the target chip, which is beneficial to the integrity of information when backpropagation is performed on the target chip.

[0160] Step S5.3.3: Before updating the model parameters, divide the gradient values ​​obtained by backpropagation based on the loss scaling factor to adapt to the computational accuracy required to continue the training task of the model on the target chip.

[0161] In some embodiments, after backpropagation, the model parameters can be updated based on the obtained gradient values. Since the parameter update is performed by the model's optimizer, the optimizer's computational precision is the same as that of the source chip but different from that of the target chip, requiring the gradient values ​​to be re-adapted to the required precision.

[0162] In some embodiments, the accuracy of the loss value is adjusted by multiplying the loss value by a loss scaling factor. Therefore, the gradient value after backpropagation needs to be divided by the loss scaling factor to restore the accuracy of the gradient value.

[0163] Furthermore, the optimizer can update the model parameters based on the gradient values ​​restored by the computational precision, thereby enabling multiple adaptations of computational precision based on the loss scaling factor within a single training step.

[0164] For example, suppose the loss value during the training process is 3 × 10. -8 The smallest normalized number that can be represented in FP16 format is approximately 6.1 × 10⁻⁶. -5 The smallest positive normal number is approximately 5.96 × 10⁻⁶. -8 Due to 3×10 -8 If a number smaller than the smallest positive normal number of FP16 is directly represented as FP16 without adjusting the calculation precision, the loss value will underflow to 0, and the corresponding parameter update information will be completely lost.

[0165] An initial loss scaling factor can be injected. =2 12=4096, then the equivalent amplification of the loss value in backpropagation is 3×10 -8 ×4096≈1.229×10 -4 It is higher than the minimum normalization number of FP16, which is 6.1 × 10. -5 It can be represented with normal precision in FP16.

[0166] Furthermore, the equivalent amplification value is divided by [missing value] before the parameter update. After restoration, a gradient value with the same calculation accuracy as the original gradient value is obtained: ≈3×10⁻⁶. -8 This avoids the gradient value being incorrectly cleared to zero due to cross-precision migration.

[0167] The beneficial technical effect of step S5.3.3 is that by performing a division operation on the gradient values ​​obtained by backpropagation before updating the model parameters, the calculation accuracy of the gradient values ​​is restored, so that the model can update the model parameters based on the restored gradient values, and the loss of parameter update information can be avoided when updating the model parameters.

[0168] In some embodiments, step S5.3.1 includes at least one of the following steps: Step S5.3.1.1: For the t-th training step, in response to the gradient overflow of the gradient value of the t-th training step, reduce the loss scaling factor of the t-th training step to obtain the loss scaling factor of the (t+1)-th training step, and discard the gradient value of the t-th training step.

[0169] In some embodiments, the loss scaling factor can be dynamically updated for each training step so that the loss scaling factor automatically converges to a suitable value under the current numerical state during the training process.

[0170] In some embodiments, the loss scaling factor for the next training step can be increased or decreased based on whether gradient overflow occurred in the previous training step.

[0171] Optionally, for the t-th training step, if the gradient value of the t-th training step overflows after loss scaling using the loss scaling factor of the t-th training step, indicating that the loss scaling factor is too large, then the loss scaling factor of the t-th training step can be reduced to obtain the loss scaling factor of the (t+1)-th training step.

[0172] In some embodiments, to avoid gradient overflow caused by excessive reduction in the loss scaling factor, a lower bound, i.e., the minimum value of the loss scaling factor, can be set, and then the loss scaling factor of the t-th training step can be reduced under the constraint of the minimum value of the loss scaling factor.

[0173] Optionally, the loss scaling factor for the (t+1)th training step can be determined based on the loss scaling factor at the t-th training step and the minimum value of the loss scaling factor.

[0174] For example, we can take half the size of the loss scaling factor at the t-th training step, compare it with the minimum value of the loss scaling factor, and determine the maximum value as the loss scaling factor for the (t+1)-th training step.

[0175] Optionally, the formula for determining the loss scaling factor at the (t+1)th training step is as follows: (3) in, This represents the loss scaling factor at the (t+1)th training step. This represents the loss scaling factor at the t-th training step. This represents the minimum value of the loss scaling factor.

[0176] In some embodiments, if gradient overflow occurs in the gradient value of the t-th training step, the gradient value of the t-th training step can be discarded to skip the parameter update of the t-th training step.

[0177] The beneficial technical effect of step S5.3.1.1 is that when gradient overflow occurs in the gradient value of the t-th training step, the loss scaling factor of the t-th training step is reduced to obtain the loss scaling factor of the (t+1)-th training step. This can effectively avoid gradient overflow in the (t+1)-th training step, thereby ensuring the smooth progress of model training and preventing model divergence.

[0178] Step S5.3.1.2: In response to the absence of gradient overflow in the gradient values ​​of K consecutive training steps, the loss scaling factor of the t-th training step is amplified to obtain the loss scaling factor of the (t+1)-th training step.

[0179] In some embodiments, training steps that do not exhibit gradient overflow can be continuously monitored. If gradient overflow is detected for K consecutive training steps, it indicates that the current loss scaling factor is too small, resulting in insufficient gradient resolution.

[0180] In some embodiments, to avoid insufficient gradient resolution affecting the accuracy of model training, the loss scaling factor of the t-th training step can be amplified to obtain the loss scaling factor of the (t+1)-th training step.

[0181] It should be noted that the t-th training step is the last training step in a series of K consecutive training steps.

[0182] In other words, if no gradient overflow occurs in the K training steps before the t-th training step, the loss scaling factor of the t-th training step can be increased.

[0183] In some embodiments, to avoid gradient overflow caused by an excessively large loss scaling factor, an upper limit, i.e., a maximum value of the loss scaling factor, can be set, thereby amplifying the loss scaling factor at the t-th training step under the constraint of the maximum value of the loss scaling factor.

[0184] Optionally, the loss scaling factor for the (t+1)th training step can be determined based on the loss scaling factor at the t-th training step and the maximum value of the loss scaling factor.

[0185] For example, we can take twice the loss scaling factor of the t-th training step and compare it with the maximum value of the loss scaling factor to determine the minimum value, which can then be used as the loss scaling factor for the (t+1)-th training step.

[0186] Optionally, the formula for determining the loss scaling factor at the (t+1)th training step is as follows: (4) in, This represents the loss scaling factor at the (t+1)th training step. This represents the loss scaling factor at the t-th training step. This represents the maximum value of the loss scaling factor.

[0187] The beneficial technical effect of step S5.3.1.2 is that by increasing the loss scaling factor of the t-th training step when there is no gradient overflow in the gradient values ​​of K consecutive training steps, the numerical resolution under low precision can be improved and the training accuracy of the model can be enhanced.

[0188] Step S5.3.1.3: In response to the absence of gradient overflow in the gradient value of the t-th training step and the fact that the t-th training step has not reached the last training step among K consecutive training steps, the loss scaling factor of the t-th training step is used as the loss scaling factor of the (t+1)-th training step.

[0189] In some embodiments, when there is no gradient overflow in the gradient value of the t-th training step, but the t-th training step is not the last training step in a series of K consecutive training steps, the gradient value neither overflows nor reaches stability. In order to avoid gradient overflow caused by adjusting the loss scaling factor, the loss scaling factor can be kept unchanged.

[0190] In other words, the loss scaling factor of the (t+1)th training step is equal to the loss scaling factor of the tth training step.

[0191] Where K is a natural number greater than or equal to 2.

[0192] The beneficial technical effect of step S5.3.1.3 is that, when the gradient value of the model does not overflow but is unstable, keeping the loss scaling factor unchanged can effectively prevent the gradient value from overflowing due to sudden changes in the loss scaling factor, which is conducive to the stability of model training.

[0193] In some embodiments, step S5 further includes: Step S5.1': Determine the first operator used by the model to perform the training task on the source chip.

[0194] It should be noted that the model needs to use operators to perform training tasks on the chip. In other words, the chip's operators can perform the model's training tasks and enable the model to be trained.

[0195] Because the model is trained using the first operator on the source chip, but the target chip may not support the first operator, it is impossible to continue training the model on the target chip.

[0196] In other words, the operators of the source chip and the target chip can be adapted so that the same operators as the first operator can be used to continue training the model on the target chip.

[0197] The beneficial technical effect of step S5.1' is that by determining the first operator used by the model to perform the training task on the source chip, it is possible to determine whether the target chip supports the first operator based on the first operator, thus providing operator support for subsequent model training on the target chip.

[0198] Step S5.2': Obtain the set of operators supported by the target chip.

[0199] In some embodiments, the set of operators supported by the target chip is related to the target chip manufacturer; that is, the set of operators supported by the target chip can be obtained from the development information of the target chip manufacturer.

[0200] In some embodiments, by obtaining the set of operators supported by the target chip, it can be determined whether a first operator exists in the set of operators. If it does, it means that the target chip supports the first operator, and the first operator in the set of operators can be used directly to continue the training task of the model.

[0201] If it does not exist, the first operator needs to be replaced to obtain an equivalent or approximate operator, so that the model training task can continue to be performed on the target chip based on the operator.

[0202] The beneficial technical effect of step S5.2' is that by obtaining the set of operators supported by the target chip, it is possible to determine whether the first operator is missing from the set of operators, thereby avoiding the problem of operator incompatibility caused by different chip types during cross-chip training from the root, and improving the success rate of model training tasks on heterogeneous chips.

[0203] Step S5.3': In response to the fact that the first operator does not exist in the operator set, determine the second operator corresponding to the first operator in order to adapt the operator required to continue the training task of the model on the target chip.

[0204] In some embodiments, when the first operator does not exist in the operator set, it can be determined that the model training task cannot be continued on the target chip using the same operator as the source chip.

[0205] Optionally, the first operator can be replaced with an operator that is equivalent to or similar to the first operator as the second operator, wherein the second operator is an operator supported by the target chip.

[0206] The beneficial technical effect of step S5.3' is that by replacing the first operator, an operator equivalent to or similar to the first operator can be used on the target chip to continue the training task of the model, so as to ensure that the training task of the model can continue to be executed on the target chip, which is conducive to continuing to execute the training task on chips of different chip types.

[0207] In some embodiments, step S5.3' includes: Step S5.3.1': Obtain the operator replacement table.

[0208] In some embodiments, an operator replacement table can be maintained in advance. The operator replacement table uses triples as the basic record unit. Each triple may include a source operator identifier, a target chip identifier, and a replacement operator identifier.

[0209] Among them, the source operator is the first operator.

[0210] In some embodiments, the operator substitution table can record the correspondence between the operators used during the training of the source chip and the equivalent or approximate operators on various target chips.

[0211] In other words, after determining the identifier of the first operator and the identifier of the target chip, the identifier of the replacement operator can be determined by querying the operator replacement table, and the operator corresponding to the replacement operator identifier can be used as the second operator.

[0212] It should be noted that the operator replacement table can be incrementally maintained as the set of operators supported by the target chip is updated.

[0213] The beneficial technical effect of step S5.3.1' is that by obtaining the operator replacement table and determining the second operator based on the operator replacement table, the training process can be correctly executed on the target chip without manual intervention in code adaptation.

[0214] Step S5.3.2': Based on the first operator, query the operator replacement table to obtain the operator combination with the same calculation logic as the first operator, and use it as the second operator.

[0215] In some embodiments, the replacement operators in the operator replacement table can be implemented by combining basic operators and have equivalent computational logic to the source operator. That is, the replacement operator can be a combination of operators with the same computational logic as the first operator.

[0216] Optionally, the operator replacement table can be queried based on the identifier of the first operator, and combined with the identifier of the target chip to locate the replacement operator identifier, so that the operator combination corresponding to the replacement operator identifier is used as the second operator.

[0217] The beneficial technical effect of step S5.3.2' is that by obtaining the operator combination with the same computational logic as the first operator from the operator replacement table as the second operator, it is possible to achieve operator adaptation of the target chip, and further realize the model training task on the target chip, ensuring the success rate of model training continuation.

[0218] Step S5.3.3': Based on the first operator, query the operator replacement table to obtain operators with similar values ​​to the first operator, and use them as the second operator.

[0219] In some embodiments, the replacement operator in the operator replacement table can be an operator whose value is similar to that of the source operator but whose computation path is different. That is, the replacement operator has a value similar to that of the first operator.

[0220] Optionally, the operator replacement table can be queried based on the identifier of the first operator, and combined with the identifier of the target chip to locate the replacement operator identifier, so that the operator corresponding to the replacement operator identifier is used as the second operator.

[0221] The beneficial technical effect of step S5.3.3' is that by obtaining a second operator with a value similar to the first operator from the operator replacement table, operator adaptation of the target chip can be achieved, and the training task of the model can be continued on the target chip, ensuring the success rate of model training.

[0222] In some embodiments, step S3 includes: Step S3.1: Obtain the memory capacity of the target chip from the capability data.

[0223] The beneficial technical effect of step S3.1 is that by obtaining the memory capacity of the target chip, the target parallel partitioning scheme can be solved under the constraint of memory capacity, so that the target parallel partitioning scheme matches the memory capacity of the target chip, thereby enabling the model training task to continue to be executed on the target chip.

[0224] Step S3.2: Determine the memory requirements of the chip-independent intermediate representation.

[0225] In some embodiments, the memory requirements of a chip-independent intermediate representation refer to the sum of the storage capacity of the logic tensor weight data, the first and second moments of the optimizer, and the state of the random number generator.

[0226] In other words, by determining the storage capacity 1 of the weight data, the storage capacity 2 of the first and second moments of the optimizer, and the storage capacity 3 of the state of the random number generator, and calculating the sum of storage capacity 1, storage capacity 2, and storage capacity 3, the memory requirements of the chip-independent intermediate representation are obtained.

[0227] The beneficial technical effect of step S3.2 is that by determining the memory requirements of the chip-independent intermediate representation, the memory requirements of the chip-independent intermediate representation can be met under the constraint of memory capacity, avoiding the loss of data due to ignoring the size of the chip-independent intermediate representation and ensuring the integrity of the chip-independent intermediate representation.

[0228] Step S3.3: Under the constraint of video memory capacity, determine multiple candidate parallel partitioning schemes based on video memory requirements.

[0229] In some embodiments, memory capacity and memory requirements can determine the number of partitions for each candidate parallel partitioning scheme.

[0230] In other words, under the constraint of video memory capacity, the number of partitions of the candidate parallel partitioning scheme can be determined based on video memory requirements, and then multiple candidate parallel partitioning schemes can be enumerated based on the number of partitions.

[0231] In some embodiments, the ratio of video memory capacity to video memory requirement can be calculated and used as the minimum value of the number of partitions.

[0232] In some embodiments, a candidate parallel partitioning scheme can be represented as (TP', PP', DP'), where the number of partitions is related to the number of TP' and PP'.

[0233] Optionally, the product of TP' and PP' represents the number of partitions in a candidate parallel partitioning scheme. The number of DP' can be determined based on the product of TP' and PP'.

[0234] In some embodiments, a value greater than the minimum number of segments can be determined as the number of segments, and based on the number of segments and the memory requirements of the chip-independent intermediate representation, it can be determined whether the memory capacity constraint is met.

[0235] Optionally, the ratio of the memory requirement of the chip-independent intermediate representation to the number of segments can be determined. If the ratio is less than the memory capacity, the memory capacity constraint is satisfied.

[0236] In other words, multiple parallel partitioning schemes with a minimum number of partitions can be enumerated, and multiple parallel partitioning schemes can be screened based on memory requirements to determine multiple candidate parallel partitioning schemes that meet the memory capacity constraints.

[0237] The beneficial technical effect of step S3.3 is that by determining multiple candidate parallel partitioning schemes based on the memory requirements under the constraint of memory capacity, data loss can be avoided when partitioning chip-independent intermediate representations, thus ensuring the integrity of the partitioning of chip-independent intermediate representations.

[0238] Step S3.4: Based on topology metadata, determine the first communication group of each of the multiple candidate parallel splitting schemes.

[0239] In some embodiments, the communication packets of the candidate parallel partitioning scheme can be TP', PP', and DP'. Topology metadata is a chip-independent intermediate representation of the communication packets partitioned on the source chip, i.e., the source partitioning scheme.

[0240] In order to standardize and align the parallelism of different dimensions in multiple candidate parallel partitioning schemes with the source partitioning scheme to generate logically unified communication packets, the communication packets of multiple candidate parallel partitioning schemes can be dimensionally aligned based on topological metadata to obtain the first communication packet of each of the multiple candidate parallel partitioning schemes.

[0241] The beneficial technical effect of step S3.4 is that by determining the first communication group of each of the multiple candidate parallel segmentation schemes, the target parallel segmentation scheme can be determined based on the first communication group, thereby realizing the automatic determination of the target parallel segmentation scheme without manual intervention, and reducing the cost of adapting the source chip and the target chip.

[0242] Step S3.5: Based on the first communication group, determine the target parallel partitioning scheme from multiple candidate parallel partitioning schemes.

[0243] In some embodiments, each first communication group has a different communication frequency. Based on the first communication group with high frequency communication, the target parallel segmentation scheme can be determined from multiple candidate parallel segmentation schemes. This is to determine the segmentation strategy with the best communication efficiency from the candidate parallel segmentation schemes, so as to maximize the release of the actual computing power of the target chip and avoid communication bottlenecks slowing down the overall training speed.

[0244] In some embodiments, the target chip can be a chip cluster, where any two chips in the chip cluster communicate with different interconnect bandwidths. To improve communication efficiency, the first communication packet of high-frequency communication can use high bandwidth for communication. Therefore, candidate parallel segmentation schemes in which the high-frequency communication packets fall within the high bandwidth domain can be used as the target parallel segmentation scheme.

[0245] The beneficial technical effect of step S3.5 is that, from multiple candidate parallel segmentation schemes, a candidate parallel segmentation scheme that makes high-frequency communication packets fall within the high bandwidth domain is determined as the target parallel segmentation scheme, which can maximize the release of the actual computing power of the target chip and avoid communication bottlenecks slowing down the overall training speed.

[0246] In some embodiments, step S3.4 includes: Step S3.4.1: Obtain the second partitioning method of tensor parallelism, pipeline parallelism and data parallelism corresponding to the topology metadata.

[0247] In some embodiments, the topology metadata includes tensor parallelism, pipeline parallelism, and data parallelism on the source chip, as well as a second partitioning method for each parallelism.

[0248] The beneficial technical effect of step S3.4.1 is that by determining the second partitioning method of each degree of parallelism in the topology metadata, it is possible to achieve dimension alignment of the parallelism of multiple candidate parallel partitioning schemes based on the second partitioning method, so as to generate communication groups that are logically consistent with the source chip.

[0249] Step S3.4.2: Based on the second segmentation method, the parallelism of multiple candidate parallel segmentation schemes is dimensionally aligned to obtain the first communication group.

[0250] In some embodiments, the parameters of the parallelism of multiple candidate parallel segmentation schemes can be converted based on the second segmentation method to convert them into a standard format of parallelism, thereby obtaining the first communication group corresponding to the candidate parallel segmentation scheme.

[0251] The beneficial technical effect of step S3.4.2 is that, based on the second segmentation method, the parallelism of multiple candidate parallel segmentation schemes is aligned in dimensions, which can generate communication groups that are logically consistent with the source chip and optimize the communication efficiency of the first communication group.

[0252] In some embodiments, step S3.5 includes: Step S3.5.1: Determine the second communication group that needs to be used for high-frequency communication from the first communication group.

[0253] In some embodiments, the first communication packet may include TP' and PP'. Since the communication frequency of tensor parallel packets is significantly higher than that of pipeline parallel packets, TP' can be identified as the second communication packet that requires high-frequency communication.

[0254] In other words, the second communication group that needs to be used for high-frequency communication can be determined based on the communication needs of the first communication group.

[0255] Optionally, communication requirements can be determined based on the volume of different first communication packets. The larger the volume, the greater the communication requirement.

[0256] The beneficial technical effect of step S3.5.1 is that by selecting the second communication group that needs to conduct high-frequency communication, the bandwidth resources of different communication groups can be accurately matched to improve the bandwidth utilization rate and thus reduce the overall communication latency.

[0257] Step S3.5.2: Determine the bandwidth domain corresponding to the second communication group, and determine the candidate parallel segmentation scheme corresponding to the second communication group with the largest bandwidth in the bandwidth domain, as the target parallel segmentation scheme.

[0258] In some embodiments, the interconnection links in the target chip cluster have bandwidth differences, for example, the maximum bandwidth is within a single chip. By determining the bandwidth domain corresponding to the second communication packet of each candidate parallel partitioning scheme, and determining the second communication packet with the largest bandwidth from the different bandwidth domains, the candidate parallel partitioning scheme corresponding to this second communication packet is taken as the target parallel partitioning scheme.

[0259] In other words, the candidate parallel segmentation schemes for the second communication packet within a single chip can be determined as the target parallel segmentation scheme.

[0260] For example, consider a model with a parameter size of 70 bytes. Logical weights are stored in BF16 (2 bytes / parameter), totaling approximately 140GB. The optimizer's first and second moments are stored in FP32 (4 bytes / parameter each), totaling approximately 560GB. The combined memory requirements for these three components are... Approximately 700GB. The target chip's video memory capacity. Given a size of 64GB, the minimum number of partitions required to satisfy the hard constraints can be calculated as follows: (5) The first stage enumerates candidate parallel partitioning schemes that satisfy the constraint, such as TP'=8 and PP'=2 (the number of partitions is 16, and the actual usage is approximately 700 / 16≈43.75GB, satisfying the hard constraint). The second stage considers the actual interconnect topology of the target chip cluster (assuming a single node of the target chip has 8 accelerator cards, and the intra-node interconnect bandwidth is significantly higher than the inter-node interconnect bandwidth). Since the communication frequency of tensor parallel packets is significantly higher than that of pipeline parallel packets, tensor parallel packets are the second communication packets requiring high-frequency communication. It can be chosen to fully allocate the tensor parallel packets with TP'=8 within the 8 accelerator cards of the same node, and deploy the pipeline parallel packets with PP'=2 across nodes. If the total number of cards in the target chip cluster is 128, then the data parallelism DP'=128 / (TP'×PP')=128 / 16=8. The above result is the target parallel partitioning scheme (TP'=8, PP'=2, DP'=8) automatically solved in this scenario, without requiring manual calculations throughout the process.

[0261] The beneficial technical effect of step S3.5.2 is that by determining the bandwidth domain corresponding to the second communication packet, a candidate parallel segmentation scheme corresponding to the second communication packet with the largest bandwidth is determined based on the bandwidth domain, which is then used as the target parallel segmentation scheme. This enables the most frequently communicating packets to fall into the bandwidth domain with the largest bandwidth, thereby improving bandwidth utilization and reducing overall communication latency.

[0262] Based on the above embodiments, a breakpoint-based training system for an intelligent computing cloud platform based on heterogeneous computing power can also be determined using a breakpoint-based training method for heterogeneous computing power. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a structural diagram of the breakpoint retraining system of the intelligent computing cloud platform based on heterogeneous computing power provided by the present invention.

[0263] Figure 2 The intelligent computing cloud platform based on heterogeneous computing power and its breakpoint resume training system includes: a task layer, a core component layer, and a data layer.

[0264] The task layer carries the training tasks that are running or have been interrupted on the source chip, as well as the resumed training tasks that need to be resumed on the target chip.

[0265] The core component layer includes four functional components: checkpoint normalizer, parallel policy remapping unit, precision compensator, and operator replacement table.

[0266] The data layer maintains chip-independent intermediate representations and chip capability data. The former stores normalized breakpoint data, while the latter stores data such as the memory capacity, precision format, interconnect bandwidth, and operator set of various chips.

[0267] Among them, the checkpoint normalizer is used to read the weights and optimizer states after physical partitioning on the source chip when saving breakpoint data on the intelligent computing cloud platform, normalize the breakpoint data into a chip-independent intermediate representation, and save the original parallel topology metadata along with the chip-independent intermediate representation.

[0268] The parallel strategy remapper is used to read the target chip's capability data and solve for the target parallel partitioning scheme using a two-stage method.

[0269] The precision compensator determines whether precision compensation needs to be enabled based on whether the precision of the logic value in the chip-independent intermediate representation matches the actual execution precision of the target chip. If the precision is inconsistent, the precision compensator automatically injects and dynamically adjusts the loss scaling factor.

[0270] The operator replacement table is used to automatically perform operator downgrading and replacement when the target chip's operator set does not support the first operator.

[0271] Based on the above embodiments, it is possible to base on Figure 2 The illustrated intelligent computing cloud platform-based breakpoint training system executes the intelligent computing cloud platform-based breakpoint training method based on heterogeneous computing power.

[0272] Figure 3 This is a flowchart of the breakpoint continuation training method for the intelligent computing cloud platform based on heterogeneous computing power provided by this invention. In other words, Figure 3 Is Figure 2 Based on this, the breakpoint continuation training method based on heterogeneous computing power is implemented on the intelligent computing cloud platform.

[0273] Figure 3 This includes: Step 1, checkpoint normalization: When the intelligent computing cloud platform saves breakpoint data, the checkpoint normalizer reads the weights and optimizer states after physical partitioning on the source chip, normalizes the breakpoint data into a chip-independent intermediate representation, and saves the original parallel topology metadata along with the chip-independent intermediate representation.

[0274] Step 2, Capability Data Reading and Joint Solving: The parallel strategy remapper reads the capability data of the target chip and uses a two-stage method to solve for the target parallel partitioning scheme. Then, based on whether the logic precision of the chip-independent intermediate representation is consistent with the actual execution precision of the target chip, it is determined whether precision compensation needs to be enabled.

[0275] Step 3, Physical Resegmentation and Distribution Loading: According to the target parallel segmentation scheme, the weights, first-order moments and second-order moments of the optimizer organized by logical tensors in the chip-independent intermediate representation are physically resegmented, distributed and loaded to each computing device of the target chip, and the state of the random number generator is restored.

[0276] Step 4, numerical compensation and operator replacement: If the precision is inconsistent, the loss scaling factor is automatically injected and dynamically adjusted by the precision compensator; if the operator set of the target chip does not support the first operator, the operator replacement table automatically completes the operator downgrading and replacement.

[0277] Step 5, Continue training: After the above steps are completed, the training task continues to be executed on the target chip from the breakpoint, without the need for manual intervention in chip specification analysis, splitting scheme design or conversion script writing.

[0278] Figure 4 This is a flowchart of the remapping process performed by the parallel strategy remapping mechanism provided by the present invention.

[0279] Figure 4 In the process, the source chip includes topology metadata of TP=4, PP=2, and DP=8, as well as breakpoint data after partitioning based on the topology metadata. By normalizing the topology metadata and breakpoint data, a chip-independent intermediate representation of the topology metadata, including the set of logical tensors, the first and second moments of the optimizer, the state of the random number generator, and the topology metadata, can be obtained. By inputting the chip-independent intermediate representation into a parallel policy remapper, and by the parallel policy remapper reading the capability data of the target chip, the target parallel partitioning scheme of the target chip, i.e., TP'=8, PP'=1, and DP'=4, can be obtained through a two-stage solution.

[0280] An example is shown below of an implementation of a breakpoint resume training system based on heterogeneous computing power using an intelligent computing cloud platform.

[0281] Example 1: If the intelligent computing cloud platform originally used chip 1 to train a model with a parameter scale of 70B using a parallel partitioning scheme of TP=4, PP=2, DP=8, and needs to switch to chip 2 for continued training at the 50,000th training step due to supply chain reasons, the checkpoint normalizer reads the weights and optimizer states after physical partitioning on chip 1 and normalizes them into a chip-independent intermediate representation. The parallel policy remapping unit reads the capability data of chip 2 and solves for a new target parallel partitioning scheme (e.g., TP'=8, PP'=1, DP'=4) while satisfying its memory capacity constraints. It also detects that the computational precision of chip 1 is inconsistent with the computational precision used by chip 2, and can automatically inject and dynamically adjust the loss scaling factor. The weights, optimizer states, and random number generator states after re-physical partitioning based on the target parallel partitioning scheme are distributed and loaded onto each computing device of chip 2. The training task resumes from the 50,000th training step within minutes, without requiring manual analysis of chip 2 specifications or manual writing of conversion scripts.

[0282] Example 2: A training task uses a proprietary fusion attention operator from the source chip manufacturer on the source chip. When the task needs to be migrated to chip 3 for continued training, a comparison with the operator set recorded in the chip 3's capability data reveals that the fusion attention operator is not included. Therefore, the operator replacement table is queried, and it is automatically replaced with equivalent computational logic implemented by a combination of the basic matrix multiplication operator and the Softmax operator already supported by chip 3. Simultaneously, because the source computational precision is inconsistent with the actual execution precision of chip 3, the precision compensator simultaneously enables automatic loss scaling compensation. Both the operator replacement and precision compensation are automatically completed during the start of the continued training task; the training script itself does not need modification.

[0283] Please see Figure 5 , Figure 5 This is a structural diagram of a breakpoint resuming training device for an intelligent computing cloud platform based on heterogeneous computing power provided by the present invention, as shown below. Figure 5 As shown, the breakpoint retraining device 500 of the intelligent computing cloud platform based on heterogeneous computing power includes: Processing module 501 is used to acquire breakpoint data of the intelligent computing cloud platform in heterogeneous computing power executing model training tasks on the source chip, and normalize the breakpoint data into chip-independent intermediate representation. The chip-independent intermediate representation includes at least the topological metadata of the training task, the weight data of the logical tensor, the first and second moments of the optimizer corresponding to the logical tensor, and the state of the random number generator during the training process. The acquisition module 502 is used to acquire the capability data of a target chip in the heterogeneous computing power that is of a different chip type than the source chip; The determining module 503 is used to determine the target parallel partitioning scheme of the chip-independent intermediate representation based on the chip-independent intermediate representation and the capability data; The segmentation module 504 is used to re-segment the chip-independent intermediate representation according to the target parallel segmentation scheme and distribute and load it to the target chip; The training continuation module 505 is used to adapt the computational precision and operators required to continue the training task of the model on the target chip based on the respective computational precision and operators of the source chip and the target chip, so as to continue the training task of the model on the target chip.

[0284] In some embodiments, the segmentation module 504 is further configured to: Determine the tensor parallelism, pipeline parallelism, and data parallelism of the target parallel partitioning scheme, as well as the first partitioning method for each parallelism. According to the tensor parallelism, pipeline parallelism, and data parallelism, as well as the first partitioning method, the weight data of the logical tensor, the first moment and the second moment of the optimizer are partitioned to obtain the partitioned data. The fragmented data is loaded onto the target chip, and the state of the random number generator is restored.

[0285] In some embodiments, the training continuation module 505 is further configured to: Obtain the logical value precision of the chip-independent intermediate representation and the actual execution precision of the target chip; In response to the inconsistency between the precision of the logical numerical value and the actual execution precision, the loss value obtained by the model through forward propagation on the target chip is acquired. During the training process of the model, the loss value is subjected to loss scaling to adapt to the computational accuracy required to continue the training of the model on the target chip.

[0286] In some embodiments, the training continuation module 505 is further configured to: Obtain the loss scaling factor for each training step; Before backpropagation of the model, the loss value is multiplied based on the loss scaling factor to obtain the loss value after the calculation accuracy is adapted. Before updating the model parameters, the gradient values ​​obtained from backpropagation are divided based on the loss scaling factor to adapt to the computational accuracy required to continue the training task of the model on the target chip.

[0287] In some embodiments, the training continuation module 505 is further configured to perform at least one of the following steps: For the t-th training step, in response to the gradient overflow of the gradient value of the t-th training step, the loss scaling factor of the t-th training step is reduced to obtain the loss scaling factor of the (t+1)-th training step, and the gradient value of the t-th training step is discarded. In response to the absence of gradient overflow in the gradient values ​​of K consecutive training steps, the loss scaling factor of the t-th training step is amplified to obtain the loss scaling factor of the (t+1)-th training step; wherein, the t-th training step is the last training step in the K consecutive training steps. In response to the absence of gradient overflow in the gradient value of the t-th training step and the fact that the t-th training step has not reached the last training step among K consecutive training steps, the loss scaling factor of the t-th training step is used as the loss scaling factor of the (t+1)-th training step.

[0288] In some embodiments, the step training module 505 is further configured to: Determine the first operator used by the model to perform the training task on the source chip; Obtain the set of operators supported by the target chip; In response to the first operator not existing in the operator set, a second operator corresponding to the first operator is determined to adapt the operator required to continue the training task of the model on the target chip.

[0289] In some embodiments, the training continuation module 505 is further configured to: Obtain the operator replacement table; Based on the first operator, query the operator replacement table to obtain an operator combination with the same computational logic as the first operator, and use it as the second operator; or... Based on the first operator, query the operator replacement table to obtain an operator with a similar value to the first operator, and use it as the second operator.

[0290] In some embodiments, the determining module 503 is further configured to: Obtain the memory capacity of the target chip from the capability data; Determine the memory requirements of the chip-independent intermediate representation; Under the constraint of the video memory capacity, and based on the video memory requirements, multiple candidate parallel partitioning schemes are determined; Based on the topology metadata, the first communication group of each of the multiple candidate parallel segmentation schemes is determined; Based on the first communication packet, the target parallel segmentation scheme is determined from the plurality of candidate parallel segmentation schemes.

[0291] In some embodiments, the determining module 503 is further configured to: Obtain the second partitioning method corresponding to the tensor parallelism, pipeline parallelism, and data parallelism of the topology metadata; Based on the second segmentation method, the parallelism of the multiple candidate parallel segmentation schemes is dimensionally aligned to obtain the first communication group.

[0292] In some embodiments, the determining module 503 is further configured to: Determine the second communication packet that needs to be used for high-frequency communication from the first communication packet; Determine the bandwidth domain corresponding to the second communication packet, and determine the candidate parallel segmentation scheme corresponding to the second communication packet with the largest bandwidth in the bandwidth domain, as the target parallel segmentation scheme.

[0293] The breakpoint training device for the intelligent computing cloud platform based on heterogeneous computing power provided by this invention can realize the various processes of the various embodiments of the above-mentioned breakpoint training method for the intelligent computing cloud platform based on heterogeneous computing power. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0294] It should be noted that the breakpoint resuming device of the intelligent computing cloud platform based on heterogeneous computing power in this invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0295] The present invention also provides an electronic device, see [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 601, a processor 602, and a program or instructions stored in the memory 601 that run on the processor 602. When the program or instructions are executed by the processor 602, they can achieve the following: Figure 1 The corresponding steps in the breakpoint training method embodiment based on heterogeneous computing power of the intelligent computing cloud platform and the achievement of the same beneficial effect will not be elaborated here.

[0296] The processor 602 can be a CPU, ASIC, FPGA or GPU.

[0297] Those skilled in the art will understand that all or part of the steps of the above-described embodiment of the breakpoint resume training method based on heterogeneous computing power for intelligent computing cloud platforms can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0298] The present invention also provides a readable storage medium on which a computer program is stored, and which, when executed by a processor, can perform the above-described functions. Figure 1 The corresponding intelligent computing cloud platform, based on the breakpoint resume training method embodiment of heterogeneous computing power, can achieve the same technical effect, and will not be elaborated here to avoid repetition. The storage medium mentioned includes, for example, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0299] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The various processes of the corresponding intelligent computing cloud platform based on the breakpoint training method embodiment can achieve the same technical effect, and will not be described in detail here to avoid repetition.

[0300] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0301] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0302] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of this application.

[0303] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for resuming training from breakpoints on an intelligent computing cloud platform based on heterogeneous computing power, characterized in that, The method includes: Step S1: Obtain the breakpoint data of the intelligent computing cloud platform in the heterogeneous computing power executing the model training task on the source chip, and normalize the breakpoint data into a chip-independent intermediate representation. The chip-independent intermediate representation includes at least the topological metadata of the training task, the weight data of the logical tensor, the first and second moments of the optimizer corresponding to the logical tensor, and the state of the random number generator during the training process. Step S2: Obtain the capability data of the target chip in the heterogeneous computing power that is of a different chip type than the source chip; Step S3: Based on the chip-independent intermediate representation and the capability data, determine the target parallel partitioning scheme for the chip-independent intermediate representation; Step S4: Re-segment the chip-independent intermediate representation according to the target parallel segmentation scheme and distribute and load it to the target chip; Step S5: Based on the computational precision and operators of the source chip and the target chip respectively, adapt the computational precision and operators required to continue the training task of the model on the target chip, so as to continue the training task of the model on the target chip.

2. The method according to claim 1, characterized in that, Step S4 includes: Step S4.1: Determine the tensor parallelism, pipeline parallelism, and data parallelism of the target parallel partitioning scheme, as well as the first partitioning method for each parallelism. Step S4.2: According to the tensor parallelism, pipeline parallelism, and data parallelism, as well as the first partitioning method, partition the weight data of the logic tensor, the first moment and the second moment of the optimizer to obtain the partitioned data. Step S4.3: Load the fragmented data onto the target chip and restore the state of the random number generator.

3. The method according to claim 1, characterized in that, Step S5 includes: Step S5.1: Obtain the logical value precision of the chip-independent intermediate representation and the actual execution precision of the target chip; Step S5.2: In response to the inconsistency between the logical numerical precision and the actual execution precision, obtain the loss value obtained by the model through forward propagation on the target chip; Step S5.3: During the process of the model performing the training task, the loss value is subjected to loss scaling processing to adapt to the computational accuracy required to continue performing the training task of the model on the target chip.

4. The method according to claim 3, characterized in that, Step S5.3 includes: Step S5.3.1: Obtain the loss scaling factor for each training step; Step S5.3.2: Before the backpropagation of the model, the loss value is multiplied based on the loss scaling factor to obtain the loss value after the calculation accuracy is adapted. Step S5.3.3: Before updating the model parameters, perform a division operation on the gradient values ​​obtained by backpropagation based on the loss scaling factor to adapt to the computational accuracy required to continue the training task of the model on the target chip.

5. The method according to claim 4, characterized in that, Step S5.3.1 includes at least one of the following steps: Step S5.3.1.1: For the t-th training step, in response to the gradient overflow of the gradient value of the t-th training step, reduce the loss scaling factor of the t-th training step to obtain the loss scaling factor of the (t+1)-th training step, and discard the gradient value of the t-th training step. Step S5.3.1.2: In response to the absence of gradient overflow in the gradient values ​​of K consecutive training steps, the loss scaling factor of the t-th training step is amplified to obtain the loss scaling factor of the (t+1)-th training step; wherein, the t-th training step is the last training step in the K consecutive training steps. Step S5.3.1.3: In response to the absence of gradient overflow in the gradient value of the t-th training step and the fact that the t-th training step has not reached the last training step in the K consecutive training steps, the loss scaling factor of the t-th training step is used as the loss scaling factor of the (t+1)-th training step.

6. The method according to any one of claims 1-5, characterized in that, Step S5 further includes: Step S5.1': Determine the first operator used by the model to perform the training task on the source chip; Step S5.2': Obtain the set of operators supported by the target chip; Step S5.3': In response to the first operator not existing in the operator set, determine the second operator corresponding to the first operator to adapt the operator required to continue the training task of the model on the target chip.

7. The method according to claim 6, characterized in that, Step S5.3' includes: Step S5.3.1': Obtain the operator replacement table; Step S5.3.2': Based on the first operator, query the operator replacement table to obtain an operator combination with the same calculation logic as the first operator, and use it as the second operator; or, Step S5.3.3': Based on the first operator, query the operator replacement table to obtain an operator with a similar value to the first operator, and use it as the second operator.

8. The method according to any one of claims 1-5, characterized in that, Step S3 includes: Step S3.1: Obtain the memory capacity of the target chip from the capability data; Step S3.2: Determine the memory requirements of the chip-independent intermediate representation; Step S3.3: Under the constraint of the video memory capacity, based on the video memory requirements, determine multiple candidate parallel partitioning schemes; Step S3.4: Based on the topology metadata, determine the first communication group for each of the multiple candidate parallel splitting schemes; Step S3.5: Based on the first communication packet, determine the target parallel segmentation scheme from the plurality of candidate parallel segmentation schemes.

9. The method according to claim 8, characterized in that, Step S3.4 includes: Step S3.4.1: Obtain the second partitioning method of tensor parallelism, pipeline parallelism and data parallelism corresponding to the topology metadata; Step S3.4.2: Based on the second segmentation method, the parallelism of the multiple candidate parallel segmentation schemes is dimensionally aligned to obtain the first communication group.

10. The method according to claim 8, characterized in that, Step S3.5 includes: Step S3.5.1: Determine the second communication group that needs to be used for high-frequency communication from the first communication group; Step S3.5.2: Determine the bandwidth domain corresponding to the second communication group, and determine the candidate parallel segmentation scheme corresponding to the second communication group with the largest bandwidth in the bandwidth domain, as the target parallel segmentation scheme.

11. A breakpoint resume training device for an intelligent computing cloud platform based on heterogeneous computing power, characterized in that, The device includes: The processing module is used to acquire breakpoint data of the heterogeneous computing power intelligent computing cloud platform executing model training tasks on the Zhongyuan chip, and normalize the breakpoint data into a chip-independent intermediate representation. The chip-independent intermediate representation includes at least the topological metadata of the training task, the weight data of the logical tensor, the first and second moments of the optimizer corresponding to the logical tensor, and the state of the random number generator during the training process. The acquisition module is used to acquire the capability data of a target chip in the heterogeneous computing power that is of a different chip type than the source chip; The determination module is used to determine a target parallel partitioning scheme for the chip-independent intermediate representation based on the chip-independent intermediate representation and the capability data; The segmentation module is used to re-segment the chip-independent intermediate representation according to the target parallel segmentation scheme and distribute and load it onto the target chip; The training continuation module is used to adapt the computational precision and operators required to continue the training task of the model on the target chip based on the respective computational precision and operators of the source chip and the target chip, so as to continue the training task of the model on the target chip.

12. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the breakpoint resuming training method for the intelligent computing cloud platform based on heterogeneous computing power as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the breakpoint retraining method for the intelligent computing cloud platform based on heterogeneous computing power as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps of the breakpoint retraining method for the intelligent computing cloud platform based on heterogeneous computing power as described in any one of claims 1 to 10.

Images