Model training method and device based on heterogeneous hardware and electronic equipment

Abstract

The invention relates to a model training method and device based on heterogeneous hardware and electronic equipment. Determining a parallel strategy, dividing the model training task into training tasks corresponding to multiple pieces of heterogeneous hardware by using the parallel strategy, and allocating the training tasks to the corresponding heterogeneous hardware, so that each piece of heterogeneous hardware executes the corresponding training task; when the heterogeneous hardware executes the task, task execution information representing the resource utilization condition is obtained, and resource scheduling is conducted on the heterogeneous hardware through the task execution information. And training task allocation is realized through a parallel strategy, the training task execution efficiency and effect of each piece of heterogeneous hardware are ensured, and the resource utilization rate and load optimization is realized through respective resource scheduling of each piece of heterogeneous hardware. Therefore, in a model training scene based on heterogeneous hardware, the task execution efficiency and effect of each piece of heterogeneous hardware are improved, the resource utilization rate and the load are optimized, and then the efficiency and effect of model training are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a model training method, device and electronic device based on heterogeneous hardware. Background Art

[0002] With the development of artificial intelligence technology and computer technology, the scale of deep learning models has gradually expanded. In the scenario where the scale of deep learning models is gradually expanding, multiple devices can be used for distributed computing. For example, in the training scenario of the model, multiple distributed devices can perform computing to complete the training of the model. Summary of the invention

[0003] The purpose of the present invention is to provide a model training method, device and electronic device based on heterogeneous hardware.

[0004] In order to achieve the above-mentioned objectives, in a first aspect, the present disclosure provides a model training method based on heterogeneous hardware, including: determining a parallel strategy corresponding to multiple heterogeneous hardware; according to the parallel strategy, dividing the model training task into training tasks corresponding to multiple heterogeneous hardware respectively, and assigning the training tasks corresponding to the multiple heterogeneous hardware respectively to the corresponding heterogeneous hardware, so that the multiple heterogeneous hardware respectively execute the assigned training tasks; obtaining task execution information corresponding to the multiple heterogeneous hardware respectively, and the task execution information is used to characterize the resource utilization of the heterogeneous hardware when executing tasks; according to the task execution information corresponding to the multiple heterogeneous hardware respectively, performing resource scheduling for the multiple heterogeneous hardware respectively.

[0005] Optionally, determining the parallel strategies corresponding to multiple heterogeneous hardware includes: obtaining related information that affects the parallel execution effect of the model training task; and determining the parallel strategies corresponding to the multiple heterogeneous hardware based on the related information.

[0006] Optionally, the associated information includes at least one of the following information: model information corresponding to the model training task; training requirement information corresponding to the model training task; hardware information corresponding to the multiple heterogeneous hardware; communication requirement information between the multiple heterogeneous hardware; and execution information of the simulation training task corresponding to the model training task.

[0007] Optionally, the model training task includes model training data, and according to the parallel strategy, the model training task is divided into training tasks corresponding to multiple heterogeneous hardware, including: if the parallel strategy is a data parallel strategy, the model training data is divided into multiple training data; according to the multiple training data and the model training task, the training tasks corresponding to the multiple heterogeneous hardware are generated, wherein one training task corresponds to one piece of training data.

[0008] Optionally, the model training task includes a model to be trained, and according to the parallel strategy, the model training task is divided into training tasks corresponding to multiple heterogeneous hardware, including: if the parallel strategy is pipeline parallelism, the model to be trained is divided into multiple sub-models to be trained; according to the multiple sub-models to be trained and the model training task, the training tasks corresponding to the multiple heterogeneous hardware are generated, wherein one training task corresponds to one sub-model to be trained.

[0009] Optionally, the model training task includes a model to be trained and model training data, and according to the parallel strategy, the model training task is divided into training tasks corresponding to multiple heterogeneous hardware, including: if the parallel strategy is a hybrid parallel strategy, the model to be trained is divided into multiple sub-models to be trained, and the model training data is divided into multiple training data; according to the multiple sub-models to be trained, the multiple training data and the model training task, the training tasks corresponding to the multiple heterogeneous hardware are generated, wherein one training task corresponds to one sub-model to be trained and one training data.

[0010] Optionally, the model training method also includes: obtaining resource scheduling information corresponding to the multiple heterogeneous hardware, wherein the resource scheduling information is used to characterize the resource scheduling effect of the heterogeneous hardware; adjusting the training task corresponding to at least one heterogeneous hardware according to the task execution information corresponding to the multiple heterogeneous hardware and the resource scheduling information corresponding to the multiple heterogeneous hardware.

[0011] Optionally, the model training method further includes: obtaining model training results respectively obtained by the plurality of heterogeneous hardware; and integrating the model training respectively obtained by the plurality of heterogeneous hardware according to the parallel strategy to obtain a target model training result.

[0012] Optionally, the multiple heterogeneous hardware communicate with each other through a unified communication library.

[0013] In a second aspect, the present disclosure provides a model training device based on heterogeneous hardware, comprising: a determination module, configured to determine a parallel strategy corresponding to multiple heterogeneous hardware; an allocation module, configured to divide the model training task into training tasks corresponding to multiple heterogeneous hardware according to the parallel strategy, and allocate the training tasks corresponding to the multiple heterogeneous hardware to the corresponding heterogeneous hardware, so that the multiple heterogeneous hardware respectively execute the allocated training tasks; an acquisition module, configured to acquire task execution information corresponding to the multiple heterogeneous hardware, the task execution information is used to characterize the resource utilization of the heterogeneous hardware when executing tasks; a scheduling module, configured to perform resource scheduling for the multiple heterogeneous hardware according to the task execution information corresponding to the multiple heterogeneous hardware.

[0014] In a third aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model training method based on heterogeneous hardware described in the first aspect.

[0015] In a fourth aspect, the present disclosure provides an electronic device, comprising: a memory on which a computer program is stored; and a processor for executing the computer program in the memory to implement the model training method based on heterogeneous hardware described in the first aspect.

[0016] In a fifth aspect, the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the model training method based on heterogeneous hardware described in the first aspect.

[0017] Through the above technical solution, first determine the parallel strategy, use the parallel strategy to divide the model training task into multiple training tasks corresponding to heterogeneous hardware, and assign them to the corresponding heterogeneous hardware, so that each heterogeneous hardware performs the corresponding training task; when each heterogeneous hardware performs the task, obtain the task execution information that characterizes the resource utilization, and use the task execution information to schedule the resources of each heterogeneous hardware. Among them, the training task allocation is realized through the parallel strategy, which can ensure the execution efficiency and effect of the training tasks of each heterogeneous hardware, and optimize the resource utilization and load through the resource scheduling of each heterogeneous hardware. Therefore, this technical solution can improve the task execution efficiency and effect of each heterogeneous hardware in the model training scenario based on heterogeneous hardware, and optimize the resource utilization and load, thereby improving the efficiency and effect of model training.

[0018] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings: Figure 1 It is a flowchart of a model training method based on heterogeneous hardware according to an exemplary embodiment.

[0020] Figure 2 It is an architecture diagram of a heterogeneous computing power parallel and optimization platform for a model according to an exemplary embodiment.

[0021] Figure 3 A business process diagram is shown according to an exemplary embodiment.

[0022] Figure 4It is a structural block diagram of a model training device based on heterogeneous hardware according to an exemplary embodiment.

[0023] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0024] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0025] With the development of artificial intelligence and computer technology, the scale of deep learning models has gradually expanded. In the scenario where the scale of deep learning models gradually expands, the number of model parameters can reach billions or even hundreds of billions. Models of this scale can be distributed across multiple devices for computing.

[0026] For example, in a model training scenario, a large model system and service platform for distributed training is configured to integrate the computing resources of multiple devices.

[0027] In some related technologies, a distributed computing platform across multiple devices is deployed. However, the hardware performance of these multiple devices is basically uniform and there is no differentiation. Therefore, there is a lack of effective optimization solutions for these multiple devices, making it difficult to fully utilize computing resources, which limits the efficiency of distributed training and cannot guarantee the effect of distributed training.

[0028] In some related technologies, multiple types of equipment environments are deployed, and the equipment performance varies greatly. However, these related technologies are usually unable to fully coordinate the resource allocation of different equipment. For example, stronger performance equipment may be overloaded, while weaker equipment wastes resources and fails to achieve optimal overall utilization. This unbalanced resource use is particularly evident in large-scale training tasks, resulting in slower training speed and low computing efficiency.

[0029] In some related technologies, manual debugging and optimization are performed in heterogeneous systems with multiple devices and nodes. In a mixed environment of different hardware platforms such as NVIDIA H100 and Moore's Law S4000, the performance differences and communication complexity between devices increase the difficulty of debugging. In the absence of automated tuning tools, users need to manually adjust task allocation and related configurations, which consumes a lot of time and energy and reduces development efficiency.

[0030] Based on this, the embodiment of the present disclosure provides a technical solution, which can be applied to the distributed training scenario of heterogeneous hardware. In the distributed training scenario of heterogeneous hardware, multiple heterogeneous hardware are involved. The training task allocation is realized through the parallel strategy, which can ensure the execution efficiency and effect of the training tasks of each heterogeneous hardware. Through the resource scheduling of each heterogeneous hardware, the resource utilization and load optimization can be realized. In addition, the efficiency and effect of model training can be guaranteed.

[0031] Furthermore, this technical solution does not require manual adjustment or configuration, which can reduce development efficiency.

[0032] Regarding the distributed training scenarios of heterogeneous hardware, for example, the scenarios of mixed training of heterogeneous resources of domestic intelligent computing accelerator cards and NVIDIA GPU (Graphics Processing Unit), and the pre-training technology for large language models based on Transformers architecture (such as Qwen-72B-Chat).

[0033] Furthermore, the technical solution of the disclosed embodiment can be applied to large model systems and service platforms that require efficient distributed training, and is more suitable for scenarios where different types of GPUs are mixed in the Mlops (Machine Learning Operations) process.

[0034] Therefore, the disclosed embodiment provides a heterogeneous computing power parallel simulation and optimization solution for large-scale language models, which maximizes the utilization of heterogeneous hardware resources and improves model training efficiency through core technical means such as intelligent scheduling and parallel computing optimization. This technical solution can be applied to environments containing heterogeneous computing devices such as NVIDIA H100 and domestic Moore's Law S4000, and combines parallel strategies and resource scheduling optimization to support the full process requirements from model training to deployment.

[0035] In some embodiments, the technical solution of the embodiments of the present disclosure can be applied in the form of a platform or system. Different software layers can be deployed in the platform or system to achieve different functions. The software layer deployment method of the platform or system will be introduced later in conjunction with the specific technical solution.

[0036] Figure 1 is a flowchart of a model training method based on heterogeneous hardware according to an exemplary embodiment. Figure 1 As shown, the model training method can be applied to some software layers of a platform or system, and the method includes the following steps: Step S11, determining parallel strategies corresponding to multiple heterogeneous hardware.

[0037] Step S12, according to the parallel strategy, divide the model training task into training tasks corresponding to multiple heterogeneous hardware respectively, and assign the training tasks corresponding to the multiple heterogeneous hardware respectively to the corresponding heterogeneous hardware, so that the multiple heterogeneous hardware respectively execute the assigned training tasks.

[0038] Step S13, obtaining task execution information corresponding to the plurality of heterogeneous hardware respectively, where the task execution information is used to characterize resource utilization of the heterogeneous hardware when executing tasks.

[0039] Step S14: performing resource scheduling on the plurality of heterogeneous hardware respectively according to the task execution information respectively corresponding to the plurality of heterogeneous hardware.

[0040] In some embodiments, the plurality of heterogeneous hardware are in a distributed relationship. In order to achieve efficient computing of the plurality of heterogeneous hardware, a parallel strategy of the plurality of heterogeneous hardware may be selected. The parallel strategy may determine the distributed operation mode of the plurality of heterogeneous hardware.

[0041] In some embodiments, heterogeneous hardware may include: training chips, inference chips, and general computing and distributed training storage, etc. Among them, the training chip may be the Indawei-A / H series or a domestic training chip series. The inference chip may be a domestic inference chip or Indawei RTX4090, etc. General computing and distributed training storage may include: CPU (Central Processing Unit), RAM (Random Access Memory), object storage, and SSD (Solid State Disk) storage, etc.

[0042] As an optional implementation, step S11 includes: obtaining related information that affects the parallel execution effect of the model training task; and determining the parallel strategies corresponding to multiple heterogeneous hardware based on the related information.

[0043] In this implementation, the specific parallel strategy to be adopted is analyzed from the perspective of the parallel execution effect of the model training task.

[0044] In some embodiments, the Alpa automatic parallel optimization framework can be used to implement parallel strategy selection and optimization: Hierarchical parallel optimization: Alpa combines integer linear programming and dynamic programming to generate optimal parallel strategies.

[0045] Automated optimization process: With Alpa, the system can automatically generate the best parallel optimization solution, reducing the complexity of manual debugging.

[0046] Optimality guarantee: Alpa provides theoretical optimality guarantee, ensuring that the automatically generated plans can achieve the highest execution efficiency in heterogeneous hardware environments.

[0047] In some embodiments, the associated information includes at least one of the following information: model information corresponding to the model training task; training requirement information corresponding to the model training task; hardware information corresponding to multiple heterogeneous hardware; communication requirement information between multiple heterogeneous hardware; execution information of the simulation training task corresponding to the model training task.

[0048] In some embodiments, if the model training task is a real training task, the execution information of the simulation training task corresponding to the model training task may be involved. If the model training task is a simulation training task, the associated information may not be involved.

[0049] In some embodiments, the model training task is a simulation training task or a real training task. The difference between the simulation training task and the real training task is the duration. The real training task takes longer and involves specific data. The simulation training task takes less time than the real training task and does not involve specific data. A simple prediction can be made on the task execution. The simulation training task may not necessarily be executed, and can be selected according to the actual situation.

[0050] In some embodiments, model training can be first performed based on a simulation training task, and based on the training results of the simulation training task, heterogeneous hardware can be optimized, and then the optimized heterogeneous hardware can be used to perform real training tasks.

[0051] In some embodiments, the execution information of the simulation training task can be used to allocate tasks to heterogeneous hardware in real training tasks, such as determining parallel strategies.

[0052] In some embodiments, the model information corresponding to the model training task may include: model type, model parameters, model size, etc. The model information can be determined according to the specific model targeted by the model training task.

[0053] In some embodiments, the training requirement information corresponding to the model training task can represent the requirements of the model training, such as efficiency requirements, accuracy requirements, etc. The training requirement information can be formulated by the user or can be preset information.

[0054] In some embodiments, the hardware information corresponding to the heterogeneous hardware may include: hardware model, hardware performance, etc. The hardware information is known information of the heterogeneous hardware and can be directly obtained.

[0055] In some embodiments, the communication demand information between heterogeneous hardware can characterize the communication demand between heterogeneous hardware, and may include: communication type, communication delay, communication bottleneck, etc. The communication demand information can be determined based on the specific situation of the heterogeneous hardware, or can be information preset by the user.

[0056] In some embodiments, the execution information of the simulation training task corresponding to the model training task may include: task execution efficiency, resource utilization of heterogeneous hardware, etc. The execution information is known execution information of the simulation training task and can be directly obtained.

[0057] In some embodiments, multiple parallel strategies can be preset, and various associated information conditions corresponding to the multiple parallel strategies can be preset. Based on the current associated information, the associated information conditions corresponding to the multiple parallel strategies are compared, and the parallel strategy corresponding to the associated information condition that best matches the associated information is determined as the current parallel strategy.

[0058] In the embodiments of the present disclosure, the multiple parallel strategies may include: data parallel strategy, pipeline parallel strategy and hybrid parallel strategy.

[0059] In some embodiments, the data parallel strategy can be understood as: multiple heterogeneous hardware processes data in parallel, then the training data of the model training task can be divided into multiple heterogeneous hardware. The data parallel strategy is suitable for parallel processing of large batches of data, so the corresponding associated information conditions include large model parameters, model size, etc.

[0060] In some embodiments, the pipeline parallel strategy can be understood as: multiple heterogeneous hardware performs segmented parallel computing. Then, the model training task can be divided into multiple parts, and different heterogeneous hardware is used to execute different parts.

[0061] Pipeline parallel strategies, such as: BPipe: By balancing the memory load of each heterogeneous hardware, it avoids resource waste and uneven distribution, and is suitable for dynamic load adjustment in heterogeneous computing environments. GPipe: Utilizes micro-batch flow for segmented parallel computing, which is suitable for devices with limited memory to perform large-scale model training tasks.

[0062] Therefore, different pipeline parallel strategies can configure different associated information conditions according to their characteristics.

[0063] In some embodiments, the hybrid parallel strategy can be understood as a combination of the data parallel strategy and the pipeline parallel strategy. The hybrid parallel strategy can maximize resource utilization.

[0064] In the hybrid parallel strategy, data parallel strategy and pipeline parallel strategy can be used to achieve different functions. For example, for large models such as Qwen-72B-Chat, data parallel strategy is used for batch data processing, while pipeline parallel strategy is used to distribute different parts of the model to multiple heterogeneous hardware for execution. Under this strategy, not only can the communication overhead between heterogeneous hardware be reduced, but also the utilization rate of various types of heterogeneous hardware can be maximized in a heterogeneous hardware environment.

[0065] Therefore, in the associated information conditions corresponding to the hybrid parallel strategy, corresponding configurations can be made for training requirements and communication requirements.

[0066] In step S12, the model training tasks may be divided and allocated according to the parallel strategy.

[0067] In some embodiments, a model may be loaded from a preset model training library, and then a model training task may be generated. In the model training task, the model to be trained and the training data required for the model training may be configured.

[0068] In some embodiments, different parallel strategies may adopt different task division methods.

[0069] Therefore, as a first optional implementation, step S12 includes: if the parallel strategy is a data parallel strategy, dividing the model training data into multiple training data; generating training tasks corresponding to multiple heterogeneous hardware according to the multiple training data and model training tasks, wherein one training task corresponds to one training data.

[0070] In this implementation, the model training data is divided based on the data parallel strategy. Specifically, the training data can be divided into multiple parts according to the performance of each heterogeneous hardware. For example, the better the performance of the heterogeneous hardware, the larger the amount of training data corresponding to it.

[0071] In some embodiments, after the training data is divided, in each training task, the model is the same, but the amount of training data may be different, and the specific training data involved may also be different. Thus, for each heterogeneous hardware, one model is trained, but different training data is used.

[0072] As a second optional implementation, step S12 includes: if the parallel strategy is pipeline parallelism, dividing the model to be trained into multiple sub-models to be trained; generating multiple training tasks corresponding to heterogeneous hardware according to the multiple sub-models to be trained and the model training tasks, wherein one training task corresponds to one sub-model to be trained.

[0073] In some embodiments, the multiple sub-models to be trained may be multiple model parts split from the model to be trained.

[0074] In some embodiments, multiple sub-models to be trained may be divided according to the performance of each heterogeneous hardware. For example, the better the performance of the heterogeneous hardware, the higher the training difficulty of the corresponding sub-model to be trained.

[0075] In some embodiments, the sub-models to be trained corresponding to different heterogeneous hardware may be the same or different, and there may be heterogeneous hardware corresponding to different sub-models to be trained.

[0076] In some embodiments, after the model to be trained is divided, the model is different in each training task. Therefore, for each heterogeneous hardware, different parts of a model are trained. For example, the first heterogeneous hardware is trained first, and the training results obtained can be transmitted to the next heterogeneous hardware, and the next heterogeneous hardware continues to train.

[0077] As a third optional implementation, step S12 includes: the parallel strategy is a hybrid parallel strategy, the model to be trained is divided into multiple sub-models to be trained, and the model training data is divided into multiple training data; according to the multiple sub-models to be trained, the multiple training data and the model training tasks, a plurality of training tasks corresponding to heterogeneous hardware are generated, wherein one training task corresponds to one sub-model to be trained and one training data.

[0078] In this implementation, both the model to be trained and the model training data are divided to obtain multiple sub-models to be trained and multiple training data.

[0079] In some embodiments, the division of models to be trained and model training data may refer to the performance of different heterogeneous hardware.

[0080] In some embodiments, after the models to be trained and the model training data are divided, the sub-models to be trained and the training data involved in each training task may be different.

[0081] Thus, for each heterogeneous hardware, different parts of a model are trained based on different training data. For example, a model includes a feature extraction part and a prediction part, the feature extraction part is trained based on the training data corresponding to the feature extraction, and the prediction part is trained based on the training data corresponding to the prediction.

[0082] In some embodiments, the training tasks corresponding to different heterogeneous hardware may be the same or different. For example, two heterogeneous hardware with substantially the same performance may have the same corresponding training tasks, while two heterogeneous hardware with a large performance gap may have different corresponding training tasks.

[0083] In some embodiments, when dividing the training tasks, the heterogeneous hardware performance requirements required by each training task can be specified, and then, when allocating the training tasks, the matching heterogeneous hardware is allocated according to the corresponding heterogeneous hardware performance requirements.

[0084] Furthermore, after completing the allocation of training tasks, multiple heterogeneous hardware will respectively execute the corresponding allocated tasks.

[0085] Therefore, in step S13, task execution information corresponding to a plurality of heterogeneous hardwares can be obtained, and the task execution information is used to characterize the resource utilization of the heterogeneous hardware when executing tasks.

[0086] Regarding resources, it may involve: CPU, GPU, memory, network, etc.

[0087] For example, the task execution information may include: resource utilization of heterogeneous hardware, if multiple resources are involved, then the resource utilization corresponding to the multiple resources respectively. Alternatively, it may also include: bottlenecks of heterogeneous hardware, such as insufficient resources or excessive load.

[0088] Furthermore, in step S14, resource scheduling is performed on the plurality of heterogeneous hardwares respectively according to the task execution information respectively corresponding to the plurality of heterogeneous hardwares.

[0089] In some embodiments, the execution priority of tasks can be adjusted according to resource utilization, insufficient resources, overload, etc., as well as task allocation and task offloading of resources.

[0090] For example, if resource A is overloaded, the tasks of resource A can be unloaded and the unloaded tasks can be scheduled to other resources. For example, if the resource utilization rate of resource A is low, the tasks with higher priority can be adjusted to resource A for execution.

[0091] Alternatively, other resource scheduling methods may be used, and specific reference may be made to mature technologies in the art.

[0092] In some embodiments, resources may be dynamically configured by combining actual hardware performance data with automated tools, including: Hardware resource configuration: DeepSpeed's profiling tool automatically measures the FLOPs (Floating Point Operations Per Second) of large models in heterogeneous hardware environments. Combined with actual test data, the platform can dynamically adjust the hardware resource allocation strategy to ensure that the performance differences of different devices are effectively utilized.

[0093] Configuration method: Use profiling data to identify performance differences and bottlenecks, adjust task scheduling, and ensure that resources are maximized.

[0094] Optimization goal: Reduce hardware idle or overload conditions and improve overall training efficiency.

[0095] In some embodiments, after resource scheduling is performed, the parallel strategy may be further adjusted.

[0096] Therefore, as an optional implementation, the model training method also includes: obtaining resource scheduling information corresponding to multiple heterogeneous hardware, the resource scheduling information is used to characterize the resource scheduling effect of the heterogeneous hardware; according to the task execution information corresponding to the multiple heterogeneous hardware and the resource scheduling information corresponding to the multiple heterogeneous hardware, adjusting the training task corresponding to at least one heterogeneous hardware.

[0097] In this implementation, the resource scheduling effect may represent the change in resource utilization after resource scheduling. For example, if the utilization of resources with lower resource utilization is improved after resource scheduling, the resource scheduling effect is good.

[0098] Therefore, the resource scheduling information may include resource utilization changes, load changes, etc. corresponding to multiple heterogeneous hardware.

[0099] Furthermore, by combining the task execution information corresponding to each heterogeneous hardware and the resource scheduling information corresponding to multiple heterogeneous hardware, it can be selected whether to adjust the training tasks corresponding to each heterogeneous hardware to optimize the parallel strategy.

[0100] For example, if the task execution information corresponding to some heterogeneous hardware indicates poor resource utilization and heavy load, after resource scheduling, it is determined through the resource scheduling information that both resource utilization and load are optimized, and the task execution information corresponding to some heterogeneous hardware indicates poor resource utilization and heavy load, after resource scheduling, it is determined through the resource scheduling information that both resource utilization and load are optimized. Then, part of the training tasks of the non-optimized heterogeneous hardware can be transferred to the optimized heterogeneous hardware.

[0101] In some embodiments, an intelligent scheduling algorithm is used to schedule tasks to appropriate heterogeneous hardware in real time according to the current load of the heterogeneous hardware and the complexity of the task.

[0102] Configuration method: The scheduling algorithm dynamically adjusts the execution order and distribution of tasks based on the actual hardware performance, device resource utilization, and task priority. It ensures that tasks are evenly distributed on different hardware to avoid overloading of certain devices or idle resources. At the same time, the scheduling strategy is adjusted in a timely manner by monitoring the execution status of tasks.

[0103] Optimization goal: Through intelligent scheduling, balance the load of each device, ensure efficient execution of tasks, and minimize task queuing time and waste of equipment resources.

[0104] In some embodiments, the model training method may further include: obtaining model training results obtained by multiple heterogeneous hardware respectively; integrating the model training obtained by multiple heterogeneous hardware respectively according to a parallel strategy to obtain a target model training result.

[0105] In this implementation, the model training results obtained by multiple heterogeneous hardwares may include: model weights.

[0106] Correspondingly, according to the parallel strategy, the weights of each model are synchronized or merged to obtain the final model weight, that is, the training result of the target model.

[0107] It is understandable that since the parallel strategy determines the parallel training mode of multiple heterogeneous hardware, the training results need to be integrated according to the integration mode that matches the parallel training mode. For example, the weights of different model parts are merged to obtain the weight of the overall model.

[0108] In some related technologies, it is impossible to optimize business configuration in real time for communication bottlenecks and performance delays in production environments: In actual production environments, large-scale distributed training models usually involve frequent communications between multiple devices and clusters, and communication delays and bandwidth bottlenecks become key issues affecting training performance. Related technologies are unable to monitor and optimize these communication problems in real time during the training process, resulting in inefficient data transmission between devices, which in turn slows down the overall training process. Due to the existence of communication bottlenecks, the performance degradation in model training cannot be adjusted and optimized in a timely manner during training through traditional methods, which seriously affects the continuous iteration and development of the business.

[0109] In the embodiments of the present disclosure, in addition to solving the problem through the aforementioned resource scheduling, the problem can also be solved by optimizing the communication method between heterogeneous hardware.

[0110] Therefore, as an optional implementation, multiple heterogeneous hardware communicate with each other through a unified communication library.

[0111] In this implementation, a unified communication library is configured for multiple heterogeneous hardware, and the unified communication library can limit the communication mode between the heterogeneous hardware, thereby avoiding the occurrence of communication bottlenecks.

[0112] For example, the unified communication library can be a communication library between domestic computing power and traditional GPU.

[0113] In the disclosed embodiment, multi-device collaborative training is supported, and a distributed training environment with multiple machines and multiple cards can be accurately simulated. Through real-time monitoring, the platform reflects the communication delay and load distribution between heterogeneous hardware, and dynamically adjusts strategies according to different task requirements.

[0114] Its application scenarios may involve: actual scenarios of multi-device parallel training, such as distributed data parallel training, cross-device communication scenarios of gradient synchronization and weight merging.

[0115] Through simulation and monitoring functions, it is possible to identify performance bottlenecks during training and automatically adjust device load and resource allocation. Taking Qwen-72B-Chat as an example, the optimized training efficiency has increased by more than 30% and the communication delay has been significantly reduced.

[0116] This series of technologies and configuration strategies enable flexible response to complex training tasks, especially in heterogeneous hardware environments, which improves task execution efficiency and resource utilization, while significantly optimizing communication overhead and load balancing.

[0117] Figure 2 It is an architecture diagram of a heterogeneous computing power parallel and optimization platform for a model according to an exemplary embodiment, and the platform involves software modules of various functions introduced in the aforementioned embodiments.

[0118] like Figure 2 As shown in the figure, the platform can be divided into: visualization monitoring layer, training task parallel computing optimization layer, unified resource management layer and heterogeneous hardware integration layer.

[0119] The visualization monitoring layer involves: FLOPS calculation and performance prediction, parallel simulation, training task simulation, real-time visualization, hardware monitoring and communication monitoring data collection and processing, task management, policy configuration, log and performance monitoring.

[0120] The parallel computing optimization layer for training tasks involves: parallel strategy engine, including: hybrid parallel drive, data parallel drive, model parallel drive and pipeline parallel drive; Alpa automated parallel framework, including: hierarchical parallel optimization and hybrid parallel optimization; intelligent scheduling algorithm, including: task allocation and load balancing, workload prediction, parallel strategy support and real-time priority adjustment; unified communication library, including: heterogeneous hardware support and collective communication operations.

[0121] Unified resource management layer, including: application gateway and authentication, network interface and management, computing resource management, storage management, workload scheduling, resource monitoring and logging, tenant authentication and quota management.

[0122] Heterogeneous hardware integration layer, including: training chips, including: NVIDIA-A / H series and domestic training chips. Inference chips, including: domestic inference chips and NVIDIA RTX4090. General computing and distributed storage, including: CPU, RAM, object storage and SSD storage.

[0123] based on Figure 2 As shown in the architecture, the entire business process can be divided into three main layers: visualization monitoring layer, training task parallel computing optimization layer, and unified resource management layer.

[0124] Figure 3 is a business process diagram according to an exemplary embodiment. Figure 3 As shown, the process description of each level is as follows: The visualization monitoring layer is responsible for initializing tasks and monitoring and simulating hardware resources during training to provide feedback and optimization information for task execution.

[0125] FLOPs calculation and performance prediction: First, the platform calculates the FLOPs performance of the device through the profiling tool and automatically configures the training task according to the hardware resources. This stage ensures that the computing power of each device is optimally configured.

[0126] Task allocation and release: Next, the tasks are distributed to various heterogeneous hardware, and the system will monitor the execution of the tasks in real time and provide performance feedback.

[0127] Log and performance monitoring: During the task execution, the system will continuously collect execution logs and perform performance analysis to provide optimization information after the task is completed.

[0128] The training task parallel computing optimization layer is responsible for task scheduling, parallel strategy selection, and dynamic optimization of tasks.

[0129] Task scheduling and optimization: First, the parallel strategy is optimized through the Alpa framework to generate a suitable execution plan based on available resources and task requirements. Then, the task is scheduled to multiple hardware for execution to ensure maximum utilization of hardware resources.

[0130] Parallel Strategy Engine: The engine automatically selects the appropriate parallel computing strategy (. The Alpa framework further dynamically optimizes tasks through ILP and DP algorithms to ensure efficient execution of training tasks.

[0131] Priority adjustment: During the execution process, the platform monitors the resource load in real time. If a bottleneck is detected, it will adjust the priority of the task, reallocate resources and schedule the task execution to ensure the smooth progress of the task.

[0132] Weight synchronization and merging: Finally, after the model training is completed, the system synchronizes and merges the model weights of each heterogeneous hardware to ensure the consistency of the training results.

[0133] The unified resource management layer is responsible for the management, monitoring, and scheduling of underlying hardware resources.

[0134] Resource scheduling and monitoring: Monitor all hardware resources (such as CPU, GPU, memory, network, etc.) in real time and dynamically adjust resource allocation according to task requirements.

[0135] Task resource feedback and adjustment: Based on real-time monitoring of resources, the execution priority of tasks can be flexibly adjusted, and tasks can be actively unloaded when insufficient resources or excessive load are detected.

[0136] As a first example, the platform can be used to continue pre-training the large-scale model Qwen-72B-Chat.

[0137] Scenario description: Continue pre-training of the Qwen-72B-Chat large-scale language model on a heterogeneous computing cluster including NVIDIA H100 and domestic Moore Thread S4000. Qwen-72B-Chat is a basic model that has undergone unsupervised training. It has learned a wide range of language representation capabilities during the initial training process, but in order to adapt to more specific task requirements, the model needs to be continued pre-trained.

[0138] In this example scenario, the following processes may be involved: Basic model loading: First, the platform loads the Qwen-72B-Chat basic model from the pre-trained model library. This model is large in scale and has undergone extensive unsupervised learning in the initial stage, and has the ability to understand a large number of language structures and contexts.

[0139] FLOPs calculation and simulation configuration: The DeepSpeed ​​Profiling tool measures the hardware performance in the cluster (such as the FLOPs of H100 and S4000) and automatically configures training tasks based on this performance.

[0140] Parallel strategy selection: Combining data parallelism and pipeline parallelism, the platform automatically optimizes the distribution and execution of tasks through the Alpa framework. BPipe is used to ensure that the memory and computing load of each hardware are balanced to avoid resource waste.

[0141] Task execution and monitoring: Tasks are assigned to each device, and the communication delay between devices is optimized using a unified communication library. The task execution status is monitored in real time, and the task priority and resource scheduling are adjusted.

[0142] Weight synchronization and merging: After the task is completed, the platform synchronizes and merges the weights of the model to ensure that the training results of each device are consistent, and finally outputs the model after further pre-training.

[0143] Performance improvement summary: After optimization, the training efficiency of the Qwen-72B-Chat model has increased by about 15%, the communication delay has been significantly reduced by about 10%, and the training process has become more efficient.

[0144] As a second example, the platform can be used for distributed training of the GPT-3 model.

[0145] Scenario description: Distributed training of the GPT-3 basic model is performed in a multi-machine, multi-card heterogeneous cluster. GPT-3 is a widely used large-scale language model that has undergone unsupervised pre-training and can generate high-quality natural language text. Training large models in a distributed environment requires the use of parallel strategies and communication optimization to improve training efficiency.

[0146] In this example scenario, the following processes may be involved: Basic model loading: The platform loads the GPT-3 basic model from the model library. GPT-3 is an unsupervised pre-trained general language model that contains billions of parameters and has the ability to handle tasks such as natural language generation, translation, and conversation.

[0147] FLOPs calculation and simulation configuration: The platform uses the Profiling tool to detect the computing power (such as FLOPs) of the hardware device and generates preliminary task configuration based on this data.

[0148] Parallel strategy selection: Using a hybrid parallel strategy, data parallelism is combined with GPipe pipeline parallelism. The platform optimizes task allocation through the Alpa framework to ensure efficient execution of model training tasks on different devices.

[0149] Communication optimization: Use unified communication libraries and RDMA (Remote Direct Memory Access) communication to reduce communication delays between devices and ensure the efficiency of gradient synchronization and weight updates. Tasks monitor resource utilization in real time, dynamically schedule task priorities, and optimize hardware utilization.

[0150] Weight synchronization and merging: After the task is completed, the platform synchronizes and merges the weights of the model to ensure that the training results of each hardware are consistent, and finally outputs the model after further pre-training.

[0151] Task completion and optimization summary: After the training task is completed, logs and performance reports are generated, showing that the training time of the GPT-3 model is shortened by 15% and the communication delay is reduced by about 10%.

[0152] As a third example, the platform can be used to perform online supervised fine-tuning of the BERT model.

[0153] Scenario description: Perform online supervised fine-tuning on the BERT model in a heterogeneous computing cluster. BERT is a basic model that has undergone unsupervised pre-training and is widely used in natural language understanding tasks (such as sentiment analysis, text classification, etc.). In order to improve the performance of the model in specific tasks, it is necessary to use supervised data for fine-tuning.

[0154] In this example scenario, the following processes may be involved: Basic model loading: Load the BERT pre-trained model from the model library. The BERT model already has the ability to understand a wide range of language patterns, but requires supervised fine-tuning on task-specific data.

[0155] FLOPs calculation and performance configuration: The platform detects the computing performance of H100 and S4000 devices through profiling tools, automatically generates hardware resource allocation strategies, and ensures optimal resource utilization during task execution.

[0156] Parallel strategy and task execution: Parallel strategy selection: The Alpa framework automatically selects a suitable parallel strategy. In this scenario, the LoRA (low-rank adaptation) method is used to fine-tune BERT. LoRA reduces the parameter scale of training by introducing low-rank matrices, thereby improving training efficiency. Task execution: The platform combines data parallelism with pipeline parallelism, using data parallelism to process large batches of supervised data, while pipeline parallelism is responsible for the segmented execution of the model.

[0157] Intelligent scheduling and communication optimization: Dynamically adjust task priorities and resource scheduling based on the real-time load of the hardware, reduce communication overhead between hardware through a unified communication library and RDMA technology, and optimize the gradient synchronization and weight update process.

[0158] Weight synchronization and merging: After the task is completed, the platform synchronizes and merges the weights obtained from each hardware training to ensure that the weights of the fine-tuned model are consistent.

[0159] Performance summary: Through the application of LoRA technology, the computing resource usage of fine-tuning BERT is reduced by about 40%, the training efficiency is improved by 10%, the communication delay is reduced by about 10%, and the performance of the final model on the task is significantly improved.

[0160] Due to differences in the actual performance of heterogeneous hardware, task complexity, and data set size, the specific improvement effect may fluctuate. In order to ensure the scientificity and accuracy of the performance evaluation, the above performance improvements are explained in a conservative estimation method, and the actual effect will vary depending on the task and environment.

[0161] These optimization measures provide more efficient resource utilization and execution strategies for the training and fine-tuning of large models, ensuring stable performance improvements in various hardware environments.

[0162] Figure 4 is a structural block diagram of a model training device based on heterogeneous hardware according to an exemplary embodiment. Figure 4 As shown, the device comprises: The determination module 401 is configured to determine parallel strategies corresponding to multiple heterogeneous hardware.

[0163] The allocation module 402 is configured to divide the model training task into training tasks corresponding to multiple heterogeneous hardware according to the parallel strategy, and allocate the training tasks corresponding to the multiple heterogeneous hardware to the corresponding heterogeneous hardware, so that the multiple heterogeneous hardware respectively execute the allocated training tasks.

[0164] The acquisition module 403 is configured to acquire task execution information corresponding to the plurality of heterogeneous hardware respectively, wherein the task execution information is used to characterize resource utilization of the heterogeneous hardware when executing tasks.

[0165] The scheduling module 404 is configured to perform resource scheduling on the plurality of heterogeneous hardware respectively according to the task execution information respectively corresponding to the plurality of heterogeneous hardware.

[0166] Optionally, the acquisition module 403 is further configured to obtain related information that affects the parallel execution effect of the model training task; the determination module 401 is further configured to determine the parallel strategy corresponding to the multiple heterogeneous hardware based on the related information.

[0167] Optionally, the associated information includes at least one of the following information: model information corresponding to the model training task; training requirement information corresponding to the model training task; hardware information corresponding to the multiple heterogeneous hardware; communication requirement information between the multiple heterogeneous hardware; and execution information of the simulation training task corresponding to the model training task.

[0168] Optionally, the allocation module 402 is also configured to divide the model training data into multiple training data if the parallel strategy is a data parallel strategy; generate training tasks corresponding to the multiple heterogeneous hardware respectively according to the multiple training data and the model training tasks, wherein one training task corresponds to one training data.

[0169] Optionally, the allocation module 402 is also configured to divide the model to be trained into multiple sub-models to be trained if the parallel strategy is pipeline parallelism; generate training tasks corresponding to the multiple heterogeneous hardware respectively according to the multiple sub-models to be trained and the model training tasks, wherein one training task corresponds to one sub-model to be trained.

[0170] Optionally, the allocation module 402 is also configured to divide the model to be trained into multiple sub-models to be trained, and divide the model training data into multiple training data if the parallel strategy is a hybrid parallel strategy; generate training tasks corresponding to the multiple heterogeneous hardware respectively according to the multiple sub-models to be trained, the multiple training data and the model training tasks, wherein one training task corresponds to one sub-model to be trained and one training data.

[0171] Optionally, the acquisition module 403 is further configured to: obtain resource scheduling information corresponding to the multiple heterogeneous hardware, and the resource scheduling information is used to characterize the resource scheduling effect of the heterogeneous hardware; the allocation module 402 is further configured to: adjust the training task corresponding to at least one heterogeneous hardware according to the task execution information corresponding to the multiple heterogeneous hardware and the resource scheduling information corresponding to the multiple heterogeneous hardware.

[0172] Optionally, the acquisition module 403 is further configured to: obtain the model training results respectively obtained by the multiple heterogeneous hardware; the device also includes an integration module, which is configured to: integrate the model training respectively obtained by the multiple heterogeneous hardware according to the parallel strategy to obtain the target model training result.

[0173] Optionally, the multiple heterogeneous hardware communicate with each other through a unified communication library.

[0174] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0175] Figure 5 FIG. 5 is a block diagram of an electronic device 500 according to an exemplary embodiment. Figure 5 As shown, the electronic device 500 may include: a processor 501 , a memory 502 . The electronic device 500 may also include one or more of a multimedia component 503 , an input / output (I / O) interface 504 , and a communication component 505 .

[0176] The processor 501 is used to control the overall operation of the electronic device 500 to complete all or part of the steps in the above-mentioned model training method based on heterogeneous hardware. The memory 502 is used to store various types of data to support the operation of the electronic device 500, and these data may include, for example, instructions for any application or method for operating on the electronic device 500, and application-related data, such as contact data, sent and received messages, pictures, audio, video, etc. The memory 502 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk. The multimedia component 503 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone, which is used to receive external audio signals. The received audio signal may be further stored in the memory 502 or sent through the communication component 505. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 504 provides an interface between the processor 501 and other interface modules, and the above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 505 is used for wired or wireless communication between the electronic device 500 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 505 may include: Wi-Fi module, Bluetooth module, NFC module.

[0177] In an exemplary embodiment, the electronic device 500 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned heterogeneous hardware-based model training method.

[0178] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned model training method based on heterogeneous hardware are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 502 including program instructions, and the above-mentioned program instructions can be executed by the processor 501 of the electronic device 500 to complete the above-mentioned model training method based on heterogeneous hardware.

[0179] In another exemplary embodiment, a computer program product is also provided, which includes a computer program that can be executed by a processor, and when the computer program is executed by the processor, the steps of the above-mentioned model training method based on heterogeneous hardware are implemented.

[0180] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, the technical solution of the present disclosure can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present disclosure.

[0181] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0182] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A model training method based on heterogeneous hardware, characterized in that: include: Determine parallel strategies for multiple heterogeneous hardware; According to the parallel strategy, the model training task is divided into training tasks corresponding to a plurality of heterogeneous hardwares, and the training tasks corresponding to the plurality of heterogeneous hardwares are allocated to corresponding heterogeneous hardwares, so that the plurality of heterogeneous hardwares respectively execute the allocated training tasks; Acquire task execution information corresponding to the plurality of heterogeneous hardware respectively, wherein the task execution information is used to characterize resource utilization of the heterogeneous hardware when executing tasks; According to the task execution information respectively corresponding to the plurality of heterogeneous hardware, resource scheduling is performed on the plurality of heterogeneous hardware respectively.

2. The model training method according to claim 1, characterized in that: The determining of the parallel strategies corresponding to the plurality of heterogeneous hardware includes: Obtaining relevant information that affects the parallel execution effect of the model training task; According to the association information, parallel strategies corresponding to the plurality of heterogeneous hardware are determined.

3. The model training method according to claim 2, characterized in that: The associated information includes at least one of the following information: Model information corresponding to the model training task; Training requirement information corresponding to the model training task; Hardware information corresponding to the plurality of heterogeneous hardware respectively; Communication requirement information between the plurality of heterogeneous hardware; The execution information of the simulation training task corresponding to the model training task.

4. The model training method according to claim 1, characterized in that: The model training task includes model training data, and the model training task is divided into a plurality of training tasks corresponding to the heterogeneous hardware according to the parallel strategy, including: If the parallel strategy is a data parallel strategy, the model training data is divided into multiple training data; According to the multiple training data and the model training tasks, training tasks corresponding to the multiple heterogeneous hardware are generated, wherein one training task corresponds to one training data.

5. The model training method according to claim 1, characterized in that: The model training task includes a model to be trained, and the model training task is divided into a plurality of training tasks corresponding to the heterogeneous hardware according to the parallel strategy, including: If the parallel strategy is pipeline parallelism, the model to be trained is divided into multiple sub-models to be trained; According to the multiple sub-models to be trained and the model training tasks, training tasks corresponding to the multiple heterogeneous hardware are generated, wherein one training task corresponds to one sub-model to be trained.

6. The model training method according to claim 1, characterized in that: The model training task includes a model to be trained and model training data. According to the parallel strategy, the model training task is divided into a plurality of training tasks corresponding to heterogeneous hardware, including: If the parallel strategy is a hybrid parallel strategy, the model to be trained is divided into a plurality of sub-models to be trained, and the model training data is divided into a plurality of training data; According to the multiple sub-models to be trained, the multiple training data and the model training tasks, training tasks corresponding to the multiple heterogeneous hardware are generated, wherein one training task corresponds to one sub-model to be trained and one training data.

7. The model training method according to any one of claims 1 to 6, characterized in that: The model training method also includes: Obtain resource scheduling information corresponding to the plurality of heterogeneous hardware respectively, wherein the resource scheduling information is used to characterize the resource scheduling effect of the heterogeneous hardware; According to the task execution information respectively corresponding to the multiple heterogeneous hardware and the resource scheduling information respectively corresponding to the multiple heterogeneous hardware, the training task corresponding to at least one heterogeneous hardware is adjusted.

8. The model training method according to any one of claims 1 to 6, characterized in that: The model training method also includes: Obtaining model training results respectively obtained by the plurality of heterogeneous hardware; According to the parallel strategy, the model trainings obtained respectively by the plurality of heterogeneous hardware are integrated to obtain the target model training result.

9. The model training method according to any one of claims 1 to 6, characterized in that: The multiple heterogeneous hardware communicate with each other through a unified communication library.

10. A model training device based on heterogeneous hardware, characterized in that: include: A determination module is configured to determine a parallel strategy corresponding to a plurality of heterogeneous hardware; An allocation module is configured to divide the model training task into training tasks corresponding to a plurality of heterogeneous hardwares according to the parallel strategy, and allocate the training tasks corresponding to the plurality of heterogeneous hardwares to corresponding heterogeneous hardwares, so that the plurality of heterogeneous hardwares respectively execute the allocated training tasks; An acquisition module is configured to acquire task execution information corresponding to the plurality of heterogeneous hardwares respectively, wherein the task execution information is used to characterize resource utilization of the heterogeneous hardwares when executing tasks; The scheduling module is configured to perform resource scheduling on the plurality of heterogeneous hardware respectively according to the task execution information respectively corresponding to the plurality of heterogeneous hardware.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the model training method based on heterogeneous hardware described in any one of claims 1 to 9 is implemented.

12. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, used to execute the computer program in the memory to implement the model training method based on heterogeneous hardware as described in any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model training method based on heterogeneous hardware described in any one of claims 1 to 9 is implemented.

Images