Model training method and device based on distributed computing, equipment and program product

Abstract

The invention discloses a model training method and device based on distributed computing, equipment and a program product. According to the scheme, Monte Carlo simulation is executed on a mixed precision layer in a neural network to evaluate the comprehensive performance of multiple candidate mixed precision configuration strategies, the mixed precision layer adopts at least two kinds of numerical precision to execute calculation in training, and the comprehensive performance of the mixed precision layer is evaluated through indexes such as the precision standard-reaching rate, resource consumption and training efficiency. And selecting a target mixing precision configuration strategy from the plurality of candidate mixing precision configuration strategies according to the comprehensive performance of each candidate mixing precision configuration strategy on the mixing precision neural network layer, inputting the target mixing precision configuration strategy into the hypergraph convolutional neural network for learning rate optimization, and determining a target learning rate matched with the target mixing precision configuration strategy. And finally, distributing the target strategy and the target learning rate to each distributed computing node so as to guide the configuration of the distributed computing nodes and cooperatively complete the training task of the neural network. The efficiency and accuracy of large-scale model training are effectively improved.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a model training method, apparatus, device, and program product based on distributed computing. Background Technology

[0002] As the scale of deep learning models continues to expand, memory and computational resource consumption during training has become a key bottleneck restricting model development and application. To alleviate this pressure, mixed-precision training techniques have emerged. By combining data types of different precisions (such as FP32 and FP16 / BF16), these techniques significantly reduce memory usage and improve computational efficiency while maintaining model accuracy, and have become the mainstream practice for large-scale model training. However, most existing mixed-precision configuration strategies rely on manual experience or static configuration, making it difficult to dynamically adapt to data characteristics, network structure, and hardware environment. This results in suboptimal training efficiency, especially in handling complex data relationships and achieving stable training at lower precision. Furthermore, in large-scale distributed training scenarios, the aforementioned static or manual strategies often cannot adapt to the heterogeneity and dynamic load changes between different computing nodes, easily causing imbalances in computation and communication between nodes, further limiting the overall scalability and resource utilization efficiency of the training system. Therefore, how to achieve dynamic optimization and collaborative configuration of mixed-precision configuration strategies in distributed training environments has become a key issue in improving the performance of large-scale model training. Summary of the Invention

[0003] This application proposes a model training method, apparatus, device, and program product based on distributed computing, aiming to solve key problems such as the static nature and poor adaptability of existing mixed-precision training strategies, as well as the inability to dynamically and collaboratively optimize in a distributed environment, resulting in low training efficiency and insufficient resource utilization. Accordingly, the technical solution of this application is as follows: Firstly, a model training method based on distributed computing is provided, including: For each of a plurality of candidate mixed-precision configuration strategies, a Monte Carlo simulation is performed on a mixed-precision neural network layer in the neural network to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer; wherein the mixed-precision neural network layer is configured to: during the training of the neural network, select and use at least two corresponding numerical computation precisions to perform computations according to the configured mixed-precision configuration strategy; the overall performance is evaluated based on multiple performance metrics including precision achievement rate, resource consumption, and training efficiency; Based on the comprehensive performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer, a target mixed-precision configuration strategy is selected from the plurality of candidate mixed-precision configuration strategies; The mixed-precision neural network layer adopting the target mixed-precision configuration strategy is imported into the hypergraph convolutional neural network for learning rate tuning to determine the target learning rate that is compatible with the target mixed-precision configuration. The target mixed precision configuration strategy and the target learning rate are distributed to distributed computing nodes, so that the distributed computing nodes configure and execute the training task of the neural network according to the target mixed precision configuration strategy and the target learning rate.

[0004] Secondly, a model training device based on distributed computing is provided, comprising: A simulation execution module is used to perform Monte Carlo simulations on a mixed-precision neural network layer in a neural network for each of a plurality of candidate mixed-precision configuration strategies, in order to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer; wherein the mixed-precision neural network layer is configured to: select and use at least two corresponding numerical computation precisions to perform computations according to the configured mixed-precision configuration strategy during the training of the neural network; the overall performance is evaluated based on multiple performance indicators including accuracy achievement rate, resource consumption, and training efficiency; A strategy selection module is used to select a target mixed-precision configuration strategy from the plurality of candidate mixed-precision configuration strategies based on the comprehensive performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer. The learning rate tuning module is used to import the mixed-precision neural network layer that adopts the target mixed-precision configuration strategy into the hypergraph convolutional neural network for learning rate tuning, and determine the target learning rate that is compatible with the target mixed-precision configuration. The training configuration module is used to distribute the target mixed precision configuration strategy and the target learning rate to distributed computing nodes, so that the distributed computing nodes can configure and execute the training task of the neural network according to the target mixed precision configuration strategy and the target learning rate.

[0005] Thirdly, embodiments of this application provide an electronic device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method described in the first aspect.

[0006] Fourthly, a computer program product is provided, the computer program product including a computer-readable storage medium storing a computer program operable to cause a computer to perform the method described in the first aspect.

[0007] In this embodiment, firstly, for each of the multiple candidate mixed-precision configuration strategies, a Monte Carlo simulation is performed on the mixed-precision neural network layer in the neural network to estimate its comprehensive performance based on multiple indicators such as accuracy achievement rate, resource consumption, and training efficiency. Secondly, based on the comprehensive performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer, a target mixed-precision configuration strategy is selected from the multiple candidate mixed-precision configuration strategies. Next, the mixed-precision neural network layer adopting the target mixed-precision configuration strategy is imported into a hypergraph convolutional neural network for learning rate tuning to determine a target learning rate that is compatible with the target mixed-precision configuration. Finally, the target mixed-precision configuration strategy and the target learning rate are distributed to distributed computing nodes so that the distributed computing nodes configure and execute the training task of the neural network according to the target mixed-precision configuration strategy and the target learning rate. This solution automates and quantitatively evaluates and optimizes multiple candidate mixed-precision configuration strategies using Monte Carlo simulation, effectively avoiding the inefficiency and subjective biases of traditional manual parameter tuning. Building upon this, a hypergraph convolutional neural network is introduced to model the complex dependencies between neural network layers, enabling fine-grained adaptation of the target learning rate and significantly improving the model's convergence stability and training speed. Subsequently, the optimized target mixed-precision configuration strategy and its corresponding learning rate are collaboratively distributed to distributed computing nodes. While ensuring consistency in training configuration across nodes, this fully unleashes the parallel computing potential of the distributed system, ultimately achieving a synergistic optimization effect of significantly shortening the training cycle and comprehensively improving system resource utilization. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a flowchart illustrating the model training method based on distributed computing according to an embodiment of this application.

[0010] Figure 2 This is a schematic diagram of a hypergraph convolutional neural network in the distributed computing-based model training method of this application embodiment.

[0011] Figure 3 This is a schematic diagram of the structure of a distributed computing-based model training device according to an embodiment of this application.

[0012] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0014] As the scale of deep learning models continues to expand, memory and computational overhead during training have become core bottlenecks restricting model development and deployment. Mixed-precision training, by fusing different precision data types such as FP32 and FP16 / BF16, significantly reduces memory usage and improves computational efficiency while ensuring model accuracy, and has become the mainstream method for large-scale training. However, most current mixed-precision configuration strategies rely on manual experience or static configuration, making it difficult to dynamically adjust according to data characteristics, network structure, and hardware environment. This not only results in insufficient stability in complex models and low-precision scenarios, but also leads to unbalanced computation and communication due to node heterogeneity and load fluctuations in distributed training, severely limiting system scalability and resource utilization efficiency. To address this, this application proposes a model training scheme based on distributed computing, aiming to solve the problems of low training efficiency and resource waste caused by the static nature and poor adaptability of existing mixed-precision configuration strategies and their lack of collaborative optimization in distributed scenarios. The technical solutions provided by various embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0015] One embodiment of this application provides a model training method based on distributed computing. Figure 1 This is a flowchart illustrating the model training method, including: S101, for each of the multiple candidate mixed-precision configuration strategies, a Monte Carlo simulation is performed on the mixed-precision neural network layer in the neural network to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer; wherein, the mixed-precision neural network layer is configured to: select and use at least two corresponding numerical computation precisions to perform computation according to the configured mixed-precision configuration strategy during the training of the neural network; the overall performance is evaluated based on multiple performance indicators including accuracy achievement rate, resource consumption and training efficiency.

[0016] Neural networks are mathematical models that simulate biological neural networks and are widely used in modeling and predicting various complex tasks. In this embodiment, a neural network training task needs to be performed to serve application scenarios such as real-time perception and decision-making in autonomous driving systems, recommendation model iteration in online recommendation systems, or product defect detection in industrial quality inspection systems.

[0017] A mixed-precision neural network layer refers to a specific network layer that, during the training process of a neural network, can select and use at least two corresponding numerical computation precisions (such as FP32, FP16, BF16, or FP8) to perform computations according to a configured mixed-precision configuration strategy.

[0018] This embodiment performs Monte Carlo simulations on a mixed-precision neural network layer, aiming to systematically evaluate the potential performance of different candidate mixed-precision configuration strategies at that layer using probabilistic statistical methods. Its significance lies in replacing traditional strategy selection methods that rely on human experience or static configurations, providing objective, data-driven decision-making basis for subsequent strategy optimization through quantitative simulation.

[0019] Specifically, the Monte Carlo simulation is implemented as follows: First, for each candidate mixed-precision configuration strategy, this embodiment constructs a performance calculation model for its interaction with the mixed-precision neural network layer. This performance calculation model can simulate and output performance metrics related to accuracy achievement rate, resource consumption, and training efficiency based on input random variables (such as data distribution, initial weight perturbation, etc.). Then, for each candidate mixed-precision configuration strategy, multiple (e.g., 100) Monte Carlo simulation experiments are performed. In each simulation experiment, random input data conforming to a preset probability distribution is generated for the random variables in the performance calculation model, and the performance calculation model is run to obtain a corresponding set of performance metric results. Finally, based on a large number of performance metric samples obtained from multiple simulation experiments for each candidate mixed-precision configuration strategy, statistical analysis (such as calculating the mean and variance) is performed to evaluate the overall performance of the strategy on the mixed-precision neural network layer.

[0020] As an example, the formula for evaluating overall performance is: ; in, For accuracy compliance rate; This is the resource consumption growth rate, used to characterize resource consumption; Training time is used to characterize training efficiency; , The baseline training time; , and All of these are preset weighting coefficients.

[0021] Furthermore, in this embodiment, the hybrid precision neural network layer should also possess the ability to interpolate the input data. This interpolation process specifically includes: identifying target data segments with uneven sampling intervals in the input data; determining the corresponding interpolation interval N (N is an integer, typically between 1 and 3 depending on the degree of numerical difference within the data segment) based on the data fluctuation or data change gradient of the target data segment; performing interpolation calculations in units of every N adjacent data points in the target data segment, and filling the corresponding interpolation positions with the interpolation calculation results. This process can be specifically implemented based on the following interpolation formula:

[0022] in, The coordinates of the position to be interpolated; , , for The coordinates of three adjacent known points; , and They are respectively , , The corresponding value; This is the calculated interpolation result.

[0023] It should be understood that the introduction of this type of structured interpolation aims to address the numerical jumps and information discontinuities caused by uneven sampling or local missing data in the training data. By adaptively selecting the number of interpolation points (N=1~3) based on the degree of data fluctuation and applying the corresponding interpolation algorithm, transitional values ​​can be smoothly generated while preserving the original data distribution characteristics as much as possible, thereby enhancing the regularity and continuity of the input data. This provides a more stable and consistent numerical environment for subsequent mixed-precision calculations, helping to reduce rounding errors and gradient anomalies caused by sudden input changes in low-precision calculations, and improving the numerical stability of the training process and the generalization performance of the final model.

[0024] S102, select the target mixed-precision configuration strategy from multiple candidate mixed-precision configuration strategies based on the comprehensive performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer.

[0025] The reason this embodiment selects candidate strategies based on overall performance is that different mixed-precision configuration strategies have significant trade-offs in accuracy, resources, and efficiency. Relying on manual experience or fixed rules for selection is not only time-consuming and laborious, but also makes it difficult to make globally optimal decisions under dynamically changing training environments and diverse hardware conditions. Through the quantitative evaluation based on Monte Carlo simulation in step S101, each candidate strategy has obtained an objective and comparable overall performance score. The significance of selecting strategies based on this is that it transforms the strategy decision-making process from subjective experience-driven to data-driven, thereby systematically and efficiently identifying the configuration scheme with the best overall performance in a specific task scenario and computing environment, providing an optimized and validated starting point for subsequent training phases.

[0026] In terms of the specific implementation method, this embodiment typically selects the candidate mixed-precision configuration strategy with the highest overall performance score as the target mixed-precision configuration strategy. For example, if the scores of candidate strategies A, B, and C are 0.72, 0.68, and 0.70 respectively, then strategy A is selected as the target strategy because of its highest score. This selection mechanism ensures that the adopted strategy achieves a relatively optimal balance in terms of accuracy achievement rate, resource consumption control, and training efficiency improvement. After determining the target mixed-precision configuration strategy, this strategy will serve as the core configuration basis for learning rate tuning in subsequent steps and the final distributed training execution, thereby connecting the complete optimization chain from strategy evaluation and selection to deployment and application, laying a key foundation for improving overall training efficiency.

[0027] S103, the mixed-precision neural network layer with the target mixed-precision configuration strategy is imported into the hypergraph convolutional neural network for learning rate tuning, and the target learning rate that is adapted to the target mixed-precision configuration is determined.

[0028] This embodiment further optimizes the learning rate after selecting the target mixed-precision configuration strategy because changes in numerical precision during mixed-precision training directly affect the magnitude and stability of gradient updates. A fixed learning rate is difficult to adapt to the dynamic gradient distribution changes brought about by different combinations of precision layers, which may lead to slow convergence, oscillations, or even divergence. Therefore, this embodiment imports the mixed-precision neural network layer with the configured target strategy into a hypergraph convolutional neural network, utilizing its powerful high-order correlation modeling capabilities to represent and analyze the complex interactions between layers and the training dynamics, thereby finding the learning rate that best matches the specific precision configuration. The significance of this step is to achieve synergistic optimization of the mixed-precision configuration strategy and the training hyperparameters (learning rate), ensuring that while improving computational efficiency, stable convergence and the final accuracy of the model are maintained during the training process, thus solving the typical problem of difficulty in balancing efficiency and stability in mixed-precision training.

[0029] The specific implementation method for learning rate optimization is as follows: First, within the hypergraph convolutional neural network framework, this embodiment randomly selects a certain size of data samples (e.g., extracting 2000 training data points) and divides them into a learning group and a validation group. Training experiments are performed on the learning group, and model performance is evaluated on the validation group, thus testing the performance differences under different learning rate values. In this process, the hypergraph convolutional neural network, with its hyperedge structure, can effectively model the complex relationships between different precision regions in a mixed-precision neural network layer, thereby more accurately capturing the overall impact of learning rate changes on training dynamics.

[0030] During the experiment, this embodiment selected a group of representative candidate learning rates for comparative testing based on the characteristics of the target mixed accuracy configuration strategy. For example, for scenarios that require balancing training speed and numerical stability, multiple learning rate values, including 0.05, 0.005, and 0.001, can be tested, and the accuracy achieved on the validation set and the training time consumed are recorded respectively, as shown in the table below:

[0031] Based on the table above, it can be seen that when the learning rate is 0.005, the model achieves the highest accuracy (99.87%) on the validation set, while the training time (127 seconds) is relatively short. Therefore, this is determined as the target learning rate in this experiment. Finally, through the systematic experimental evaluation based on the hypergraph convolutional neural network described above, this embodiment can match a fully validated optimal learning rate to the target mixed-precision configuration strategy, thereby optimizing training efficiency while ensuring model performance.

[0032] in, Figure 2 This paper illustrates the hypergraph convolutional neural network structure used in an embodiment of this application. This structure comprises two parallel processing paths: the left path takes the user embedding matrix as input, and through layer-by-layer feature transformations from layer 1 to layer L-1 of the hypergraph convolutional layers, sequentially generates a high-order embedding representation P. 1 P 2 ... P L-1 And finally output P L The right-hand path takes the region embedding matrix as input and processes it through the same multi-layer hypergraph convolutional layers to obtain the corresponding region embedding representation Q. 1 Q 2 ... Q L -1 Q LThe two representations are then input into a graph convolutional model to further extract local structured features. Next, a user / region representation fusion module deeply fuses the global high-order correlation features extracted by the hypergraph convolutional path with the local features extracted by the graph convolutional path. The fused representation is finally fed into a region prediction module, outputting the prediction result for the training precision region. The core principle of this structure lies in utilizing the ability of a hypergraph (rather than a regular graph) to flexibly represent multi-level relationships, thereby more effectively capturing the complex many-to-many high-order dependencies between users and regions, and between regions, laying the foundation for learning accurate representations. In the learning rate tuning task, this embodiment uses this structure as a modeling tool: the features of the neural network layer adopting the target mixed precision configuration strategy are used as input. This hypergraph network encodes and infers the complex high-order relationships between "neural network layer - precision configuration - data features," and then dynamically derives the learning rate most suitable for the current specific mixed precision configuration, thereby achieving an optimized balance between accuracy and convergence speed during training.

[0033] S104, the target mixed precision configuration strategy and target learning rate are distributed to the distributed computing nodes so that the distributed computing nodes can configure and execute the neural network training task according to the target mixed precision configuration strategy and target learning rate.

[0034] In this embodiment, after completing the aforementioned strategy optimization and learning rate tuning, the target mixed-precision configuration strategy and the target learning rate are distributed to each distributed computing node participating in the training. This step is crucial for achieving large-scale, efficient training. Its significance lies in ensuring that all distributed computing nodes can execute computational tasks based on a unified, optimized configuration benchmark (i.e., the target mixed-precision configuration strategy and the target learning rate) during collaborative training. This effectively avoids problems such as training discrepancies, gradient synchronization difficulties, or unstable convergence caused by inconsistent node configurations, thereby guaranteeing the overall coordination of distributed training and the consistency of the final model's performance. In this way, this embodiment can systematically extend the performance gains obtained through centralized optimization to the entire distributed computing cluster, maximizing the overall training efficiency and resource utilization of the cluster to complete the neural network training task.

[0035] It's important to note that the training samples for neural networks vary depending on the specific task: In real-time perception and decision-making tasks of autonomous driving systems, training samples primarily consist of multimodal time-series data (such as image sequences, point cloud frames, and radar signals) collected by sensors like cameras, LiDAR, and millimeter-wave radar, labeled with the location, category, and motion status information of target objects such as vehicles, pedestrians, and traffic signs; in the iterative recommendation model scenarios of online recommendation systems, training samples are typically user behavior log data, covering user clicks, browsing, purchases, ratings, and other interaction records, as well as user attributes, item characteristics, and contextual information; in product defect detection tasks of industrial quality inspection systems, training samples are mostly image or video data of the product's surface or internal structure, collected by optical cameras, X-rays, or ultrasonic equipment, labeled with defect types (such as cracks, scratches, and deformations) and their location, dimensions, and other quality information. These samples, after preprocessing and labeling, form the basis for model training, supporting the high-precision and high-efficiency intelligent perception and decision-making needs of the corresponding scenarios.

[0036] In practical implementation, to further improve the efficiency of data exchange during distributed training, this embodiment can dynamically allocate network forwarding priorities to the data packets to be transmitted generated by each distributed computing node during the execution of training tasks based on the computing power efficiency index of each node. The computing power efficiency index characterizes the computing power per unit power consumption of the corresponding distributed computing node; a higher value indicates higher computing energy efficiency. Based on this index, data packets from high-efficiency nodes can be assigned higher network forwarding priorities, thus prioritizing the transmission of gradient or model update information that contributes more to the overall training progress. In the specific network implementation, the transmission of the data packets to be transmitted is completed based on the IPv6 protocol. The network forwarding priority is identified by the Priority Code Point (PCP) field in the 802.1Q tag encapsulated in the IPv6 packet. To implement the above scheduling logic, this embodiment maps the computing power efficiency index to the corresponding value range of the PCP field (e.g., 8–15) and establishes a reverse mapping relationship: the higher the computing power efficiency, the smaller the corresponding PCP value, and the higher the network forwarding priority. For example, the data packets of the node with the highest computational efficiency can be marked as PCP=8, while those with lower computational efficiency can be marked with a larger value (such as 15). This enables differentiated forwarding and scheduling of data packets at the IP network layer, ensuring that data from high-value computing nodes can be transmitted first, and ultimately improving the overall efficiency of the distributed neural network training task.

[0037] In summary, the method of this embodiment first performs Monte Carlo simulations on the mixed-precision neural network layer in the neural network for each of the multiple candidate mixed-precision configuration strategies to estimate its comprehensive performance based on multiple indicators such as accuracy achievement rate, resource consumption, and training efficiency. Second, based on the comprehensive performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer, a target mixed-precision configuration strategy is selected from the multiple candidate mixed-precision configuration strategies. Next, the mixed-precision neural network layer adopting the target mixed-precision configuration strategy is imported into a hypergraph convolutional neural network for learning rate tuning to determine a target learning rate suitable for the target mixed-precision configuration. Finally, the target mixed-precision configuration strategy and the target learning rate are distributed to distributed computing nodes, so that the distributed computing nodes configure and execute the training task of the neural network according to the target mixed-precision configuration strategy and the target learning rate. This solution automates and quantitatively evaluates and optimizes multiple candidate mixed-precision configuration strategies using Monte Carlo simulation, effectively avoiding the inefficiency and subjective biases of traditional manual parameter tuning. Building upon this, a hypergraph convolutional neural network is introduced to model the complex dependencies between neural network layers, enabling fine-grained adaptation of the target learning rate and significantly improving the model's convergence stability and training speed. Subsequently, the optimized target mixed-precision configuration strategy and its corresponding learning rate are collaboratively distributed to distributed computing nodes. While ensuring consistency in training configuration across nodes, this fully unleashes the parallel computing potential of the distributed system, ultimately achieving a synergistic optimization effect of significantly shortening the training cycle and comprehensively improving system resource utilization.

[0038] In addition, corresponding to Figure 1 In addition to the method shown, another embodiment of this example also provides a model training device based on distributed computing. Figure 3 This is a schematic diagram of the memory management device 300, including: The simulation execution module 310 is configured to perform Monte Carlo simulations on a mixed-precision neural network layer in a neural network for each of a plurality of candidate mixed-precision configuration strategies, in order to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer; wherein the mixed-precision neural network layer is configured to: select and use at least two corresponding numerical computation precisions to perform computations according to the configured mixed-precision configuration strategy during the training of the neural network; the overall performance is evaluated based on multiple performance indicators including accuracy achievement rate, resource consumption, and training efficiency.

[0039] The strategy selection module 320 is used to select a target mixed precision configuration strategy from the plurality of candidate mixed precision configuration strategies based on the comprehensive performance of each candidate mixed precision configuration strategy on the mixed precision neural network layer.

[0040] The learning rate tuning module 330 is used to import the mixed-precision neural network layer that adopts the target mixed-precision configuration strategy into the hypergraph convolutional neural network for learning rate tuning, and determine the target learning rate that is compatible with the target mixed-precision configuration.

[0041] The training configuration module 340 is used to distribute the target mixed precision configuration strategy and the target learning rate to distributed computing nodes, so that the distributed computing nodes configure and execute the training task of the neural network according to the target mixed precision configuration strategy and the target learning rate.

[0042] Optionally, the simulation execution module 310 performs Monte Carlo simulations on the mixed-precision neural network layer of the neural network for each of the multiple candidate mixed-precision configuration strategies to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer. This includes: constructing a performance calculation model for each candidate mixed-precision configuration strategy on the mixed-precision neural network layer, wherein the performance calculation model is used to simulate and output performance indicators related to accuracy achievement rate, resource consumption, and training efficiency based on input random variables; performing multiple Monte Carlo simulation experiments for each candidate mixed-precision configuration strategy; wherein, in each Monte Carlo simulation experiment, corresponding random input data is generated for the random variables in the performance calculation model, and the performance calculation model is run to obtain the corresponding performance indicators; and performing statistical analysis based on the performance indicators obtained by each candidate mixed-precision configuration strategy in the multiple simulation experiments to evaluate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer.

[0043] Optionally, the formula for evaluating the overall performance is: ;in, For accuracy compliance rate; This is the resource consumption growth rate, used to characterize resource consumption; Training time is used to characterize training efficiency; , The baseline training time; , and All of these are preset weighting coefficients.

[0044] Optionally, the hybrid precision neural network layer is configured to: perform interpolation processing on the input data; wherein the interpolation processing includes: identifying target data segments in the input data with uneven sampling intervals; determining the corresponding interpolation interval N, where N is an integer, based on the data fluctuation degree and / or data change gradient of the target data segment; performing interpolation calculations in units of every N adjacent data points in the target data segment, and filling the corresponding interpolation positions with the interpolation calculation results.

[0045] Optionally, there are multiple distributed computing nodes; after distributing the target mixed precision configuration strategy and the target learning rate to the distributed computing nodes, the training configuration module 340 is further configured to: allocate network forwarding priorities to the data packets to be transmitted generated by the distributed computing nodes during the execution of the training task based on the computing power efficiency index of each distributed computing node; wherein, the computing power efficiency index is used to characterize the computing power of the corresponding distributed computing node per unit power consumption; and schedule and transmit the data packets to be transmitted of each distributed computing node according to the allocated network forwarding priority to complete the training task of the neural network.

[0046] Optionally, the transmission of the data packet to be transmitted is implemented based on the IPv6 protocol; the network forwarding priority is identified by the priority field of the 802.1Q tag encapsulated in the IPv6 packet.

[0047] Optionally, the training task of the neural network is used for any of the following application scenarios: Real-time perception and decision-making in autonomous driving systems; Recommendation model iteration in online recommendation systems; Product defect detection in industrial quality inspection systems.

[0048] It should be noted that the model training device in this embodiment can be used as... Figure 1 The execution body of the method shown is therefore able to achieve... Figure 1 The steps and functions of the method shown are illustrated.

[0049] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to it. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0050] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0051] Memory is used to store computer programs. Specifically, the computer program may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides the computer program to the processor.

[0052] Specifically, the processor reads the corresponding computer program from non-volatile memory into memory and then runs it, forming the above-mentioned logical structure. Figure 3 The illustrated model training device is based on distributed computing. Correspondingly, the processor executes the program stored in memory and specifically performs the following operations: For each of a plurality of candidate mixed-precision configuration strategies, a Monte Carlo simulation is performed on a mixed-precision neural network layer in the neural network to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer; wherein the mixed-precision neural network layer is configured to: select and use at least two corresponding numerical computation precisions to perform computation according to the configured mixed-precision configuration strategy during the training of the neural network; the overall performance is evaluated based on multiple performance indicators including accuracy achievement rate, resource consumption, and training efficiency.

[0053] Based on the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer, a target mixed-precision configuration strategy is selected from the plurality of candidate mixed-precision configuration strategies.

[0054] The mixed-precision neural network layer adopting the target mixed-precision configuration strategy is imported into the hypergraph convolutional neural network for learning rate tuning to determine the target learning rate that is compatible with the target mixed-precision configuration.

[0055] The target mixed precision configuration strategy and the target learning rate are distributed to distributed computing nodes, so that the distributed computing nodes configure and execute the training task of the neural network according to the target mixed precision configuration strategy and the target learning rate.

[0056] The above is as described in this instruction manual. Figure 1 The distributed computing-based model training method disclosed in the illustrated embodiments can be applied to a processor and implemented by the processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0057] Of course, in addition to software implementation, the electronic device described in this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0058] Furthermore, embodiments of this application also propose a computer program product, including a computer-readable storage medium storing one or more computer programs, the one or more computer programs including instructions.

[0059] When the aforementioned instructions are executed by a portable electronic device that includes multiple applications, they enable the portable electronic device to perform... Figure 1 The steps in the method shown include: For each of a plurality of candidate mixed-precision configuration strategies, a Monte Carlo simulation is performed on a mixed-precision neural network layer in the neural network to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer; wherein the mixed-precision neural network layer is configured to: select and use at least two corresponding numerical computation precisions to perform computation according to the configured mixed-precision configuration strategy during the training of the neural network; the overall performance is evaluated based on multiple performance indicators including accuracy achievement rate, resource consumption, and training efficiency.

[0060] Based on the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer, a target mixed-precision configuration strategy is selected from the plurality of candidate mixed-precision configuration strategies.

[0061] The mixed-precision neural network layer adopting the target mixed-precision configuration strategy is imported into the hypergraph convolutional neural network for learning rate tuning to determine the target learning rate that is compatible with the target mixed-precision configuration.

[0062] The target mixed precision configuration strategy and the target learning rate are distributed to distributed computing nodes, so that the distributed computing nodes configure and execute the training task of the neural network according to the target mixed precision configuration strategy and the target learning rate.

[0063] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0065] The above are merely embodiments of this specification and are not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification. Furthermore, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this document.

Claims

1. A model training method based on distributed computing, characterized in that, include: For each of a plurality of candidate mixed-precision configuration strategies, a Monte Carlo simulation is performed on a mixed-precision neural network layer in the neural network to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer; wherein the mixed-precision neural network layer is configured to: during the training of the neural network, select and use at least two corresponding numerical computation precisions to perform computations according to the configured mixed-precision configuration strategy; the overall performance is evaluated based on multiple performance metrics including precision achievement rate, resource consumption, and training efficiency; Based on the comprehensive performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer, a target mixed-precision configuration strategy is selected from the plurality of candidate mixed-precision configuration strategies; The mixed-precision neural network layer adopting the target mixed-precision configuration strategy is imported into the hypergraph convolutional neural network for learning rate tuning to determine the target learning rate that is compatible with the target mixed-precision configuration. The target mixed precision configuration strategy and the target learning rate are distributed to distributed computing nodes, so that the distributed computing nodes configure and execute the training task of the neural network according to the target mixed precision configuration strategy and the target learning rate.

2. The method according to claim 1, characterized in that... For each of a plurality of candidate mixed-precision configuration strategies, Monte Carlo simulations are performed on a mixed-precision neural network layer in the neural network to estimate the combined performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer, including: For each candidate mixed precision configuration strategy, a performance calculation model is constructed on the mixed precision neural network layer. The performance calculation model is used to simulate and output performance indicators related to accuracy achievement rate, resource consumption and training efficiency based on the input random variables. For each candidate mixed precision configuration strategy, multiple Monte Carlo simulation experiments are performed; wherein, in each Monte Carlo simulation experiment, corresponding random input data is generated for the random variables in the performance calculation model, and the performance calculation model is run to obtain the corresponding performance index. Statistical analysis was performed on the performance metrics obtained from the multiple simulation experiments for each candidate mixed-precision configuration strategy to evaluate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer.

3. The method according to claim 1, characterized in that, The formula for evaluating the overall performance is as follows: ； in, For accuracy compliance rate; This is the resource consumption growth rate, used to characterize resource consumption; Training time is used to characterize training efficiency; , The baseline training time; , and All of these are preset weighting coefficients.

4. The method according to claim 1, characterized in that, The mixed-precision neural network layer is configured to: interpolate the input data; wherein the interpolation includes: Identify target data segments with uneven sampling intervals in the input data; Based on the degree of data fluctuation and / or the gradient of data change in the target data segment, the corresponding interpolation interval N is determined, where N is an integer; Interpolation calculations are performed on a unit of N adjacent data points in the target data segment, and the interpolation calculation results are used to fill the corresponding interpolation positions.

5. The method according to claim 1, characterized in that, The distributed computing nodes are multiple; after distributing the target mixed precision configuration strategy and the target learning rate to the distributed computing nodes, the method further includes: Based on the computing power efficiency index of each distributed computing node, network forwarding priority is assigned to the data packets to be transmitted generated by the distributed computing node during the execution of training tasks; wherein, the computing power efficiency index is used to characterize the computing power of the corresponding distributed computing node per unit power consumption. Based on the assigned network forwarding priority, the data packets to be transmitted from each distributed computing node are scheduled and transmitted to complete the training task of the neural network.

6. The method according to claim 5, characterized in that, The transmission of the data packets to be transmitted is based on the IPv6 protocol; the network forwarding priority is identified by the priority field of the 802.1Q tag encapsulated in the IPv6 packet.

7. The method according to any one of claims 1 to 6, characterized in that, The training task of the neural network is used for any of the following application scenarios: Real-time perception and decision-making in autonomous driving systems; Recommendation model iteration in online recommendation systems; Product defect detection in industrial quality inspection systems.

8. A model training device based on distributed computing, characterized in that, include: A simulation execution module is used to perform Monte Carlo simulations on a mixed-precision neural network layer in a neural network for each of a plurality of candidate mixed-precision configuration strategies, in order to estimate the overall performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer; wherein the mixed-precision neural network layer is configured to: select and use at least two corresponding numerical computation precisions to perform computations according to the configured mixed-precision configuration strategy during the training of the neural network; the overall performance is evaluated based on multiple performance indicators including accuracy achievement rate, resource consumption, and training efficiency; A strategy selection module is used to select a target mixed-precision configuration strategy from the plurality of candidate mixed-precision configuration strategies based on the comprehensive performance of each candidate mixed-precision configuration strategy on the mixed-precision neural network layer. The learning rate tuning module is used to import the mixed-precision neural network layer that adopts the target mixed-precision configuration strategy into the hypergraph convolutional neural network for learning rate tuning, and determine the target learning rate that is compatible with the target mixed-precision configuration. The training configuration module is used to distribute the target mixed precision configuration strategy and the target learning rate to distributed computing nodes, so that the distributed computing nodes can configure and execute the training task of the neural network according to the target mixed precision configuration strategy and the target learning rate.

9. An electronic device, comprising: processor; And a memory arranged to store computer-executable instructions, characterized in that, when executed, the executable instructions cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer program product, the computer program product comprising a computer-readable storage medium storing a computer program, characterized in that, The computer program is operable to cause the computer to perform the method as described in any one of claims 1 to 7.

Images