Model training method and device, equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a model training method and device, equipment and a storage medium, wherein the method comprises: a loss function scaler creation interface creating a loss function scaler based on a scaler construction parameter; in a forward propagation process of each round of mixed precision training of a deep learning model, a loss function scaler scaling interface amplifying a loss value output by the deep learning model in the current round of mixed precision training based on the loss function scaler to obtain an amplified loss value; in a backward propagation process of each round of mixed precision training, a loss function optimizer iteration interface updating network parameters in the deep learning model based on the amplified loss value using a set optimizer and the loss function scaler to obtain a trained deep learning model.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, and storage medium. Background Technology

[0002] During the training of deep learning models, the data type of network parameters is typically uniform, generally 32-bit floating-point (fp32), i.e., full-precision floating-point. To reduce memory consumption or improve training speed, some network parameters can be converted to 16-bit floating-point (fp16), i.e., half-precision floating-point. However, if all parameters in the model are fp16, insufficient precision and data overflow can easily occur, affecting the normal progress or effectiveness of model training. Related technologies employ mixed-precision training techniques to train deep learning models. This involves converting some network parameters to fp16, allowing some computations to be performed with fp32 precision and others with fp16 precision during training. This improves model accuracy while reducing storage space usage and execution time during training.

[0003] However, in related technologies, training deep learning models with mixed precision usually requires users to manually develop code in the algorithm, or to use third-party tools or training frameworks, resulting in diverse implementation methods that are difficult to unify and reuse. Summary of the Invention

[0004] In view of the above, the present disclosure provides at least one model training method, apparatus, device, and storage medium.

[0005] The technical solution of this disclosure embodiment is implemented as follows:

[0006] This disclosure provides a model training method, the method comprising:

[0007] The loss function scaler creation interface creates a loss function scaler based on the scaler construction parameters;

[0008] During the forward propagation of each round of mixed-precision training of the deep learning model, the loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed-precision training based on the loss function scaler, to obtain the amplified loss value.

[0009] During the backpropagation process of each round of mixed-precision training, the loss function optimizer iterative interface uses the set optimizer and the loss function scaler to update the network parameters in the deep learning model based on the amplified loss value, thereby obtaining the trained deep learning model.

[0010] This disclosure provides a model training apparatus, the apparatus comprising:

[0011] The loss function scaler creation interface is used to create a loss function scaler based on the scaler construction parameters;

[0012] The loss function scaler scaling interface is used to amplify the loss value output by the deep learning model in the current round of mixed precision training during the forward propagation process of each round of mixed precision training, based on the loss function scaler, to obtain the amplified loss value.

[0013] The loss function optimizer iteration interface is used to update the network parameters in the deep learning model based on the amplified loss value during the backpropagation process of each round of mixed-precision training, using a set optimizer and the loss function scaler, to obtain the trained deep learning model.

[0014] This disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0015] This disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above-described method.

[0016] This disclosure provides a computer program including computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0017] This disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method.

[0018] In this embodiment, a loss function scaler is created based on scaler construction parameters through a loss function scaler creation interface. During the forward propagation of each round of mixed-precision training on the deep learning model, the loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed-precision training based on the loss function scaler, obtaining an amplified loss value. During the backpropagation of each round of mixed-precision training, the loss function optimizer iteration interface uses the set optimizer and loss function scaler to update the network parameters in the deep learning model based on the amplified loss value, obtaining the trained deep learning model. Thus, on the one hand, by utilizing the loss function scaler creation interface, loss function scaler scaling interface, and loss function optimizer iteration interface, mixed-precision training of the deep learning model can be conveniently and quickly implemented to improve the accuracy of the trained deep learning model while reducing storage space usage and execution time during model training. On the other hand, the implementation process of mixed-precision training can be simplified, unified, and standardized through interface reuse, thereby improving the generalizability of mixed-precision training applications. Furthermore, by using a unified loss function scaler, the loss value output by the deep learning model in the current round of mixed-precision training is amplified during the forward propagation process of each round of mixed-precision training. In the backpropagation process of each round of mixed-precision training, the network parameters in the deep learning model are updated based on the amplified loss value. This can further simplify the implementation process of mixed-precision training and improve the model training efficiency.

[0019] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0021] Figure 1 A schematic diagram of the architecture of a computer vision algorithm interface model provided in an embodiment of this disclosure;

[0022] Figure 2 A schematic diagram illustrating the implementation process of a model training method provided in this embodiment of the disclosure;

[0023] Figure 3 A schematic diagram illustrating the implementation process of a model training method provided in this embodiment of the disclosure;

[0024] Figure 4 A schematic diagram illustrating the implementation process of a model training method provided in this embodiment of the disclosure;

[0025] Figure 5 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0031] To better understand the embodiments of this disclosure, the computer vision interface model is briefly described below:

[0032] 1) Algorithm Interface Goals

[0033] By defining the interface adaptation layer, the following goals can be achieved: a) the algorithm implementation can use different deep learning frameworks as backends and can be switched; b) it can run on different hardware, such as servers and distributed clusters; c) the algorithm can read different data through a unified data interface.

[0034] 2) Algorithm Interface Model

[0035] The relationship between data, algorithms, and models is described from three levels: system resources, interface adaptation, and algorithm application. For example... Figure 1The computer vision algorithm interface model shown in this standard mainly specifies the technical requirements for the interface adaptation layer: a) System resource layer: including hardware and software, providing the necessary storage, computation, and inference functions for the computer vision system. The specific implementation of the system resource layer varies among vendors, and this standard does not further describe or specify it; b) Interface adaptation layer: providing service interfaces such as data interfaces, optimization interfaces, distributed interfaces, and model interfaces to the upper-layer system processes, ensuring efficient and flexible model training and model migration between different frameworks; c) Algorithm application layer: completing the model training and model inference processes, requiring the use of various interfaces defined in this standard during model training.

[0036] The interface requirements for computer vision systems (mandatory and optional requirements, and appropriate usage) are as follows:

[0037] 1. Data and Model Structure

[0038] 1) Data Structure

[0039] Datasets can be in formats such as images, videos, and binary data. The dataset annotation file is in JSON format and contains annotation information for all samples in the dataset. If the annotation results include other auxiliary files, such as mask layer information, the relative paths of these auxiliary files are stored in the JSON file. Common data types are represented as follows:

[0040] Supports category labels, such as integers, where 0 represents background and positive integers represent foreground; supports bounding boxes, such as using the coordinates of the top-left and bottom-right vertices in the order (top-left x-coordinate, top-left y-coordinate, bottom-right x-coordinate, bottom-right y-coordinate); supports annotation files being parsed into lists or arrays after passing through the data reading interface, where each element is a dictionary or key-value pair container containing all relevant information for a sample, accessible by the dataset using an index; supports loading specific dataset formats and custom datasets; supports annotation files in mainstream open-source dataset formats, such as COCO and PASCAL VOC.

[0041] 2) Model Structure

[0042] a) Basic operators should be supported, including but not limited to "+", "-", "*", " / ", convolution operations, etc.; b) The meaning and values ​​of the parameters of the operator should be defined, and the computational logic for obtaining the output from the input during forward operation should be defined; c) Basic operators should support the construction of computational graphs through operator chaining and function nesting; d) The computational graph should support differentiation through backpropagation using the chain rule; e) The computational graph should support construction through conditional judgments or loops; f) User-defined operators should be compiled and allowed to be added to the computational graph; g) Serialization and deserialization of model parameters should be supported.

[0043] 2. Training Interface

[0044] 1) Optimizer Interface

[0045] The optimizer interface should support network model optimization, updating model parameters according to different optimization algorithms such as SGD, Adam, and Momomtum. Specifically, it should implement functions such as gradient calculation and backpropagation, parameter updates, optimization algorithms, and learning rate updates.

[0046] The optimizer interface should implement: a) support implementation as a class whose constructor parameters are the network model or a list of model parameters, as well as other required parameters, such as the learning rate; b) support implementation of the `step()` function for performing a single parameter optimization update. After this function is called, the model parameters should be updated based on the accumulated gradients.

[0047] 2) Mixed Precision Training Interface

[0048] The mixed-precision training interface provides unified support for mixed-precision training of algorithms, which can reduce memory consumption and improve training speed when the graphics card supports it. This interface should implement the following functions:

[0049] a) Supports precision conversion, converting model parameters to fp16, except for special layers (such as BN layers), while retaining a copy of the fp32 parameters; b) Supports input forward propagation, converting input data to fp16 for forward propagation and loss calculation; c) Supports loss amplification, amplifying the calculated loss with both fixed and dynamic amplification modes; d) Supports gradient calculation, calculating and backpropagating gradients in fp16 mode, then converting them to fp32 and scaling them back to the actual scale proportionally according to the amplification factor in c); e) Supports parameter update, updating parameters in the fp32 parameter copy based on the gradient calculated in d), and then assigning the updated parameters to the fp16 model.

[0050] 3) Distributed Interface

[0051] Through a distributed interface, the framework can complete the data transfer between multiple processes in a distributed training scenario with multiple machines and multiple GPUs. This interface should have the following core functionalities:

[0052] The distributed interface should cover functions such as `bcast()`, `reduce()`, `scatter()`, `gather()`, `allreduce()`, `allgateher()`, and `sync()`. This set of interfaces should implement the following: support broadcasting data from the main process to each process; support reducing data from each process to the main process; support scattering a set of data from the main process to each process; support collecting data scattered across each process into the main process as a set; support reducing data from each process and then broadcasting it to each process; support collecting scattered data from each process into a set and then broadcasting it to each process; and support ensuring that all previously issued communication commands have been completed.

[0053] 4) Quantization training interface

[0054] The quantization training interface enables algorithms to perceive the information loss caused by model quantization during neural network training. During training, quantized weights are approximated using floating-point weights, allowing the quantized model to be simulated during forward propagation. The floating-point error is then calculated and backpropagated to update the weights. Quantization helps accelerate model inference and reduce storage requirements. This interface should implement the following functionalities:

[0055] a) Supports input quantization, converting input from 32-bit floating-point type to 8-bit or custom-bit fixed-point type; b) Supports quantization and dequantization of convolution and addition operators; c) Supports pseudo-quantization nodes, which should include functions for quantizing and dequantizing floating-point weights; d) Supports error backpropagation, calculating the error through b) during forward propagation, updating the floating-point weights, and then quantizing; e) Supports quantized model output, allowing the quantized trained model to be converted into a fixed-point model for storage.

[0056] 5) Data processing interface

[0057] The data interface should support the conversion of data into formats required by the module, such as tensors. The data interface should have both outer and inner interfaces. The framework should be able to prepare continuous training data for algorithm training in an iterative manner. This interface should have the following core functionalities:

[0058] Supports the implementation of an iterable data loader type, with each iteration returning a batch of data;

[0059] It supports sampling data from the dataset according to training requirements; it supports constructing dataset objects based on the dataset path and related parameters; it supports reading part or all of the dataset data from storage devices or services, such as annotation files and data samples; and it supports preprocessing operations on the data, such as image scaling, flipping, and color perturbation.

[0060] 6) Visual Interface

[0061] The training visualization interface provides visualizations of model structure, parameters, gradients, and features during algorithm training. This interface should support: visualization of model structure diagrams; visualization of feature maps; visualization of weight histograms; visualization of scalar changes; and visualization of convolutional kernels.

[0062] 7) Distillation interface

[0063] The distillation interface supports the use of a teacher network to guide the student network during training, thereby improving the training accuracy of the student model: it should support target distillation method; it should support feature distillation method.

[0064] 8) Graphical computation fusion interface

[0065] Graph-computation fusion optimizes the overall network execution time by analyzing and optimizing the existing network computation graph logic, reducing overhead during operator execution intervals, and improving the utilization of device computing resources. It supports operations such as splitting, reorganizing, and merging existing computation logic; and allows enabling graph-computation fusion by modifying the context parameter in the training script.

[0066] 3. Inference Interface

[0067] 1) Process orchestration interface

[0068] The computer vision system supports workflow orchestration, and its interface meets the following requirements:

[0069] It should be able to combine key processes such as image acquisition, image decoding, image scaling, object detection, image cropping, image classification, and serialization;

[0070] It is advisable to support the plug-in approach for key processes, with configurable properties for each plug-in.

[0071] Users should be able to mount metadata.

[0072] It should support configuration file-based process orchestration management and have management components;

[0073] It should support specifying accelerators for particular processes;

[0074] It should support orchestration of multiple request and multiple output processes;

[0075] The following models should be supported for orchestration: YOLOv3, YOLOv3-tiny, ResNet50, Faster R-CNN, YOLOv4, SSD-VGG16, SSD MobileNet v1 FPN, CRNN, YOLOv5, Faster R-CNN-FPN / Cascade R-CNN-FPN, ResNet-18, DeepLabv3+, CTPN, DeepLabv3, BERT-Base (Uncased), DeepLabv3+, U-Net, Mask R-CNN, FaceNet, SSD MobileNet v1 FPN, OpenPose, Unet++, RetinaNet;

[0076] It should support single-input, single-output, multiple-input, and multiple-output orchestration.

[0077] 2) Data processing interface

[0078] The computer vision system has a data processing interface that meets the following requirements:

[0079] It should support reading data from image files and moving it into a pre-configured cache;

[0080] It should support JPG / JPEG / BMP format image decoding, with a resolution range of (32*32, 8192*8192);

[0081] It should at least support JPG image encoding, with a resolution range of (32*32, 8192*8192);

[0082] Image scaling with specified target width and height should be supported, and image width and height should be scaled and aligned to the step size.

[0083] It should support specifying the expansion ratio in the four directions (top, bottom, left, and right) to expand the area of ​​the target bounding box for cropping.

[0084] It should support H264 / H265 video decoding, with a resolution range of (128*128, 4096*4096).

[0085] Width and height scaling alignment should be supported for step-based scaling.

[0086] It should support resolutions ranging from 128*128 to 1920*1920, and both H264 MP and H265 MP.

[0087] Image normalization, center cropping, affine transformation, and rotation should be supported;

[0088] It should support data transfer between key processes and preferably support the multiple distribution of a single input;

[0089] It should support data transfer between processor memory and main memory;

[0090] It should support frame skipping processing of video data;

[0091] Serialization should be supported.

[0092] 3) Plug-in interface

[0093] The computer vision system should support the development and use of visual processing plugins and meet the following requirements: it should support user-developed plugins, registration, and compilation; it should support interfaces for plugin initialization, deinitialization, execution, attribute registration, and retrieval; it should support interfaces for defining variable and immutable ports for plugin input and output; and it should support interfaces for defining and throwing business logic exceptions.

[0094] It should support a streaming plugin interface to achieve the following functions: a) sending data of a specified type or channel to different ports; b) outputting data from multiple ports in sequence through a single port.

[0095] It should support multiple instantiation interfaces for plugins of the same type; it should support plugin caching mechanisms and interfaces to enable the transfer of business data (such as decoded video and image data) between plugins; it should support description interfaces for plugin metadata (such as classification information and target information), and implement the transfer by relying on the plugin cache; it should support single-input, single-output, multi-input, and multi-output plugin interfaces; it should support inference plugin interfaces, supporting target classification, detection, and tensor-based (input) inference; it should support model post-processing plugin interfaces, enabling it to interface with models for target detection, classification, semantic segmentation, text generation, text box detection, pose detection, etc.

[0096] The video analytics plugin interface should be supported to implement the following functions: a) multi-target (including machine, non-human, and face) path recording; b) face alignment (correcting detected face images); c) video quality diagnosis.

[0097] It should support a debugging plugin interface to enable data export (e.g., JSON format) and data loading and restoration; it should also support a screen display plugin interface to enable drawing basic units on images, such as drawing frames, lines, circles, and writing text.

[0098] 4) Block detection interface

[0099] The computer vision system supports a block detection interface and meets the following requirements:

[0100] It supports filtering duplicate targets in overlapping areas after segmentation; it supports user-defined parameters such as the number / size of segments and overlap, and automatically generates target boxes for image segments; it supports merging images of segmented inference results; and during multi-level inference, it supports filtering post-processing results based on the selection of maximum and minimum area, upper and lower area limits, and confidence thresholds.

[0101] 4. Module

[0102] A Module is a fundamental module in a neural network. Neural network modules are built upon this base class to construct graphs. A Module provides the following functionalities:

[0103] 1) Forward computation of the module: a) Interface name: forward; b) Interface function description: The module performs a forward computation and returns the computation result of the module. If it is in training state, a computation graph is constructed during the forward computation process to calculate the gradient of the module parameters.

[0104] 2) Get trainable parameters of the module: a) Interface name: get_parameters; b) Interface function description: Returns the trainable parameters of the module.

[0105] 3) Retrieve Modules and Submodules: a) Interface Name: get_modules; b) Interface Function Description: Optional. This interface returns an iterator that iterates through the module itself and its submodules. Duplicate modules are returned only once.

[0106] 4) Get module state: a) Interface name: get_state_dict; b) Interface function description: Returns the module state in key-value pairs, including the module parameters and buffer.

[0107] 5) Loading module status: a) Interface name: load_state_dict; b) Interface function description: Loading module status, including module parameters and buffers.

[0108] 6) Module Backward Computation: a) Interface Name: backward(grad_input, grad_output); b) Interface Function Description: The module performs a backward computation and returns the computation result. If in training mode, the gradient of the module parameters is calculated during the backward computation process. This function is automatically generated by the computation graph and can also be registered later using register_backward_function.

[0109] 7) Forward Computation of Module: a) Interface Name: register_backward_function; b) Interface Function Description: The module performs a backward computation and returns the computation result of the module. If it is in training state, the gradient of the module parameters is calculated during the backward computation process.

[0110] This disclosure provides a model training method that can be executed by a processor of a computer device. The computer device can refer to a server, laptop computer, tablet computer, desktop computer, smart TV, set-top box, mobile device (e.g., mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or any other device with data processing capabilities. Figure 2 This is a schematic diagram illustrating the implementation process of a model training method provided in an embodiment of this disclosure, as shown below. Figure 2 As shown, the method includes the following steps S101 to S103:

[0111] Step S101: The loss function scaler creation interface creates a loss function scaler based on the scaler construction parameters.

[0112] Here, the loss function scaler is a predefined function, method, or object used to amplify and / or reduce the loss value in each epoch of mixed-precision training during mixed-precision training. In implementation, the loss function scaler can be implemented in any suitable manner. For example, it can be implemented using a function in the C language, or a static class or object in Python, Java, or C++. This disclosure does not limit this approach.

[0113] The scaler constructor parameters are the parameters required to create a loss function scaler, and may include, but are not limited to, at least one of the following: scaling strategy for scaling the loss value, scaling value, scaling frequency, etc.

[0114] In some implementations, the loss function scaler may employ a fixed target scaling value to amplify and / or reduce the loss value in each epoch of mixed-precision training during mixed-precision training. This fixed scaling value may be included in the scaler's construction parameters. In practice, those skilled in the art can set appropriate values ​​for the fixed scaling value according to actual circumstances; this is not a limitation.

[0115] In some implementations, the loss function scaler may employ a dynamically adjusted target scaling value to amplify and / or reduce the loss value in each epoch of mixed-precision training during mixed-precision training. The scaler construction parameters may include an initial value for the target scaling value and an adjustment strategy or parameters for adjusting that target scaling value. In implementation, those skilled in the art can use appropriate scaler construction parameters according to the actual situation; this is not limited to any particular parameter.

[0116] The loss function scaler creation interface is a predefined interface for creating loss function scalers. Using this interface, a loss function scaler can be created based on the scaler construction parameters of the loss function scaler to be created. In implementation, this interface can be implemented in any suitable manner; for example, it can be implemented using at least one programming language such as Python, Java, C, or C++. This disclosure does not limit this approach. The input to the loss function scaler creation interface can be scaler construction parameters, and the output is the loss function scaler created based on those parameters.

[0117] In step S102, during the forward propagation process of each round of mixed precision training of the deep learning model, the loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed precision training based on the loss function scaler, and obtains the amplified loss value.

[0118] Here, the deep learning model can be any suitable neural network model to be trained, and this disclosure does not limit it. For example, the deep learning model can be at least one of the following: object detection model, image segmentation model, speech recognition model, text recognition model, image classification model, etc.

[0119] A portion of the network parameters in the deep learning model to be trained can be converted to fp16 type, while another portion can be converted to fp32 type to obtain a precision-converted deep learning model. The original fp32 type network parameters are then backed up. During training, a portion of the computations in the precision-converted deep learning model are performed using fp32 precision, and a portion using fp16 precision, thus achieving mixed-precision training of the deep learning model. In implementation, at least one round of mixed-precision training can be performed on the deep learning model to be trained, depending on the actual situation; this embodiment is not limited in this respect.

[0120] During the forward propagation of each round of mixed-precision training of the deep learning model, the input sample data can be converted to fp16 type. The deep learning model, after precision conversion, then predicts the fp16 type sample data to obtain the predicted label and the loss value between the predicted label and the target label. Since the data type of this loss value is fp16, and fp32 data has a smaller representation range and lower precision, directly using the loss value between the predicted and target labels for backpropagation and updating the network parameters in the deep learning model can easily lead to data overflow. Data overflow can include overflow and underflow. Overflow occurs when the data is too large, causing fp16 to be unable to represent the full range, while underflow occurs when the low precision of fp16 data results in smaller data being represented as 0. Underflow issues are common during mixed-precision training. Therefore, a loss function scaler interface can be used to amplify the loss value of the deep learning model in the current round of mixed-precision training. This amplified loss value can reduce underflow issues during backpropagation based on the loss value and during the updating of network parameters in the deep learning model.

[0121] The loss function scaler scaling interface is a predefined interface used to scale the loss value according to the target scaling value in the loss function scaler. The input to this loss function scaler creation interface can be the loss value to be scaled and the loss function scaler, and the output is the scaled loss value. After inputting the loss value output by the deep learning model to be trained in the current epoch of mixed-precision training and the already created loss function scaler into the loss function scaler scaling interface, the interface can scale the input loss value based on the current target scaling value in the input loss function scaler and output the scaled loss value. In implementation, the loss function scaler scaling interface can be implemented in any suitable manner. For example, it can be implemented using at least one programming language such as Python, Java, C, C++, etc., and this disclosure embodiment is not limited in this respect.

[0122] In step S103, during the backpropagation process of mixed precision training in each round, the loss function optimizer iteration interface uses the set optimizer and the loss function scaler to update the network parameters in the deep learning model based on the amplified loss value, thereby obtaining the trained deep learning model.

[0123] Here, during the backpropagation process of each round of mixed-precision training on the deep learning model, the amplified loss value can be backpropagated using the precision-converted deep learning model to obtain the half-precision floating-point gradient value of each network layer in the deep learning model. This half-precision floating-point gradient value is then converted to a full-precision floating-point gradient value. The loss function optimizer iteration interface can use the target scaling value in the loss function scaler to reduce the full-precision floating-point gradient value, obtaining a reduced full-precision floating-point gradient value. Using the set optimizer, the network parameters in the deep learning model are updated based on the reduced full-precision floating-point gradient value, resulting in the trained deep learning model. In some implementations, the fp32 type network parameters in the backup deep learning model can be updated based on the reduced full-precision floating-point gradient value, resulting in updated fp32 type network parameters. The trained deep learning model can then be obtained based on these updated fp32 type network parameters. In practice, based on the amplified loss value, the learnable parameters in the deep learning model can be updated. The learnable parameters can include, but are not limited to, at least one of the weights and offsets of the network layers.

[0124] The input to the loss function optimizer iteration interface can be the deep learning model to be trained, the set optimizer, and the loss function scaler, and the output is the trained deep learning model. In implementation, this loss function optimizer iteration interface can be implemented in any suitable manner. For example, it can be implemented using at least one programming language such as Python, Java, C, or C++. This disclosure does not limit this approach.

[0125] In this embodiment, a loss function scaler is created based on scaler construction parameters through a loss function scaler creation interface. During the forward propagation of each round of mixed-precision training on the deep learning model, the loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed-precision training based on the loss function scaler, obtaining an amplified loss value. During the backpropagation of each round of mixed-precision training, the loss function optimizer iteration interface uses the set optimizer and loss function scaler to update the network parameters in the deep learning model based on the amplified loss value, obtaining the trained deep learning model. Thus, on the one hand, by utilizing the loss function scaler creation interface, loss function scaler scaling interface, and loss function optimizer iteration interface, mixed-precision training of the deep learning model can be conveniently and quickly implemented to improve the accuracy of the trained deep learning model while reducing storage space usage and execution time during model training. On the other hand, the implementation process of mixed-precision training can be simplified, unified, and standardized through interface reuse, thereby improving the generalizability of mixed-precision training applications. Furthermore, by using a unified loss function scaler, the loss value output by the deep learning model in the current round of mixed-precision training is amplified during the forward propagation process of each round of mixed-precision training. In the backpropagation process of each round of mixed-precision training, the network parameters in the deep learning model are updated based on the amplified loss value. This can further simplify the implementation process of mixed-precision training and improve the model training efficiency.

[0126] It should be noted that this model training method has a specific technical connection with the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects of improving the internal performance of computer systems in accordance with natural laws.

[0127] This disclosure provides a model training method that can be executed by a processor of a computer device. For example... Figure 3 As shown, the method includes the following steps S201 to S203:

[0128] Step S201: The loss function scaler creation interface creates the loss function scaler based on the initial value of the target scaling value and the scaling update parameters of the target scaling value in the loss function scaler to be created.

[0129] Here, when creating a loss function scaler, the loss function scaler creation interface can determine the initial value of the target scaling value and the scaling update parameters of the target scaling value in the loss function scaler. The scaling update parameters may include at least one of the following: growth period, growth factor, and backoff factor.

[0130] In some implementations, the target scaling value interface in the loss function scaler is multiplied by a growth factor after a certain number of growth cycles of mixed-precision training to avoid underflow. Whenever overflow occurs, the loss function scaler multiplies the target scaling value by a backoff factor to back off the target scaling value (i.e., reduce the target scaling value), thereby achieving the effect of reducing loss value overflow globally.

[0131] Step S202: During the forward propagation process of each round of mixed precision training of the deep learning model, the loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed precision training based on the loss function scaler, and obtains the amplified loss value.

[0132] In step S203, during the backpropagation process of mixed precision training in each round, the loss function optimizer iteration interface uses the set optimizer and the loss function scaler to update the network parameters in the deep learning model based on the amplified loss value, thereby obtaining the trained deep learning model.

[0133] Here, steps S202 to S203 correspond to steps S102 to S103 mentioned above, and specific implementation methods of steps S102 to S103 can be referred to during implementation.

[0134] In this embodiment of the disclosure, a loss function scaler is created based on the initial value of the target scaling value and the scaling update parameters of the target scaling value in the loss function scaler to be created, through the loss function scaler creation interface. In this way, by defining the initial value of the target scaling value and the scaling update parameters of the target scaling value in the loss function scaler through the loss function scaler creation interface, the loss function scaler can be created conveniently and quickly.

[0135] In some embodiments, the above method may further include at least one of the following steps S211 and S212:

[0136] Step S211: The loss function scaler scaling interface updates the current target scaling value based on the scaling update parameter to obtain the updated target scaling value.

[0137] In step S212, the loss function optimizer iterative interface updates the current target scaling value based on the scaling update parameter to obtain the updated target scaling value.

[0138] In some embodiments, when the scaling update parameters include a growth period and a growth coefficient, step S211 above may include steps S221 to S222 as follows:

[0139] Step S221, the loss function scaler scaling interface determines the number of the first round of mixed precision training in which none of the currently obtained amplified second loss values ​​have overflowed;

[0140] Here, the first round number refers to the number of mixed-precision training rounds in which the amplified second loss value has not overflowed in the current consecutive 100 rounds of mixed-precision training. For example, if the amplified second loss value has not overflowed in the current consecutive 100 rounds of mixed-precision training, the first round number is 100.

[0141] In step S222, when the first round reaches the growth cycle, the loss function scaler scaling interface amplifies the current target scaling value based on the growth coefficient to obtain the amplified target scaling amount.

[0142] In some implementations, the growth factor can be a number greater than 1, and the current target scaling value can be multiplied by the growth factor to obtain the scaled target value.

[0143] In some implementations, the growth factor can be a positive number less than 1, and the current target scaling value can be divided by the growth factor to obtain the scaled target value.

[0144] In some implementations, the growth factor can be a positive number, and the current target scaling value can be added to the growth factor to obtain the scaled target value.

[0145] In some implementations, the growth factor can be a negative number, and the current target scaling value can be subtracted from the growth factor to obtain the scaled target value.

[0146] In some embodiments, when the scaling update parameter includes a rollback coefficient, step S211 above may include step S231 as follows:

[0147] In step S231, if the second loss value amplified in the current round of mixed precision training overflows, the loss function scaler scaling interface reduces the current target scaling value based on the backoff coefficient to obtain the reduced target scaling amount, and stops updating the network parameters in the deep learning model in the current round of mixed precision training.

[0148] In some implementations, the rollback factor can be a positive number less than 1, and the current target scaling value can be multiplied by the rollback factor to obtain the reduced target scaling amount.

[0149] In some implementations, the rollback factor can be a number greater than 1, and the current target scaling value can be divided by the rollback factor to obtain the reduced target scaling amount.

[0150] In some implementations, the rollback factor can be a positive number, which can be subtracted from the current target scaling value to obtain the reduced target scaling amount.

[0151] In some implementations, the rollback factor can be a negative number, and the current target scaling value can be added to the rollback factor to obtain the reduced target scaling amount.

[0152] In some embodiments, when the scaling update parameters include a growth period and a growth coefficient, step S212 may include steps S241 to S242 as follows:

[0153] Step S241, the loss function optimizer iterative interface determines the number of the second round of mixed precision training in which the network parameters have not overflowed during the current continuous updating of the network parameters in the deep learning model;

[0154] Here, the second round number refers to the number of mixed-precision training rounds in which the network parameters of the deep learning model have not overflowed during the current consecutive updates. For example, if the network parameters of the deep learning model have not overflowed during the current 200 consecutive mixed-precision training rounds, the second round number is 200.

[0155] In step S242, when the second round number reaches the growth cycle, the loss function optimizer iteration interface amplifies the current target scaling value based on the growth coefficient to obtain the amplified target scaling amount.

[0156] In some embodiments, when the scaling update parameter includes a rollback coefficient, step S212 above may include the following step S251:

[0157] In step S251, if the network parameters overflow during the process of updating the network parameters in the deep learning model in the current round of mixed precision training, the loss function optimizer iteration interface reduces the current target scaling value based on the backoff coefficient to obtain the reduced target scaling amount, and stops updating the network parameters in the deep learning model in the current round of mixed precision training.

[0158] This disclosure provides a model training method that can be executed by a processor of a computer device. For example... Figure 4 As shown, the method includes the following steps S301 to S304:

[0159] Step S301: Based on the set mixed precision switch parameters, the precision conversion interface converts the full-precision network parameters of the network layers in the deep learning model into half-precision network parameters to obtain the precision-converted deep learning model, and backs up the full-precision network parameters.

[0160] Here, full-precision network parameters refer to network parameters whose data type is full-precision floating-point. Half-precision network parameters refer to network parameters whose data type is half-precision floating-point.

[0161] The precision conversion interface can take the deep learning model to be trained and mixed precision switch parameters as input, and output the precision-converted deep learning model. Using this interface, the mixed precision training switch of the deep learning model to be trained can be turned on or off by setting the mixed precision switch parameters. During model training, the state of the mixed precision training switch can be used to determine whether to convert the precision type of the network parameters of the deep learning model. In implementation, this precision conversion interface can be implemented in any suitable manner. For example, it can be implemented using at least one programming language such as Python, Java, C, or C++. This disclosure does not limit this approach.

[0162] Step S302: The loss function scaler creation interface creates a loss function scaler based on the scaler construction parameters.

[0163] Step S303: During the forward propagation process of each round of mixed precision training of the deep learning model, the loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed precision training based on the loss function scaler, so as to obtain the amplified loss value.

[0164] Here, steps S302 to S303 correspond to steps S101 to S102, respectively. In practice, the specific implementation of steps S101 to S102 can be referred to.

[0165] In step S304, during the backpropagation process of mixed-precision training in each round, the loss function optimizer iterative interface uses the set optimizer and the loss function scaler to update the backed-up full-precision network parameters based on the amplified loss value, thereby obtaining the trained deep learning model.

[0166] In this embodiment, the precision conversion interface converts the full-precision network parameters of the network layers in the deep learning model into half-precision network parameters based on the set mixed precision switch parameters, thereby obtaining the precision-converted deep learning model and backing up the full-precision network parameters. During the forward propagation of each round of mixed precision training of the deep learning model, the loss function scaler scaling interface amplifies the loss value output by the precision-converted deep learning model in the current round of mixed precision training based on the loss function scaler, thereby obtaining the amplified loss value. During the backpropagation of each round of mixed precision training, the loss function optimizer iteration interface uses the set optimizer and loss function scaler to update the backed-up full-precision network parameters based on the amplified loss value, thereby obtaining the trained deep learning model. In this way, on the one hand, based on the set mixed precision switching parameters, the full-precision network parameters of the network layers set in the deep learning model can be converted into half-precision network parameters according to actual needs, so as to obtain the deep learning model after precision conversion, which can better meet the needs of mixed precision training in actual application scenarios; on the other hand, updating the backed-up full-precision network parameters based on the amplified loss value can improve the precision of the updated network parameters, thereby improving the precision of the trained deep learning model.

[0167] In some embodiments, step S301 may include the following step S311:

[0168] Step S311: When the mixed precision switch parameter indicates that mixed precision training is enabled, the precision conversion interface converts the full-precision network parameters of the network layers set in the deep learning model into half-precision network parameters to obtain the precision-converted deep learning model, and backs up the full-precision network parameters.

[0169] In some embodiments, the loss function scaler includes a target scaling value; the loss function scaler scaling interface described in step S303 above, based on the loss function scaler, amplifies the loss value output by the precision-converted deep learning model in the current round of mixed-precision training to obtain the amplified loss value, and may include the following step S321:

[0170] Step S321: The loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed precision training based on the current target scaling value in the loss function scaler, to obtain the amplified loss value.

[0171] In some embodiments, the loss function optimizer iteration interface described in step S304 above uses a set optimizer and the loss function scaler to update the backed-up full-precision network parameters based on the amplified loss value to obtain the trained deep learning model, and may include the following steps S331 to S333:

[0172] Step S331: The loss function optimizer iterative interface obtains the half-precision gradient value of each network layer in the deep learning model obtained by backpropagating the amplified loss value in the deep learning model after precision conversion.

[0173] Step S332: The loss function optimizer iterative interface performs precision conversion on the half-precision gradient value of each network layer to obtain the full-precision gradient value of each network layer.

[0174] Here, full-precision gradient values ​​refer to gradient values ​​whose data type is full-precision floating-point. Half-precision gradient values ​​refer to gradient values ​​whose data type is half-precision floating-point.

[0175] Step S333: The loss function optimizer iteration interface uses the set optimizer and the loss function scaler to update the full-precision network parameters of the corresponding network layer in the deep learning model based on the full-precision gradient value of each network layer, so as to obtain the trained deep learning model.

[0176] In some embodiments, step S333 may include steps S341 to S342 as follows:

[0177] Step S341: The loss function optimizer iteration interface reduces the full-precision gradient value of each network layer based on the current target scaling value in the loss function scaler, and obtains the reduced full-precision gradient value corresponding to each network layer.

[0178] In step S342, the loss function optimizer iteration interface uses the set optimizer to update the full-precision network parameters of the corresponding network layer in the deep learning model based on the reduced full-precision gradient value corresponding to each network layer, thereby obtaining the trained deep learning model.

[0179] The following describes the application of the model training method provided in this disclosure in a real-world scenario, taking the application of the model training method in a mixed-precision training interface based on computer vision algorithms as an example.

[0180] This disclosure provides a mixed-precision training interface based on computer vision algorithms, offering unified support for mixed-precision training. This reduces memory consumption and, with sufficient GPU support, improves training speed, mitigating the poor training performance caused by entirely low-precision data. The interface integrates various functionalities required for mixed-precision training, providing a unified and standardized approach for mixed-precision training algorithms. Specifically, the interface implements the following functions during mixed-precision training:

[0181] a) Supports precision conversion function, converting network parameters in the network layers set in the deep learning model into fp16 type, except for special layers (e.g., batch-normalized (BN) layers, etc.), while retaining a backup of the parameters in fp32 type;

[0182] b) Supports input forward propagation, converting input data into fp16 type data, performing forward propagation, and calculating the loss value obtained from forward propagation;

[0183] c) Supports loss amplification function, which amplifies the calculated loss value, and supports two amplification modes: fixed factor and dynamic factor.

[0184] Here, the fixed-factor scaling mode refers to scaling the loss value using a fixed target scaling value. The dynamic-factor scaling mode refers to scaling the loss value using a dynamically adjusted target scaling value.

[0185] d) Support gradient calculation function. Calculate the gradient values ​​of each network layer in the deep learning model in fp16 mode and backpropagate them to obtain the fp16 type gradient values ​​of each network layer. Then, convert the fp16 type gradient values ​​of each network layer into fp32 type gradient values. Based on the target scaling value for amplifying the loss value in step c), reduce the fp32 type gradient values ​​to obtain the reduced fp32 type gradient values.

[0186] e) Support parameter update function. Based on the scaled-down fp32 type gradient value calculated in step d), update the network parameters in the fp32 type parameter backup, and then assign the updated network parameters to the fp16 model after precision conversion.

[0187] Based on the principles and functions of mixed-precision training, this disclosure defines four interfaces (i.e., mixed-precision training interfaces) required for mixed-precision training: a precision conversion interface, a loss function scaler creation interface, a loss function scaler scaling interface, and a loss function optimizer iteration interface. By defining the parameter tables and functional implementation methods of each mixed-precision training interface, the use of mixed-precision training interfaces for various frameworks by deep learning algorithms is standardized.

[0188] The following sections describe the precision conversion interface, the loss function scaler creation interface, the loss function scaler scaling interface, and the loss function optimizer iteration interface.

[0189] 1) Precision Conversion Interface

[0190] A. Interface Name: aoocast

[0191] Define the precision conversion interface as: autocast(model, enable).

[0192] B. Interface Function Description:

[0193] This precision conversion interface enables or disables mixed-precision training of the model.

[0194] C. Interface parameter list:

[0195] The parameters of this precision conversion interface are shown in Table 1 below:

[0196] Table 1. List of Precision Conversion Interface Parameters

[0197]

[0198] D. Interface exception handling:

[0199] If there are no errors, the operation is returned as successful.

[0200] E. Other additional notes: None

[0201] F. Implementation principle: By turning the mixed precision switch of the deep learning model on or off, mixed precision training can be turned on or off. During the actual training process, the deep learning model will determine whether to convert the precision type of the network parameters according to the state of the mixed precision switch.

[0202] 2) Loss function scaler creation interface

[0203] A. Interface name: construct_gradscaler

[0204] Define the loss function scaler creation interface as follows:

[0205] construct_gradscaler(init_scale,growth_factor,backoff_factor,growth_interval,grad_scaler).

[0206] B. Interface Function Description:

[0207] This loss function scaler creation interface is used to create loss function scalers. Loss function scalers can scale loss values ​​to reduce overflow (data is too large, exceeding the range that half-precision floating-point types can represent) and underflow (data precision requirements are too high, causing it to become 0 in half-precision floating-point types) during backpropagation after mixed-precision training. The interface has an initial scaling value `init_scale` (the initial value of the target scaling value in the loss function scaler). After mixed-precision training for a certain number of growth intervals (`growth_interval`), the scaler multiplies the target scaling value by a growth factor `growth_factor` to reduce underflow. If overflow occurs during the current round of mixed-precision training, the interface multiplies the target scaling value by a backoff factor `backoff_factor` to back off the scaling value, thus reducing loss value overflow globally.

[0208] C. Interface parameter list:

[0209] The parameters for creating the loss function scaler interface are shown in Table 2 below:

[0210] Table 2 List of parameters for the loss function scaler creation interface

[0211]

[0212] D. Interface exception handling:

[0213] If there are no errors, the operation is returned as successful.

[0214] E. Other additional notes: None

[0215] F. Implementation Principle:

[0216] The loss function scaler creation interface receives parameters such as initial scaling value, growth period, growth coefficient, and backoff coefficient to create a loss function scaler and save the relevant parameters. Then, according to the description in the interface function description, the loss function scaler automatically adjusts the internally stored scaling value during execution.

[0217] 3) Loss function scaler scaling interface

[0218] A. Interface name: grad_scale(loss, grad_scaler)

[0219] Define the loss function scaler interface as: grad_scale(loss, grad_scaler)

[0220] B. Interface Function Description:

[0221] The loss function scaler scaling interface is used to scale the loss value based on the scaling value inside the loss function scaler grad_scaler.

[0222] C. Interface parameter list:

[0223] The parameters of the loss function scaler scaling interface are shown in Table 3 below:

[0224] Table 3 List of parameters for the loss function scaler scaling interface

[0225]

[0226] D. Interface exception handling:

[0227] If there are no errors, the operation is returned as successful.

[0228] E. Other additional notes: None

[0229] F. Implementation Principle:

[0230] A new loss value is obtained by multiplying the target scaling value from the loss function scaler grad_scaler with the loss value.

[0231] 4) Loss Function Optimizer Iteration Interface

[0232] A. Interface Name: update_learnable_parameter

[0233] Define the loss function, scaler, optimizer, and iterative interface as follows:

[0234] update_learnable_parameter(model,optimizer,grad_scaler)

[0235] B. Interface Function Description:

[0236] The loss function scaler optimizer iterative interface is used to iteratively update the learnable parameters (such as weights) of each network layer in the deep learning model based on the loss function scaler. Based on the deep learning model, after the gradient values ​​of each network layer have been calculated in backpropagation, the gradient values ​​of each network layer are proportionally reduced according to the current target scaling value in the loss function scaler. Then, the optimization strategy in the optimizer is used to update the learnable parameters of the corresponding network layer in the deep learning model based on the reduced gradient values ​​of each network layer.

[0237] If the learnable parameters in the model overflow after being updated, the network parameters in the deep learning model are stopped in the current round of mixed-precision training, and the target scaling value is multiplied by the backoff factor to back off. If the number of rounds of mixed-precision training with continuous network parameter updates without overflow reaches the number of growth intervals, the target scaling value is multiplied by the growth factor.

[0238] C. Interface parameter list:

[0239] The parameters of the loss function optimizer iteration interface are shown in Table 4 below:

[0240] Table 4. List of parameters for the loss function optimizer iteration interface

[0241]

[0242] D. Interface exception handling:

[0243] If there are no errors, the operation is returned as successful.

[0244] E. Other additional notes: None

[0245] F. Implementation Principle:

[0246] The network parameters of the deep learning model are updated by calling the optimizer. If parameter overflow occurs during the process of updating network parameters in the current round of mixed precision training, the target scaling value is updated in the loss function scaler.

[0247] Mixed-precision training can reduce memory usage during model training and improve training efficiency. The mixed-precision training interface provided in this disclosure helps algorithms and algorithm frameworks better call mixed-precision related interfaces for mixed-precision training.

[0248] Users can utilize the model training methods and mixed-precision training interfaces provided in this disclosure within the algorithm. These methods and interfaces can be used by companies or individuals engaged in AI model production to reduce GPU memory usage and improve training efficiency. Furthermore, they can be used in algorithm and algorithm framework development, as well as in deep learning applications derived from these algorithms. Additionally, the methods and interfaces provided in this disclosure can be applied to algorithm optimization and acceleration tools, and in deep learning products derived from these tools.

[0249] Based on the foregoing embodiments, this disclosure provides a model training device, which includes the included units and the modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0250] Figure 5 This is a schematic diagram of the composition structure of a model training device provided in an embodiment of the present disclosure, as shown below. Figure 5 As shown, the model training device 400 includes: a loss function scaler creation interface 410, a loss function scaler scaling interface 420, and a loss function optimizer iteration interface 430, wherein:

[0251] The loss function scaler creation interface 410 is used to create a loss function scaler based on the scaler construction parameters;

[0252] The loss function scaler scaling interface 420 is used to amplify the loss value output by the deep learning model in the current round of mixed precision training during the forward propagation process of each round of mixed precision training of the deep learning model, based on the loss function scaler, to obtain the amplified loss value.

[0253] The loss function optimizer iteration interface 430 is used to update the network parameters in the deep learning model based on the amplified loss value during the backpropagation process of each round of mixed precision training using a set optimizer and the loss function scaler, so as to obtain the trained deep learning model.

[0254] In some embodiments, the loss function scaler includes a target scaling value, and the scaler construction parameters include an initial value of the target scaling value and scaling update parameters of the target scaling value; the loss function scaler creation interface is further configured to: create the loss function scaler based on the initial value and the scaling update parameters; wherein the scaling update parameters include at least one of the following: growth period, growth coefficient, and backoff coefficient.

[0255] In some embodiments, the loss function scaler scaling interface is further configured to: update the current target scaling value based on the scaling update parameter to obtain the updated target scaling value.

[0256] In some embodiments, the loss function optimizer iteration interface is further configured to: update the current target scaling value based on the scaling update parameter to obtain the updated target scaling value.

[0257] In some embodiments, when the scaling update parameters include a growth period and a growth coefficient, the loss function scaler scaling interface is further configured to: determine the first epoch of mixed-precision training in which none of the currently obtained amplified second loss values ​​have overflowed; and when the first epoch reaches the growth period, amplify the current target scaling value based on the growth coefficient to obtain the amplified target scaling amount.

[0258] In some embodiments, when the scaling update parameter includes a backoff coefficient, the loss function scaler scaling interface is further configured to: reduce the current target scaling value based on the backoff coefficient when the amplified second loss value overflows in the current round of mixed precision training, to obtain the reduced target scaling amount, and stop updating the network parameters in the deep learning model in the current round of mixed precision training.

[0259] In some embodiments, when the scaling update parameters include a growth period and a growth coefficient, the loss function optimizer iteration interface is further configured to: determine the second round number of mixed-precision training in which the network parameters have not overflowed during the current continuous updating of the network parameters in the deep learning model; and when the second round number reaches the growth period, amplify the current target scaling value based on the growth coefficient to obtain the amplified target scaling amount.

[0260] In some embodiments, when the scaling update parameter includes a backoff coefficient, the loss function optimizer iteration interface is further configured to: when the network parameters in the deep learning model overflow during the updating process of the network parameters in the current round of mixed precision training, the loss function optimizer iteration interface reduces the current target scaling value based on the backoff coefficient to obtain the reduced target scaling amount, and stops updating the network parameters in the deep learning model in the current round of mixed precision training.

[0261] In some embodiments, the apparatus further includes: a precision conversion interface, configured to convert the full-precision network parameters of a network layer in the deep learning model into half-precision network parameters based on a set mixed-precision switching parameter, to obtain the precision-converted deep learning model, and to back up the full-precision network parameters; the loss function scaler scaling interface is further configured to: amplify the loss value output by the precision-converted deep learning model in the current round of mixed-precision training based on the loss function scaler, to obtain the amplified loss value; the loss function optimizer iteration interface is further configured to: update the backed-up full-precision network parameters based on the amplified loss value using a set optimizer and the loss function scaler, to obtain the trained deep learning model.

[0262] In some embodiments, the precision conversion interface is further configured to: convert the full-precision network parameters of the network layers set in the deep learning model into half-precision network parameters when the mixed precision switch parameter characterizes the activation of mixed precision training, thereby obtaining the precision-converted deep learning model, and backing up the full-precision network parameters.

[0263] In some embodiments, the loss function scaler includes a target scaling value; the loss function scaler scaling interface is further configured to: based on the current target scaling value in the loss function scaler, amplify the loss value output by the precision-converted deep learning model in the current round of mixed precision training to obtain the amplified loss value.

[0264] In some embodiments, the loss function optimizer iteration interface is further configured to: obtain the half-precision gradient value of each network layer in the deep learning model obtained by backpropagating the amplified loss value in the precision-converted deep learning model; convert the half-precision gradient value of each network layer to obtain the full-precision gradient value of each network layer; and update the full-precision network parameters of the corresponding network layer in the deep learning model based on the full-precision gradient value of each network layer using the set optimizer and the loss function scaler to obtain the trained deep learning model.

[0265] In some embodiments, the loss function optimizer iteration interface is further configured to: reduce the full-precision gradient value of each network layer based on the current target scaling value in the loss function scaler, to obtain the reduced full-precision gradient value corresponding to each network layer; and update the full-precision network parameters of the corresponding network layer in the deep learning model based on the reduced full-precision gradient value corresponding to each network layer using a set optimizer, to obtain the trained deep learning model.

[0266] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or included modules or interfaces of the apparatus provided in this disclosure can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0267] It should be noted that, in the embodiments of this disclosure, if the above-described model training method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk. Thus, the embodiments of this disclosure are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0268] This disclosure provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0269] This disclosure provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium may be transient or non-transient.

[0270] This disclosure provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0271] This disclosure provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0272] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referenced interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0273] It should be noted that, Figure 6 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this disclosure, such as... Figure 6 As shown, the hardware entity of the computer device 500 includes: a processor 501, a communication interface 502, and a memory 503, wherein:

[0274] Processor 501 typically controls the overall operation of computer device 500.

[0275] Communication interface 502 enables computer devices to communicate with other terminals or servers via a network.

[0276] The memory 503 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 501 and various modules in the computer device 500. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 501, the communication interface 502, and the memory 503 can be performed via bus 504.

[0277] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0278] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0279] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0280] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0281] In addition, each functional unit in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0282] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0283] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0284] The above description is merely an embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A model training method, characterized in that, The method includes: The loss function scaler creation interface creates a loss function scaler based on the scaler construction parameters; During the forward propagation of each round of mixed-precision training of the deep learning model, the loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed-precision training based on the loss function scaler, to obtain the amplified loss value. During the backpropagation process of each round of mixed precision training, the loss function optimizer iterative interface uses the set optimizer and the loss function scaler to update the network parameters in the deep learning model based on the amplified loss value, thereby obtaining the trained deep learning model. The loss function scaler includes a target scaling value. The scaler construction parameters include an initial value of the target scaling value and a scaling update parameter for the target scaling value. The target scaling value is used to amplify and / or reduce the loss value output by the deep learning model in each round of mixed precision training. The scaling update parameter is used to update the target scaling value. The scaling update parameter includes at least one of the following: growth period, growth coefficient, and backoff coefficient.

2. The method according to claim 1, characterized in that, The loss function scaler creation interface creates a loss function scaler based on scaler construction parameters, including: The loss function scaler creation interface creates the loss function scaler based on the initial value and the scaling update parameter.

3. The method according to claim 2, characterized in that, The method further includes at least one of the following: The loss function scaler scaling interface updates the current target scaling value based on the scaling update parameter to obtain the updated target scaling value; The loss function optimizer iterative interface updates the current target scaling value based on the scaling update parameter to obtain the updated target scaling value.

4. The method according to claim 3, characterized in that, When the scaling update parameters include a growth period and a growth coefficient, the loss function scaler scaling interface updates the current target scaling value based on the scaling update parameters to obtain the updated target scaling value, including: The loss function scaler scaling interface determines the number of the first round of mixed-precision training in which none of the currently obtained amplified second loss values ​​have overflowed. When the first round reaches the growth cycle, the loss function scaler scaling interface amplifies the current target scaling value based on the growth coefficient to obtain the amplified target scaling amount.

5. The method according to claim 3, characterized in that, When the scaling update parameters include a backoff coefficient, the loss function scaler scaling interface updates the current target scaling value based on the scaling update parameters to obtain the updated target scaling value, including: If the second loss value, amplified during the current round of mixed-precision training, overflows, the loss function scaler scaling interface reduces the current target scaling value based on the backoff coefficient to obtain the reduced target scaling amount, and stops updating the network parameters in the deep learning model during the current round of mixed-precision training.

6. The method according to claim 3, characterized in that, When the scaling update parameters include a growth period and a growth coefficient, the loss function optimizer iterative interface updates the current target scaling value based on the scaling update parameters to obtain the updated target scaling value, including: The loss function optimizer iterative interface determines the number of the second round of mixed-precision training in which the network parameters have not overflowed during the current continuous update of the network parameters in the deep learning model. When the second round number reaches the growth cycle, the loss function optimizer iteration interface amplifies the current target scaling value based on the growth coefficient to obtain the amplified target scaling amount.

7. The method according to claim 3, characterized in that, When the scaling update parameters include a backoff coefficient, the loss function optimizer iterative interface updates the current target scaling value based on the scaling update parameters to obtain the updated target scaling value, including: If the network parameters overflow during the updating of the network parameters in the deep learning model in the current round of mixed precision training, the loss function optimizer iteration interface reduces the current target scaling value based on the backoff coefficient to obtain the reduced target scaling amount, and stops updating the network parameters in the deep learning model in the current round of mixed precision training.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The precision conversion interface converts the full-precision network parameters of the network layers in the deep learning model into half-precision network parameters based on the set mixed precision switch parameters, thereby obtaining the precision-converted deep learning model, and backs up the full-precision network parameters. The loss function scaler scaling interface, based on the loss function scaler, amplifies the loss value output by the deep learning model in the current round of mixed-precision training to obtain the amplified loss value, including: The loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed precision training after precision conversion, based on the loss function scaler, to obtain the amplified loss value. The loss function optimizer iteration interface uses a predefined optimizer and the loss function scaler to update the network parameters in the deep learning model based on the amplified loss value, thereby obtaining the trained deep learning model, including: The loss function optimizer iteration interface uses the set optimizer and the loss function scaler to update the backed-up full-precision network parameters based on the amplified loss value, thereby obtaining the trained deep learning model.

9. The method according to claim 8, characterized in that, The precision conversion interface, based on set mixed precision switching parameters, converts the full-precision network parameters of the network layers in the deep learning model into half-precision network parameters, obtaining the precision-converted deep learning model, and backs up the full-precision network parameters, including: When the mixed precision switch parameter indicates that mixed precision training is enabled, the precision conversion interface converts the full-precision network parameters of the network layers set in the deep learning model into half-precision network parameters, thereby obtaining the precision-converted deep learning model, and backs up the full-precision network parameters.

10. The method according to claim 8, characterized in that, The loss function scaler includes a target scaling value; The loss function scaler scaling interface, based on the loss function scaler, amplifies the loss value output by the precision-converted deep learning model in the current round of mixed-precision training to obtain the amplified loss value, including: The loss function scaler scaling interface amplifies the loss value output by the deep learning model in the current round of mixed precision training based on the current target scaling value in the loss function scaler, thus obtaining the amplified loss value.

11. The method according to claim 8, characterized in that, The loss function optimizer iteration interface uses a set optimizer and the loss function scaler to update the backed-up full-precision network parameters based on the amplified loss value, thereby obtaining the trained deep learning model, including: The loss function optimizer iterative interface obtains the half-precision gradient value of each network layer in the deep learning model by backpropagating the amplified loss value in the precision-transformed deep learning model. The loss function optimizer iterative interface performs precision conversion on the half-precision gradient value of each network layer to obtain the full-precision gradient value of each network layer. The loss function optimizer iteration interface uses the set optimizer and the loss function scaler to update the full-precision network parameters of the corresponding network layer in the deep learning model based on the full-precision gradient value of each network layer, so as to obtain the trained deep learning model.

12. The method according to claim 11, characterized in that, The loss function optimizer iterative interface utilizes a predefined optimizer and the loss function scaler to update the full-precision network parameters of the corresponding network layers in the deep learning model based on the full-precision gradient values ​​of each network layer, thereby obtaining the trained deep learning model, including: The loss function optimizer iteration interface reduces the full-precision gradient value of each network layer based on the current target scaling value in the loss function scaler, thereby obtaining the reduced full-precision gradient value corresponding to each network layer. The loss function optimizer iterative interface uses a set optimizer to update the full-precision network parameters of the corresponding network layer in the deep learning model based on the reduced full-precision gradient value corresponding to each network layer, thereby obtaining the trained deep learning model.

13. A model training device, characterized in that, include: The loss function scaler creation interface is used to create a loss function scaler based on the scaler construction parameters; The loss function scaler scaling interface is used to amplify the loss value output by the deep learning model in the current round of mixed precision training during the forward propagation process of each round of mixed precision training, based on the loss function scaler, to obtain the amplified loss value. The loss function optimizer iteration interface is used to update the network parameters in the deep learning model based on the amplified loss value during the backpropagation process of each round of mixed precision training using a set optimizer and the loss function scaler, so as to obtain the trained deep learning model. The loss function scaler includes a target scaling value. The scaler construction parameters include an initial value of the target scaling value and a scaling update parameter for the target scaling value. The target scaling value is used to amplify and / or reduce the loss value output by the deep learning model in each round of mixed precision training. The scaling update parameter is used to update the target scaling value. The scaling update parameter includes at least one of the following: growth period, growth coefficient, and backoff coefficient.

14. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 12.

Images