Image classification method and system fusing Dear and Zero methods, terminal and storage medium

Abstract

The invention discloses an image classification method and system fusing a Dear method and a Zero method, a terminal and a storage medium. The method comprises the following steps: constructing a deep convolutional neural network, collecting historical image data, preprocessing the historical image data, and taking the preprocessed historical image data as a training data set; deploying the deep convolutional neural network to a distributed training system, and according to the training data set, performing distributed training on the deep convolutional neural network on the distributed training system through a DZ optimizer obtained by fusing a Dear method and a Zero method to obtain a target visual neural network; and obtaining to-be-processed visual image data, inputting the visual image data into the target visual neural network for processing, and outputting a classification result of the visual image data. According to the method, excellent balance between video memory efficiency and calculation performance is realized, the problems of video memory occupation and overlarge communication overhead of model distributed training are solved, and the image processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image data processing, and particularly relates to a Dear and Zero method fused image classification method, system, terminal and storage medium. BACKGROUND

[0002] With the continuous expansion of the size of deep learning models, its application in high-resolution image classification, target detection and other computer vision tasks puts unprecedented requirements on computing and memory resources. Data parallel method has become one of the mainstream distributed training schemes by distributing training data to multiple devices and using All-Reduce to synchronize gradient information. However, there are two key challenges in large-scale model training: memory occupancy efficiency and communication overhead. For example, when training a classic convolutional neural network such as ResNet or an updated Vision Transformer model on a large dataset such as ImageNet, memory occupancy and communication overhead become the main bottleneck of training efficiency and model performance.

[0003] Traditional data parallel methods have high memory occupancy; ZeRO (Zero Redundancy Optimizer) method optimizes memory but has low communication efficiency; DeAR (Decentralized and Adaptive Resource-aware training) method improves communication efficiency but introduces additional memory overhead. These methods cannot solve the dual bottleneck problem of memory and communication at the same time, which leads to the fact that the memory occupancy and communication overhead of model distributed training are too large when classifying image data, which limits the efficiency of image processing.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a Dear and Zero method fused image classification method, system, terminal and computer readable storage medium, which aims to solve the problem of excessive memory occupancy and communication overhead of model distributed training when classifying image data in the prior art, which limits the efficiency of image processing.

[0006] To achieve the above purpose, the present application provides a Dear and Zero method fused image classification method, which includes the following steps:

[0007] constructing a deep convolutional neural network, collecting historical visual image data, preprocessing the historical visual image data, and taking the preprocessed historical image data as a training data set;

[0008] deploying the deep convolutional neural network to a distributed training system, performing distributed training on the deep convolutional neural network on the distributed training system according to the training data set through a DZ optimizer, and obtaining a target visual neural network;

[0009] The DZ optimizer is obtained by fusing a Dear method and a Zero method.

[0010] obtaining visual image data to be processed, inputting the visual image data into the target visual neural network for processing, and outputting a classification result of the visual image data.

[0011] Optionally, the image classification method of fusing the Dear and Zero methods, wherein the obtaining the target visual neural network by performing distributed training on the deep convolutional neural network on the distributed training system according to the training data set through the DZ optimizer further includes:

[0012] performing one global initialization on the distributed training system using the DZ optimizer, and dispersing a model state of the deep convolutional neural network to each data-parallel computing node to complete optimizer state fragmentation and communication structure establishment, so as to reduce static memory occupation of a single node to the maximum extent.

[0013] The model state includes a model parameter, a gradient, and an optimizer state.

[0014] Optionally, the image classification method of fusing the Dear and Zero methods, wherein the obtaining the target visual neural network by performing distributed training on the deep convolutional neural network on the distributed training system according to the training data set through the DZ optimizer specifically includes:

[0015] performing forward propagation on the distributed training system based on a communication and calculation overlap mechanism through the DZ optimizer according to the training data set, calculating a parameter of each layer of the deep convolutional neural network, and calculating a forward propagation loss value when the forward propagation ends.

[0016] According to the training data set, by the DZ optimizer, based on the communication and calculation overlapping mechanism, the back propagation is carried out on the distributed training system, the gradient of each layer parameter is calculated according to the forward propagation loss value, the gradient is put into the buffer group of tensor fusion, when the buffer group is filled, an asynchronous communication is initiated to the buffer group, and the back propagation is continued at the same time of the asynchronous communication;

[0017] When the back propagation and all asynchronous communications are completed, the gradient slices of each node in the deep convolutional neural network are obtained based on the communication results, the local optimizer state and the model parameter corresponding to each node are updated according to the gradient slices, and the next round of training is continued until the training round reaches a preset number, so that the distributed training of the deep convolutional neural network is completed, and a target visual neural network is obtained.

[0018] Optionally, the image classification method of the fusion Dear and Zero method fuses the Dear and Zero method, wherein the forward propagation is carried out on the distributed training system according to the training data set by the DZ optimizer based on the communication and calculation overlapping mechanism, and the method further comprises the following steps:

[0019] The back propagation mechanism of the deep convolutional neural network is expanded to obtain a Hook mechanism, the Hook mechanism refers to registering a callback function at a specific position in a calculation graph, and the callback function is automatically called to insert a custom operation when the back propagation reaches the specific position;

[0020] The Hook mechanism comprises a gradient hook and a forward propagation front hook;

[0021] The gradient hook is called after the gradient of a tensor is calculated, and is used to add the gradient of each layer parameter to the corresponding communication group;

[0022] The forward propagation front hook is called before the forward propagation reaches a certain layer, and is used to realize cross-layer parameter synchronization.

[0023] Optionally, the image classification method of the fusion Dear and Zero method fuses the Dear and Zero method, wherein the forward propagation is carried out on the distributed training system according to the training data set by the DZ optimizer based on the communication and calculation overlapping mechanism, and the method further comprises the following steps:

[0024] Before the forward propagation starts, an asynchronous All-Gather communication operation is started, and the complete model parameter updated in the last iteration is aggregated from each node to each node;

[0025] selecting a layer of the deep convolutional neural network as a target layer, registering a forward propagation pre-hook for the target layer, and performing forward propagation on the distributed training system according to the training data set through the DZ optimizer;

[0026] After the forward propagation reaches the target layer, if the All-Gather of the last iteration is completed and the current layer is the first layer after grouping, an All-Gather operation for the parameters of the next layer is started, and the updated parameter slices on each node are aggregated into complete parameters.

[0027] Optionally, the image classification method fusing the Dear and Zero methods, wherein the back propagation on the distributed training system according to the training data set through the DZ optimizer based on the communication and computation overlap mechanism, the calculation of the gradient of each layer parameter according to the forward propagation loss value, the placement of the gradient into a buffer group for tensor fusion, the initiation of an asynchronous communication for the buffer group when the buffer group is filled, and the continuation of the back propagation at the same time as the asynchronous communication, specifically include:

[0028] The back propagation on the distributed training system according to the training data set through the DZ optimizer, and the copying of the gradient to a global buffer group for tensor fusion by calling a gradient hook as soon as the gradient of one parameter is calculated;

[0029] When the buffer group is filled with gradients, an asynchronous Reduce-Scatter collective communication is initiated for the global buffer group, and the back propagation calculation of the gradient is continued at the same time as the asynchronous communication to fill the next buffer group;

[0030] After all the back propagation and Reduce-Scatter collective communication are completed, the back propagation is completed, and each node obtains its own gradient slice.

[0031] Optionally, the image classification method fusing the Dear and Zero methods, wherein the updating of the local optimizer state and the model parameter corresponding to each node according to the gradient slice, specifically includes:

[0032] After receiving the messages of all the completed Reduce-Scatter collective communications, the gradient slice of each node is checked and gradient clipped through a preset number of lightweight All-Reduce communications, and a target gradient slice is obtained;

[0033] The target gradient slice locally held by each node is used to update the corresponding parameter slice and optimizer state slice.

[0034] In addition, to achieve the above object, the application further provides a fusion Dear and Zero method image classification system, comprising:

[0035] a model construction and data collection module, configured to construct a deep convolutional neural network, collect historical visual image data, pre-process the historical visual image data, and use the pre-processed historical image data as a training data set;

[0036] a distributed training module, configured to deploy the deep convolutional neural network to a distributed training system, and perform distributed training on the deep convolutional neural network on the distributed training system according to the training data set through a DZ optimizer to obtain a target visual neural network, wherein the DZ optimizer is obtained by fusing a Dear method and a Zero method;

[0037] a visual fusion Dear and Zero method image classification module, configured to acquire visual image data to be processed, input the visual image data to the target visual neural network for processing, and output a classification result of the visual image data.

[0038] In addition, to achieve the above object, the application further provides a terminal, comprising a memory, a processor, and a fusion Dear and Zero method image classification program stored in the memory and executable on the processor, wherein the fusion Dear and Zero method image classification program, when executed by the processor, implements the steps of the fusion Dear and Zero method image classification method as described above.

[0039] In addition, to achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a fusion Dear and Zero method image classification program, and the fusion Dear and Zero method image classification program, when executed by a processor, implements the steps of the fusion Dear and Zero method image classification method as described above.

[0040] In the present application, by innovatively combining the communication optimization of DeAR and the video memory optimization of ZeRO, on the one hand, the optimizer state is divided into multiple computing nodes, which significantly reduces the redundant video memory occupation, and the gradient is discarded after being calculated to reduce unnecessary video memory overhead; on the other hand, the All-Reduce communication operation is decoupled into Reduce-Scatter and All-Gather, and is respectively implemented with high efficient overlap of backward calculation and forward calculation, and the tensor fusion technology is adopted to reduce the communication startup overhead. The DZ method realizes an excellent balance between video memory efficiency and computing performance, solves the problems of large video memory occupation and communication overhead of model distributed training, improves the efficiency of image processing, and can provide a practical and efficient end-to-end solution for constructing an image classification system with high performance fusion Dear and Zero method. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of a preferred embodiment of the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method of the present application;

[0042] Figure 2 is a system architecture diagram of DZ in the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method of the present application;

[0043] Figure 3 is a schematic diagram of a single-step training timeline in the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method of the present application;

[0044] Figure 4 is an example diagram of video memory analysis in the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method of the present application;

[0045] Figure 5 is a single-step time consumption comparison diagram of different distributed optimizers on a 2080Ti cluster in the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method of the present application;

[0046] Figure 6 is a single-step time consumption comparison diagram of different distributed optimizers on an A6000 cluster in the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method of the present application;

[0047] Figure 7 is a video memory overhead comparison diagram of different optimization methods on a 2080Ti cluster in the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method of the present application;

[0048] Figure 8is a memory consumption comparison chart of different optimization methods in the image classification method of the fusion Dear and Zero method of the present application on an A6000 cluster;

[0049] Figure 9 is a variation curve diagram of training set loss and validation set Top-1 accuracy in the image classification method of the fusion Dear and Zero method of the present application;

[0050] Figure 10 is a structure diagram of a preferred embodiment of the image classification system of the fusion Dear and Zero method of the present application;

[0051] Figure 11 is a structure diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0052] The present application provides an image classification method, system and terminal of a fusion Dear and Zero method, in order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0053] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood as having meanings consistent with those in the context of the prior art, and unless specifically defined as such, should not be interpreted in an idealized or overly formal sense.

[0054] In addition, if the present application embodiments involve "first", "second" and the like, the "first", "second" and the like are only for description purposes, and should not be understood as indicating or implying the relative importance of the technical features indicated, or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection claimed by the present application.

[0055] The image classification method of the fusion Dear and Zero method of the preferred embodiment of the present application, as Figure 1and Figure 2 As shown, the image classification method of the fusion Dear and Zero method comprises the following steps:

[0056] Step S10, constructing a deep convolutional neural network, collecting historical visual image data, preprocessing the historical visual image data, and taking the preprocessed historical image data as a training data set.

[0057] Specifically, a deep convolutional neural network is constructed, which can be a ResNet50 image classification model (an important member of the Residual Network family, one of the basic architectures in the field of computer vision) to solve the image classification task of the fusion Dear and Zero method.

[0058] Further, historical visual image data (which can be medical image data, road scene image data, or laser radar point cloud projection image, etc.) and corresponding labels are collected. Before training, the input image undergoes a standard preprocessing procedure, including decoding, random cropping, horizontal flipping, and normalization, to enhance the generalization ability of the model. The preprocessed historical image data and corresponding labels are used as a training data set to train the deep convolutional neural network.

[0059] Step S20, deploying the deep convolutional neural network to a distributed training system, and training the deep convolutional neural network on the distributed training system through a DZ optimizer according to the training data set to obtain a target visual neural network.

[0060] It can be understood that as the size of deep learning models continues to expand, their application in high-resolution image classification, target detection, and other computer vision tasks has made unprecedented demands on computing and memory resources. Data Parallelism method has become one of the mainstream distributed training solutions by distributing training data to multiple devices and using All-Reduce to synchronize gradient information. However, in large-scale model training, there are two major challenges: memory occupancy efficiency and communication overhead. For example, when training a classic convolutional neural network like ResNet or an updated Vision Transformer model on a large dataset like ImageNet, memory occupancy and communication overhead become the main bottleneck restricting training efficiency and model performance. Therefore, developing a training method that can optimize memory and communication efficiency simultaneously is crucial for promoting the application of advanced image recognition technology.

[0061] One of the mainstream methods for data parallel memory optimization is to use the ZeRO (Zero Redundancy Optimizer) method, which mainly aims to equally divide the model state among different GPUs. The model state includes the optimizer state, gradient, and model parameters. For the Adam optimizer, the memory overhead of the optimizer state is twice that of the model parameters and gradients. It occupies a large overhead. ZeRO optimizes these three model states step by step, i.e., three stages. Stage 1 saves the equally divided optimizer state on each GPU, and each GPU saves the complete gradient and parameter information. Stage 2 saves the cut optimizer state and gradient on each GPU, and the parameter information is complete. Stage three cuts all three model states. The memory overhead of these stages decreases step by step. However, in terms of communication overhead, ZeRO's 1st and 2nd stages maintain the same communication volume as DDP in PyTorch, but DDP uses a communication overlap mechanism, which makes it have an advantage in single-step time consumption over ZeRO. ZeRO3 increases the communication volume, and the communication overhead is larger.

[0062] The DeAR (Decentralized Autogradient with Resilience) method proposes a more innovative communication optimization strategy, which decouples the All-Reduce primitive into two basic communication operations: Reduce-Scatter and All-Gather. On this basis, DeAR realizes a more fine-grained overlap with the calculation: Reduce-Scatter overlaps with backpropagation, All-Gather overlaps with forward propagation, and Tensor fusion technology is used to aggregate multiple layers of gradient information into larger tensors. This design enables communication operations to achieve more efficient parallelism with forward and backward propagation calculations, greatly improving the utilization of GPU resources. Compared with WFBP and other methods that only overlap backpropagation, DeAR achieves more comprehensive communication-computation overlap by using the forward propagation stage for communication. However, in order to ensure model convergence, this method increases additional memory overhead for model synchronization after tensor fusion, which may cause memory explosion in the case of large models.

[0063] Therefore, the DZ (Dear+Zero) method (DZ optimizer) is obtained by fusing the Dear method and the Zero method, the training process of a model is optimized using the DZ method, and specifically, after initialization of a distributed environment, the DZ optimizer replaces a standard optimizer. It first shards the optimizer state (such as the first moment and the second moment of Adam) and distributes it to each computing node together with model parameter shards, thereby significantly reducing the static memory occupation of a single node. In the training iteration process, DZ hides the communication overhead through its communication and computation overlap mechanism. The DZ method achieves an excellent balance between memory efficiency and computing performance, and provides a practical and efficient solution for large-scale model training.

[0064] Specifically, the method innovatively combines the communication optimization of DeAR and the memory optimization of the 1st and 2nd stages of ZeRO, on the one hand, the optimizer state (such as the first moment and the second moment of Adam) is cut into multiple computing nodes, the redundant memory occupation is significantly reduced, and the gradient is discarded after the gradient is calculated to reduce unnecessary memory overhead; on the other hand, the All-Reduce communication operation is decoupled into Reduce-Scatter and All-Gather, and efficient overlap is realized with backward calculation and forward calculation respectively, and the tensor fusion technology is used to reduce the communication startup overhead.

[0065] Further, the distributed training of the deep convolutional neural network on the distributed training system through the DZ optimizer according to the training data set to obtain the target visual neural network further includes that:

[0066] The DZ optimizer is used for global initialization of the distributed training system once, and the model state of the deep convolutional neural network is dispersed to each data-parallel computing node, the optimizer state sharding and the communication structure establishment are completed, and the static memory occupation of a single node is maximally reduced;

[0067] The model state includes model parameters, gradients and optimizer states.

[0068] In the embodiment, before the formal training starts, the DZ optimizer performs global initialization and state allocation once. The core goal is to disperse the complete model state (including parameters, gradients and optimizer states) to each data-parallel computing node, thereby maximally reducing the static memory occupation of a single node.

[0069] After initialization, the entire distributed training system enters the following stable state:

[0070] Optimizer states are sharded: Corresponding to the parameters, the first and second moment states required by the Adam optimizer are also sharded. Each node is only responsible for maintaining and updating the part of the optimizer state corresponding to its parameter shard. This makes the GPU memory overhead of the optimizer state also evenly divided among the N nodes.

[0071] Communication structure setup: Communication buffers for subsequent gradient Reduce-Scatter and parameter All-Gather, as well as Fused Tensors, have been created and initialized according to the preset strategy (such as the fusion threshold THRESHOLD). Backpropagation and forward propagation hooks are registered. This prepares for efficient and pipelined communication in the training step.

[0072] Further, according to the training data set, the DZ optimizer is used to perform forward propagation on the distributed training system based on a communication and computation overlap mechanism, calculate the parameters of each layer of the deep convolutional neural network, and calculate the forward propagation loss value at the end of the forward propagation.

[0073] The forward propagation based on the communication and computation overlap mechanism on the distributed training system by the DZ optimizer according to the training data set further includes:

[0074] The backpropagation mechanism of the deep convolutional neural network is expanded to obtain a Hook mechanism. The Hook mechanism refers to registering a callback function when a specific position in the computation graph is reached. When backpropagation reaches the specific position, the callback function is automatically called to insert custom operations.

[0075] The Hook mechanism includes a Gradient Hook and a Pre-Forward Hook.

[0076] The Gradient Hook is called after the gradient of a tensor is calculated, and is used to add the gradients of the parameters of each layer to the corresponding communication group.

[0077] The Pre-Forward Hook is called before forward propagation reaches a certain layer, and is used to implement cross-layer parameter synchronization.

[0078] It can be understood that in the single-step training process of data parallelism, the entire data set is first divided into several subsets, which are allocated to different computing nodes. Each node holds a complete copy of the model and independently performs forward propagation and backward propagation operations on its corresponding data subset to calculate the loss function and gradient. When all nodes complete gradient calculation, the All-Reduce operation is used to aggregate and synchronize the gradient. The traditional method is to overlap the All-Reduce communication operation with the backward propagation calculation process, that is, while completing the backward propagation calculation of the current layer, the communication of the layer that has completed the backward propagation is started, thereby effectively reducing the time overhead of calculation and communication. This method has been widely used in multiple deep learning frameworks, such as the distributed data parallel (DDP) in PyTorch (an open-source Python machine learning library) and the one-stage and two-stage of ZeRO in DeepSpeed, which all use the overlap mechanism to accelerate the deep learning training process and improve training efficiency and scalability.

[0079] The DeAR method is to decompose the All-Reduce communication primitive into two basic communication primitives, Reduce-Scatter and All-gather. Then overlap the two communication operations with forward propagation and backward propagation respectively, and the training process is referred to (a) in Figure 3 Taking single-step training as an example, after the L-layer backward propagation calculation is completed, the L-1 layer backward propagation calculation is performed, and at the same time, the Reduce-Scatter communication operation is performed, and there is no coupling relationship between the two, which can be successfully overlapped. The latter layers perform similar operations, and after all the Reduce-scatter operations are performed, each node contains a slice of gradient synchronization information, and then the first layer All-Gather is performed, which can aggregate the gradient information of all nodes to ensure that the gradient information of each node is consistent, and then update the parameters according to the gradient information after All-Gather. After the first layer of All-Gather communication is performed, the parameter update task can be performed, and after completion, the forward calculation is performed, which can be performed at the same time as the second layer All-Gather, realizing the overlap of communication and calculation again. In this way, the communication resources and calculation resources are fully utilized in one training to speed up the training efficiency. DeAR also realizes the tensor fusion mechanism, as shown in (b) in Figure 3 ​where a and b are the communication delay and the communication time of a single element, respectively. With the gradient fusion mechanism, reducing the start-up overhead, i.e., reducing a, can better realize the overlap of computation and communication, improve the computation efficiency, reduce the communication delay, optimize the resource utilization, and improve the training speed, under the condition that the total amount of communication is unchanged.

[0080] Similar to the DeAR method, DZ also adopts the overlap mechanism and the gradient fusion mechanism, as shown in (c) in Figure 3 However, it differs in the communication strategy. Unlike DeAR, which only exchanges gradient information during the communication process, DZ synchronizes the model parameter information in the All-Gather operation. Specifically, after the Reduce-Scatter operation completes the synchronization of gradient information, DZ updates part of the model parameters using the sliced gradient. In this way, each computing node only needs to save its corresponding sliced optimizer state, thereby effectively reducing the redundant memory occupation while ensuring that the training efficiency is not affected.

[0081] In the present embodiment, it is mentioned in the above-mentioned communication and computation overlap strategy that during the backpropagation process, the gradients of each layer need to be aggregated and communicated in groups by Reduce-Scatter. In order to realize this function, and considering that PyTorch adopts a dynamic computation graph design, PyTorch records the calculation dependency relationship between each tensor during forward propagation and constructs a dynamic computation graph. During backpropagation, the system starts from the loss function and automatically calculates the gradient of each tensor in the graph using the chain rule. The present embodiment extends the backpropagation mechanism of PyTorch:

[0082] In order to support the grouped communication of gradients, the present embodiment introduces a set of Hook mechanisms in the backpropagation process of PyTorch. Hook refers to registering a callback function at a certain position in the computation graph, such as the inter-layer boundary of the model. When backpropagation reaches this position, the registered callback function will be automatically called, so that some custom operations can be inserted.

[0083] In this embodiment, the following two types of Hook are mainly used: (1) Gradient Hook: This type of Hook is called after the gradient of a tensor is calculated. This embodiment uses this Hook to add the gradient of each layer parameter to the corresponding communication group, preparing for the subsequent Reduce-Scatter. The specific method is that for each parameter that needs to be reduced, a gradient hook is registered. When the gradient calculation of this parameter is completed, the Hook will put its gradient into the corresponding communication group. Immediately after putting into the communication group, the gradient is released, and the required data of the model is updated by the communication group. When all the gradients in the group are ready, a Reduce-Scatter operation is triggered, and the gradients of the group are reduced between nodes. In this way, the grouped Reduce-Scatter operation is seamlessly embedded into the backpropagation process.(2) Pre-forward Hook: This type of Hook is called before the forward propagation reaches a certain layer. This Hook is used to implement cross-layer parameter synchronization. Specifically, for each layer of the model, this embodiment registers a pre-forward Hook on it. When the forward propagation reaches this layer, if the asynchronous communication condition is met (i.e. the All-Gather of the last iteration has been completed, and the current layer is the first layer after grouping), an All-Gather operation for the parameters of the next layer can be started, thereby aggregating the updated parameter slices on each node into a complete parameter. Whether or not to start communication, the parameters of the current layer need to be updated to achieve parameter synchronization between different parts of the model. In this way, forward propagation and All-Gather operation are also overlapped.

[0084] As can be seen, by inserting these two Hooks in the backpropagation and forward propagation processes of PyTorch, the gradient communication operation is ingeniously integrated into the automatic differentiation process, ensuring correctness and minimizing additional overhead. At the same time, the Hook mechanism gives great flexibility, allowing custom operations to be inserted at various stages of backpropagation as needed without modifying the original training code.

[0085] The forward propagation of the deep convolutional neural network is performed on the distributed training system based on the communication and computation overlap mechanism according to the training data set through the DZ optimizer, and each layer parameter of the deep convolutional neural network is calculated, specifically comprising:

[0086] Before the forward propagation starts, an asynchronous All-Gather communication operation is started to aggregate the updated complete model parameters from each node to each node in the last iteration;

[0087] selecting a layer of the deep convolutional neural network as a target layer, registering a forward propagation pre-hook for the target layer, and performing forward propagation on the distributed training system according to the training data set through the DZ optimizer;

[0088] After the forward propagation reaches the target layer, if the All-Gather of the last iteration is completed and the current layer is the first layer after grouping, an All-Gather operation for the next layer of parameters is started, and the updated parameter slices on each node are aggregated into complete parameters.

[0089] It can be understood that before the forward propagation starts, an asynchronous All-Gather communication operation is started to aggregate the complete model parameters updated in the last iteration from each node to each node. The communication operation is performed in overlap with the next forward calculation, thereby avoiding additional waiting overhead.

[0090] Further, the back propagation is performed on the distributed training system according to the training data set through the DZ optimizer based on the communication and calculation overlap mechanism, the gradient of each layer of parameters is calculated according to the forward propagation loss value, the gradient is put into a buffer group for tensor fusion, when the buffer group is filled, an asynchronous communication is initiated for the buffer group, and the back propagation is continued at the same time as the asynchronous communication.

[0091] Specifically, the back propagation is performed on the distributed training system according to the training data set through the DZ optimizer, and each time a gradient of a parameter is calculated, a gradient hook is called to copy the gradient to a global buffer group for tensor fusion; when the buffer group is filled with gradients, an asynchronous Reduce-Scatter collective communication is initiated for the global buffer group, and the back propagation is continued at the same time as the asynchronous communication to calculate the gradient to fill the next buffer group; when all the back propagation and Reduce-Scatter collective communication are completed, the back propagation is completed, and each node obtains its own gradient slice.

[0092] In this embodiment, the gradient of each parameter is first calculated according to the loss value calculated in the forward direction (in contrast to the forward propagation, the backward propagation is calculated layer by layer from the output layer to the input layer). Whenever the gradient of a parameter is calculated, the backward_hook is called, which is immediately copied to a preset global buffer group (pad_buffer) being filled. When the buffer group is filled, the system immediately initiates an asynchronous Reduce-Scatter collective communication for the group. Since the communication is asynchronous, the calculation unit of the GPU does not need to wait and can continue to calculate the gradient for the previous layers and fill the next buffer group. This makes the time-consuming gradient communication be covered by the calculation, thereby hiding the delay. After all the backward propagation and Reduce-Scatter communication are completed, each node i obtains its own gradient shard (shard_buffers).

[0093] It can be understood that in the backward propagation process, the gradient is divided into several continuous groups according to the layer structure of the model. The gradient in each group is first reduced among the nodes through the Reduce-Scatter communication primitive, and each node is only responsible for aggregating and updating the partial gradient assigned to itself. Similarly, the Reduce-Scatter operation overlaps with the calculation of the local gradient, further reducing the synchronization waiting time. When all the gradient calculations are completed, the main thread calls a synchronization primitive to wait for the completion of all the previously initiated Reduce-Scatter and AllGather communications.

[0094] Further, when the backward propagation and all the asynchronous communications are completed, the gradient shards of each node in the deep convolutional neural network are obtained based on the communication results, and the local optimizer state and the model parameters corresponding to each node are updated according to the gradient shards, and the next round of training is continued until the training round reaches a preset number, thereby completing the distributed training of the deep convolutional neural network and obtaining a target visual neural network.

[0095] The updating of the local optimizer state and the model parameters corresponding to each node according to the gradient shards specifically includes:

[0096] When receiving the message that all the Reduce-Scatter collective communications are completed, the gradient shards of each node are checked and gradient clipping is performed through a preset number of lightweight All-Reduce communications, thereby obtaining target gradient shards;

[0097] The target gradient shards locally held by each node are used to update the corresponding parameter shards and optimizer state shards.

[0098] It can be understood that after all the Reduce-Scatter and AllGather communications are completed, each node updates the local optimizer state and model parameters maintained by itself based on the sub-gradient obtained from the communication results (this step is completely local and has no additional communication overhead), and prepares for the next round of iteration. Before the start of the forward propagation of the next iteration, the asynchronous AllGather is started again, and the training continues until the training round reaches the preset number, the distributed training of the deep convolutional neural network is completed, and the target visual neural network is obtained.

[0099] In general, the splitting strategy of the present application is different from the traditional ZeRO splitting strategy. First, ZeRO usually splits in units of single parameter tensors, while the method of the present application splits in units of module groups combined with a tensor fusion strategy, so that logically related parameters can be processed together; second, ZeRO generates a large number of independent communication operations when dealing with a large number of small parameters, while the method of the present application greatly reduces the number of communications through tensor fusion; third, the implementation of the present application optimizes the parameter prefetch mechanism in the forward propagation process, providing better computation-communication overlap than the ZeRO implementation; fourth, in terms of memory organization, the present application uses continuous buffer and index structure to reduce memory fragmentation and index overhead. Through this fusion design, the method of the present application not only reduces the optimizer state demand to 1 / N of the original as the memory efficiency of the traditional parameter splitting method, but also improves the computing efficiency when dealing with large-scale small parameter models due to the reduction of memory fragmentation and index operations. At the same time, tensor fusion significantly reduces the number of communication operations, and the communication efficiency is also improved. Through the improvement of all aspects, the method of the present application improves the overall training speed compared with the traditional parameter splitting method, and allows the training of larger scale models, providing an effective solution for super large scale model training.

[0100] Step S30, obtaining visual image data to be processed, inputting the visual image data to the target visual neural network for processing, and outputting the classification result of the visual image data.

[0101] Specifically, in computer vision tasks, first, raw visual image data that has not been processed is obtained, which may come from real-time pictures captured by a camera, historical image data, or visual information captured by other devices. Then, the visual image data to be processed is standardized according to the format and preprocessing requirements required by the target visual neural network, including but not limited to adjusting the image size, normalizing the pixel value, performing data enhancement, etc., to ensure that the data can adapt to the network structure.

[0102] Subsequently, the pre-processed visual image data is sequentially input into the trained target visual neural network, which extracts features, analyzes features and understands semantics of the image data through its internal multi-layer convolutional layer, pooling layer, fully connected layer and other structures, and finally outputs the classification result of the visual image data.

[0103] Further, to verify the effectiveness of the scheme, the generated model is evaluated on the ImageNet2012 validation set, and the Top-1 classification accuracy reaches 74.8%. This result reaches the industry benchmark level, which powerfully proves that the DZ method does not sacrifice the final convergence accuracy while greatly improving the training efficiency and reducing the memory occupation. The model produced by the training can be directly used for the image classification task of fusing the Dear and Zero methods, and accurately predicts the category of new input pictures. Therefore, the application example clearly shows how the DZ method supports a complete and efficient technical solution from raw data to a high-performance available model as a core component.

[0104] To evaluate the performance difference between the proposed DZ optimizer and existing distributed training methods, a series of comprehensive experiments are designed. The experiments are conducted in two different GPU cluster environments: (1) NVIDIA RTX 2080Ti cluster, each node equipped with 4 GPU cards; (2) NVIDIA A6000 cluster, each node equipped with 8 GPU cards. PyTorch and NCCL are used as the communication backend for all experiments. A variety of deep learning models with different scales and architectures are selected for testing, including: ResNet50: a convolutional neural network with 23.4 million parameters; Bert-base: a Transformer model with 110 million parameters; Bert-large: a Transformer model with 340 million parameters; GPT2-small: an autoregressive Transformer model with 124 million parameters; GPT2-medium: an autoregressive Transformer model with 355 million parameters; LLama1B: a large language model with 1 billion parameters (only tested on the A6000 cluster). For each model, randomly generated training datasets are used for performance evaluation, and a fixed random seed is used to ensure the reproducibility of the results.

[0105] The application compares the performance of the following five distributed optimizers: DDP (Distributed Data Parallel): the standard data parallel training method of PyTorch; Zero1: the first stage of ZeRO optimizer implemented by DeepSpeed, which shards the optimizer state; Zero2: the second stage of ZeRO optimizer implemented by DeepSpeed, which shards the optimizer state and gradients; Dear: an efficient distributed training method proposed before; DZ (Dear_Zero2): a new method proposed by the application that combines the advantages of Dear and ZeRO.

[0106] Single-step training time (seconds / step) is mainly used as a performance evaluation indicator, which directly reflects the computational efficiency of distributed training. For each configuration, 100 training steps are executed and the first 10 steps are excluded to eliminate the preheating effect, and then the average execution time and standard deviation of the remaining steps are calculated. All experiments are performed under a fixed number of worker nodes (32 GPUs) and batch size (batch size per GPU is 1) to ensure consistency and comparability of the results.

[0107] As Figure 4 shown, in order to more intuitively compare the memory usage between different distributed training methods, the application gives an example diagram of memory analysis. The diagram shows a distributed training system containing two nodes (Node0 and Node1), each equipped with two GPUs. Different colored blocks in the diagram represent different types of memory usage: blue represents model parameters, green represents gradients, yellow represents optimizer state, light red represents padding cache, purple represents gradient cache, orange represents tensor cache, and light blue represents tensor cache. From Figure 2 it can be seen that DDP, as a standard data parallel method, saves complete model parameters, gradients and optimizer state on each GPU, so the memory usage is high. The DeAR method introduces additional padding cache and tensor cache to achieve communication optimization, resulting in an increase in memory overhead compared to DDP. ZeRO1 significantly reduces the memory usage of the optimizer state by splitting the optimizer state, thereby reducing the overall memory overhead. ZeRO2 further splits the gradient information, further reducing the memory usage. The DZ method combines the ZeRO strategy, splits the optimizer state, deletes redundant gradient information, and introduces tensor cache. Compared with DeAR, DZ reduces the memory usage of the optimizer state, but still needs additional tensor cache. When the number of nodes is sufficient, the model state memory usage can be close to the effect of ZeRO1.

[0108] As Figure 5 and Figure 6As shown, the present application compares and analyzes the performance of different distributed optimizers on 2080Ti and A6000 clusters. The experimental results show that Dear and DZ optimizers exhibit significant performance advantages in all test models. As the model complexity increases, this performance difference becomes more pronounced, especially when processing large models such as Bert-large, GPT2_medium, and llama1B, Dear and DZ can reduce the single-step execution time by about 33% compared to Zero1 and Zero2 optimizers. In addition, it is observed that models based on Transformer are more sensitive to optimizer selection compared to CNN architectures (ResNet50). Although there are differences in the absolute performance of the two GPU platforms, the relative performance ranking of the optimizers remains consistent, confirming that these performance differences are due to the algorithm itself rather than hardware characteristics.

[0109] As shown in Figure 7 and Figure 8 , the memory usage characteristics of different optimization methods on various models can be clearly observed. On the 2080Ti cluster, for compute-intensive models such as BERT-Large and GPT-2Medium, traditional DDP and Dear methods exhibit higher memory consumption (about 7GB and 8GB, respectively), while the DZ method proposed by the present application effectively reduces the memory consumption to about 3.5-4GB. The Zero2 method exhibits the best memory efficiency, requiring only about 2.2-3.8GB. In the A6000 cluster test, the optimization effect on large-scale models such as LLaMA-1B is more significant, with the DDP method consuming about 13.5GB of memory, while the DZ method reduces it to about 6.1GB, and the Zero2 further reduces it to 3.9GB. All test models exhibit consistent memory consumption order: DDP ≈ Dear > Zero1 > DZ > Zero2. It is worth noting that the DZ method of the present application is stably positioned between Zero1 and Zero2 in terms of memory efficiency, which indicates that DZ successfully combines the computational efficiency of Dear and the memory optimization capability of Zero methods, providing a good compromise for large-scale distributed training. Compared to the benchmark DDP method, the DZ method can achieve about 45-55% memory savings when processing large models, which is of great significance for model training in resource-constrained environments.

[0110] Overall, the experimental results show that the DZ optimizer can provide excellent performance under various model sizes and batch size configurations, making it particularly suitable for large-scale distributed training scenarios that consider both computational efficiency and memory efficiency.

[0111] As shown in Figure 9As shown in the figure, the curves of training set loss and validation set Top-1 accuracy during the training process are shown. As can be seen from the figure, the whole training process shows good stability. The training loss steadily decreases with the increase of the number of iterations, while the accuracy on the validation set continues to rise steadily, and finally converges. This shows that the communication and computation overlap, optimizer state fragmentation and other strategies adopted by the DZ optimizer do not introduce instability to the gradient flow and parameter update of model training, ensuring a healthy training dynamic.

[0112] After the training is completed, the ResNet50 model finally reaches a Top-1 classification accuracy of 74.8% on the ImageNet2012 validation set. This precision result is comparable to the industry benchmark level that can be achieved by training using standard data parallel methods (such as DDP), which powerfully proves that the DZ method can ensure that the model converges to a high-quality solution while achieving significant performance optimization. This experimental result confirms that the DZ method is not only an efficient training acceleration tool, but also a complete solution that is reliable and can produce high-performance models in practical applications.

[0113] Further, as Figure 10 shown, based on the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method, the present application also correspondingly provides an image classification system of the fusion Dear and Zero method of the fusion Dear and Zero method, wherein the image classification system of the fusion Dear and Zero method of the fusion Dear and Zero method comprises:

[0114] A model construction and data acquisition module 51 is configured to construct a deep convolutional neural network, acquire historical visual image data, pre-process the historical visual image data, and use the pre-processed historical image data as a training data set.

[0115] A distributed training module 52 is configured to deploy the deep convolutional neural network to a distributed training system, perform distributed training on the deep convolutional neural network on the distributed training system according to the training data set through a DZ optimizer, and obtain a target visual neural network, wherein the DZ optimizer is obtained by fusing a Dear method and a Zero method.

[0116] An image classification module 53 of the fusion Dear and Zero method of the visual fusion Dear and Zero method is configured to acquire visual image data to be processed, input the visual image data to the target visual neural network for processing, and output a classification result of the visual image data.

[0117] Further, as Figure 11As shown, based on the image classification method and system of the fusion Dear and Zero method of the fusion Dear and Zero method, the application also correspondingly provides a terminal, which comprises a processor 10, a memory 20 and a display 30. Figure 11 Only part of the components of the terminal is shown, but it should be understood that all the shown components are not required to be implemented, and more or less components can be alternatively implemented.

[0118] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Further, the memory 20 can include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the image classification program 40 of the fusion Dear and Zero method of the fusion Dear and Zero method is stored on the memory 20, which can be executed by the processor 10, so as to realize the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method.

[0119] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, which is used to run program codes or process data stored in the memory 20, such as to execute the image classification method of the fusion Dear and Zero method of the fusion Dear and Zero method, etc.

[0120] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 30 is used to display information of the terminal and to display a visual user interface. The components of the terminal communicate with each other through a system bus.

[0121] In an embodiment, the following steps are implemented when the processor 10 executes the image classification program 40 of the fused Dear and Zero method in the memory 20:

[0122] constructing a deep convolutional neural network, collecting historical visual image data, preprocessing the historical visual image data, and taking the preprocessed historical image data as a training data set;

[0123] deploying the deep convolutional neural network to a distributed training system, performing distributed training on the deep convolutional neural network on the distributed training system according to the training data set through a DZ optimizer, and obtaining a target visual neural network;

[0124] wherein the DZ optimizer is fused from the Dear method and the Zero method;

[0125] obtaining visual image data to be processed, inputting the visual image data to the target visual neural network for processing, and outputting a classification result of the visual image data.

[0126] wherein the obtaining a target visual neural network by performing distributed training on the deep convolutional neural network on the distributed training system according to the training data set through a DZ optimizer further includes:

[0127] performing a global initialization on the distributed training system using the DZ optimizer, and dispersing a model state of the deep convolutional neural network to each data-parallel computing node, to complete optimizer state fragmentation and communication structure establishment, so as to reduce static memory occupation of a single node to the maximum extent;

[0128] wherein the model state includes a model parameter, a gradient, and an optimizer state.

[0129] wherein the obtaining a target visual neural network by performing distributed training on the deep convolutional neural network on the distributed training system according to the training data set through a DZ optimizer specifically includes:

[0130] performing forward propagation on the distributed training system through the DZ optimizer based on a communication and computation overlap mechanism according to the training data set, calculating a parameter of each layer of the deep convolutional neural network, and calculating a forward propagation loss value when the forward propagation ends;

[0131] According to the training data set, through the DZ optimizer, based on the communication and calculation overlapping mechanism, the back propagation is carried out on the distributed training system, the gradient of each layer parameter is calculated according to the forward propagation loss value, the gradient is put into the buffer group of tensor fusion, when the buffer group is filled, an asynchronous communication is initiated to the buffer group, and the back propagation is continued at the same time of the asynchronous communication;

[0132] When the back propagation and all asynchronous communications are completed, the gradient slices of each node in the deep convolutional neural network are obtained based on the communication results, the local optimizer state and the model parameter corresponding to each node are updated according to the gradient slices, and the next round of training is continued until the training round reaches a preset number, so that the distributed training of the deep convolutional neural network is completed, and a target visual neural network is obtained.

[0133] According to the training data set, through the DZ optimizer, based on the communication and calculation overlapping mechanism, the back propagation is carried out on the distributed training system, and before the back propagation, the following is further included:

[0134] The back propagation mechanism of the deep convolutional neural network is expanded to obtain a Hook mechanism, the Hook mechanism refers to registering a callback function at a specific position in a calculation graph, and when the back propagation reaches the specific position, the callback function is automatically called to insert a custom operation;

[0135] The Hook mechanism includes a gradient hook and a forward propagation pre-hook;

[0136] The gradient hook is called after the gradient of a tensor is calculated, and is used to add the gradient of each layer parameter to the corresponding communication group;

[0137] The forward propagation pre-hook is called before the forward propagation reaches a certain layer, and is used to realize the parameter synchronization across layers.

[0138] According to the training data set, through the DZ optimizer, based on the communication and calculation overlapping mechanism, the back propagation is carried out on the distributed training system, and before the back propagation, the following is further included:

[0139] Before the forward propagation starts, an asynchronous All-Gather communication operation is started, and the complete model parameters updated after the last iteration are aggregated from each node to each node;

[0140] A forward propagation pre-hook is registered for a target layer of the deep convolutional neural network, and according to the training data set, the forward propagation is carried out on the distributed training system through the DZ optimizer;

[0141] After the current forward propagation reaches the target layer, if the All-Gather of the last iteration is completed and the current layer is the first layer after grouping, an All-Gather operation for the next layer parameters is started, and the updated parameter slices on each node are aggregated into complete parameters.

[0142] Wherein, according to the training data set, the DZ optimizer is used to perform back propagation on the distributed training system based on the communication and calculation overlap mechanism, calculate the gradient of each layer parameter according to the forward propagation loss value, put the gradient into the buffer group of tensor fusion, and when the buffer group is filled, initiate an asynchronous communication to the buffer group, and continue to perform back propagation at the same time.

[0143] According to the training data set, the DZ optimizer is used to perform back propagation on the distributed training system, and every time a parameter gradient is calculated, a gradient hook is called to copy the gradient to a global buffer group for tensor fusion;

[0144] When the buffer group is filled with gradients, an asynchronous Reduce-Scatter collective communication is initiated to the global buffer group, and at the same time, the back propagation is continued to calculate the gradient to fill the next buffer group;

[0145] When all the back propagation and Reduce-Scatter collective communication are completed, the back propagation is completed, and each node obtains its own gradient slice.

[0146] Wherein, the gradient slice is used to update the local optimizer state and model parameter corresponding to each node, specifically including:

[0147] When receiving the message of all Reduce-Scatter collective communication completion, the gradient slice of each node is checked and gradient clipped through a preset number of lightweight All-Reduce communication, and the target gradient slice is obtained;

[0148] The target gradient slice held locally by each node is used to update the corresponding parameter slice and optimizer state slice.

[0149] The present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a fusion Dear and Zero method image classification program, and the fusion Dear and Zero method image classification program is executed by a processor to realize the steps of the fusion Dear and Zero method image classification method as described above.

[0150] In summary, the present application proposes an image classification method, system, terminal and storage medium of the fusion Dear and Zero method, which combines the communication optimization of DeAR and the video memory optimization of ZeRO innovatively. On the one hand, the optimizer state is divided into multiple computing nodes, which significantly reduces the redundant video memory occupation. After calculating the gradient, it is discarded to reduce unnecessary video memory overhead. On the other hand, the All-Reduce communication operation is decoupled into Reduce-Scatter and All-Gather, and is respectively implemented with backward calculation and forward calculation to achieve efficient overlap. At the same time, the tensor fusion technology is adopted to reduce the communication startup overhead. The DZ method successfully realizes the balance between video memory optimization and computing efficiency, inherits the advantages of efficient calculation and communication overlap of Dear, and integrates the video memory optimization mechanism of ZeRO, which is especially suitable for large-scale model training in resource-constrained environment. This balanced performance makes DZ an effective solution to solve the problem of balancing memory bottleneck and computing efficiency in current deep learning. The subsequent work will further optimize the parameter division strategy and explore the combination with other parallel technologies to support the efficient training needs of larger scale models, improve the efficiency of image processing, and provide a practical and efficient end-to-end solution for constructing an image classification system of the fusion Dear and Zero method.

[0151] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or terminal. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of another identical element in the process, method, article or terminal including the element.

[0152] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the computer program can include the processes of the above-mentioned embodiments of the method. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0153] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall fall within the protection scope of the appended claims of the present application.

Claims

1. An image classification method integrating the Dear and Zero methods, characterized in that, The image classification method that fuses the Dear and Zero methods includes: Construct a deep convolutional neural network, collect historical visual image data, preprocess the historical visual image data, and use the preprocessed historical image data as a training dataset. The deep convolutional neural network is deployed to a distributed training system. Based on the training dataset, the deep convolutional neural network on the distributed training system is trained in a distributed manner using the DZ optimizer to obtain the target visual neural network. The DZ optimizer is obtained by fusing the Dear method and the Zero method. The visual image data to be processed is acquired, the visual image data is input into the target visual neural network for processing, and the classification result of the visual image data is output.

2. The image classification method that fuses the Dear and Zero methods according to claim 1, characterized in that, The step of distributively training a deep convolutional neural network on the distributed training system using a DZ optimizer based on the training dataset to obtain a target visual neural network, prior to which the following steps are also included: The DZ optimizer is used to perform a global initialization of the distributed training system and distribute the model state of the deep convolutional neural network to each data-parallel computing node to complete the optimizer state sharding and communication structure establishment, so as to minimize the static memory usage of a single node. The model state includes model parameters, gradients, and optimizer state.

3. The image classification method that fuses the Dear and Zero methods according to claim 1, characterized in that, The step of distributively training a deep convolutional neural network on the distributed training system using a DZ optimizer based on the training dataset to obtain a target visual neural network specifically includes: Based on the training dataset, the DZ optimizer performs forward propagation on the distributed training system using a communication and computation overlap mechanism to calculate the parameters of each layer of the deep convolutional neural network, and calculates the forward propagation loss value when the forward propagation ends. Based on the training dataset, backpropagation is performed on the distributed training system using the DZ optimizer and based on the communication and computation overlap mechanism. The gradient of each layer parameter is calculated based on the forward propagation loss value, and the gradient is placed into the tensor fusion buffer. When the buffer is full, an asynchronous communication is initiated to the buffer, and backpropagation continues during the asynchronous communication. After backpropagation and all asynchronous communication are completed, the gradient slices of each node in the deep convolutional neural network are obtained based on the communication results. After updating the local optimizer state and model parameters corresponding to each node according to the gradient slices, the next round of training continues until the training rounds reach the preset number, thus completing the distributed training of the deep convolutional neural network and obtaining the target visual neural network.

4. The image classification method that fuses the Dear and Zero methods according to claim 3, characterized in that, The step of performing forward propagation on the distributed training system based on the training dataset and the DZ optimizer, using a communication and computation overlap mechanism, also includes: The backpropagation mechanism of the deep convolutional neural network is extended to obtain the Hook mechanism. The Hook mechanism refers to registering a callback function at a specific position in the computation graph. When the backpropagation reaches the specific position, the callback function is automatically called to insert a custom operation. The hook mechanism includes gradient hooks and forward propagation hooks; The gradient hook is invoked after the gradient of a tensor is calculated, and is used to add the gradients of the parameters of each layer to the corresponding communication group. The forward propagation pre-hook is invoked before the forward propagation reaches a certain layer, and is used to achieve parameter synchronization across layers.

5. The image classification method that fuses the Dear and Zero methods according to claim 4, characterized in that, The step of calculating the parameters of each layer of the deep convolutional neural network based on the training dataset, through the DZ optimizer and based on the communication and computation overlap mechanism in the distributed training system, specifically includes: Before forward propagation begins, an asynchronous All-Gather communication operation is initiated to aggregate the complete model parameters updated in the last iteration from each node to each node. Select a layer of the deep convolutional neural network as the target layer, register a forward propagation hook for the target layer, and perform forward propagation on the distributed training system through the DZ optimizer based on the training dataset. After the forward propagation reaches the target layer, if the All-Gather operation of the previous iteration has been completed and the current layer is the first layer after grouping, then the All-Gather operation for the parameters of the next layer is started, and the updated parameter slices on each node are aggregated into complete parameters.

6. The image classification method that fuses the Dear and Zero methods according to claim 4, characterized in that, The process involves using the DZ optimizer, based on a communication and computation overlap mechanism, to perform backpropagation on the distributed training system using the training dataset. The gradient of each layer's parameters is calculated based on the forward propagation loss value, and this gradient is placed into a tensor fusion buffer. Once the buffer is full, an asynchronous communication is initiated with the buffer, and backpropagation continues simultaneously with this asynchronous communication. Specifically, this includes: Based on the training dataset, the DZ optimizer performs backpropagation on the distributed training system. Whenever the gradient of a parameter is calculated, the gradient hook is invoked to copy the gradient to a global buffer for tensor fusion. Once the buffer group is filled with gradients, an asynchronous Reduce-Scatter collection communication is initiated to the global buffer group. While the asynchronous communication is in progress, backpropagation continues to calculate gradients to fill the next buffer group. Once all backpropagation and Reduce-Scatter collection communication is complete, backpropagation is finished, and each node obtains its own gradient shard.

7. The image classification method that fuses the Dear and Zero methods according to claim 4, characterized in that, The step of updating the local optimizer state and model parameters corresponding to each node according to the gradient partitioning specifically includes: After receiving the message that all Reduce-Scatter collection communication is complete, the gradient shards of each node are checked and the gradients are pruned through a preset number of lightweight All-Reduce communications to obtain the target gradient shards. Update the corresponding parameter slices and optimizer state slices using the target gradient slices held locally on each node.

8. An image classification system that integrates the Dear and Zero methods, characterized in that, The image classification system that fuses the Dear and Zero methods includes: The model building and data acquisition module is used to build a deep convolutional neural network, acquire historical visual image data, preprocess the historical visual image data, and use the preprocessed historical image data as a training dataset. A distributed training module is used to deploy the deep convolutional neural network to a distributed training system. Based on the training dataset, the deep convolutional neural network on the distributed training system is trained in a distributed manner using a DZ optimizer to obtain a target visual neural network. The DZ optimizer is obtained by fusing the Dear method and the Zero method. The image classification module, which integrates the visual fusion Dear and Zero methods, is used to acquire visual image data to be processed, input the visual image data into the target visual neural network for processing, and output the classification result of the visual image data.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and an image classification program that integrates the Dear and Zero methods, stored in the memory and executable on the processor. When executed by the processor, the image classification program that integrates the Dear and Zero methods implements the steps of the image classification method that integrates the Dear and Zero methods as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image classification program that integrates the Dear and Zero methods, which, when executed by a processor, implements the steps of the image classification method that integrates the Dear and Zero methods as described in any one of claims 1-7.

Images