Training method, device, medium and program product of neural network model

Abstract

The specification provides a method, device, medium and program product for training a neural network model, a target network layer of the model registers a callback function for calling a temporarily encapsulated sparsification layer; the method comprises: in a forward propagation process, sequentially performing forward calculation of each network layer; when the target network layer is executed, intercepting an output tensor of the target network layer; calling the sparsification layer to perform the following sparsification processing: shrinking specified non-zero elements in the output tensor to zero through a differentiable soft threshold operator to obtain a sparse tensor, and taking the sparse tensor as an input tensor of a next network layer of the target network layer; a deviation between the specified non-zero elements and an overall distribution level of each non-zero element in the output tensor is greater than a deviation between other non-zero elements in the output tensor and the overall distribution level, and the specified non-zero elements are less than the overall distribution level; in a back propagation process, determining gradients of each network layer, and updating network parameters based on the gradients of each network layer.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of artificial intelligence technology, and more particularly to a method, apparatus, medium, and program product for training a neural network model. Background Technology

[0002] In related technologies, the feature tensors passed between network layers in Artificial Intelligence (AI) models typically contain a large number of feature elements, but only a small subset plays a crucial role in subsequent computations. To reduce computational and storage overhead and improve inference efficiency, activation pruning is often employed. This involves retaining only a few feature elements with large amplitudes from the feature tensors output by the current network layer for computation in the next layer. This approach can effectively maintain model accuracy while compressing computation. However, this operation requires sorting the feature elements, and the sorting process is non-differentiable, preventing gradients from being effectively backpropagated during backpropagation, thus making end-to-end training of the model difficult. Summary of the Invention

[0003] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a method for training a neural network model is proposed. The neural network model includes multiple network layers, and a target network layer among the multiple network layers has a pre-registered callback function. When the callback function is triggered, it calls a temporarily encapsulated sparsity layer to insert the processing of the sparsity layer after the target network layer has finished executing and before the next layer of the target network layer has finished executing. The method includes: During the forward propagation process, the forward computation of each of the multiple network layers is performed sequentially; wherein: When execution reaches the target network layer, the output tensor of the target network layer is intercepted; The callback function is executed to invoke the sparsification layer, causing the sparsification layer to perform the following sparsification process: shrinking a specified non-zero element in the output tensor to zero using a differentiable soft thresholding operator to obtain a sparse tensor, and using the sparse tensor as the input tensor of the next network layer of the target network layer; the deviation between the specified non-zero element and the overall distribution level of each non-zero element in the output tensor is greater than the deviation between other non-zero elements in the output tensor and the overall distribution level, and the specified non-zero element is smaller than the overall distribution level; When the execution reaches the output layer of the neural network model, the loss function is determined based on the output tensor of the output layer; During backpropagation, backpropagation calculations are performed based on the loss function to determine the gradients of each network layer; Based on the gradients of each network layer, update the network parameters of the corresponding network layer.

[0004] According to a second aspect of one or more embodiments of this specification, an image processing method is provided, the method comprising: Obtain the image to be processed; The image to be processed is input into a neural network model; the neural network model includes a visual language model. Obtain the natural language description related to the image to be processed generated by the neural network model; The neural network model is trained based on the method described in the first aspect of one or more embodiments of this specification.

[0005] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method described in the first or second aspect of one or more embodiments of this specification by executing the executable instructions.

[0006] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method described in the first or second aspect of one or more embodiments of this specification.

[0007] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first or second aspect of one or more embodiments of this specification.

[0008] As can be seen from the above embodiments, this specification breaks away from the framework of sorting by comparing numerical values ​​and swapping element indices, and instead adopts a feature importance discrimination mechanism based on the degree of relative deviation. This method temporarily encapsulates a sparsification layer after the target network layer that needs pruning. A callback function is registered to call this sparsification layer, which uses a differentiable soft thresholding operator to shrink non-zero feature elements that deviate significantly from the overall distribution level of the non-zero elements in the output tensor to 0, resulting in a sparse tensor. This reduces the number of effective elements that subsequent network layers need to process, thereby reducing computational complexity and storage overhead, and improving training efficiency. Since the shrinkage operation performed by the sparsification layer is a differentiable operation and does not involve any discrete sorting or hard thresholding operations, the process of pruning the output tensor of the target network layer can be transformed into a differentiable process. Therefore, during backpropagation, the gradient can be propagated from the next network layer of the target network layer to the sparsification layer, and then back to the target network layer, avoiding gradient propagation interruptions caused by non-differentiable sorting operations. This achieves end-to-end training of the entire model, including the target network layer. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of a system architecture provided in an exemplary embodiment.

[0010] Figure 2 This is a flowchart illustrating a training method for a neural network model, provided in an exemplary embodiment.

[0011] Figure 3 This is an exemplary embodiment of a general flowchart.

[0012] Figure 4 This is a flowchart of an iterative process provided in an exemplary embodiment.

[0013] Figure 5 This is a flowchart of an image processing method provided in an exemplary embodiment.

[0014] Figure 6 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0016] Figure 1 A schematic diagram of the system architecture during the neural network model training process is shown. (For example...) Figure 1 As shown, the system architecture includes a processing unit and a storage unit, which interact with each other via a bus or interconnect structure. The processing unit can be a Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), or Neural Network Processing Unit (NPU), etc., used to perform the core computational tasks in the forward propagation and backward gradient update of the neural network. The processing unit integrates a control module, an instruction scheduler, and computational units, such as the Arithmetic Logic Unit (ALU) and Tensor Core within a GPU, or the Systolic Array within a TPU. During training, the processing unit, based on the topology of each layer of the neural network model, reads the input tensor and corresponding network parameters of the current network layer from the storage unit through the control module and instruction scheduler, distributes the data to the internal computational units for processing, and writes the resulting output tensor back to the storage unit as the input for the next network layer. Storage units can include, but are not limited to, off-chip high-bandwidth memory (HBM) or double data rate synchronous dynamic random access memory (DDR SDRAM), as well as on-chip shared memory, registers, and multi-level caches, etc., for hierarchical storage of network parameters, input tensors, intermediate computation results, and output tensors, and reduce access latency and improve overall training throughput through data reuse and memory access optimization.

[0017] In neural network architectures, the number of neurons in a single layer is typically large, resulting in a high-dimensional output tensor. However, only a small subset of these neurons truly play a crucial role in subsequent computations, meaning the importance distribution of elements in the output tensor exhibits sparsity. Therefore, during training, to reduce computational and storage overhead and improve inference efficiency, a common practice is to retain only a few elements with large amplitudes in the current layer's output tensor, setting other elements to zero or discarding them, thus enabling the tensor to participate in the next layer's computation as a sparse tensor. However, this operation usually involves sorting the elements in the output tensor by amplitude. The sorting process (involving numerical comparisons and index permutations) is mathematically non-differentiable, preventing gradients from being effectively backpropagated during backpropagation, making end-to-end training of the model difficult.

[0018] Based on this, this specification proposes a scheme to pre-register forward and backward callback functions for the target network layer of the neural network model, serving as forward hook procedures and backward hook procedures, respectively. During forward propagation, the forward hook procedures are executed to perform operations such as determining the baseline value, calculating the difference tensor, and element shrinkage. These operations are all differentiable; therefore, during backward propagation, the gradient of the target network layer can be directly determined based on the derivative of the loss function with respect to the output tensor of the target network layer, and this gradient is then used for further backward propagation, ultimately achieving end-to-end training of the entire neural network model. The implementation details of this specification's embodiments are illustrated below.

[0019] like Figure 2 and Figure 3 The diagram illustrates a training method for a neural network model according to an embodiment of this specification. The neural network model includes multiple network layers. A target network layer among these layers has a pre-registered callback function. When triggered, this callback function calls a temporarily encapsulated sparsity layer to insert sparsity layer processing after the target network layer has finished executing and before the next layer of the target network layer has finished executing. The method includes: Step S12: During the forward propagation process, the forward computation of each of the plurality of network layers is performed sequentially; wherein: Step S1202: When the target network layer is reached, intercept the output tensor of the target network layer; Step S1204: Execute the callback function to invoke the sparsification layer, so that the sparsification layer performs the following sparsification process: Step S1206: Shrink the specified non-zero elements in the output tensor to zero using a differentiable soft threshold operator to obtain a sparse tensor; the deviation between the specified non-zero element and the overall distribution level of each non-zero element in the output tensor is greater than the deviation between other non-zero elements in the output tensor and the overall distribution level, and the specified non-zero element is smaller than the overall distribution level. Step S1208: Use the sparse tensor as the input tensor of the next network layer of the target network layer; Step S1210: When the output layer of the neural network model is reached, the loss function is determined based on the output tensor of the output layer; Step S14: During backpropagation, backpropagation calculation is performed based on the loss function to determine the gradient of each network layer; Step S16: Update the network parameters of the corresponding network layer based on the gradient of each network layer.

[0020] The neural network models described in this specification can encompass various types of network architectures, which can be selected based on the application scenario and task requirements. For example, a neural network model can be a language model, such as a text-based language model (e.g., BERT, GPT series), used to handle natural language understanding and generation tasks; or a visual language model (e.g., CLIP, Flamingo), used to achieve cross-modal understanding and reasoning between images and text. Furthermore, neural network models can also include, but are not limited to, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and their variants (e.g., Long Short-Term Memory Networks (LSTM), Gated Recurrent Units (GRUs)).

[0021] Neural network models typically consist of multiple network layers stacked according to a predefined topology, forming a hierarchical computational path from input to output. Specifically, a network layer may include an input layer, several hidden layers, and an output layer. The type of hidden layer can be designed according to the task the model performs. For example, hidden layers may include, but are not limited to, fully connected layers, convolutional layers, normalization layers, activation layers, and attention layers. In practical deployment, each network layer processes the input tensor sequentially: the output tensor of the previous network layer becomes the input tensor of the current network layer. After processing by the network parameters and computational logic of the current network layer, a new output tensor is generated and passed to the next network layer, thus propagating forward layer by layer until the output layer. Correspondingly, during backpropagation, the gradient generated by the loss function is propagated back along each network layer to update the parameters of each layer.

[0022] In step S12, sample data can be input into the input layer of the neural network model, which converts the sample data into an input tensor format that meets the model's processing requirements. In some embodiments, the neural network model is a visual language model, and correspondingly, the sample data is image data; or, the neural network model is a text-based language model, and correspondingly, the sample data is text data. Subsequently, according to the topology of the neural network model, the input tensors required by the current network layer are sequentially loaded from storage units (e.g., off-chip high-bandwidth memory or on-chip shared memory) into the cache or registers inside the processing unit; the processing unit calls the corresponding computation unit to perform forward computation according to the type of the current network layer. When the sample data is text data, the input tensor can be text features; when the sample data is image data, the input tensor can be a feature map. For example, for convolutional layers, the processing unit loads the input feature map and corresponding convolutional kernel weights into the computation unit (such as a tensor core in a GPU or a systolic array in a TPU), performs a sliding window-style multiply-accumulate operation, and generates an output feature map. For activation layers, the processing unit calls the arithmetic logic unit to perform a nonlinear transformation on each element of the input tensor (such as the ReLU operation: setting negative values ​​to zero and positive values ​​to remain unchanged). For pooling layers, the computation unit performs a downsampling operation on the maximum or average value. For normalization layers, the computation unit calculates the mean and variance of the input tensor and performs scaling and offsetting. For fully connected layers, the computation unit performs matrix multiplication and addition operations, multiplying the input feature vector with the weight matrix and adding a bias term. After each network layer completes the forward computation, the processing unit writes the generated output tensor back to the storage unit as the input tensor for the next network layer. Simultaneously, this output tensor can be temporarily stored in the on-chip cache for reuse by subsequent layers, reducing off-chip memory access overhead. This process continues layer by layer until the output layer generates the final prediction result, completing the entire forward propagation process.

[0023] Unlike the traditional training process of neural network models, the embodiments in this specification pre-register callback functions for the target network layer of the neural network model. The target network layer can include one or more network layers in the neural network model, and can be flexibly configured according to optimization needs and model structure. For example, for deep neural networks containing dozens or even hundreds of network layers, convolutional layers or fully connected layers with a high computational cost can be selected as the target network layer; while for network layers with lower computational cost, such as normalization layers or activation layers, they can be excluded from the target network layer to maintain their original computational flow and preserve model accuracy. The target network layer can be specified by the developer through configuration files or an Application Programming Interface (API), such as specifying a network layer of a specific number (e.g., the 5th convolutional layer) or a network layer of a specific type (e.g., all convolutional layers) as the target network layer. Furthermore, the selection of the target network layer can also be based on the depth level of the network layer, for example, only the shallow parts (closer to the input layer) or the deep parts (closer to the output layer) of the network model can be identified as target network layers.

[0024] Callback functions can act as hooks, invoked when triggered to temporarily encapsulate a sparsification layer. This allows for the insertion of sparsification processing after the target network layer completes its forward computation and before its output tensor is passed to the next network layer for forward computation. This processing performs custom sparsification on the target network layer's output tensor, shrinking smaller elements to zero. During forward propagation, after the processing unit completes the forward computation of the target network layer, the output tensor is not directly passed to the next network layer but is first input to the sparsification layer. The processing unit calls the forward hook to perform custom sparsification in this sparsification layer, obtaining a sparse tensor. This sparse tensor then serves as the input tensor for the next network layer, continuing the subsequent forward propagation. In this way, the sparsification layer logically acts as a virtual layer inserted between the target network layer and the next network layer, implementing sparsification of the target network layer's output tensor without modifying the internal structure of either the target or next network layer.

[0025] Based on this, when the forward propagation process reaches the target network layer, step S1202 is executed to intercept the output tensor of the target network layer. The intercepted output tensor can be temporarily stored in a storage unit for later retrieval during processing. In related technologies, after the processing unit completes the forward computation of the target network layer, the output tensor generated by that network layer should normally be directly passed to the next network layer as input. However, in this embodiment, through a hook mechanism established by a pre-registered callback function, the control module in the processing unit intercepts the output tensor before it is forward propagated to the next network layer of the target network layer. The intercepted output tensor is temporarily stored in a designated area in the storage unit, such as on-chip shared memory or a buffer allocated in off-chip high-bandwidth memory, for subsequent forward callback functions to read and modify when performing custom sparsification operations. In some embodiments, this temporary storage area can be isolated from the storage area of ​​the normal forward propagation data stream to ensure that the original output tensor is not consumed by the next network layer before the callback function finishes execution, thereby providing a safe operating window for the callback function. Through the aforementioned interception and temporary storage mechanisms, the embodiments of this specification can insert custom processing logic into the forward propagation path of the target network layer, providing support for subsequent implementation of differentiable sparsity processing.

[0026] In step S1204, the processing unit may execute a callback function to invoke the sparsification layer, so that the sparsification layer performs sparsification processing based on a predefined sparsification processing flow. The sparsification processing flow is described below.

[0027] In step S1206, a differentiable soft thresholding operator shrinks specified non-zero elements in the output tensor to zero, resulting in a sparse tensor. Specifically, the deviation between the specified non-zero element and the overall distribution level of all non-zero elements in the output tensor is greater than the deviation between other non-zero elements in the output tensor and this overall distribution level, and the specified non-zero element is smaller than this overall distribution level. In other words, only negative outliers that are below the overall distribution level and deviate significantly from it are selected. Once these two conditions are met, the soft thresholding operator directly shrinks these specified non-zero elements to zero, while other non-zero elements with smaller deviations or larger deviations but values ​​greater than the overall distribution level are retained. Through this mechanism, smaller outliers in the output tensor are sparsified to zero, thus serving to sort and filter features with smaller values.

[0028] In some embodiments, the above-described shrinkage process can be implemented as follows: First, the output tensor of the target network layer is used as the input tensor of the sparsification layer, and a baseline value is determined based on the differentiable central tendency statistic of each non-zero element in the input tensor. Here, a central tendency statistic refers to a statistical measure used to describe the centrality of a set of data in its distribution. Common central tendency statistics include the mean, median, and mode. In neural network training, to ensure that gradients can propagate back through the network layers of the neural network model, the central tendency statistic used needs to be differentiable. A differentiable central tendency statistic is a central tendency statistic that meets the above differentiability requirement. For example, the arithmetic mean, weighted mean, or smoothed quantile of each non-zero element in the input tensor of the target network layer can be determined as a differentiable central tendency statistic. Taking the arithmetic mean of each non-zero element as an example, this differentiable central tendency statistic... It can be written as: ; in, This represents the set of all non-zero elements in the input tensor of the target network layer. This represents the arithmetic mean. In some embodiments, before determining the trend statistics within the differentiable set, the input tensor of the target network layer can also be mapped to a nonnegative space; this process is called energy flow mapping. For example, each element in the input tensor can be directly mapped to its absolute value, or the elements in the input tensor can be mapped using the ReLU function.

[0029] After determining the differentiable central tendency statistic, it can be directly used as the benchmark value, or a weighted average can be applied to the differentiable central tendency statistic, and the weighted average can be used as the benchmark value. The benchmark value reflects the overall level or central tendency of the non-zero elements in the output tensor of the target network layer. It can serve as a threshold for dynamically and adaptively filtering the elements in the output tensor of the target network layer. Therefore, the benchmark value can also be called the global suppression field.

[0030] Then, the deviations (also known as competition dissipation) between each non-zero element in the input tensor and the benchmark value can be determined to obtain the difference tensor. The deviation between any non-zero element and the benchmark value is positively correlated with the magnitude of the non-zero element itself. Therefore, by calculating the deviations, non-zero elements with larger values ​​can be screened out, thereby identifying key elements that contribute significantly to the computation of subsequent network layers.

[0031] Next, a sparse tensor can be generated by shrinking non-zero elements in the difference tensor whose deviation is less than a preset threshold (e.g., 0) to 0 using a differentiable soft thresholding operator. The soft thresholding operator performs a predefined nonlinear transformation on the difference tensor. Through this nonlinear transformation, elements greater than or equal to the preset threshold remain unchanged, while elements less than the preset threshold are shrunk to 0. A differentiable soft thresholding operator is one that is differentiable. In some embodiments, the differentiable soft thresholding operator can be a ReLU operator, where, for any input parameter x, the output parameter obtained after processing by the ReLU operator can be denoted as: .

[0032] Based on the aforementioned differentiable soft thresholding operator, the difference tensor It can be written as: ; in, The weighting parameter is used to weight the trend statistics of differentiable ensembles. This weighting parameter can be a learnable parameter (scalar) in the sparse layer, and it can be adjusted during the training of the neural network model. Used to characterize the shrinkage strength for non-zero elements. The larger the value, the greater the contraction strength, and the fewer non-zero elements are ultimately retained. This embodiment addresses this by setting learnable weight parameters. Compared to using a fixed weight approach, this allows the neural network model to adaptively determine the number of features to retain, thereby enabling on-demand computation and end-to-end optimization.

[0033] In step S1208, the sparse tensor can be used as the input tensor of the next network layer of the target network layer, and forward propagation calculation can continue until the output tensor of the output layer of the neural network model is obtained.

[0034] In step S1210, when the forward propagation reaches the output layer of the neural network model, the loss function can be determined based on the output tensor of that output layer. The type of loss function can be designed according to the task performed by the neural network model; for example, it can be a cross-entropy loss function, a mean squared error loss function, etc.

[0035] At this point, the forward propagation process is complete. In step S14, the backward propagation process can begin. During backward propagation, the processing unit starts with the loss function L and traverses each network layer in reverse order of the forward propagation, beginning from the Nth layer (output layer). Each network layer needs to complete two tasks: first, calculate the gradient of the loss function with respect to its own learnable parameters; and second, continue to extrapolate the gradient forward. The following explanation uses a neural network model containing N (N is a positive integer) network layers as an example to illustrate this process.

[0036] First, calculate the loss function with respect to the learnable parameters of the Nth layer (such as weight parameters). and bias parameters gradient of ) and The processing unit, according to the chain rule, uses the loss function L to process the output tensor of the Nth layer. The gradient and the local derivatives of the output with respect to the learnable parameters in the forward computation of this layer are calculated using the following formula: .

[0037] Simultaneously, the processing unit calculates the loss on the Nth layer input tensor according to the chain rule. gradient This gradient is used as an error signal and is propagated back to the previous network layer (i.e., the (N-1)th layer). The calculation formula is as follows: .

[0038] Next, the error signal is propagated from layer N to layer N-1. The error signal is treated by the (N-1)th layer as the gradient of the loss function on its output tensor, i.e.: .

[0039] Then, obtain the learnable parameters of the loss function for the (N-1)th layer (such as weight parameters). and bias parameters gradient of ) and The specific method can be found in layer N, and will not be repeated here. This process is repeated layer by layer in reverse until the target network layer is reached.

[0040] When backpropagating to the target network layer, it can intercept error signals that should originally be directly propagated back to the target network layer from the next layer below it. Specifically, assuming the target network layer is the m-th layer of the neural network model, without sparsification layers, the error signal calculated by the (m+1)-th layer (i.e., the next layer after the target network layer) during backpropagation... It will be directly used as the gradient of the output tensor of the m-th layer. This is used to calculate the gradient of the learnable parameters of the m-th layer. In this embodiment, because a temporary sparsity layer is inserted between the target network layer and the next network layer during forward propagation, the error signal propagation path during backpropagation is changed accordingly: the processing unit intercepts the error signal returned from the (m+1)-th layer and uses it as the output gradient of the sparsity layer. Then, the processing unit, also according to the chain rule, determines the gradient of the loss function with respect to the learnable parameters of the sparsity layer based on the gradient of the loss function with respect to the output parameters of the sparsity layer and the gradient of the output parameters of the sparsity layer with respect to the learnable parameters of the sparsity layer, and determines the gradient of the loss function with respect to the input tensor of the sparsity layer. This gradient is returned to the target network layer as an error signal, serving as the gradient of the loss function with respect to the output tensor of the target network layer. Subsequent processing is similar to that of the Nth network layer and will not be described in detail here.

[0041] The following example illustrates the calculation process of the gradient of the sparsification layer.

[0042] In some embodiments, the learnable parameters of the sparsity layer may include weight parameters; therefore, the gradient of the loss function with respect to the learnable parameters of the sparsity layer may include the gradient of the loss function with respect to the weight parameters of the sparsity layer.

[0043] In other embodiments, during forward propagation, sparsification of the target network layer's output tensor is achieved by shrinking non-zero elements with deviations less than a preset threshold in the difference tensor to zero. This result serves as the input tensor for the next network layer. Since this shrinkage operation alters the numerical distribution of the original output tensor while retaining some elements, the statistical scale (e.g., mean or amplitude range) of the next network layer's input tensor undergoes a compressive shift compared to the original output of the target network layer. This may affect the computational stability and model convergence performance of subsequent network layers. Therefore, a learnable compensation parameter can be introduced into the sparsification layer to compensate for the sparse tensor (i.e., scale recovery). This compensation operation, using multiplication or addition, maintains statistical consistency between the numerical distribution of the sparse tensor and the distribution of the target network layer's output tensor, reducing the adverse effects of the distribution shift introduced by sparsification on model training. This process is called energy conservation compensation. The compensation process in some embodiments is as follows: ; in, The output tensor of the sparse layer. To compensate for parameters, It is a sparse tensor.

[0044] Based on this, the learnable parameters of the sparsity layer can include the aforementioned compensation parameters, and correspondingly, the gradient of the loss function with respect to the learnable parameters of the sparsity layer can include the gradient of the loss function with respect to the compensation parameters.

[0045] In some embodiments, such as Figure 4 As shown, sparsity can also be achieved through multiple iterations, a process known as dynamic screening loops. Specifically, in the first iteration, the output tensor of the target network layer is used as the input tensor for the sparsification layer in the first iteration. Subsequently, the baseline value for the first iteration is determined based on the differentiable lumbar trend statistics of the non-zero elements in the input tensor during the first iteration; the deviation between each non-zero element in the input tensor during the first iteration and the baseline value is determined to obtain the difference tensor for the first iteration; and the non-zero elements with deviations less than a preset threshold in the difference tensor of the first iteration are shrunk to 0 using a differentiable soft threshold operator to generate the sparse tensor for the first iteration.

[0046] In the (i+1)th iteration, the sparse tensor obtained in the ith iteration can be used as the input tensor for the (i+1)th iteration of the sparsification layer. Subsequently, the baseline value for the (i+1)th iteration can be determined based on the differentiable maxima of the non-zero elements in the input tensor of the (i+1)th iteration. The deviations between each non-zero element in the input tensor of the (i+1)th iteration and the baseline value are determined to obtain the difference tensor for the (i+1)th iteration. Finally, the non-zero elements in the difference tensor of the (i+1)th iteration with deviations less than a preset threshold are shrunk to 0 using a differentiable soft threshold operator, generating the sparse tensor for the (i+1)th iteration. Here, i is a positive integer.

[0047] The above process draws on the idea of ​​dynamic systems, treating each element as a state variable that evolves with time or iteration step size. By constructing an iterative update equation, strong signals are preserved while weak signals gradually decay to zero, thus replacing the traditional non-differentiable sorting operation with a differentiable dynamic evolution method.

[0048] In some embodiments, a maximum number of iterations can be preset. Once the maximum number of iterations is reached, iteration stops; otherwise, the next iteration begins. In other embodiments, after each iteration, the proportion of non-zero elements in the sparse tensor obtained in that iteration can be determined. If this proportion is less than a preset threshold, that iteration is determined as the last iteration and iteration stops; otherwise, the next iteration begins. Alternatively, the current iteration can be determined as the last iteration if either the number of iterations reaches a preset threshold or the proportion of non-zero elements in the sparse tensor obtained in that iteration is less than a preset threshold. If neither of these conditions is met, the next iteration begins. The sparse tensor obtained in the last iteration can be used as the input tensor of the next network layer of the target network layer. Further, in the example where compensation parameters are used to compensate the sparse tensor, compensation can be applied only to the sparse tensor obtained in the last iteration, and the compensated sparse tensor can be used as the input tensor of the next network layer of the target network layer.

[0049] In the embodiment employing multiple iterations, the differentiable central tendency statistic at iteration t can be denoted as: ; in, Let be the differentiable axial trend statistic at iteration t, which can be abbreviated as , Let t be the set of non-zero elements in the input tensor of the sparsified layer at the t-th iteration. When t equals 1, the input tensor of this iteration is the output tensor of the target network layer; when t is greater than 1, the input tensor of this iteration is the sparse tensor obtained in the previous iteration.

[0050] The sparse tensor at the (t+1)th iteration can be denoted as: ; The final output tensor of the sparsification layer can be denoted as: ; This output tensor It can be used as the input tensor of the next network layer after the target network layer. This is the sparse tensor output during the last iteration, and T is the preset maximum number of iterations.

[0051] In an embodiment employing multiple iterations, the gradient of the sparsification layer can be calculated for each iteration, and the gradient can be backtracked from the last iteration to the first iteration.

[0052] During the backpropagation phase, assume the gradient propagated back from the next network layer after the target network layer to the output of the sparse layer is... For the compensation parameter G, since the last step of the forward propagation is According to the chain rule, the gradient of G is obtained by passing through... The weighted sum of global gradients, i.e., for the last iteration, determines the output tensor of the sparsed layer after that iteration (i.e., the loss function). The gradients of each non-zero element in the tensor are used to calculate the gradients of the sparse tensor after the iteration (i.e., ...). The gradient is weighted by each non-zero element in the formula, and the weighted gradients are summed to obtain the gradient of the loss function with respect to the compensation parameter G. This process can be written as: ; Where Q is the total number of non-zero elements in the output tensor of the sparsification layer. Let be the eigenvalue of the i-th non-zero element in the sparse tensor obtained from the last iteration of the sparsification layer. It is the i-th non-zero element in the output tensor of the sparse layer.

[0053] In multiple iterations (rounds t=1, 2, ..., T), the difference tensor . The gradient is the feedback accumulated across all iterations and all non-zero elements. That is, for each iteration, the gradient of the loss function over each non-zero element in the difference tensor of that iteration can be obtained separately. Based on the trend statistics of the differentiable ensemble of that iteration, the gradients of each non-zero element in the difference tensor are weighted, and the weighted gradients over each non-zero element are summed to obtain the result. The gradient in that iteration, then, for The gradients in each iteration are summed to obtain... The gradient. Specifically, it can be written as: ; in, Let i be the i-th non-zero element in the difference tensor of the t-th iteration. Let be the differentiable central tendency statistic for the t-th iteration.

[0054] After obtaining the two gradients mentioned above, the sparsification layer will call the optimizer (such as Adam or SGD) to update the learning rate according to the preset learning rate. The parameters G and G enable the operator to adaptively learn the optimal value. And G.

[0055] In order to backpropagate the gradient to the target network layer, it is necessary to calculate the loss function L for the input of the sparsified layer. The gradient is calculated by working backwards from round T to round 1 inside the sparsity layer. The gradient of the loss function with respect to the output tensor of the sparsity layer can be obtained, and this gradient can be weighted by the compensation parameter G to obtain the initial gradient entering the iteration loop from the output layer, specifically denoted as: ; For any iteration t, given the gradient of the next iteration... The difference tensor of the current round The gradient can be determined by adjusting the gradient of the next round using the compensation parameter G. Perform weighted gradation, and calculate the derivative of the weighted gradient with respect to the differentiable soft thresholding operator (for example, when the differentiable soft thresholding operator is ReLU, its derivative is an indicator function, denoted as ). The Hadamard product between the two is used as the difference tensor. The gradient. Specifically, it can be written as: .

[0056] because This caused all non-zero elements to be mutually coupled. The received gradients include not only the paths directly propagated backward (i.e., the intermediate tensors of the loss function for this iteration) The gradient), also includes through The generated global feedback gradient can be determined as follows: Obtain the gradient of the loss function for each element in the intermediate tensor of this iteration; then, use the mask of each element (the mask is 1 if the element is greater than 0, and 0 otherwise) and the weight parameter to perform a weighted sum of the gradients of the corresponding elements to obtain the global feedback gradient. Specifically, it can be denoted as: ; in, This is a mask to determine whether the j-th element is greater than 0. Let Q be the total number of non-zero elements in the output tensor at the current iteration, and let Q be the total number of elements in the intermediate tensor. In related technologies, once a feature is not selected or is set to zero at a certain stage, its backpropagation gradient immediately becomes zero, causing the model to be unable to correct the eliminated features (i.e., the "dead neuron" problem). However, in the embodiments of this specification, according to the above formula, even if an element j fails to compete after the t-th iteration, resulting in... (i.e., direct gradient path) (Blocked), because it still satisfies at the beginning of round t. And participated The element can still be calculated through The following partial derivatives receive negative feedback from the global gradient: .

[0057] This means that even if a feature is eliminated during the iteration, the model can still calculate a reasonable compensating gradient for it and propagate it back to the target network layer. This completely breaks the technical bias of traditional sorting or selection algorithms that "the gradient breaks once eliminated," greatly improving the stability and optimization ability of the model during end-to-end training.

[0058] When the reverse iteration within the sparsified layer reaches the first layer, the gradient of the loss function relative to the output tensor of the sparsified layer at the time of the first iteration is obtained, i.e. At this point, the output of the target network layer can be directly obtained. The gradient is: ; Then, the gradients of the learnable parameters of the target network layer can be calculated using the standard chain rule of calculus, assuming its internal calculation logic is as follows: ,in, For the output tensor of the target network layer, For the input tensor of the target network layer, For the weight parameters of the target network layer, Let the bias parameters be those of the target network layer, then: .

[0059] After obtaining the gradients of the learnable parameters of each network layer, in step S16, the learnable parameters of the corresponding network layer can be adjusted using the gradients of the learnable parameters of each network layer. Because the operator output provided in this application... It is precise and continuous, and the target network layer can accurately know "which features are generated too weakly and need to be strengthened, and which features are generated too strong and need to be suppressed" based on the final loss function, thus achieving end-to-end training.

[0060] The above process controls the threshold of the number of iterations or the intensity of feature shrinkage through learnable weight parameters, so that the number of non-zero elements decreases dynamically as the number of iterations increases, eventually approaching the expected number of significant features retained (i.e., achieving differentiable Top-K screening).

[0061] like Figure 5 As shown in the embodiments of this specification, an image processing method is also provided, the method comprising: Step S22: Obtain the image to be processed; Step S24: Input the image to be processed into the neural network model; the neural network model includes a visual language model; Step S26: Obtain the natural language description related to the image to be processed generated by the neural network model; The neural network model is trained based on the method described in any of the foregoing embodiments.

[0062] The method in this embodiment can be used in image question-and-answer scenarios.

[0063] In step S22, the image to be processed can be an image captured by an image acquisition device on an electronic device (such as a mobile phone, tablet computer, etc.), or an image pre-stored on the electronic device. After acquiring the image to be processed, it can also be decoded to convert it into a standard tensor format (such as RGB or grayscale matrix), and normalization processing (such as mean-variance normalization) can be performed to adapt to the input requirements of the neural network model.

[0064] In step S24, the electronic device can input the preprocessed tensor into a neural network model, which can be a visual language model. This neural network model can be deployed locally on the electronic device or in the cloud, connected to the electronic device via a network. Furthermore, the electronic device can also input text information into the neural network model. This text information can be acquired through an audio acquisition device on the electronic device, input by a user through the user input interface of an image acquisition device, or pre-configured as default information in the electronic device. In some embodiments, the text information is a question about the image to be processed, such as a query about the category of a subject in the image. In some embodiments, the neural network model can employ a dual-encoder architecture of a visual encoder and a language encoder. The visual encoder extracts image feature sequences using a Transformer or convolutional neural network (e.g., ViT, ResNet), while the language encoder processes the text input (such as user question text) based on a BERT / GPT architecture.

[0065] In step S26, the neural network model can generate a natural language description related to the image to be processed. Specifically, the decoder in the neural network model can decode based on the features output by the encoder to obtain a sequence of natural language descriptions containing semantic information. For example, the natural language description can be the recognition result obtained by the neural network model after recognizing the subject in the image to be processed. After obtaining the natural language description, the neural network model can return the natural language description to the electronic device and play it through the audio playback device on the electronic device, or display it through the display interface on the electronic device.

[0066] Figure 6 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 6 As shown, device 400 mainly consists of a communication interface 402, a user interface 404, a processor 406, and a data storage 408. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 410. The communication interface 402 enables device 400 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 402 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 402 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 402 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 402 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0067] User interface 404 includes receiving user input and providing output to the user. Therefore, user interface 404 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 404 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 404 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 400 may support remote access from other devices via communication interface 402 or another physical interface (not shown). User interface 404 may be configured to receive user input, the position and movement of which may be indicated by an indicator or cursor described herein. User interface 404 may also be configured as a display device for rendering or displaying text fragments.

[0068] Processor 406 may contain one or more general-purpose processors and / or special-purpose processors.

[0069] Data storage 408 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 406. Data storage 408 may include removable and non-removable components.

[0070] Processor 406 is capable of executing program instructions 418 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 408 to perform the various functions described herein. Data storage 408 may comprise a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 400, enable device 400 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 406 executing program instructions 418 may result in processor 406 using data 412.

[0071] For example, program instructions 418 may include an operating system 422 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 400 and one or more applications 420 (e.g., a browser, social application, or game application). Similarly, data 412 may include operating system data 416 and application data 414. Operating system data 416 is primarily accessible to the operating system 422, while application data 414 is primarily accessible to one or more applications 420. Application data 414 may reside in a file system visible or hidden from the user of device 400.

[0072] Application 420 can communicate with operating system 422 through one or more application programming interfaces (APIs). These APIs help application 420 read and / or write application data 414, transmit or receive information via communication interface 402, receive or display information on user interface 404, etc.

[0073] In some terminology, application 420 may be simply referred to as "app". Furthermore, application 420 can be downloaded to device 400 through one or more online app stores or app markets. However, applications can also be installed on device 400 in other ways, such as through a web browser or a physical interface on device 400 (e.g., a USB port).

[0074] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0075] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0076] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0077] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0078] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0079] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0080] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0081] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0082] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0083] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A method for training a neural network model, the neural network model comprising multiple network layers, wherein a target network layer among the multiple network layers has a pre-registered callback function, and when the callback function is triggered, it calls a temporarily encapsulated sparsity layer to insert the processing of the sparsity layer after the target network layer has finished executing and before the next layer of the target network layer has finished executing; the method includes: During the forward propagation process, the forward computation of each of the multiple network layers is performed sequentially; wherein: When execution reaches the target network layer, the output tensor of the target network layer is intercepted; The callback function is executed to invoke the sparsification layer, causing the sparsification layer to perform the following sparsification process: shrinking a specified non-zero element in the output tensor to zero using a differentiable soft thresholding operator to obtain a sparse tensor, and using the sparse tensor as the input tensor of the next network layer of the target network layer; the deviation between the specified non-zero element and the overall distribution level of each non-zero element in the output tensor is greater than the deviation between other non-zero elements in the output tensor and the overall distribution level, and the specified non-zero element is smaller than the overall distribution level; When the execution reaches the output layer of the neural network model, the loss function is determined based on the output tensor of the output layer; During backpropagation, backpropagation calculations are performed based on the loss function to determine the gradients of each network layer; Based on the gradients of each network layer, update the network parameters of the corresponding network layer.

2. The method according to claim 1, wherein shrinking a specified non-zero element in the output tensor to zero using a differentiable soft thresholding operator to obtain a sparse tensor comprises: Use the output tensor of the target network layer as the input tensor of the sparsification layer; The baseline value is determined based on the differentiable central tendency statistics of each non-zero element in the input tensor. The deviations between each non-zero element in the input tensor and the reference value are determined to obtain the difference tensor; The non-zero elements in the difference tensor with deviations less than a preset threshold are shrunk to 0 by using a differentiable soft threshold operator to generate a sparse tensor.

3. The method according to claim 2, wherein the sparsification process includes multiple rounds of iterative processing; wherein: In the first iteration, the output tensor of the target network layer is used as the input tensor of the sparsification layer; In the (i+1)th iteration, the sparse tensor output in the i-th iteration is used as the input tensor of the sparsification layer; i is a positive integer; The sparse tensor obtained in the last iteration is used as the input tensor of the next network layer of the target network layer.

4. The method according to claim 3, further comprising: After any iteration, if the number of iterations reaches a preset threshold, or if the proportion of non-zero elements in the sparse tensor obtained in this iteration is less than a preset threshold, this iteration will be determined as the last iteration. Otherwise, proceed to the next iteration.

5. The method according to claim 3, wherein the sparsification layer includes learnable weight parameters to characterize the shrinkage strength of non-zero elements; the baseline value of any iteration is obtained by weighting the differentiable central tendency statistics of the current iteration with the weight parameters of the current iteration.

6. The method according to claim 5, wherein the weight parameter of any iteration is positively correlated with the number of iterations in this iteration.

7. The method of claim 1, wherein the sparsification layer further comprises learnable compensation parameters; and the method further comprises, before using the sparse tensor as the input tensor of the next network layer of the target network layer: The sparse tensor is compensated based on the compensation parameters.

8. The method according to claim 1, further comprising, before determining the benchmark value based on the differentiable axial trend statistics of each non-zero element in the input tensor: Map the input tensor to non-negative values.

9. An image processing method, the method comprising: Obtain the image to be processed; The image to be processed is input into the neural network model; The neural network model includes a visual language model; Obtain the natural language description related to the image to be processed generated by the neural network model; The neural network model is trained based on the method described in any one of claims 1 to 8.

10. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-9 by executing the executable instructions.

11. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

12. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-9.

Images