Field-oriented generalized gradient adaptive quantization perception training method, device and equipment

Abstract

The application provides a field generalization-oriented gradient adaptive quantization perception training method, device and equipment, relates to the technical field of computers, and aims to improve the generalization capability of a quantization model in unknown fields. The method comprises the following steps: acquiring a training data set and a quantization convolutional neural network model; inputting each sample image in the training data set into the quantization convolutional neural network model for processing, so as to obtain task gradients and smooth gradients of each scale factor in different training domains; evaluating the gradient confusion degree of the task gradients of each scale factor in different training domains, wherein the gradient confusion degree represents the consistency degree of the direction of the task gradients; determining the learning gradients of each scale factor in different training domains according to the gradient confusion degree, and updating the parameters of the quantization convolutional neural network model based on the learning gradients, so as to obtain a trained quantization convolutional neural network model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a gradient adaptive quantization perception training method, device and equipment for domain generalization. BACKGROUND

[0002] Deep learning models perform well in various computer vision tasks, such as classification tasks, object detection tasks, and semantic segmentation tasks. In practical applications, these models often show significant performance degradation due to domain shift, which means poor generalization ability for previously unseen data distribution. Domain generalization aims to solve this problem by enabling a model trained on a known source domain to effectively generalize to an unknown target domain.

[0003] However, existing domain generalization schemes are only applicable to full-precision models. In many real-world scenarios, especially in environments involving edge device deployment and susceptible to domain shift, models often need to run under very limited computing resources. Therefore, the models generated based on the current domain generalization scheme do not have high practicality in actual deployment. The low-precision computing method commonly used by edge devices (i.e., quantization method) only considers the same distribution assumption, and the application of the current domain generalization scheme to the quantization model does not work well. Due to the distribution shift of unseen data in actual applications, the generalization performance of the quantization model cannot be guaranteed.

[0004] Therefore, how to improve the generalization ability of the quantization model in the unknown domain is a technical problem to be solved. SUMMARY

[0005] In view of the above problems, the embodiments of the present application provide a gradient adaptive quantization perception training method, device and equipment for domain generalization, so as to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect of the embodiments of the present application, a gradient adaptive quantization perception training method for domain generalization is disclosed, and the method comprises:

[0007] obtaining a training data set and a quantization convolutional neural network model, the training data set comprising a plurality of sample images of equal amount of different training domains;

[0008] inputting each sample image in the training data set into the quantization convolutional neural network model for processing to obtain task gradients and smooth gradients of each scale factor in different training domains, the scale factor representing a feature of weight and activation value distribution of the quantization convolutional neural network model, the task gradient being used to optimize the task performance of the quantization convolutional neural network model, and the smooth gradient being used to optimize the generalization of the quantization convolutional neural network model;

[0009] evaluate gradient confusion degrees of the task gradients of the respective scale factors in different training domains, the gradient confusion degrees representing consistency degrees of directions of the task gradients;

[0010] determine learning gradients of the respective scale factors in different training domains according to the gradient confusion degrees, and update parameters of the quantized convolutional neural network model based on the learning gradients, to obtain a trained quantized convolutional neural network model, the learning gradients at least including the smoothed gradients.

[0011] Optionally, the evaluating of the gradient confusion degrees of the task gradients of the respective scale factors in different training domains comprises:

[0012] evaluate the gradient confusion degrees of the task gradients of the respective scale factors in different training domains every target number of training steps;

[0013] determine learning gradients of the respective scale factors in different training domains according to the gradient confusion degrees, and update parameters of the quantized convolutional neural network model based on the learning gradients, comprises:

[0014] determine learning gradients of the respective scale factors in different training domains within a next target number of training steps according to the gradient confusion degrees, and update parameters of the quantized convolutional neural network model based on the learning gradients within the next target number of training steps.

[0015] Optionally, the evaluating of the gradient confusion degrees of the task gradients of the respective scale factors in different training domains every target number of training steps comprises:

[0016] determine a task gradient sequence according to the task gradients of the scale factors in the training domains within the target number of training steps; wherein the task gradient of the scale factor in the training domain within each training step is one task gradient in the task gradient sequence;

[0017] obtain the gradient confusion degrees according to direction changes of adjacent two task gradients in the task gradient sequence.

[0018] Optionally, the task gradient sequence comprises a first task gradient sequence and a second task gradient sequence, and two task gradients with the same order in the first task gradient sequence and the second task gradient sequence represent two task gradients of adjacent training steps;

[0019] obtain the gradient confusion degrees according to direction changes of adjacent two task gradients in the task gradient sequence, comprises:

[0020] sequentially compare directions of two task gradients with the same order in the first task gradient sequence and the second task gradient sequence, to determine a number of two task gradients of adjacent training steps in different directions;

[0021] a ratio of the number and the number of the target training steps as the gradient chaos degree.

[0022] Optionally, according to the gradient chaos degree, determining the learning gradient of each scale factor in different training domains comprises:

[0023] for each scale factor in each training domain, if the gradient chaos degree of the task gradient of the scale factor in the training domain is less than or equal to a gradient chaos degree threshold, determining the smoothed gradient as the learning gradient of the scale factor in the training domain;

[0024] if the gradient chaos degree of the task gradient of the scale factor in the training domain is greater than the gradient chaos degree threshold, determining the smoothed gradient and the task gradient as the learning gradient of the scale factor in the training domain.

[0025] Optionally, inputting each sample image in the training data set to the quantized convolutional neural network model for processing to obtain the task gradient and the smoothed gradient of each scale factor in different training domains comprises:

[0026] constructing a loss function according to the quantization result of the quantized convolutional neural network model and the sample image, the loss function comprising a task loss and a smoothed loss;

[0027] determining the task gradient of each scale factor in different training domains according to the task loss, and determining the smoothed gradient of each scale factor in different training domains according to the smoothed loss.

[0028] A second aspect of the embodiments of the present application discloses an image processing method, the method comprising:

[0029] obtaining an image to be processed;

[0030] inputting the image to be processed into a trained quantized convolutional neural network model to obtain a target visual task result, the trained quantized convolutional neural network model being trained according to the gradient self-adaptive quantization perception training method for domain generalization of the first aspect of the embodiments of the present application.

[0031] A third aspect of the embodiments of the present application discloses a gradient self-adaptive quantization perception training device for domain generalization, the device comprising:

[0032] an obtaining module configured to obtain a training data set and a quantized convolutional neural network model, the training data set comprising a plurality of sample images of equal amounts of different training domains;

[0033] a quantization module configured to input each sample image in the training data set into the quantized convolutional neural network model for processing to obtain a task gradient and a smoothing gradient of each scale factor in different training domains, the scale factor representing a feature of a distribution of a weight and an activation value of the quantized convolutional neural network model, the task gradient being used to optimize a task performance of the quantized convolutional neural network model, and the smoothing gradient being used to optimize a generalization of the quantized convolutional neural network model;

[0034] an evaluation module configured to evaluate a gradient confusion degree of the task gradient of each scale factor in different training domains, the gradient confusion degree representing a consistency degree of a direction of the task gradient;

[0035] an updating module configured to determine a learning gradient of each scale factor in different training domains according to the gradient confusion degree, and update parameters of the quantized convolutional neural network model based on the learning gradient to obtain a trained quantized convolutional neural network model, the learning gradient at least including the smoothing gradient.

[0036] In a third aspect, an electronic device is disclosed, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the field generalization-oriented gradient adaptive quantization-aware training method according to the first aspect of the present application or the steps of the image processing method according to the second aspect of the present application when executing the computer program.

[0037] In a fourth aspect, a computer-readable storage medium is disclosed, which stores a computer program, and the computer program implements the steps of the field generalization-oriented gradient adaptive quantization-aware training method according to the first aspect of the present application or the steps of the image processing method according to the second aspect of the present application when executed by a processor.

[0038] In a fifth aspect, a computer program product is disclosed, which includes a computer program, and the computer program implements the steps of the field generalization-oriented gradient adaptive quantization-aware training method according to the first aspect of the present application or the steps of the image processing method according to the second aspect of the present application when executed by a processor.

[0039] The embodiments of the present application have the following advantages:

[0040] In the embodiments of the present application, the smooth gradient is introduced to update the parameters of the quantized convolutional neural network model, so that the quantization process and model smoothness can be jointly optimized, and the generalization ability of the model is improved. In addition, the gradient confusion degree of the task gradient of each scale factor in different training domains is evaluated during the training process, and the task gradient is selectively used for learning based on the gradient confusion degree, which enhances the stability of model training and ensures the global convergence of overall performance. In this way, the conflict between the smooth gradient and the task gradient of the quantized model is fine-tuned at the granularity of the domain, the generalization ability of the quantized model in unknown domains is improved, and the generalization ability of the deployed model is enhanced on resource-constrained devices, so that the model is more robust when facing data distribution changes. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 is a step flow chart of a gradient adaptive quantization-aware training method for domain generalization provided by an embodiment of the present application;

[0043] Figure 2 is a step flow chart of another gradient adaptive quantization-aware training method for domain generalization provided by an embodiment of the present application;

[0044] Figure 3 is an architecture schematic diagram of a gradient adaptive quantization-aware training method for domain generalization provided by an embodiment of the present application;

[0045] Figure 4 is a step flow chart of an image processing method provided by an embodiment of the present application;

[0046] Figure 5 is a structure schematic diagram of a gradient adaptive quantization-aware training device for domain generalization provided by an embodiment of the present application;

[0047] Figure 6 is a structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0049] In practical applications, when deploying machine learning models, the test data distribution may be different from the training distribution, which is a common phenomenon called distribution shift. Domain generalization (DG) aims to enhance the model's generalization to unseen domains. Common strategies include domain alignment, meta-learning, data augmentation, decoupled representation learning, and leveraging causal relationships. However, even with these complex techniques, under carefully controlled experimental conditions, basic empirical risk minimization can still achieve comparable out-of-distribution generalization performance.

[0050] At the same time, there is increasing attention on the geometry of the loss function landscape, especially on shared perception minimization that seeks flatter minima during training. Flatter minima can lead to smaller domain generalization gaps. Inspired by the study of flat minima, flatness-based generalization methods have started to gain attention and have shown significant performance in domain generalization. For example, SAGM, a domain generalization scheme, improves generalization by optimizing the angle of the weight gradient. Although flatness-based generalization methods have shown significant effects in improving out-of-distribution generalization performance, they are limited to full-precision training. This means that the models generated by current methods are not very practical in real-world deployment and do not take into account the unique factors of quantization. In other words, in many real-world scenarios, especially in environments involving edge device deployment and susceptible to domain shift, models often need to run under very limited computational resources.

[0051] Quantization-Aware Training (QAT) involves inserting simulated quantization nodes and retraining the model. QAT enables weights to perceive numerical changes by simulating quantization during forward-backward processes, achieving a better balance between accuracy and compression ratio to improve runtime efficiency. One quantization-aware training method uses low-precision weights and activations in forward propagation and uses STE technology to estimate the gradient of the piecewise quantization function in backward propagation. Another quantization-aware training method adjusts the quantization function by introducing a learnable step scaling factor. There are also some quantization-aware training methods that further improve the possibility of quantization performance by freezing unstable weights.

[0052] However, there are still challenges in implementing quantization-aware training for domain generalization, specifically in the following two aspects: 1) target conflict: low-precision computation aims to reduce model complexity, but this is in conflict with maintaining generalization performance; 2) training instability: how to ensure the correct convergence of low-precision weights is still a difficult problem, as both analog quantization and sharpness-aware minimization (SAM) involve specific gradient approximation.

[0053] The embodiments of the present application find that when the domain generalization sharpness-aware minimization method is directly applied to quantization-aware training, the model generalization performance may unexpectedly decrease. For example, when 4-bit quantization is performed on the PACS dataset, the out-of-distribution average performance of the model decreases by 28.36%. Through in-depth analysis of the gradient behavior of the quantizer, the significant conflict between the task loss (empirical loss) and the smoothing loss caused by gradient approximation leads to a decrease in the generalization ability of the trained model, and even performs worse than a model that only optimizes a single target.

[0054] In summary, the existing domain generalization scheme is only applicable to full-precision models, while the low-precision computation method commonly used in edge devices (i.e., quantization method) only considers the same distribution assumption, and the application of the current domain generalization scheme to quantization models is not good. Since the distribution of the data is not seen in actual applications, the generalization performance of the quantization model cannot be guaranteed.

[0055] In order to overcome the limitations of the related art, the embodiments of the present application provide a gradient adaptive quantization-aware training method for domain generalization, which defines a gradient confusion degree for describing the inconsistency of the gradient direction in the training process to quantify the degree of gradient conflict. Specifically, the smoothing gradient is introduced into the quantized convolutional neural network model (quantizer), which enables the joint optimization of the quantization process and the model smoothing, and improves the generalization ability of the model. Moreover, for the two different gradients (task gradient and smoothing gradient) received by the quantized convolutional neural network model from the quantization target and the shared awareness minimization target, by evaluating the gradient confusion degree of the task gradient of each scale factor in different training domains, the task gradient is selectively used for learning based on the gradient confusion degree, which enhances the stability of model training and ensures the global convergence of overall performance. In this way, the smoothing gradient and task gradient conflict problem of the quantization model is fine-tuned at the domain granularity, which improves the generalization ability of the quantization model in unknown domains, enhances the generalization ability of the deployed model on resource-constrained devices, and makes the model more robust when facing data distribution changes.

[0056] In order to better understand the technical solutions of the present application, first, the "quantization" and "flat minimum in domain generalization" involved in the embodiments of the present application are described.

[0057] Quantization: The embodiments of the present application consider a uniform quantization function for the weights and activations of the layer: wherein represents a rounding operator, is a learnable scaling factor in quantization-aware training (QAT), clip is a function that ensures the value remains within the limits In b-bit quantization, for activation quantization, set and ; for weight quantization, set and In addition, to overcome the non-differentiability of the rounding operation, a straight-through estimator (STE) is employed to approximate the gradient:

[0058] Flat minima in domain generalization: the embodiments of the present application employ three optimization objectives for sharpness-aware minimization: (a) empirical risk , (b) perturbation loss , and (c) proxy gap . Among them, minimizing and can find low-loss regions, while minimizing ensures flat minima. The combination of such optimization objectives improves training performance and generalization. Therefore, the overall optimization is: where is a hyperparameter, which can be further rewritten as: where, .

[0059] The field generalization-oriented gradient adaptive quantization-aware training method of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0060] Referring to FIG. 1, Figure 1 is a step flowchart of a field generalization-oriented gradient adaptive quantization-aware training method provided by the embodiments of the present application. As shown in Figure 1 , the field generalization-oriented gradient adaptive quantization-aware training method can include steps S110 to S140: Figure 1

[0061] Step S110: obtaining a training data set and a quantized convolutional neural network model, the training data set including multiple sample images of equal amounts of different training domains.

[0062] Among them, the quantized convolutional neural network model can be a quantized convolutional neural network model used for processing various computer vision tasks.

[0063] ​The training data set is used to train the quantized convolutional neural network model, and the training data set includes multiple sample images of equal quantity in different training domains, that is, the number of sample images corresponding to each training domain in the training data set is the same, for example, there are 3 training domains, and the number of sample images of each training domain is 32, so the training data set has a total of 96 sample images. The training domain refers to the distribution of data, and the data distribution of one training domain is the same, and the data distribution between different training domains is different.

[0064] Step S120: input each sample image in the training data set into the quantized convolutional neural network model for processing to obtain the task gradient and the smoothing gradient of each scale factor in different training domains, the scale factor representing the characteristics of the weight and activation value distribution of the quantized convolutional neural network model, the task gradient being used to optimize the task performance of the quantized convolutional neural network model, and the smoothing gradient being used to optimize the generalization of the quantized convolutional neural network model.

[0065] The quantized convolutional neural network module includes multiple scale factors, the scale factor is used to describe (represent) the characteristics of the weight and activation value distribution of the quantized convolutional neural network model, has great sensitivity to perturbation loss, and obvious convergence of the scale factor reaching a suboptimal state does not necessarily represent satisfactory convergence, and can have a negative impact on the performance outside the training domain. For each convolutional layer of the quantized convolutional neural network model, there is a scale factor corresponding to the weight and a scale factor corresponding to the activation value, for example, if the quantized convolutional neural network model has 5 convolutional layers, there are a total of 10 scale factors.

[0066] The task gradient refers to the task-related gradient of the quantized convolutional neural network model, and the smoothing gradient refers to the gradient related to flatness. The sample image is input into the quantized convolutional neural network model for processing, and each individual training domain will cause the task gradient and the smoothing gradient of the scale factor to have characteristics, thereby obtaining the task gradient and the smoothing gradient of each scale factor in different training domains. For example, there are 5 scale factors and 3 training domains, and the sample image is input into the quantized convolutional neural network model for processing, and the task gradient and the smoothing gradient of the 5 scale factors in the 3 training domains are obtained.

[0067] Step S130: evaluate the gradient confusion degree of the task gradient of each scale factor in different training domains, the gradient confusion degree representing the consistency degree of the direction of the task gradient.

[0068] In the embodiments of the present application, the inconsistency of the task gradient direction in the training process is quantified by the gradient confusion degree of the task gradient of each scale factor in different training domains. The smaller the gradient confusion degree is, the more consistent the direction of the task gradient of the scale factor in the training domain is, which means more stable training. The larger the gradient confusion degree is, the more inconsistent the direction of the task gradient of the scale factor in the training domain is, which means less stable training. It should be noted that although high gradient confusion degree does not necessarily mean incorrect gradient, low gradient confusion degree can provide some guarantee of the correctness of the gradient.

[0069] Step S140: determining the learning gradient of each scale factor in different training domains according to the gradient confusion degree, and updating the parameters of the quantized convolutional neural network model based on the learning gradient to obtain a trained quantized convolutional neural network model, wherein the learning gradient at least includes the smoothed gradient.

[0070] In the embodiments of the present application, the quantized convolutional neural network model processes the sample image, and the task gradient and the smoothed gradient of each scale factor in different training domains are obtained, that is, two different gradients of the scale factor in the training domain are obtained, and the learning of the task gradient may interfere with the learning of the smoothed gradient. In order to avoid the conflict between the task gradient and the smoothed gradient, the learning gradient of each scale factor in different training domains is determined according to the gradient confusion degree, and the learning gradient at least includes the smoothed gradient, that is, according to the gradient confusion degree, the task gradient is discarded on some scales, and only the smoothed gradient is selected for learning, so as to reduce the conflict between the task gradient and the smoothed gradient, and ensure the global convergence of the overall performance.

[0071] The updating of the parameters of the quantized convolutional neural network model based on the learning gradient means that the corresponding scale factor is updated based on the learning gradient. For example, if the learning gradient of a certain scale factor in a certain training domain is the smoothed gradient, the scale factor is updated based on the smoothed gradient, and if the learning gradient of a certain scale factor in a certain training domain is the smoothed gradient and the task gradient, the scale factor is updated based on the smoothed gradient and the task gradient. In this way, the training of multiple training steps is performed according to the above steps, and after the training end condition is met, the trained quantized convolutional neural network model is obtained.

[0072] By adopting the technical solutions of the embodiments of the present application, the smooth gradient is introduced to update the parameters of the quantized convolutional neural network model, so that the quantization process and the model smoothness can be jointly optimized, and the generalization ability of the model is improved. In addition, the gradient confusion degree of each scale factor in the task gradient of different training domains is evaluated during the training process, and the task gradient is selectively used for learning based on the gradient confusion degree, thereby enhancing the stability of the model training and ensuring the global convergence of the overall performance. In this way, the conflict between the smooth gradient and the task gradient of the quantized model is finely optimized at the granularity of the domain, the generalization ability of the quantized model in unknown domains is improved, and the generalization ability of the deployed model is enhanced on the resource-constrained device, so that the model is more robust when facing changes in data distribution.

[0073] In combination with the above embodiments, in an embodiment, the embodiments of the present application also provide a gradient adaptive quantization perception training method for domain generalization. In the method, the step S120 of "inputting each sample image in the training data set into the quantized convolutional neural network model for processing to obtain the task gradient and the smooth gradient of each scale factor in different training domains" specifically includes the following sub-steps S120-1 to S120-2:

[0074] Step S120-1: constructing a loss function according to the quantization result of the quantized convolutional neural network model and the sample image, wherein the loss function includes a task loss and a smooth loss.

[0075] Step S120-2: determining the task gradient of each scale factor in different training domains according to the task loss, and determining the smooth gradient of each scale factor in different training domains according to the smooth loss.

[0076] In the embodiments of the present application, a smooth target (smooth loss) is introduced into the loss function of the quantized convolutional neural network model to perform generalization optimization in the latent weight space.

[0077] For example, the loss function can be represented as:

[0078]

[0079] wherein, represents the task loss, represents the smooth loss, represents the parameters of the quantized convolutional neural network model, represents the scaling factor of the weight, represents the quantization process of the quantized convolutional neural network model, represents the gradient of the task loss, represents the gradient adjustment parameter, represents the sample image.

[0080] Compared with full-precision training, there are several scale factors in the loss function Each scale factor corresponds to two optimization objectives (i.e., task objectives and smoothing objectives), thus generating two sets of gradients, one set of task gradients from task loss, i.e., task loss from , and the other set of smoothing gradients from smoothing loss, i.e., smoothing gradient from .

[0081] By adopting the technical solutions of the embodiments of the present application, the smoothing objectives are introduced into the loss function of the quantized convolutional neural network model, so that two different gradients from the quantization objectives (task loss) and the perceptual minimization objectives (smoothing loss) are simultaneously received, the quantization process and the model smoothing are jointly optimized, and the generalization ability of the model is improved.

[0082] In combination with the above embodiments, in an implementation manner, the embodiments of the present application further provide a gradient adaptive quantization perceptual training method for domain generalization. Specifically:

[0083] In the method, the "evaluating the gradient confusion degree of the task gradients of each scale factor in different training domains" in step S130 includes: evaluating the gradient confusion degree of the task gradients of each scale factor in different training domains every target number of training steps.

[0084] In the method, the "determining the learning gradients of each scale factor in different training domains according to the gradient confusion degree, and updating the parameters of the quantized convolutional neural network model based on the learning gradients" in step S140 includes: determining the learning gradients of each scale factor in different training domains in the next target number of training steps according to the gradient confusion degree, and updating the parameters of the quantized convolutional neural network model based on the learning gradients in the next target number of training steps.

[0085] In the embodiments of the present application, the training of each sample image corresponds to one training step, and there are multiple training steps for training the quantized convolutional neural network model using the training data set. In the training process, the gradient confusion degree of the task gradients of each scale factor in different training domains is evaluated every target number of training steps, and then the learning gradients of each scale factor in different training domains in the next target number of training steps are determined according to the gradient confusion degree, so that the quantized convolutional neural network model updates the parameters based on the learning gradients in the next target number of training steps.

[0086] For example, for a training step number T and an evaluation interval (target training steps) K, the gradient chaos degree of the task gradient of each scale factor in different training domains is evaluated every K training steps during the training process, and then the learning gradient of each scale factor in different training domains in the next K training steps is determined according to the gradient chaos degree, and the parameters of the quantized convolutional neural network model are updated based on the learning gradient in the next K training steps.

[0087] In this way, by periodically (every target training steps) re-evaluating the gradient chaos degree of the task gradient of each scale factor in different training domains and adjusting the learning gradient, the method allows the smooth gradient to continue training while reducing the adverse effects of gradient conflict, improves overall convergence and enhances the generalization performance of the model.

[0088] In an optional embodiment, the gradient chaos degree of the task gradient of each scale factor in different training domains is evaluated every target training steps, including steps A1 to A2:

[0089] Step A1: determining a task gradient sequence according to the task gradient of the scale factor in the training domain within the target training steps; wherein the task gradient of the scale factor in the training domain within each training step is one task gradient in the task gradient sequence.

[0090] Step A2: obtaining the gradient chaos degree according to the direction change of the adjacent two task gradients in the task gradient sequence.

[0091] In the embodiments of the present application, for each scale factor in each training domain, a task gradient sequence can be determined, and then based on the direction change of the adjacent two task gradients in the task gradient sequence, the gradient chaos degree of the task gradient of the scale factor in the training domain can be determined.

[0092] Wherein, according to the direction change of the adjacent two task gradients in the task gradient sequence, the gradient chaos degree can be the number of direction changes of the adjacent two task gradients in the task gradient sequence as the gradient chaos degree; or the ratio of the number of direction changes of the adjacent two task gradients in the task gradient sequence to the number of task gradients in the task gradient sequence as the gradient chaos degree.

[0093] For example, for the task gradient sequence [-1, 1, 2], the direction of the task gradient -1 and the task gradient 1 is inconsistent, and there is 1 direction deflection (direction change), the direction of the task gradient 1 and the task gradient 2 is consistent, and there is no direction deflection, that is, there is 1 direction deflection in the task gradient sequence [-1, 1, 2] The direction of the adjacent two task gradients coexists, and the task gradient sequence has 3 task gradients, so the gradient chaos degree can be 1 / 3.

[0094] Further, the task gradient sequence includes a first task gradient sequence and a second task gradient sequence, and two task gradients with the same order in the first task gradient sequence and the second task gradient sequence represent two task gradients of adjacent training steps.

[0095] The step A1 further includes the following steps: comparing the directions of two task gradients with the same order in the first task gradient sequence and the second task gradient sequence, respectively, to determine the number of two task gradients of adjacent training steps with different directions; and taking the ratio of the number to the number of target training steps as the gradient confusion degree.

[0096] For example, for K training steps, and at each training step j, the corresponding task gradient is The task gradients of the first training step to the K-1th training step can be taken as the first task gradient sequence, and the task gradients of the second training step to the Kth training step can be taken as the second task gradient sequence. The first task gradient sequence is represented as: The second task gradient sequence is represented as: .

[0097] The gradient confusion degree can be represented as:

[0098]

[0099] wherein, represents an element-wise sign function, is an indicator function, represents the proportion of the number of steps with the gradient direction opposite to that of the previous step (i.e., the gradient confusion degree).

[0100] The technical solution of the embodiments of the present application can quantify the gradient confusion degree of the task gradient, and then discard the task gradient in some scales according to the gradient confusion degree, and only select the smooth gradient for learning, so as to reduce the conflict between the task gradient and the smooth gradient, and ensure the global convergence of the overall performance.

[0101] In combination with the above embodiments, in an embodiment, the embodiments of the present application further provide a gradient adaptive quantization perception training method for domain generalization. In the method, the step S140 of determining the learning gradient of each scale factor in different training domains according to the gradient confusion degree includes the following sub-steps S140-1 to S140-2:

[0102] Step S140-1: For each scaling factor's learning gradient in each training domain, if the gradient disorder of the scaling factor's task gradient in the training domain is less than or equal to the gradient disorder threshold, determine the smoothed gradient as the scaling factor's learning gradient in the training domain.

[0103] Step S140-2: If the gradient disorder of the task gradient of the scale factor in the training domain is greater than the gradient disorder threshold, determine the smooth gradient and the task gradient as the learning gradient of the scale factor in the training domain.

[0104] Each training domain has a corresponding gradient disorder threshold, and the gradient disorder thresholds for different training domains can be the same or different.

[0105] Specifically, for every K training steps (target number of steps), the gradient disorder of the task gradient for each scale factor in different training domains is evaluated, and the gradient disorder thresholds for different training domains are determined. If the scaling factor Gradient disorder in the j-th training domain at training step t Less than the gradient disorder threshold Then, in the next K training steps, the smooth gradient is used as the learning gradient, and the j-th training domain data pair is frozen. gradient Otherwise, the learning gradient in the training domain is scaled by the smooth gradient and the task gradient.

[0106] The technical solution of this application embodiment realizes a dynamic selective freezing scheme, which selectively uses task gradients for learning based on gradient disorder, reduces the adverse effects of gradient conflict, enhances the stability of model training, and ensures global convergence of overall performance.

[0107] The following specific embodiment illustrates the gradient adaptive quantization-aware training method for domain-specific generalization proposed in this application. (Refer to...) Figure 2 As shown, Figure 2 This is a flowchart illustrating another domain-generalized gradient adaptive quantization-aware training method provided in this application embodiment, which includes steps S210 to S240:

[0108] Step S210: Obtain the training dataset and quantize the convolutional neural network model, wherein the training dataset includes an equal number of sample images from different training domains.

[0109] Step S220: input each sample image in the training data set to the quantized convolutional neural network model for processing, construct a loss function according to the quantization result of the quantized convolutional neural network model and the sample image, the loss function including a task loss and a smoothing loss; determine the task gradient of each scale factor in different training domains according to the task loss, and determine the smoothing gradient of each scale factor in different training domains according to the smoothing loss.

[0110] Step S230: every target training step, evaluate the gradient confusion degree of the task gradient of each scale factor in different training domains, the gradient confusion degree representing the consistency degree of the direction of the task gradient.

[0111] Specifically, according to the task gradient of the scale factor in the training domain within the target training step, a task gradient sequence is determined; wherein the task gradient of the scale factor in the training domain within each training step is one task gradient in the task gradient sequence; according to the direction change of the adjacent two task gradients in the task gradient sequence, the gradient confusion degree is obtained.

[0112] Step S240: according to the gradient confusion degree, determine the learning gradient of each scale factor in different training domains within the next target training step, and update the parameters of the quantized convolutional neural network model based on the learning gradient within the next target training step.

[0113] Specifically, for the learning gradient of each scale factor in each training domain, in the case that the gradient confusion degree of the task gradient of the scale factor in the training domain is less than or equal to a gradient confusion degree threshold, the smoothing gradient is determined as the learning gradient of the scale factor in the training domain; in the case that the gradient confusion degree of the task gradient of the scale factor in the training domain is greater than the gradient confusion degree threshold, the smoothing gradient and the task gradient are determined as the learning gradient of the scale factor in the training domain.

[0114] In the embodiments of the present application, by introducing the smoothing gradient to update the parameters of the quantized convolutional neural network model, the quantization process and the model smoothing can be jointly optimized, and the generalization ability of the model is improved; and the gradient confusion degree of the task gradient of each scale factor in different training domains is evaluated during the training process, and the task gradient is selectively used for learning based on the gradient confusion degree, which enhances the stability of the model training and ensures the global convergence of the overall performance. In this way, the smoothing gradient and the task gradient conflict problem of the quantized model is fine-tuned at the domain granularity, the generalization ability of the quantized model in unknown fields is improved, the generalization ability of the deployed model is enhanced on the resource-constrained device, and the model is more robust when facing data distribution changes.

[0115] For example, the field generalization-oriented gradient adaptive quantization perception training method of the embodiment of the present application can be implemented through the dynamic selective freezing strategy of the scale factor in Table 1.

[0116] Table 1: Dynamic selective freezing strategy of scale factor

[0117]

[0118] For example, Figure 3 is an architecture schematic diagram of a field generalization-oriented gradient adaptive quantization perception training method provided by the embodiment of the present application. Compared with the full-precision weight gradient, the tensor-level scale gradient of the field generalization-oriented gradient adaptive quantization perception training method has only two directions, i.e., positive and negative. The task-related scale gradient (i.e., task gradient) newly introduced is selectively frozen. The disorder degree of each scale to the task gradient of each training domain is evaluated, and those task gradients with a task gradient disorder degree lower than a gradient disorder degree threshold are selectively frozen to improve the generalization ability of the model.

[0119] Specifically, each sample image in the training data set is input to the quantized convolutional neural network model for processing to obtain the task gradient and the smooth gradient of each scale factor in different training domains. Every K training steps, the gradient disorder degree Disorder of the task gradient of each scale factor in different training domains is evaluated through the first task gradient sequence and the second task gradient sequence. Then, in the case that the gradient disorder degree of the task gradient of the scale factor in the training domain is less than or equal to the gradient disorder degree threshold r (i.e. ), the task gradient is frozen, and the smooth gradient is taken as the learning gradient of the scale factor in the training domain; in the case that the gradient disorder degree of the task gradient of the scale factor in the training domain is greater than the gradient disorder degree threshold (i.e. ), the smooth gradient and the task gradient are taken as the learning gradient of the scale factor in the training domain.

[0120] In this way, the conflict problem of the smooth gradient and the task gradient of the quantized model is fine-tuned at the granularity of the domain, the generalization ability of the quantized model in unknown domains is improved, the generalization ability of the deployed model is enhanced on the resource-constrained device, and the model is more robust when facing data distribution changes.

[0121] The embodiment of the present application also provides an image processing method. Referring to Figure 4 , Figure 4 is a step flowchart of an image processing method provided by the embodiment of the present application. The image processing method includes steps S410 to S420.

[0122] Step S410: Obtain the image to be processed.

[0123] Step S420: Input the image to be processed into the trained quantized convolutional neural network model to obtain the target visual task result. The trained quantized convolutional neural network model is trained according to the domain-oriented generalized gradient adaptive quantization perception training method of this application embodiment.

[0124] In this embodiment, the gradient-adaptive quantization-aware training method for domain generalization introduces smooth gradients to update the parameters of the quantized convolutional neural network model, enabling joint optimization of the quantization process and model smoothness, thus improving the model's generalization ability. Furthermore, during training, the gradient disorder of each scale factor in different training domains is evaluated, and task gradients are selectively used for learning based on gradient disorder, enhancing the stability of model training and ensuring global convergence of overall performance. Therefore, according to the gradient-adaptive quantization-aware training method for domain generalization in this embodiment, the trained quantized convolutional neural network model exhibits good generalization and task performance. Processing the image to be processed based on the trained quantized convolutional neural network model can yield accurate results for the target visual task.

[0125] This application also provides a gradient-adaptive quantization perception training device for domain-specific generalization, referring to... Figure 5 As shown, Figure 5 This is a schematic diagram of a gradient adaptive quantization perception training device for domain-specific generalization provided in an embodiment of this application. The device includes:

[0126] The acquisition module 510 is used to acquire a training dataset and a quantized convolutional neural network model, wherein the training dataset includes an equal number of sample images from different training domains;

[0127] The quantization module 520 is used to input each sample image in the training dataset into the quantization convolutional neural network model for processing, and to obtain the task gradient and smoothing gradient of each scale factor in different training domains. The scale factor characterizes the weight and activation value distribution of the quantization convolutional neural network model. The task gradient is used to optimize the task performance of the quantization convolutional neural network model, and the smoothing gradient is used to optimize the generalization of the quantization convolutional neural network model.

[0128] Evaluation module 530 is used to evaluate the gradient disorder of task gradients of each scale factor in different training domains, wherein the gradient disorder characterizes the consistency of task gradient directions.

[0129] The updating module 540 is configured to determine learning gradients of the scale factors in different training domains according to the gradient chaos degree, and update parameters of the quantized convolutional neural network model based on the learning gradients to obtain a trained quantized convolutional neural network model, wherein the learning gradients at least include the smoothed gradients.

[0130] In an optional embodiment, the evaluation module is further configured to evaluate the gradient chaos degree of the task gradients of the scale factors in different training domains every target number of training steps.

[0131] The updating module is further configured to determine learning gradients of the scale factors in different training domains in the next target number of training steps according to the gradient chaos degree, and update parameters of the quantized convolutional neural network model based on the learning gradients in the next target number of training steps.

[0132] In an optional embodiment, the evaluation module includes:

[0133] The sequence determining module is configured to determine a task gradient sequence according to the task gradients of the scale factors in the training domains in the target number of training steps, wherein the task gradient of the scale factor in each training step in the training domain is a task gradient in the task gradient sequence.

[0134] The chaos degree determining module is configured to obtain the gradient chaos degree according to direction changes of two adjacent task gradients in the task gradient sequence.

[0135] In an optional embodiment, the task gradient sequence includes a first task gradient sequence and a second task gradient sequence, and two task gradients with the same order in the first task gradient sequence and the second task gradient sequence represent two task gradients of adjacent training steps.

[0136] The chaos degree determining module is further configured to compare directions of two task gradients with the same order in the first task gradient sequence and the second task gradient sequence in sequence to determine a number of two task gradients of adjacent training steps with different directions, and take a ratio of the number to a number of the target number of training steps as the gradient chaos degree.

[0137] In an optional embodiment, the updating module includes:

[0138] The learning gradient determination module is configured to determine, for each scale factor, the smooth gradient as the learning gradient of the scale factor in each training domain if the gradient confusion degree of the task gradient of the scale factor in the training domain is less than or equal to a gradient confusion degree threshold; and determine the smooth gradient and the task gradient as the learning gradient of the scale factor in the training domain if the gradient confusion degree of the task gradient of the scale factor in the training domain is greater than the gradient confusion degree threshold.

[0139] In an optional embodiment, the quantization module comprises:

[0140] The construction module is configured to construct a loss function according to the quantization result of the quantized convolutional neural network model and the sample image, wherein the loss function comprises a task loss and a smooth loss.

[0141] The gradient determination module is configured to determine, for each scale factor, a task gradient of the scale factor in different training domains according to the task loss, and determine a smooth gradient of the scale factor in different training domains according to the smooth loss.

[0142] Embodiments of the present application also provide an electronic device, which refers to Figure 6 , Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 6 , the electronic device 600 comprises a memory 610 and a processor 620, the memory 610 and the processor 620 are communicatively connected through a bus, the memory 610 stores a computer program, the computer program can run on the processor 620, and thus the steps of the gradient adaptive quantization-aware training method for domain generalization provided by the embodiments of the present application are implemented.

[0143] Embodiments of the present application also provide a computer readable storage medium, which stores a computer program, the computer program is executed by a processor to implement the steps of the gradient adaptive quantization-aware training method for domain generalization provided by the embodiments of the present application.

[0144] Embodiments of the present application also provide a computer program product, which comprises a computer program, the computer program is executed by a processor to implement the steps of the gradient adaptive quantization-aware training method for domain generalization provided by the embodiments of the present application.

[0145] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts of each embodiment can be referred to each other.

[0146] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. It is understood that each flow and / or block in the flowcharts and / or block diagrams, and a combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or a processor of other programmable data processing terminals to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminals generate a device implemented in the flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application.

[0147] These computer program instructions can also be stored in a computer readable memory capable of guiding the computer or other programmable data processing terminal to work in a specific manner, so that the instructions stored in the computer readable memory produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application.

[0148] These computer program instructions can also be loaded into the computer or other programmable data processing terminal, so that a series of operation steps are performed on the computer or other programmable terminal to produce a computer implemented process, so that the instructions executed on the computer or other programmable terminal provide steps for implementing the functions specified in the flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application. Figure 1 The flowcharts and / or block diagrams of the methods, apparatuses (systems) according to the embodiments of the present application.

[0149] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0150] Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0151] The above provides a detailed introduction to the field generalization-oriented gradient adaptive quantization perception training method, device and equipment provided by the present application. The principles and implementation modes of the present application are described in this paper by applying specific examples. The above example is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A field-oriented generalized gradient adaptive quantization perception training method, characterized in that, The method comprises the following steps: obtaining a training data set and a quantized convolutional neural network model, the training data set comprising a plurality of sample images of equal amounts of different training domains; inputting each sample image in the training data set into the quantized convolutional neural network model for processing to obtain task gradients and smoothing gradients of each scale factor in different training domains, the scale factor representing a feature of weight and activation value distribution of the quantized convolutional neural network model, the task gradient being used for optimizing the task performance of the quantized convolutional neural network model, and the smoothing gradient being used for optimizing the generalization of the quantized convolutional neural network model; evaluating gradient chaos degrees of the task gradients of each scale factor in different training domains, the gradient chaos degree representing the consistency degree of the direction of the task gradient; determining learning gradients of each scale factor in different training domains according to the gradient chaos degrees, and updating parameters of the quantized convolutional neural network model based on the learning gradients to obtain a trained quantized convolutional neural network model, the learning gradient at least comprising the smoothing gradient.

2. The method of claim 1, wherein, The step of evaluating the gradient chaos degrees of the task gradients of each scale factor in different training domains comprises: evaluating the gradient chaos degrees of the task gradients of each scale factor in different training domains every target number of training steps; The step of determining the learning gradients of each scale factor in different training domains according to the gradient chaos degrees and updating the parameters of the quantized convolutional neural network model based on the learning gradients comprises: determining the learning gradients of each scale factor in different training domains within the next target number of training steps according to the gradient chaos degrees, and updating the parameters of the quantized convolutional neural network model based on the learning gradients within the next target number of training steps.

3. The method of claim 2, wherein, The step of evaluating the gradient chaos degrees of the task gradients of each scale factor in different training domains every target number of training steps comprises: determining a task gradient sequence according to the task gradients of the scale factor in the training domain within the target number of training steps, wherein the task gradient of the scale factor in the training domain within each training step is one task gradient in the task gradient sequence; obtaining the gradient chaos degree according to the direction change of the adjacent two task gradients in the task gradient sequence.

4. The method of claim 3, wherein, The task gradient sequence comprises a first task gradient sequence and a second task gradient sequence, and the two task gradients with the same order in the first task gradient sequence and the second task gradient sequence represent two task gradients of adjacent training steps; The step of obtaining the gradient chaos degree according to the direction change of the adjacent two task gradients in the task gradient sequence comprises: comparing the directions of the two task gradients with the same order in the first task gradient sequence and the second task gradient sequence in sequence to determine the number of two task gradients of adjacent training steps in different directions; taking the ratio of the number and the number of the target number of training steps as the gradient chaos degree.

5. The method according to any of claims 1 to 4, characterized in that, The step of determining the learning gradients of each scale factor in different training domains according to the gradient chaos degrees comprises: determining the smooth gradient as the learning gradient of the scale factor in the training domain when the gradient chaos degree of the task gradient of the scale factor in the training domain is less than or equal to a gradient chaos degree threshold; determining the smooth gradient and the task gradient as the learning gradient of the scale factor in the training domain when the gradient chaos degree of the task gradient of the scale factor in the training domain is greater than the gradient chaos degree threshold.

6. The method according to any one of claims 1 to 4, characterized in that, inputting each sample image in the training data set into the quantized convolutional neural network model for processing to obtain the task gradient and the smooth gradient of each scale factor in different training domains, wherein the scale factor represents a feature of a weight and an activation value distribution of the quantized convolutional neural network model, the task gradient is used to optimize the task performance of the quantized convolutional neural network model, and the smooth gradient is used to optimize the generalization of the quantized convolutional neural network model; constructing a loss function according to the quantization result of the quantized convolutional neural network model and the sample image, wherein the loss function comprises a task loss and a smooth loss; determining the task gradient of each scale factor in different training domains according to the task loss, and determining the smooth gradient of each scale factor in different training domains according to the smooth loss.

7. An image processing method characterized by, comprising: obtaining an image to be processed; inputting the image to be processed into the trained quantized convolutional neural network model to obtain a target visual task result, wherein the trained quantized convolutional neural network model is trained according to the field generalization-oriented gradient adaptive quantization perception training method in any one of claims 1-6.

8. A device for field-oriented generalized gradient adaptive quantization-aware training, comprising: comprising: an obtaining module, configured to obtain a training data set and a quantized convolutional neural network model, wherein the training data set comprises a plurality of sample images in equal amounts and different training domains; a quantization module, configured to input each sample image in the training data set into the quantized convolutional neural network model for processing to obtain a task gradient and a smooth gradient of each scale factor in different training domains, wherein the scale factor represents a feature of a weight and an activation value distribution of the quantized convolutional neural network model, the task gradient is used to optimize the task performance of the quantized convolutional neural network model, and the smooth gradient is used to optimize the generalization of the quantized convolutional neural network model; an evaluation module, configured to evaluate a gradient chaos degree of the task gradient of each scale factor in different training domains, wherein the gradient chaos degree represents a consistency degree of a task gradient direction; an updating module, configured to determine a learning gradient of each scale factor in different training domains according to the gradient chaos degree, and update parameters of the quantized convolutional neural network model based on the learning gradient to obtain a trained quantized convolutional neural network model, wherein the learning gradient at least comprises the smooth gradient.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the field generalization-oriented gradient adaptive quantization perception training method in any one of claims 1-6, or the steps of the image processing method in claim 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the field generalization-oriented gradient adaptive quantization perception training method in any one of claims 1-6, or the steps of the image processing method in claim 7.

Images