Image semantic analysis method and system based on knowledge distillation and quantization collaborative optimization

Abstract

The application discloses a kind of based on knowledge distillation and quantization collaborative optimization image semantic analysis method and system, comprising: S1, constructs double-branch network model, inserts simulated quantization pseudo-operation node when training, generates each layer intermediate value and network parameter;S2, after monitoring the quantization error after analog quantization after each training batch forward propagation, generate dynamic distillation temperature and distillation loss;S3, under the condition, calculate double-branch network model feature map mutual information, compared with preset double threshold to obtain bit width adjustment decision;S4, receive forward propagation loss, construct joint loss function, introduce learnable parameter to generate collaborative gradient, update student model parameter, complete a batch training;If there is next batch training, the current model is step S1 parameter, repeat steps S1 to S4;If not, bit width quantization obtains final model and deploys.The application realizes that network model calculation complexity is significantly reduced and slows down quantization precision degradation, suitable for extremely high real-time requirement edge computing scene.

Description

Technical Field

[0001] This invention relates to deep learning model compression and image processing technology, and in particular to an image semantic analysis method and system based on knowledge distillation and quantization co-optimization. Background Technology

[0002] With the rapid development of deep learning in computer vision, image semantic analysis models based on convolutional neural networks and Transformer architectures have made breakthroughs in tasks such as autonomous driving scene parsing, medical image diagnosis, and remote sensing ground object classification. However, in order to achieve the ultimate pixel-level classification accuracy, mainstream models in the industry tend to build deeper and wider network structures, resulting in a model parameter count that often reaches billions, leading to persistently high computational complexity. This massive parameter scale makes the models heavily reliant on powerful graphics processing unit clusters, making it difficult to deploy them directly in edge computing environments with limited storage space and stringent power requirements, such as drones, automotive embedded systems, or mobile terminals.

[0003] In real-time video inspection applications of power transmission and distribution channels, inspection equipment (such as drones and inspection robots) needs to perform real-time semantic analysis on targets such as trees, construction machinery, and broken conductor strands within the transmission line corridor in complex field environments. This places extremely stringent demands on the model's inference speed and power consumption. Therefore, achieving extreme compression and acceleration of the model while maintaining its semantic analysis performance has become a key challenge for both academic research and industrial application.

[0004] In existing model compression techniques, knowledge distillation and model quantization are two mainstream and complementary strategies. Knowledge distillation constructs a "teacher-student" framework, allowing the student model to learn the soft labels (class probability distributions) output by the teacher model, thereby transferring the implicit knowledge learned by the teacher model. Traditional distillation methods often operate at a constant distillation temperature, failing to consider the student model's receptiveness under low-precision constraints. When the student model's capacity is too small or quantization constraints are imposed, the fixed softness can easily lead to overfitting to noise. Model quantization, on the other hand, aims to map model weights and activations from 32-bit floating-point numbers to 8-bit or even 4-bit integers to reduce memory usage and accelerate operations using integers. However, simple quantization methods typically treat the model as an independent object. When using post-training quantization or quantization-aware training, only the numerical error of the tensor before and after quantization is minimized, neglecting the fidelity of the model's internal semantic features. This is especially problematic at very low bit widths, such as below 4 bits, where severe semantic information loss and boundary degradation occur.

[0005] Current combined approaches are mostly pipelined models of "distillation first, quantization later" or "quantization first, distillation later." This fragmented approach has two inherent drawbacks: First, the distillation process is unaware of the impact of subsequent quantization, resulting in the student model learning a weight distribution that is not quantization-friendly, leading to severe rounding errors in the post-quantization stage. Second, the bit width configuration strategy for quantization is often globally uniform or a simple heuristic rule based on weight size, ignoring the differences in the contribution of different network layers to semantic representation, which can easily destroy the key semantic perception regions carried in the teacher model. These shortcomings make it difficult for existing technologies to achieve semantic analysis accuracy similar to that of the teacher model while maintaining high inference efficiency. Therefore, there is an urgent need for a novel method that can deeply bidirectionally couple distillation and quantization and conduct collaborative training to overcome the energy efficiency and accuracy bottlenecks of current image semantic analysis models when deployed at edges.

[0006] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art. Summary of the Invention

[0007] Based on the aforementioned technical problems, the purpose of this invention is to construct a collaborative framework of "two-way coupling and feedback adjustment" for quantization and distillation. The quantization error serves as a constraint for knowledge distillation. The mutual information estimate of the distillation process is used to guide the adaptive allocation of the quantization bit width. A joint loss function is designed to drive network optimization and updates, ultimately achieving Pareto optimality for model accuracy and inference efficiency.

[0008] To achieve the above objectives, this invention provides an image semantic analysis method based on knowledge distillation and quantization collaborative optimization, comprising the following steps: Step S1: Construct a dual-branch network model. During training, insert simulated quantization pseudo-operation nodes during forward propagation to generate intermediate values ​​and network parameters for each layer. Step S2: In each training batch, after forward propagation, monitor the quantization error of the simulated quantization pseudo-operation node and generate dynamic distillation temperature and distillation loss. Step S3: Train to the set conditions, calculate the mutual information between the feature maps of the teacher model and the student model, and compare it with the preset dual thresholds to obtain the bit width adjustment decision; Step S4: Receive the forward propagation loss, construct the joint loss function, introduce learnable parameters to generate collaborative gradients, update the student model parameters, and complete one batch of training; if there is a next batch of training, the current model is used as the parameter of step S1, and steps S1 to S4 are repeated; if not, the model is truly quantized according to the target bit width to obtain the final model and deploy it.

[0009] Specifically, the insertion of the simulated quantization pseudo-operation node in step S1 includes: Simulated quantization pseudo-operation nodes are inserted into the convolutional layers of the student model in the dual-branch network model, and the weight tensors of each layer to be quantized in the student model are... and target bit width Bits, quantization scale factor ,in It is a constant; Weight tensors of the quantization layer Perform quantization, rounding, and dequantization operations to obtain pseudo-quantized weights:

[0010] The activation tensor of each layer to be quantized in the student model is Target width Bits, the quantization scaling factor for calculating activation and zero point : ; The activation tensor of the quantization layer is Performing quantization, rounding, and dequantization operations yields pseudo-quantization activation:

[0011] in, This indicates rounding to the nearest integer. This is an element-by-element clipping operation.

[0012] Specifically, the generation of dynamic distillation temperature in step S2 includes: Calculate the weights of the layer to be quantized within the current training batch. With pseudo-quantization weights The mean square error between them, the activation of the quantization layer With pseudo-quantization activation The mean square error between them yields the weighted average quantization error generated by all simulated quantization pseudo-operation nodes. ; Define a dynamic distillation temperature adjustment function and calculate the adjusted dynamic distillation temperature. ; Utilizing the adjusted dynamic distillation temperature The logic value output by the teacher model is softened to obtain the dynamic distillation temperature dependence. Distillation loss , used to construct the joint loss function.

[0013] Specifically, the dynamic distillation temperature adjustment function adopts a linear form:

[0014] in, Based on the base temperature, A positive regulatory factor; Alternatively, the dynamic distillation temperature adjustment function may take a nonlinear form:

[0015] in, The preset upper temperature limit, This is a normalized reference value. It is a hyperbolic tangent function used to smooth temperature growth when the quantization error is extremely large.

[0016] Specifically, step S3 includes: For the student model, the first In the evaluation phase, a simulated quantization pseudo-operation node is introduced, and the teacher model is calculated using a mutual information neural estimator. Layer feature map With student model Layer pseudo-quantization feature map Mutual information between ; Set mutual information lower limit threshold With upper limit threshold ; like If the current bit width has not reached the preset hardware limit, it is determined that the knowledge loss of this layer is serious, and an instruction to increase the weight and activated quantization bit width of this layer by one level is triggered. like If the current bit width has reached the preset hardware limit, then the update of the quantization bit width of this layer will be paused, and this layer will not participate in pseudo-quantization operations in subsequent training to avoid further performance degradation until the next re-evaluation. like If the information in that layer is deemed redundant, an instruction is triggered to reduce the quantization bit width of the weight and activation of that layer by one level. like If so, the bit width of that layer remains unchanged.

[0017] Specifically, step S4 includes: The expression for the joint loss function is:

[0018] in, The cross-entropy loss is the true label of the task. For dynamic distillation temperature dependence Distillation loss, To incorporate the quantization error loss guided by layer-by-layer mutual information, and These are the preset hyperparameters used to balance the weights of each loss term.

[0019] In backpropagation, the joint loss function is analyzed. Calculate the distillation loss respectively. Quantization error loss True label cross-entropy loss Regarding the first Layer weights The gradient is calculated by using the sum of the gradients of the distillation loss and the true label cross-entropy loss as the first gradient component. The gradient component of the quantization error loss is used as the second gradient component. ; Learnable equilibrium parameters Calculate the cooperative gradient: and utilize the cooperative gradient Update weights ; Among them, the learnable equilibrium parameter To minimize the first gradient component With the second gradient component Gradient conflicts between parameters are used to update the optimization objective.

[0020] Specifically, the gradient conflict is resolved through the negative inner product of the two gradient components. The learnable equilibrium parameter is measured. The update rules are as follows:

[0021] in, For learnable equilibrium parameters learning rate

[0022] Or the learnable equilibrium parameter The update rules are as follows:

[0023] in, The function limits the data range to between 0.1 and 0.9.

[0024] The mutual information is estimated using a mutual information neural estimator based on variational information maximization, for each layer. Teacher feature map and student pseudo-quantization feature map The data is concatenated along the channel dimension as a sampled representation of the joint distribution, while also incorporating teacher feature maps. and unrelated feature maps from another random image Concatenated along the channel dimension, serving as a sampled representation of the marginal distribution product; mutual information. The estimate is as follows:

[0025] By training the discriminator This maximizes the terms on the right-hand side of the formula, resulting in a reliable estimate of mutual information.

[0026] This invention also provides an image semantic analysis system based on knowledge distillation and quantization collaborative optimization. The system includes: a network construction and forward pseudo-quantization module, a dynamic distillation temperature adjustment module for quantization perception, a layer-by-layer collaborative quantization bit width allocation module based on information bottleneck, and a gradient-level collaborative balancing and multi-objective joint optimization convergence module. The network construction and forward pseudo-quantization module outputs the intermediate values ​​of each layer of the current dual-branch network and the network parameters. The dynamic distillation temperature adjustment module with quantization perception is connected to the network construction and forward pseudo-quantization module and the gradient-level collaborative balancing and multi-objective joint optimization convergence module. It receives quantization error and calculates dynamic distillation temperature and distillation loss. The layer-by-layer collaborative quantization bit width allocation module based on information bottlenecks calculates the mutual information between the current network construction and the dual-branch network model of the forward pseudo-quantization module and performs a dual threshold comparison to give the corresponding layer bit width adjustment decision. The gradient-level collaborative balancing and multi-objective joint optimization convergence module receives the parameters and output results of the dual-branch network model of the network construction and forward pseudo-quantization module, as well as the quantization error loss and distillation loss of the quantization-aware dynamic distillation temperature adjustment module. It calculates the joint loss and solves the collaborative gradient, updates the parameters of the current dual-branch network model of the network construction and forward pseudo-quantization module using gradient descent, and quantizes and derives the trained student model.

[0027] Compared with the prior art, the present invention has the following advantages: 1. The present invention provides an image semantic analysis method and system based on knowledge distillation and quantization co-optimization. By constructing a co-optimization framework of "bidirectional coupling and feedback adjustment", quantization noise is used as a constraint condition for knowledge transfer, and mutual information is used to guide the adaptive allocation of quantization bit width in reverse, ultimately achieving Pareto optimality of model accuracy and inference efficiency.

[0028] 2. This invention provides an image semantic analysis method and system based on knowledge distillation and quantization co-optimization. It proposes a dynamic distillation temperature adjustment mechanism based on quantization perception, which dynamically adjusts the distillation temperature according to the real-time quantization error, softens the output label of the teacher model, and effectively prevents the student model from overfitting to quantization noise in low-precision training.

[0029] 3. The present invention provides an image semantic analysis method and system based on knowledge distillation and quantization synergistic optimization. It introduces the information bottleneck theory, calculates the mutual information between feature maps of each layer of the teacher-student model, and dynamically and layer by layer allocates the quantization bit width to ensure that the key layer retains high precision and the non-key layer achieves extreme compression.

[0030] 4. The present invention provides an image semantic analysis method and system based on knowledge distillation and quantization co-optimization. It designs a gradient-level co-balancing strategy, which adaptively adjusts the mixing ratio of distillation gradient and quantization gradient during backpropagation through a learnable balancing parameter, so as to resolve multi-objective conflicts in optimization and ensure stable convergence of the model. Attached Figure Description

[0031] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the drawings in the following description are one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is an overall flowchart of the image semantic analysis method based on knowledge distillation and quantization co-optimization of the present invention; Figure 2 This is an example flowchart of the dynamic distillation temperature regulation mechanism based on quantitative sensing of the present invention. Figure 3 This is a flowchart of the layer-by-layer collaborative quantization bit width allocation based on information bottlenecks in this invention; Figure 4 This is a flowchart illustrating the convergence of gradient-level collaborative balancing and multi-objective joint optimization in this invention; Figure 5 This is a diagram showing the module composition of the image semantic analysis system based on the collaborative optimization of knowledge distillation and quantization according to the present invention. Detailed Implementation

[0032] The following is in conjunction with the appendix Figures 1-5The following detailed description of the image semantic analysis method and system based on knowledge distillation and quantization synergistic optimization proposed in this invention will further illustrate the invention. The advantages and features of the invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, used only to facilitate and clarify the illustration of the embodiments of the invention. Please refer to the accompanying drawings to make the objectives, features, and advantages of the invention more apparent and understandable. It should be understood that the structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes to aid those skilled in the art and are not intended to limit the implementation conditions of the invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to the size, without affecting the effects and objectives achieved by the invention, should still fall within the scope of the technical content disclosed in this invention.

[0033] Example 1: A complete collaborative compression process based on a semantic segmentation backbone network; In this embodiment 1, an image semantic analysis method based on knowledge distillation and quantization co-optimization is provided for the power transmission and distribution channel scene analysis task. This method aims to transfer a high-precision semantic segmentation network to a lightweight quantized student model. This lightweight quantized student model can run in real time on edge devices such as drones or inspection robots, enabling real-time identification of key targets such as power lines, insulators, towers, trees along the channel, and construction machinery. This embodiment uses the DeepLabV3+ series architecture and a self-built power transmission and distribution channel image dataset as the implementation object, but the method of this invention is not limited to this specific network and data.

[0034] In this embodiment 1, the complete process, including network construction, forward and backward propagation in a certain training batch, and obtaining the final lightweight quantized student model after completing all training, will be described in detail. Figure 1 As shown, it includes the following steps: Step S1: Construction of the teacher-student dual-branch network model and forward pseudo-quantization; In step S1, the specific structures of the teacher model and the student model being trained are first determined and the network parameters are initialized. Then, the self-built power transmission and distribution channel image dataset is used as the input of the dual-branch network model to start the training of the dual-branch network model. The images are put into the dual-branch network model in batches, and the dual-branch network model will extract image features layer by layer, that is, the forward propagation of the dual-branch network model training. At this time, the intermediate values ​​generated by the feature extraction of each layer of the teacher model, i.e., the feature maps of each layer, the logical values ​​of the final layer output of the teacher and student network models, and various parameters of the current dual-branch network model, including the weights, activation, target bit width, etc. of the student model, will be provided.

[0035] Unlike traditional training, step S1 of this method, based on the aforementioned forward propagation, implants simulated quantization pseudo-operation nodes into the trained student model. This involves performing pseudo-quantization operations on the weights and activations of each layer to be quantized in the trained student model, including quantization, rounding, and dequantization pseudo-operations. Therefore, this step also provides the pseudo-quantized weights and activations of the trained student model, as well as pseudo-quantized feature maps.

[0036] In the described teacher-student dual-branch network model, the first branch is the fully trained teacher model, which is characterized by high accuracy but complex structure. All weights and activations are not quantized and remain in 32-bit floating-point format, and no parameter updates are performed during co-training (hereinafter referred to as training). The second branch is the student model, which is the object model for subsequent training and quantization. Therefore, it is designed as a compact and lightweight network structure, initially in 32-bit floating-point format. After training, it is quantized to obtain a lightweight quantized student model, which is then deployed on edge devices such as drones or inspection robots.

[0037] Specifically, step S1 includes: S1.1, Teacher model structure determined: DeepLabV3+ was selected as the basic architecture of the teacher model. Its backbone network uses ResNet-101 pre-trained on the ImageNet-1K dataset. To obtain denser feature maps, the dilation rate of the 3×3 convolutions in the last residual stage of ResNet-101 was adjusted to 2, and the overall output stride was set to 16. Following the backbone network, a dilated spatial pyramid pooling module is connected, containing one 1×1 convolutional branch, three 3×3 dilated convolutional branches with dilation rates of 6, 12, and 18, respectively, and a global average pooling branch. The output channels of all branches are set to 256 dimensions. This teacher model has been trained to full precision on a power transmission and distribution channel image dataset, achieving an average intersection-over-union (IoU) ratio of 81.2% on the validation set, with approximately 62.7 million parameters and approximately 279 billion floating-point operations.

[0038] S1.2, Student model structure determined: The student model uses the lightweight MobileNetV2 network as its feature extraction backbone and constructs a simplified semantic segmentation head. All inverted residual module sequences of MobileNetV2 are retained, but the channel expansion factor of each layer is uniformly reduced from 6 times to 3 times to further compress the number of parameters. The input resolution of the student model remains consistent with that of the teacher model. In the final output layer of the MobileNetV2 backbone, a feature map with a stride of 16 is taken, and the number of channels is set to 320 dimensions. The segmentation head uses a 3×3 convolution to reduce the number of channels in the backbone feature map to 128 dimensions, and then concatenates it with a low-level feature map from the backbone network with a stride of 4. This low-level feature map is also compressed to 32 dimensions using a 3×3 convolution. The concatenated 160-dimensional hybrid feature is refined sequentially through two depthwise separable convolutions, and finally a 1×1 pointwise convolution maps the number of channels to the number of target categories. A prediction map containing target category information for each pixel is obtained through bilinear interpolation upsampling. The total number of parameters in the student model was reduced to approximately 3.8 million, and the number of full-precision floating-point calculations was approximately 34 billion, about one-eighth that of the teacher model.

[0039] After setting up the dual-branch network model structure in step S1.2, the self-built power transmission and distribution channel image dataset is used as the input to the dual-branch network model, and the iterative training process is officially started. The self-built power transmission and distribution channel image dataset will be added to the current dual-branch network model in batches. It should be noted that the steps after the start of the current training and the subsequent steps S2, S3, and S4 are all executed in each training batch or according to conditions. Step S1.2 is responsible for completing the forward propagation of each batch of training, that is, performing image feature extraction layer by layer of the dual-branch network model until the output layer outputs the results. Finally, it will provide the intermediate values ​​generated after feature extraction of each layer of the teacher model, i.e., the feature maps of each layer, the final layer output logical values ​​of the teacher and student network models, and various network parameters of the current dual-branch network model, including the weights, activations, target bit widths, etc. of the current student model, for use by steps S2, S3, and S4. At this point, the forward propagation of the current batch is completed.

[0040] S1.3, Insert simulated quantization pseudo-operation nodes into the student model: During forward propagation, pseudo-quantization operations are inserted into all convolutional layers of the student model (including components of standard convolution, pointwise convolution, and depthwise separable convolution) and the final 1×1 pointwise convolutional layer of the segmentation head. This invention employs an asymmetric uniform quantization scheme based on channel-wise weighted quantization and layer-wise activation quantization.

[0041] For the weight tensor of each layer to be quantized in the student model Given a target bit width of b bits, the pseudo-quantization process is described below. First, the quantization scale factor for this layer is calculated. It is important to note the quantization scaling factor. This is a vector whose length is equal to the number of output channels, containing the quantization scale factor for each channel, and employing the maximum absolute value method combined with a small constant. Preventing division by zero error, i.e., quantization scaling factor Then, treat the weight tensors of the quantization layer. Perform quantization, rounding, and dequantization operations as shown in formula (1):

[0042] For activation values, asymmetric uniform quantization is also used. Let the activation tensor of a certain layer be... Target width For each bit, first calculate the quantization scale factor of the activation. and zero point : The activation of pseudo-quantization is shown in equation (1b):

[0043] in, This indicates rounding to the nearest integer. For element-wise pruning, integer values ​​are strictly limited to the range representable by the target bit width. During the training phase, pseudo-quantized weights... and pseudo-quantization activation All parameters are kept in floating-point form to support backpropagation gradient flow, with gradients bypassing non-differentiable parameters via a pass-through estimator. The function is passed.

[0044] Step S2, quantitative sensing of dynamic distillation temperature adjustment; Traditional distillation relies on a constant temperature coefficient, i.e., distillation temperature, to soften the output logic value of the teacher model, generating labels containing inter-class similarity information, i.e., soft labels. However, when the student model is subject to low-bit quantization constraints, its representation space becomes distorted due to severe rounding noise. If a fixed distillation temperature is still used for knowledge transfer in this case, the student model is highly likely to misjudge rounding noise as valid knowledge and fit it. To address this, this invention proposes a quantization-aware dynamic distillation temperature adjustment mechanism.

[0045] like Figure 2 As shown, after step S1 completes the first stage of training in batches, i.e., the forward propagation of the dual-branch network model training, step S2 is performed once in each training batch. First, based on the weights of the layers to be quantized obtained in step S1... Pseudo-quantized weights recovered by pseudo-quantization Activation of the layer to be quantized Pseudo-quantization activation recovered by pseudo-quantization The real-time global quantization error is calculated to obtain the dynamic distillation temperature for this batch. Then, the output logical values ​​of each pixel's target category after the forward propagation of the latest dual-branch network model in step S1 are obtained. Using the dynamic distillation temperature and the output logical values, the distillation loss is finally calculated and output. This loss is used in step S4 to complete the second stage of training and guide the parameter updates of subsequent student models, i.e., the backpropagation of the dual-branch network model training. It is important to note that the dynamic distillation temperature is the same for the same training batch.

[0046] S2.1, Batch-level global quantization error metric: For the l-th layer, its weight quantization error Defined as weights to be quantized With pseudo-quantization weights Mean square error; activation quantization error Defined as activation to be quantified With pseudo-quantization activation The mean squared error. This embodiment uses a layer-by-layer weighted averaging strategy to calculate the global average quantization error. :

[0047] in, This represents the total number of elements in the weight tensor of this layer. This represents the total number of elements in the activated tensor of this layer. The total number of layers of pseudo-quantization operations were inserted into the student model.

[0048] S2.2, Calculating the dynamic distillation temperature: This embodiment employs a nonlinear dynamic distillation temperature adjustment function with a smooth upper bound. A reference temperature is introduced. Set to 4.0; introduce a temperature upper limit. The value is 10.0. The dynamic distillation temperature adjustment function is shown in formula (3):

[0049] in, This is the amplitude scaling factor, with a value of 1.5; This is a normalization reference value, set to 0.05, and its function is to account for the quantization error of the current batch. Normalized to the main sensitive region of the tanh function (typically 0~1), 0.05 here corresponds to the expected average quantization error level in the initial training phase; tanh is a hyperbolic tangent function, and its saturation characteristics ensure that the dynamic distillation temperature approaches the upper temperature limit when the global average quantization error is maximized. Instead of growing indefinitely.

[0050] S2.3, Calculation of distillation loss: Get the dynamic distillation temperature of the current batch Then, it was applied to soften the output of the teacher and student models.

[0051] Suppose the unnormalized logic value output by the teacher model is The output logic value corresponding to the student model is The teacher model is obtained through the temperature-controlled Softmax function. Soft label probabilities for each semantic category: Similarly, the student model is obtained. Soft label probability of each semantic category Distillation loss The KL divergence form is adopted, and multiplied according to the theory of Hinton et al. The scaling factor is used to balance the gradient magnitude, as shown in Equation (4):

[0052] Where K is the number of semantic categories. As the quantization error increases, the dynamic distillation temperature... Automatic elevation results in a smoother probability distribution for soft labels, preventing students from becoming overconfident in noisy hard labels generated with low precision.

[0053] Step S3: Layer-by-layer collaborative quantization bit width allocation based on information bottleneck; To overcome the limitations of globally unified quantization, training is paused when the training batch t reaches a certain set condition. Step S3 proceeds, using independent validation set images as input into the current teacher-student dual-branch network model with pseudo-quantization. The key feature extraction layers of the currently trained student model with pseudo-quantization are evaluated. Specifically, the mutual information estimate of the feature maps for each layer is calculated for multiple key feature extraction layers of the teacher model and the student model with pseudo-quantization. Based on the magnitude of the mutual information estimate, the redundancy of the semantic representation of that layer is evaluated. Based on the evaluation results, the target bit width adjustment decision for the currently trained student model is finally output, the quantization configuration of each layer is updated, and subsequent training continues.

[0054] like Figure 3 As shown, this step presents the complete engineering process and mathematical implementation from calculating the mutual information estimate to deriving the target bit width adjustment decision.

[0055] S3.1, Collect feature maps of teachers and pseudo-quantified feature maps of students at the same level: In this embodiment, evaluation is first triggered after the student model training process reaches 25% of the total number of epochs, which is batch n. Subsequent evaluations are performed in batches n+200, n+400, etc. When evaluation is triggered, multiple key feature extraction layers of common interest to both the teacher and student models are sampled. These commonly interested extraction layers are layers in the student model that have the same semantic mapping as the teacher model. For example, the output of the inverted residual module of the student MobileNetV2 corresponds to the output of the corresponding residual block of the teacher ResNet-101. In this embodiment, mutual information estimation and dynamic bit width adjustment are performed only on the 19 selected key layers. The remaining unselected layers (such as low-level convolutional layers, BN layers, etc.) retain their current quantization bit width, with an initial bit width uniformly set to 8 bits. The initial weight and activation quantization bit width of all layers in the student model (including the 19 layers to be evaluated and other layers) are set to 8 bits. Subsequently, based on the feedback from mutual information, the quantization bit width of the 19 layers to be evaluated was adjusted step by step (each adjustment was 2 bits, for example, from 8 bits to 6 bits or up to 10 bits), while the quantization bit width of other layers remained unchanged at 8 bits. Specifically, this embodiment selects the outputs of 17 inverted residual modules of MobileNetV2 and the inputs of two pointwise convolutional layers in the segmentation head as the layers to be evaluated, totaling 19 layers. The teacher model (ResNet-101 backbone + DeepLabV3 + head) correspondingly selects its 17 residual stages (including...) to The output of the key residual block in the model, and the input of the pointwise convolutional layer after its dilated spatial pyramid pooling module, totaling 19 layers, are used as alignment layers for mutual information computation. The size of the teacher feature map is adjusted to match the student pseudo-quantized feature map using bilinear interpolation. Figure 1 This is to facilitate channel stitching and mutual information estimation. For each layer... Input 256 images from the validation set, and obtain teacher feature map sets respectively. and student pseudo-quantized feature map set .

[0056] S3.2, Mutual information calculation based on neural network estimator: To adapt to mutual information estimation in high-dimensional feature map spaces, this invention employs a mutual information neural estimator based on variational information maximization. For layers... Teacher feature map and student pseudo-quantization feature map The data is concatenated along the channel dimension to serve as a sampled representation of the joint distribution, while also incorporating the teacher feature map. and unrelated feature maps from another random image The feature maps are concatenated along the channel dimension as a sampled representation of the marginal distribution product. These concatenated feature maps are then fed into a discriminator network. In the process, the discriminator network is trained. This approach distinguishes between teacher-student feature map pairs from the same input image that are jointly distributed, and irrelevant feature map pairs from different input images that are products of marginal distributions, thereby maximizing the lower bound of mutual information. Mutual information The lower bound is estimated as shown in formula (5): (5)

[0057] in, This represents the channel splicing operation. The discriminator network is trained accordingly. This maximizes the term on the right-hand side of equation (5), resulting in a reliable estimate of mutual information. .

[0058] S3.3, Dual Threshold Feedback and Bit Width Adjustment Decision: Preset lower threshold and upper limit threshold For each layer l: If the mutual information estimate is less than the lower threshold, i.e. If the current bit width has not reached the preset hardware limit, it indicates that the knowledge distillation of this layer has encountered a bottleneck and can be adjusted. Therefore, the current weight and activation quantization bit width of this layer will be forcibly increased by one level.

[0059] If the mutual information estimate is less than the lower threshold, i.e. If the current bit width has reached the preset hardware limit, then the update of the quantization bit width of this layer will be paused, and this layer will not participate in pseudo-quantization operations in subsequent training to avoid further performance degradation until the next re-evaluation.

[0060] If the mutual information estimate is greater than the upper threshold, then... If this indicates that there is a lot of redundancy in the semantic representation of this layer, then the current weights and activation quantization bit width of this layer will be forcibly reduced by one level.

[0061] If the mutual information estimate is greater than or equal to the lower threshold and less than or equal to the upper threshold, then... The layer width remains unchanged.

[0062] In this embodiment, each "level" of bit width adjustment refers to 2 bits (e.g., adjusting from 8 bits to 6 bits or 10 bits). The above operations complement each other and together constitute the complete strategy of quantization bit width feedback control.

[0063] The target bit width adjustment decision will take effect at the start of the next training batch, i.e., in the forward propagation step S1 used for training the dual-branch network model for the next training batch.

[0064] This layer-by-layer collaborative quantization bit width allocation strategy allows the key feature extraction layer to maintain high-precision computation, while the smoothing or dimensionality reduction layer is extremely compressed.

[0065] Step S4: Gradient-level collaborative balancing and multi-objective joint optimization converge; like Figure 4 As shown, in each training batch, after S1 and S2, the loss values ​​of each target are received, the joint loss of the multi-target is calculated, and the gradient components of the distillation loss and the quantization error loss are decomposed. A trainable balancing parameter is introduced to create a cooperative gradient with respect to the distillation and quantization errors. By training the balancing parameter, the update conflict between the two gradient components in different directions is reduced. Finally, the cooperative gradient is used to perform a gradient descent update on the weights to obtain the latest dual-branch network model.

[0066] This step details how to resolve the optimization conflict caused by the two gradient directions of distillation and quantization during the backpropagation phase, and finally complete the update of the student model parameters.

[0067] S4.1, the specific form of the multi-objective joint loss function: Multi-objective joint loss function driving student model training As described in formula (6).

[0068]

[0069] In this embodiment, For image semantic analysis tasks, a ground truth label cross-entropy loss method is used to ensure the model's basic task performance. To account for quantization error loss, Set to 0.7, As the training progresses, the cosine annealing is gradually increased from 0.1 to 0.5.

[0070] S4.2, Gradient Separation and Co-weighting: In the automatic differentiation framework, distillation losses are calculated separately. and quantization error loss Regarding the student model Layer weights The gradient is used to obtain the first gradient component. Second gradient component Because both serve to improve semantic accuracy, the task loss... The gradient can be incorporated into the first gradient component. In this study, a learnable equilibrium parameter with an initial value of 0.5 and constrained to the interval [0.1, 0.9] is introduced. Cooperative gradient The calculation is shown in formula (7).

[0071]

[0072] S4.3, Model Parameter Update and Deployment: Learnable equilibrium parameters The update aims to minimize the conflict between the two gradient update directions. The gradient conflict metric C is defined as the negative inner product of the two, as shown in Equation (8):

[0073] In each training batch, the learnable equilibrium parameter λ is updated once, and then the collaborative gradient is applied. Update the current network parameters to obtain the new first gradient components in the next batch. Second gradient component Then, the learnable equilibrium parameter λ is updated in the next batch.

[0074] This invention employs a meta-gradient update strategy to measure gradient conflict. Treated as a learnable equilibrium parameter The function calculates the gradient of the gradient conflict metric with respect to the learnable equilibrium parameter using the chain rule. ,because and Since these are difficult to obtain directly, an online approximation is used, which, after updating the weights, is based on the gradient components of the current batch. and Adjust the learnable equilibrium parameters according to the following rules. : (9) In this embodiment, the learnable equilibrium parameters The updated formula is as follows: (10) Learning rate , The function restricts the data range to between 0.1 and 0.9. This formula allows the learnable balancing parameter λ to increase or decrease depending on the sign when gradients conflict, thus adaptively balancing the two gradient components. Experiments demonstrate the effectiveness of this heuristic method.

[0075] Finally, the learnable balance parameter that minimizes the gradient conflict metric C is retained. Substituting into formula (7), we obtain the cooperative gradient at this time. Using collaborative gradients Weights Perform a standard stochastic gradient descent update with momentum, thus completing the training and parameter update of one batch of student models.

[0076] After completing one batch of training, the network parameters of the student model are retained. If there is another batch of training, the latest dual-branch network model and the target bit width adjustment decision are used as the initial parameters for the next batch of training step S1, and steps S1, S2 and S4 are repeated in sequence.

[0077] It is important to note that after completing the training of all batches, the network parameters of the student model are retained, and the pseudo-operation nodes of the analog quantization are removed. The final quantization is then performed based on the target bit width of each layer at this time to obtain a lightweight quantized student model. This model is embedded in the edge device to run in real time, enabling real-time identification of key targets such as power lines, insulators, towers, trees along passageways, and construction machinery.

[0078] After approximately 60 cycles of co-training on the power transmission and distribution channel dataset, the output lightweight quantized student model achieved semantic boundary characterization capabilities almost equivalent to the full-precision student model under mixed-precision conditions of 4 to 6 bits. The inference latency on the embedded NVIDIA Jetson Orin module was about 17 times faster than the full-precision teacher model, and the memory usage was only 6% of the original model. Therefore, the final lightweight quantized student model in this embodiment fully meets the requirements of low latency and low power consumption for real-time video inspection of power transmission and distribution channels.

[0079] Furthermore, this invention provides an image semantic analysis system based on knowledge distillation and quantization co-optimization, used to implement the aforementioned image semantic analysis method based on knowledge distillation and quantization co-optimization. For example... Figure 5 As shown, the system comprises four main modules: network construction and forward pseudo-quantization module (module 1), dynamic distillation temperature adjustment module based on quantization perception (module 2), layer-by-layer collaborative quantization bit width allocation module based on information bottleneck (module 3), and gradient-level collaborative balancing and multi-objective joint optimization convergence module (module 4).

[0080] The network construction and forward pseudo-quantization module is connected to the quantization-aware dynamic distillation temperature adjustment module, the information bottleneck-based layer-by-layer collaborative quantization bit width allocation module, and the gradient-level collaborative balancing and multi-objective joint optimization convergence module. The network construction and forward pseudo-quantization module outputs the latest dual-branch network model parameters to the quantization-aware dynamic distillation temperature adjustment module, the information bottleneck-based layer-by-layer collaborative quantization bit width allocation module, and the gradient-level collaborative balancing and multi-objective joint optimization convergence module. Through the calculations of these three modules, the final target bit width adjustment of the student model output by the information bottleneck-based layer-by-layer collaborative quantization bit width allocation module is determined. The backpropagation-updated student model parameters from the decision-making and gradient-level collaborative balancing and multi-objective joint optimization convergence module are fed into the network construction and forward pseudo-quantization module as the latest dual-branch network model parameters and pseudo-quantization operation parameters. The quantization-aware dynamic distillation temperature adjustment module is connected to the gradient-level collaborative balancing and multi-objective joint optimization convergence module. It receives the weights and activations before and after the pseudo-quantization operation from the network construction and forward pseudo-quantization module, calculates the dynamic distillation temperature, softens the dual-branch network model, and calculates the distillation loss. Finally, it outputs the quantization error loss and distillation loss to the gradient-level collaborative balancing and multi-objective joint optimization module. The combined optimization convergence module is used for multi-objective joint loss calculation; the gradient-level collaborative balancing and multi-objective joint optimization convergence module receives the parameters and output results of the dual-branch network model from the network construction and forward pseudo-quantization module, as well as the quantization error loss and distillation loss from the quantization-aware dynamic distillation temperature adjustment module. It is responsible for calculating the multi-objective joint loss and solving the collaborative gradient, updating the current dual-branch network model parameters using gradient descent. If further training is needed, the updated dual-branch network model is output to the network construction and forward pseudo-quantization module for the next training iteration; otherwise, it is not. Training involves quantizing and deploying the updated student model based on the current target bit width for each layer using a gradient-level collaborative balancing and multi-objective joint optimization convergence module. This results in a lightweight quantized student model in a format usable by edge devices. When training reaches the set conditions, the layer-by-layer collaborative quantization bit width allocation module, based on information bottlenecks, pauses training, receives parameters from the dual-branch network model of the network construction and forward pseudo-quantization modules, calculates the mutual information between the teacher and student feature maps in the key feature extraction layer, and outputs a target bit width adjustment decision to the network construction and forward pseudo-quantization modules. After adjustment, subsequent training resumes.

[0081] The network construction and forward pseudo-quantization module includes: 1.1 The dual-branch network model construction module is used to execute steps S1.1 and S1.2. It is responsible for initializing and continuously managing the network parameters of the high-precision teacher model and student model, including weights, target bit width, etc., as well as the feature maps of each layer generated in the forward propagation and the output logical values, for other modules to call.

[0082] 1.2 The pseudo-quantization operation module, connected to the dual-branch network model construction module, is used to execute step S1.3, inserting simulated quantization pseudo-operation nodes in the forward propagation of the student model to perform quantization, rounding, and dequantization pseudo-operations on the weights and activations of each layer to be quantized in the student model. This module ultimately provides pseudo-quantized weights, pseudo-quantized activations, and pseudo-quantized feature maps.

[0083] The quantitative sensing dynamic distillation temperature adjustment module includes: 2.1 The global quantization error calculation module, connected to the network construction and forward pseudo-quantization module, is used to execute step S2.1, receiving the pseudo-quantization weights, pseudo-quantization activations, and weights of the layer to be quantized from the network construction and forward pseudo-quantization module. Once the quantization layer is activated, the global average quantization error is calculated and passed to the dynamic distillation temperature calculation and update module.

[0084] 2.2 The dynamic distillation temperature calculation and update module receives the global average quantization error from the global quantization error calculation module and performs step S2.2 to calculate the nonlinear dynamic distillation temperature and transmit it to the distillation loss calculation module.

[0085] 2.3 The distillation loss calculation module, connected to the dynamic distillation temperature calculation and update module and the network construction and forward pseudo-quantization module, is used to execute the above step S2.3, receive the nonlinear dynamic distillation temperature and the output logic values ​​of the teacher and student models generated by the network construction and forward pseudo-quantization module, calculate the distillation loss, and pass it to the gradient-level collaborative balancing and multi-objective joint optimization convergence module.

[0086] The gradient-level collaborative balancing and multi-objective joint optimization convergence module includes: 4.1 The multi-objective joint loss calculation module is connected to the network construction and forward pseudo-quantization module and the quantization-aware dynamic distillation temperature adjustment module. It is used to execute the above step S4.1, receive the distillation loss and quantization error loss, calculate the true label cross-entropy loss, and finally output the multi-objective joint loss, which is then passed to the gradient separation and collaborative weighting module.

[0087] 4.2 Gradient Separation and Co-weighting Module: Receives the multi-objective joint loss from the multi-objective joint loss calculation module and performs step S4.2, decomposing the multi-objective joint loss into distillation loss gradient components and quantization error loss gradient components to obtain a co-weighted gradient with learnable balancing parameters, which is then passed to the model update and deployment module.

[0088] 4.3 The model update and deployment module receives the collaborative gradient from the gradient separation and collaborative weighting module and uses it to execute step S4.3 to optimize and update the learnable equilibrium parameters, reduce the conflict between the two gradient update directions, and calculate the collaborative gradient when the gradient conflict metric is minimized. The collaborative gradient is then used to perform a standard driven stochastic gradient descent update on the weights to obtain the latest dual-branch network model.

[0089] If further training is needed, the latest dual-branch network model is passed to the network building and forward pseudo-quantization module for subsequent training; if there is no further training, the model update and deployment module quantizes the current student model to obtain the final lightweight quantized student model, which is then exported in a format usable by edge devices.

[0090] The layer-by-layer collaborative quantization bit width allocation module based on information bottlenecks includes: 3.1 The key feature extraction layer sampling module is connected to the network construction and forward pseudo-quantization module. It is used to execute the above step S3.1, input independent validation set images to the current teacher model and the student model with pseudo-quantization operation, extract feature maps of multiple key feature extraction layers, and pass them to the mutual information calculation module.

[0091] 3.2 Mutual information calculation module receives student feature maps and teacher feature maps from the key feature extraction layer sampling module, and performs step S3.2 by using a mutual information neural estimator to calculate the mutual information estimate values ​​of each layer and pass them to the dual threshold feedback and bit width adjustment module.

[0092] 3.3 The dual-threshold feedback and bit-width adjustment module receives the mutual information estimates for each layer from the mutual information calculation module and executes step S3.3. Based on the comparison between the mutual information estimate for each layer and the preset threshold, it dynamically adjusts the quantization bit width of each layer. The final target bit-width adjustment decision is then passed to the network construction and forward pseudo-quantization module, which updates the target bit width for the corresponding layer.

[0093] Furthermore, the present invention also provides an electronic device, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, it implements the above-mentioned image semantic analysis method based on knowledge distillation and quantization co-optimization.

[0094] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described image semantic analysis method based on knowledge distillation and quantization co-optimization.

[0095] Example 2: Semantic segmentation network applied to the Transformer architecture; To demonstrate the architectural generalization capability of the method of this invention, this embodiment uses a semantic segmentation network based on a Swing Transformer backbone as an example. The teacher model adopts a large-scale structure of Swing-L backbone combined with Mask2Former segmentation head, while the student model adopts Swing-T backbone combined with lightweight decoding head. Pseudo-quantization operations are simultaneously inserted into the multilayer perceptron linear layer and the QKV projection layer in the self-attention calculation of the Transformer. The mutual information evaluation layer, i.e., the key feature extraction layer, focuses on selecting the output features of each Swing Stage and the encoder-decoder cross-attention layer. In the temperature adjustment function, since the Transformer is more sensitive to quantization, It was upgraded to 5.0. The version was adjusted to 12.0. The subsequent process is basically the same as in Example 1. Experiments show that this method can also successfully compress the 32-bit floating-point Swin-T model to 4 bits for weights and 8 bits for activation. The average crossover ratio on the power transmission and distribution channel dataset decreased by only 1.9 percentage points, which is significantly better than the 7.2 percentage point decrease after direct quantization perception training. It is also applicable to the fine target recognition task in line inspection.

[0096] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0097] In the description of this invention, it should be understood that the terms "center," "height," "thickness," "upper," "lower," "vertical," "horizontal," "top," "bottom," "inner," "outer," "axial," "radial," and "circumferential," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0098] In the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0099] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0100] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. An image semantic analysis method based on knowledge distillation and quantization synergistic optimization, characterized in that, include: Step S1: Construct a dual-branch network model. During training, insert simulated quantization pseudo-operation nodes during forward propagation to generate intermediate values ​​and network parameters for each layer. Step S2: In each training batch, monitor the quantization error of the simulated quantization pseudo-operation node after forward propagation, and generate dynamic distillation temperature and distillation loss; Step S3: When training reaches the set conditions, calculate the mutual information between the feature maps of the teacher model and the student model, and compare it with the preset dual thresholds to obtain the bit width adjustment decision; Step S4: Upon receiving the forward propagation loss, construct a joint loss function, introduce learnable parameters to generate collaborative gradients, update the student model parameters, and complete one batch of training; If there is a next batch of training, the current model is used as the parameter of step S1, and steps S1 to S4 are repeated; If not, quantize according to the target bit width to obtain the final model and deploy it.

2. The image semantic analysis method based on knowledge distillation and quantization synergistic optimization according to claim 1, characterized in that, The insertion of the analog quantization pseudo-operation node in step S1 specifically includes: Simulated quantization pseudo-operation nodes are inserted into the convolutional layers of the student model in the dual-branch network model, and the weight tensors of each layer to be quantized in the student model are... and target bit width Bits, quantization scale factor ,in It is a constant; Weight tensors of the quantization layer Perform quantization, rounding, and dequantization operations to obtain pseudo-quantized weights: ， The activation tensor of each layer to be quantized in the student model is Target width Bits, the quantization scaling factor for calculating activation and zero point : ; The activation tensor of the quantization layer is Performing quantization, rounding, and dequantization operations yields pseudo-quantization activation: ， in, This indicates rounding to the nearest integer. This is an element-by-element clipping operation.

3. The image semantic analysis method based on knowledge distillation and quantization synergistic optimization according to claim 2, characterized in that, The specific steps in step S2 to generate the dynamic distillation temperature include: Calculate the weights of the layer to be quantized within the current training batch. With pseudo-quantization weights The mean square error between them, the activation of the quantization layer With pseudo-quantization activation The mean square error between them yields the weighted average quantization error generated by all simulated quantization pseudo-operation nodes. ; Define a dynamic distillation temperature adjustment function and calculate the adjusted dynamic distillation temperature. ; Utilizing the adjusted dynamic distillation temperature The logic value output by the teacher model is softened to obtain the dynamic distillation temperature dependence. Distillation loss , used to construct the joint loss function.

4. The image semantic analysis method based on knowledge distillation and quantization synergistic optimization according to claim 3, characterized in that, The dynamic distillation temperature adjustment function is in linear form: ,in, Based on the base temperature, A positive regulatory factor; Alternatively, the dynamic distillation temperature adjustment function may take a nonlinear form: ， in, The preset upper temperature limit, This is a normalized reference value. It is a hyperbolic tangent function used to smooth temperature growth when the quantization error is extremely large.

5. The image semantic analysis method based on knowledge distillation and quantization co-optimization according to claim 1, characterized in that, Step S3 specifically includes: For the student model, the first In the evaluation phase, a simulated quantization pseudo-operation node is introduced, and the teacher model is calculated using a mutual information neural estimator. Layer feature map With student model Layer pseudo-quantization feature map Mutual information between ; Set mutual information lower limit threshold With upper limit threshold ; like If the current bit width has not reached the preset hardware limit, it is determined that the knowledge loss of this layer is serious, and an instruction to increase the weight and activated quantization bit width of this layer by one level is triggered. like If the current bit width has reached the preset hardware limit, then the update of the quantization bit width of this layer will be paused, and this layer will not participate in pseudo-quantization operations in subsequent training to avoid further performance degradation until the next re-evaluation. like If the information in that layer is deemed redundant, an instruction is triggered to reduce the quantization bit width of the weight and activation of that layer by one level. like If so, the bit width of that layer remains unchanged.

6. The image semantic analysis method based on knowledge distillation and quantization synergistic optimization according to claim 1, characterized in that, The expression for the joint loss function is: ， in, The cross-entropy loss is the true label of the task. For dynamic distillation temperature dependence Distillation loss, To incorporate the quantization error loss guided by layer-by-layer mutual information, and These are the preset hyperparameters used to balance the weights of each loss term.

7. The image semantic analysis method based on knowledge distillation and quantization co-optimization according to claim 6, characterized in that: In backpropagation, according to the joint loss function Calculate the distillation loss respectively. Quantization error loss True label cross-entropy loss Regarding the first Layer weights The gradient is calculated by using the sum of the gradients of the distillation loss and the true label cross-entropy loss as the first gradient component. The gradient component of the quantization error loss is used as the second gradient component. ; Learnable equilibrium parameters Calculate the cooperative gradient: and utilize the cooperative gradient Update weights ; Among them, the learnable equilibrium parameter To minimize the first gradient component With the second gradient component Gradient conflicts between parameters are used to update the optimization objective.

8. The image semantic analysis method based on knowledge distillation and quantization synergistic optimization according to claim 7, characterized in that, The gradient conflict is resolved through the negative inner product of the two gradient components. The learnable equilibrium parameter is measured. The update rules are as follows: ， in, For learnable equilibrium parameters The learning rate; Or the learnable equilibrium parameter The update rules are as follows: ， in, The function limits the data range to between 0.1 and 0.

9.

9. The image semantic analysis method based on knowledge distillation and quantization synergistic optimization according to claim 5, characterized in that, The mutual information is estimated using a mutual information neural estimator based on variational information maximization, for each layer. Teacher feature map and student pseudo-quantization feature map The data is concatenated along the channel dimension as a sampled representation of the joint distribution, while also incorporating teacher feature maps. and unrelated feature maps from another random image Concatenated along the channel dimension, serving as a sampled representation of the marginal distribution product; mutual information. The estimate is as follows: ， By training the discriminator This maximizes the terms on the right-hand side of the formula, resulting in a reliable estimate of mutual information.

10. An image semantic analysis system based on knowledge distillation and quantization collaborative optimization, characterized in that, For implementing the image semantic analysis method based on knowledge distillation and quantization co-optimization as described in any one of claims 1 to 9, the system comprises: Network construction and forward pseudo-quantization module, dynamic distillation temperature adjustment module with quantization perception, layer-by-layer collaborative quantization bit width allocation module based on information bottleneck, gradient-level collaborative balancing and multi-objective joint optimization convergence module; The network construction and forward pseudo-quantization module outputs the intermediate values ​​of each layer of the current dual-branch network and the network parameters. The dynamic distillation temperature adjustment module with quantization perception is connected to the network construction and forward pseudo-quantization module and the gradient-level collaborative balancing and multi-objective joint optimization convergence module. It receives quantization error and calculates dynamic distillation temperature and distillation loss. The layer-by-layer collaborative quantization bit width allocation module based on information bottlenecks calculates the mutual information between the current network construction and the dual-branch network model of the forward pseudo-quantization module and performs a dual threshold comparison to give the corresponding layer bit width adjustment decision. The gradient-level collaborative balancing and multi-objective joint optimization convergence module receives the parameters and output results of the dual-branch network model of the network construction and forward pseudo-quantization module, as well as the quantization error loss and distillation loss of the quantization-aware dynamic distillation temperature adjustment module. It calculates the joint loss and solves the collaborative gradient, updates the parameters of the current dual-branch network model of the network construction and forward pseudo-quantization module using gradient descent, and quantizes and derives the trained student model.

Images