A low bit-rate semi-reference image quality inspection method and system based on self-distillation

Abstract

This application discloses a low-bitrate semi-reference image quality inspection method and system based on self-distillation, mainly relating to the field of image quality inspection technology. It addresses the problem of how to obtain sufficient reference information to support high-precision evaluation under extremely low bitrate conditions and complete quality evaluation in real time on a chip with limited computing power. The method includes: using a difference feature tensor, the teacher network obtains the quality prediction confidence index and distortion classification prediction confidence index for each image sample in the image sample set, thereby determining the type of each image sample; where the types are divided into: low uncertainty, medium uncertainty, and high uncertainty; based on the difference feature tensor and the type of each sample, the teacher network parameters are frozen to obtain a trained student network; the difference feature tensor of the test data is obtained, and using the student network, a subjective quality score and several preset distortion probability vectors are output, which are taken as the dominant distortion type of the test data.

Description

Technical Field

[0001] This application relates to the field of image quality inspection technology, and in particular to a method and system for quality inspection of low bit-rate semi-reference images based on self-distillation. Background Technology

[0002] In smart IoT scenarios, intelligent vision terminals (such as NB-IoT security cameras, remote agricultural micro cameras, and pole inspection cameras) are typically deployed in edge environments with limited uplink bandwidth and computing resources. In these scenarios, the terminals need to continuously report images or video streams, while the backend needs to monitor the image quality in real time in order to dynamically adjust the encoding bitrate, trigger error retransmissions, or issue alarm commands.

[0003] Existing image quality assessment technologies all have significant shortcomings in smart IoT environments: Full-reference image quality assessment relies on complete original images for comparison, requiring the additional transmission of hundreds of KB or even MB of lossless or high-quality reference images, exceeding the carrying capacity of IoT links; No-reference image quality assessment relies solely on the characteristics of distorted images themselves, which, although consuming the least bandwidth, makes it difficult to guarantee prediction accuracy when faced with mixed distortion scenarios such as compression artifacts, random noise, motion blur, and color drift; Existing semi-reference assessment methods use low-level statistical features such as histograms and edges, which, although having low reference bitrates, are insufficiently adaptable to mixed structure-color distortions; At the same time, the number of network parameters for difference modeling generally exceeds 20M, making it impossible to run in real time on low-power MCUs or edge gateways.

[0004] Therefore, how to obtain sufficient reference information to support high-precision evaluation under extremely low bit rate conditions and complete the quality evaluation in real time on chips with limited computing power has become a core technical problem that smart IoT terminals urgently need to solve. Summary of the Invention

[0005] This application provides a low-bitrate semi-reference image quality inspection method and system based on self-distillation, to solve the problem of how to obtain reference information sufficient to support high-precision evaluation under extremely low bitrate conditions, and to complete the quality evaluation in real time on a chip with limited computing power.

[0006] In a first aspect, this application provides a low bit-rate semi-reference image quality inspection method based on self-distillation, the method comprising:

[0007] A set of image samples from an intelligent vision terminal is acquired, and normalized images are obtained by performing standardization processing; the image sample set consists of several image samples.

[0008] A frozen basic image encoder network model is used to extract the image semantic vector of the normalized image.

[0009] The image semantic vector is multiplied and quantized to obtain the reference bitstream;

[0010] The distorted image corresponding to the image sample and the reference bitstream are sent synchronously to the receiving end. The receiving end obtains the reconstructed semantic vector corresponding to the image sample based on the reference bitstream.

[0011] Based on the reconstructed semantic vectors and the distorted image, a pseudo-reference image is generated;

[0012] Perform cross-image difference alignment between the pseudo-reference image and the distorted image within a preset salient region to obtain the difference feature tensor;

[0013] Based on the differential feature tensor, the inputs to the teacher network's quality regression subheader and distortion classification subheader are obtained. Using the differential feature tensor, the teacher network obtains the quality prediction confidence index and distortion classification prediction confidence index for each image sample in the image sample set, thereby determining the type of each image sample. The types are divided into: low uncertainty, medium uncertainty, and high uncertainty. Based on the differential feature tensor and the type of each sample, the teacher network parameters are frozen to obtain the trained student network.

[0014] Obtain the differential feature tensor of the test data, and using the trained student network, output subjective quality scores and several preset distortion probability vectors. ,Pick This is the dominant distortion type for the test data.

[0015] In one implementation of this application, a sample set of intelligent vision terminals is obtained, and normalized images are obtained by performing standardization processing, specifically including:

[0016] Terminal camera collects intelligent vision terminal sample set The resolution is fixed at 1280*720 pixels, and the frame format is YUV-420.

[0017] right A 3×3 Gaussian filter kernel is applied to each of the three chromaticity components;

[0018] Histogram equalization is applied to adjust the brightness distribution, making the pixel grayscale uniform.

[0019] The processed image is converted to RGB and normalized to the [-1, 1] interval using the following formula.

[0020] In one implementation of this application, a frozen basic image encoder network model is used to extract the image semantic vector of the normalized image, specifically including:

[0021] Normalized image Feed into the basic image encoder; via the formula:

[0022] Read the CLS Token from the last layer of the basic image encoder network to obtain the image semantic vector. ;

[0023] in, This is the frozen base image encoder network model; This represents the i-th semantic dimension.

[0024] In one implementation of this application, the image semantic vector is multiplied and quantized to obtain a reference bitstream, specifically including:

[0025] According to the preset division formula:

[0026] The image semantic vector is divided into m=16 32-dimensional sub-vectors. ;

[0027] According to the preset formula:

[0028] For each In the corresponding codebook Calculate the Euclidean distance to obtain the first-level quantization index. ;

[0029] Where K represents the number of codewords in each codebook; =[ ] , representing the m-th codebook; {1,...,256} represents the first-level quantization index;

[0030] The 16 first-level quantization indices are stored sequentially as a 16-byte unsigned integer array to obtain the first-level quantization reconstruction vector. ;

[0031] Perform the following operation on each sub-residual: ;in, This indicates a secondary residual index. This represents the first-order quantization residual of the m-th subvector. This represents the jm-th codeword in the m-th residual codebook;

[0032] The first-level quantization index of each sub-vector Sum of secondary residual indexes The index pairs are combined and then compressed into a reference bitstream b using RangeCoder entropy encoding.

[0033] In one implementation of this application, the receiving end obtains the reconstructed semantic vector based on the reference bitstream, specifically including:

[0034] The receiving end obtains reference bitstream b:

[0035] Through the formula:

[0036] Decode the reference bitstream into reconstructed semantic vectors ; This represents the decoding function.

[0037] In one implementation of this application, a pseudo-reference image is generated based on the reconstructed semantic vector and the distorted image, specifically including:

[0038] Reconstruct semantic vectors Input mapping network Output text prompt t.

[0039] With distorted images As initial noise, load the Latent Diffusion model;

[0040] Perform 5 steps of DPM-Solver sampling to obtain a pseudo-reference image. ;in, This represents the 5th step of the LatentDiffusion model, the DPM-Solver backsampler. ∈ This represents a pseudo-reference image.

[0041] In one implementation of this application, cross-image difference alignment is performed on the pseudo-reference image and the distorted image within a preset salient region to obtain a difference feature tensor, specifically including:

[0042] The pseudo-reference image and the distorted image are divided into 16×16 patches, and the token sequence is as follows: ;

[0043] Through the formula:

[0044] Calculate self-attention weights ;

[0045] And select the Top-20% Patch to form set β;

[0046] in, This represents the average attention level for the i-th patch; , This represents the head query / key weight matrix for the h-th term;

[0047] Input the token in β into a 6-layer Cross-ViT and output the differential feature tensor D.

[0048] In one implementation of this application, the teacher network uses a difference feature tensor to obtain the quality prediction confidence index and the distortion classification prediction confidence index for each sample in the sample set, thereby determining the type of each sample, specifically including:

[0049] For the same differential feature tensor D, Dropout is enabled in the teacher network and forward propagation is performed M=3 times to simulate the prediction volatility of the network output, and three sets of quality prediction results {Q(1),Q(2),Q(3)} are obtained, as well as the corresponding distortion type classification probability vector {c(1),c(2),c(3)}.

[0050] Calculate the variance of the three quality prediction scores, and obtain... , as a confidence index for quality prediction;

[0051] Through the formula:

[0052] Calculate the entropy of the three classification probability vectors. It will be used as a confidence index for distorted classification prediction;

[0053] Based on the confidence indices for quality prediction and distortion classification prediction of all samples, the distributions of the confidence indices for quality prediction and distortion classification prediction are obtained.

[0054] Based on the preset quantiles of the distribution, the quality prediction volatility threshold and the classification prediction confidence threshold are obtained;

[0055] When the confidence index of a sample's quality prediction is less than or equal to the quality prediction volatility threshold, and the confidence index of the distorted classification prediction is less than or equal to the classification prediction confidence threshold, the sample is classified as low uncertainty.

[0056] When the confidence index of a sample's quality prediction is greater than the quality prediction volatility threshold or the confidence index of a distorted classification prediction is greater than the classification prediction confidence threshold, the sample is classified as moderately uncertain.

[0057] When the confidence index of a sample's quality prediction is greater than the quality prediction volatility threshold and the confidence index of its distorted classification prediction is greater than the classification prediction confidence threshold, the sample is classified as high uncertainty.

[0058] In one implementation of this application, a student network is used to output a subjective quality score and several preset distortion probability vectors based on the differential feature tensor and the type of each sample. ,Pick The dominant distortion types include:

[0059] Input the difference feature tensor, and after mean pooling and max pooling, we get:

[0060] ;in, () is the mean pooling function. () is the max pooling function;

[0061] based on Perform a two-layer fully connected layer on the quality subheader, with the dimension changing from 384 to 64 to 1, and output the subjective quality prediction. ;

[0062] based on The classification subheader is processed by performing a two-layer fully connected layer, resulting in a dimensionality change from 384 to 64 to 8. The distortion probability is then obtained using Softmax. ;in, ;

[0063] According to the formula:

[0064] Calculate the uncertainty-weighted distillation loss;

[0065] in, , Subjective quality prediction and distortion probability of network output for teachers;

[0066] in , The weights are set as follows:

[0067] ,

[0068] ,

[0069] in, Indicates the preset adjustment factor;

[0070] Backpropagation only updates the student network and the quantization codebook; the teacher network remains frozen.

[0071] Obtain a well-trained student network;

[0072] Input the difference feature tensor into the trained student network, and the network outputs the subjective quality score and the distortion probability vector. ,Pick The dominant distortion type.

[0073] Secondly, this application provides a low bit-rate semi-reference image quality inspection system based on self-distillation, the system comprising:

[0074] The image acquisition module is used to acquire a sample set of intelligent vision terminals and perform standardization processing to obtain normalized images;

[0075] The vector extraction module is used to extract normalized image semantic vectors using a frozen base image encoder network model.

[0076] The bitstream acquisition module is used to perform product quantization on the image semantic vector to obtain a reference bitstream;

[0077] The vector reconstruction module is used to synchronously send the distorted image corresponding to the sample and the reference bitstream to the receiving end. The receiving end obtains the reconstructed semantic vector corresponding to the sample based on the reference bitstream.

[0078] The image generation module is used to generate a pseudo-reference image based on the reconstructed semantic vector and the distorted image;

[0079] The vector acquisition module is used to perform cross-image difference alignment between the pseudo-reference image and the distorted image within a preset salient region to obtain the difference feature tensor.

[0080] The teacher network module is used to obtain the input of the teacher network quality regression subheader and distortion classification subheader based on the differential feature tensor, and to obtain the quality prediction confidence index and distortion classification prediction confidence index of each image sample in the image sample set using the differential feature tensor, thereby determining the type of each image sample; among which, the types are divided into: low uncertainty, medium uncertainty and high uncertainty.

[0081] The student network module is used to freeze the teacher network parameters based on the differential feature tensor and the type of each sample to obtain a trained student network; it also obtains the differential feature tensor of the test data and uses the trained student network to output subjective quality scores and several preset distortion probability vectors. ,Pick This is the dominant distortion type for the test data.

[0082] As can be seen from the above technical solutions, this application has the following advantages:

[0083] I. Reduced reference data bandwidth usage:

[0084] This technology compresses image semantic vectors into a low-bitrate reference bitstream through product quantization. Compared to traditional full-reference methods that transmit lossless images at the hundreds of KB to MB level, it can compress the reference data volume to less than 1 / 100 of the original data volume. This compression mechanism enables real-time transmission of reference information in the limited network bandwidth environment of smart IoT, breaking through the bottleneck of IoT link capacity.

[0085] II. Improved evaluation accuracy in multi-distortion mixed scenes:

[0086] By using pseudo-reference image generation and cross-image difference alignment techniques, the system can effectively capture mixed distortion features such as compression artifacts, random noise, motion blur, and color shift. The teacher network's twin-head structure (quality regression and distortion classification) combined with an uncertainty assessment mechanism enables the system to maintain high prediction accuracy even when facing complex distortion scenarios, overcoming the shortcomings of traditional no-reference methods in terms of accuracy.

[0087] Third, real-time computing for edge devices was achieved:

[0088] By employing a self-distillation training strategy, a lightweight student network is trained by freezing the teacher network parameters, keeping the model parameter size below 5M. Combined with the efficient computational characteristics of product quantization, this application enables real-time evaluation on computationally limited MCUs or edge gateways, meeting the stringent real-time requirements of smart IoT terminals. Attached Figure Description

[0089] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 This is a flowchart of a low bit-rate semi-reference image quality inspection method based on self-distillation, provided in an embodiment of this application.

[0091] Figure 2 This is a schematic diagram of the internal structure of a low bit-rate semi-reference image quality inspection system based on self-distillation, provided in an embodiment of this application. Detailed Implementation

[0092] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0093] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0094] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0095] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0096] The embodiment provides a low bit-rate semi-reference image quality inspection method based on self-distillation, such as Figure 1 As shown in the embodiments of this application, the method mainly includes the following steps:

[0097] Step 110: Obtain the image sample set of the intelligent vision terminal and perform standardization processing to obtain the normalized image.

[0098] The image sample set consists of several image samples.

[0099] In some embodiments, this step may specifically be as follows:

[0100] Image acquisition: The terminal camera acquires reference original images. (Added to the image sample set), with a fixed resolution of 1280*720 pixels, meaning the image height is 720 pixels and the width is 1280 pixels, and the frame format is YUV-420.

[0101] Noise suppression: for A 3×3 Gaussian filter kernel is applied to each of the three chromaticity components to completely remove high-frequency noise.

[0102] Contrast Enhancement: Histogram equalization is applied to adjust the brightness distribution, making the pixel grayscale uniform.

[0103] Standardize the format: Convert the processed image to RGB and normalize it to the [-1,1] interval using the following formula.

[0104] Step 120: Using the frozen basic image encoder network model, extract the image semantic vector of the normalized image.

[0105] In some embodiments, this step may specifically be as follows:

[0106] The publicly available CLIP-ViT-Base model is used as the base image encoder, and its parameters are frozen on the terminal side. This step first uses the base image encoder to extract... The 512-dimensional semantic vector is used because, compared to traditional edges or histograms, semantic features can simultaneously encode scene category, color layout, and spatial structure, providing high-level priors for semi-reference evaluation. Furthermore, freezing the network avoids retraining on the device side, maintaining cross-scene robustness. The specific process is as follows:

[0107] Network input: will Feed it into the basic image encoder.

[0108] Semantic vector output: The semantic vector is obtained by reading the CLS Token from the last layer of the basic image encoder network, as shown in the following formula:

[0109] .

[0110] in, This is the frozen base image encoder network model; Represents the semantic vector of an image; Represents the i-th semantic dimension (a real number).

[0111] Vector normalization: L2 normalization is performed to ensure that subsequent quantization errors are minimized.

[0112] Step 130: Perform product quantization on the image semantic vector to obtain the reference bitstream.

[0113] It should be noted that this step aims to extract the semantic feature vector of the image within a bitstream of no more than 180 bytes. The image is compressed into a compact bitstream. To maintain semantic consistency between vectors during compression, the cosine similarity between the reconstructed vector and the original image semantic vector must be at least 0.94. This is achieved through a combination of Residual Product Quantization (rPQ) and entropy coding. First, for... In some embodiments, performing first-level quantization can specifically involve:

[0114] (1) Sub-vector partitioning: The 512-dimensional image semantic vector is divided into sub-vectors. Divide into m=16 32-dimensional sub-vectors The specific division formula is as follows:

[0115] ;

[0116] (2) Index determination: For each subvector In the corresponding codebook The Euclidean distance minimum matching is calculated to obtain the index:

[0117] ;

[0118] Where K represents the number of codewords in each codebook; =[ ] , representing the m-th codebook; {1,...,256} represents the first-level quantization index of the optimal codeword.

[0119] (3) Bitstream encapsulation: The 16 indices are stored sequentially as a 16-byte unsigned integer array. First-level quantization reconstructed vector Most of the semantic information was compressed, but errors still remain. To eliminate this error, second-order residual quantization is introduced: the residual is further written as... And configure a small codebook for each sub-residual. In this method, the small codebook is set to 16 dimensions. The following operations are performed on each sub-residual:

[0120] ;

[0121] This is the second-level residual index, representing the codeword index matched by the m-th residual subvector in the second-level codebook. This represents the first-order quantization residual of the m-th subvector. This represents the jm-th codeword (32-dimensional sub-residual vector) in the m-th residual codebook.

[0122] Then, the 16 sub-residual vectors are concatenated to obtain the second-level quantization residual vector. :

[0123] ;

[0124] Represents the second-order quantization residual vector. ;

[0125] To improve compression efficiency, the system uses the first-level quantization index of each subvector. Sum of secondary residual indexes These are combined into index pairs, totaling 16 pairs and occupying 19 bits. This index sequence is then compressed into the reference bitstream b using RangeCoder entropy encoding. During recovery at the receiving end, the index sequence is decoded to obtain the reconstructed semantic vector. .

[0126] Therefore, the two-stage quantization consumes a total of M(log2256+log216)=16(8+4)=192 bits of index; after RangeCoder entropy encoding, the average bitstream is compressed to 140-180 bytes, while cos( ,z)≥0.94.

[0127] This scheme proceeds with two-level quantization, and then directly performs entropy encoding on all index sequences to compress redundancy to the maximum extent, outputting a reference bitstream that meets both bandwidth and accuracy constraints.

[0128] Step 140: The distorted image corresponding to the image sample and the reference bitstream are synchronously sent to the receiving end. The receiving end obtains the reconstructed semantic vector corresponding to the image sample based on the reference bitstream.

[0129] In some embodiments, this step may specifically involve: synchronously receiving the distorted image and the compressed semantic reference bitstream at the receiving end, performing a decoding operation, and recovering and reconstructing the semantic vector. The receiver needs to simultaneously acquire the distorted image (the image after compression or channel noise) and the reference bitstream b to prepare for the subsequent generation of a pseudo-reference. The decoding process must be completely symmetrical with the encoding process to ensure no information loss. The specific process is as follows:

[0130] Image reception: The gateway receives distorted images via the MQTT protocol.

[0131] Bitstream parsing: In the S3 module, the image semantic vector of the original image. The data is compressed into a reference bitstream b, which will be decoded into a reconstructed semantic vector in this module. :

[0132] .

[0133] Step 150: Generate a pseudo-reference image based on the reconstructed semantic vector and the distorted image.

[0134] It should be noted that, in the case of only reconstructed semantic vectors Under these conditions, a "pseudo-reference" image with the same structure as the original image is rapidly reconstructed through a few steps of inverse diffusion. This allows for the acquisition of a comparative benchmark without transmitting the original image. Therefore, this step is based on reconstructing semantic vectors. Quickly generate pseudo-reference images.

[0135] In some embodiments, this step may specifically be as follows:

[0136] Text hint mapping: Input mapping network Output text prompt t.

[0137] Diffusion initialization: with The Latent Diffusion model is loaded as initial noise.

[0138] Backdiffusion: Perform 5-step DPM-Solver sampling to obtain a pseudo-reference image. .in, This represents the 5th step of the Latent Diffusion model, the DPM-Solver inverse sampler. ∈ This represents a pseudo-reference image.

[0139] Step 160: Perform cross-image difference alignment on the pseudo-reference image and the distorted image within a preset salient region to obtain the difference feature tensor.

[0140] In some embodiments, this step performs cross-image difference alignment on the pseudo-reference image and the distorted image within a preset salient region. The goal is to finely compare the structural and color differences between the pseudo-reference image and the distorted image within the salient region and output high-dimensional difference features for use by the scoring network. A sparse strategy is used to significantly reduce the computation of unimportant patches by 80%. The specific process can be as follows:

[0141] Patch embedding: The two images are divided into 16×16 patches, and the token sequence is as follows: ;

[0142] in, This represents the token sequence obtained after patch segmentation and embedding encoding of a distorted image. Indicates a pseudo-reference image The resulting token sequence after the same processing is N, which represents the number of patches into which the image is cut, and d is the embedding dimension of each patch. In this method, d is set to 512.

[0143] Salient selection: Calculating the token sequence of a distorted image Self-attention weights And select the Top-20% Patch to form a significant set β.

[0144] ;

[0145] in, This represents the average attention level for the i-th patch; , This represents the h-th head query / key weight matrix; furthermore, according to Select the indices of the Top-20% Patch to form the significant set β.

[0146] Differential encoding: Based on the salient set β, the Patch sequence within the salient region is... Input Cross-ViT for cross-image alignment and fusion encoding, and output a difference feature tensor D, which represents the difference features between the pseudo-reference image and the distorted image in salient regions.

[0147] .

[0148] Step 170: Based on the differential feature tensor, obtain the inputs of the teacher network quality regression subheader and the distortion classification subheader. Using the differential feature tensor, the teacher network obtains the quality prediction confidence index and distortion classification prediction confidence index of each image sample in the image sample set, and then determines the type of each image sample.

[0149] This step utilizes a lightweight student network to simultaneously output subjective quality scores and distortion types (blur, noise, compression artifacts, color distortion, geometric deformation, contrast distortion, brightness distortion, and edge blur). During training, a large teacher network distills the data, while during inference, only the student network runs to adapt to low-power processors, ultimately outputting the quality score and distortion type. However, intelligent vision terminals have few samples and large quality fluctuations; therefore, the teacher network's predictions for some samples are unreliable (e.g., when the pseudo-reference image is blurred), and traditional equal distillation amplifies noise. This method proposes a high-variance trigger protection mechanism for the teacher network to prevent students from learning incorrect knowledge. Therefore, this step also introduces an uncertainty weighting mechanism, using Monte Carlo Dropout to estimate the teacher network's prediction confidence for quality scores and distortion types, dynamically adjusting the knowledge distillation weights. The specific process is as follows:

[0150] Pooling operations: Perform mean pooling and max pooling on the difference feature tensor D to obtain... These two parameters serve as inputs to the quality regression subheader and the distortion classification subheader, respectively.

[0151] Teacher network uncertainty estimation: To avoid misguidance by the teacher network when sample quality fluctuates greatly or pseudo-reference images are severely distorted, this step introduces a prediction uncertainty estimation mechanism based on Dropout to dynamically adjust the distillation weights. Specifically, for the same differential feature input tensor D, Dropout is enabled in the teacher network and forward propagation is performed M=3 times to simulate the prediction volatility of the network output, obtaining 3 sets of quality prediction results {Q(1),Q(2),Q(3)}, and the corresponding distortion type classification probability vectors {c(1),c(2),c(3)}.

[0152] Specifically, for quality prediction, the variance of the three quality prediction scores is calculated to obtain... The smaller the variance, the more stable the model's judgment and the higher its reliability. This value serves as a confidence index for quality prediction and is used in subsequent distillation weight adjustments.

[0153] For distortion type classification, the entropy value of the 3rd classification probability vector is calculated. The specific formula is as follows:

[0154] ;

[0155] It is used as a confidence index for distorted classification prediction.

[0156] Uncertainty Threshold Calculation for Image Quality Prediction and Distortion Classification: To determine the uncertainty threshold for image quality prediction and distortion classification, this step selects image samples from intelligent vision terminals (such as cameras, inspection cameras, agricultural cameras, etc.) to construct a validation set. The specific steps are as follows:

[0157] Validation set sample preprocessing: The validation set images are processed through the following steps in sequence:

[0158] The S110-S160 processing steps ultimately yield the differential feature tensor D for each image. The differential feature tensor D for each sample will then be used as input to the teacher network for uncertainty assessment.

[0159] The teacher network propagated forward multiple times to collect prediction results:

[0160] For each image, the feature tensor D is repeatedly forwarded M=3 times, with Dropout enabled each time. The following are recorded: the quality prediction score sequence {Q(1),Q(2),Q(3)} and the distortion classification probability distribution sequence {c(1),c(2),c(3)}.

[0161] Calculate the variance of the quality prediction score and the distortion classification prediction entropy for each sample. Furthermore, for all image samples in the validation set, collect the variance of their quality prediction scores separately. Forming a set of distributions, classifying and predicting entropy This constitutes another distribution.

[0162] Extracting the high quantiles of the distributions as an uncertainty threshold: By statistically analyzing the 95th percentiles of the two distributions mentioned above, the following threshold is defined: Quality Prediction Volatility Threshold Classification prediction confidence threshold .

[0163] In this method, the confidence index for quality prediction is ≤ And the confidence index for distorted classification prediction is ≤ If a sample exceeds the threshold, it is classified as low uncertainty; if only one condition exceeds the threshold, it is classified as medium uncertainty; if both conditions exceed the threshold significantly, it is classified as high uncertainty.

[0164] Step 180: Based on the differential feature tensor and the type of each sample, freeze the teacher network parameters to obtain the trained student network; obtain the differential feature tensor of the test data, and use the trained student network to output the subjective quality score and several preset distortion probability vectors. ,Pick This is the dominant distortion type for the test data.

[0165] It should be noted that independent prediction is performed by the student network. In this case, the inference end no longer relies on the teacher network; all calculations are completed within the student network. The process is as follows: The student network performs the forward propagation process: First, the input D still undergoes mean pooling and max pooling, resulting in:

[0166] ;

[0167] in, () is the mean pooling function. () represents the max pooling function. Next, a two-layer fully connected layer is executed on the quality subheading, with the dimension changing from 384 to 64 to 1, outputting the subjective quality prediction as follows:

[0168] ;

[0169] in, This represents the lightweight fully connected subheader, which is the forward propagation model function of a two-layer fully connected network. Then, the classification subheader performs two fully connected layers, with the dimensionality changing from 384 to 64 to 8. The distortion probability is then obtained through Softmax, as follows:

[0170] ;

[0171] Uncertainty-weighted distillation loss:

[0172] ;

[0173] in, , The quality scores and classification results output for students; , This step generates the quality score and classification result for the teacher's network output; KL represents the Kullback-Leibler divergence. Through this step, the student network and codebook can be updated synchronously, while the teacher network parameters remain frozen.

[0174] in , The weights are set as follows:

[0175] ;

[0176] ;

[0177] Specifically, when a sample falls into the "high uncertainty" state, the distillation weights are set to zero, and the gradient update relies solely on the student's own supervision signal; when it is in the "medium uncertainty" state, the weights are adjusted according to... and The decreasing frequency naturally reduces the impact of samples with large errors on the student network; "low uncertainty" samples are kept completely distilled to make full use of teacher information. When too many "high uncertainty" samples appear within k epochs, the system temporarily freezes the corresponding subheading parameters, and automatically unfreezes them after the mixed distortion scenario is alleviated, preventing the model from being dragged off by noise gradients.

[0178] Furthermore, during training, the learning rate and weights are automatically adjusted and updated each time by calculating loss functions (such as distillation loss, mass fraction loss, and classification loss) based on the model's feedback on the loss.

[0179] Parameter update: Backpropagation only updates the student network and the quantization codebook; the teacher network remains frozen.

[0180] Student Network Independent Reasoning Phase: Pseudo-reference local generation: Terminal local reconstruction Single-step diffusion is obtained Next is the differential feature extraction: sparse HAX-ViT (6 layers, Top 20% Tokens) is run to predict student outcomes, and the final output is a subjective quality score. (0-100), and eight types (i) of distortion probability vectors ,Pick The dominant distortion type. The predicted probability representing the i-th type of distortion is obtained from the output of the student or teacher network through the Softmax layer.

[0181] This application limits the "teacher network confidence-gated distillation" to the training period, while allowing the student network to independently and deterministically perform the above mean, max pooling + lightweight fully connected inference during the inference period. This module avoids misdistillation when the teacher output is unreliable and ensures the real-time performance and stability of the terminal deployment, forming a complete closed loop.

[0182] Through the above steps, this application embodiment achieves high-precision quality assessment of images from intelligent vision terminals using only a low-bitrate reference stream. It has technical advantages such as extremely low reference information, low computational overhead, and robustness to multiple types of distortion. It can realize real-time quality monitoring under narrowband links such as NB-IoT, LoRa, and cellular Cat-M.

[0183] In addition, this application Figure 2 This application provides a low bit-rate semi-reference image quality inspection system based on self-distillation. For example... Figure 2 As shown in the embodiments of this application, the system mainly includes:

[0184] The image acquisition module 210 is used to acquire a sample set of the intelligent vision terminal and perform standardization processing to obtain a normalized image.

[0185] Vector extraction module 220 is used to extract normalized image semantic vectors using a frozen base image encoder network model;

[0186] The bitstream acquisition module 230 is used to perform product quantization on the image semantic vector to obtain a reference bitstream;

[0187] The vector reconstruction module 240 is used to synchronously send the distorted image corresponding to the sample and the reference bitstream to the receiving end, and the receiving end obtains the reconstructed semantic vector corresponding to the sample based on the reference bitstream.

[0188] Image generation module 250 is used to generate a pseudo-reference image based on reconstructed semantic vectors and distorted image;

[0189] The vector acquisition module 260 is used to perform cross-image difference alignment between the pseudo-reference image and the distorted image within a preset salient region to obtain the difference feature tensor.

[0190] The teacher network module 270 is used to obtain the input of the teacher network quality regression subheader and distortion classification subheader based on the differential feature tensor, and to obtain the quality prediction confidence index and distortion classification prediction confidence index of each image sample in the image sample set using the differential feature tensor, thereby determining the type of each image sample; among which, the types are divided into: low uncertainty, medium uncertainty and high uncertainty.

[0191] Student network module 280 is used to freeze the teacher network parameters based on the differential feature tensor and the type of each sample to obtain a trained student network; it also obtains the differential feature tensor of the test data and uses the trained student network to output subjective quality scores and several preset distortion probability vectors. ,Pick This is the dominant distortion type for the test data.

[0192] In addition, embodiments of this application also provide a non-volatile computer storage medium storing executable instructions, which, when executed, implement the self-distillation-based low bit-rate half-reference image quality inspection method described above.

[0193] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A low bit-rate semi-reference image quality inspection method based on self-distillation, characterized in that, The method includes: A set of image samples from an intelligent vision terminal is acquired, and normalized images are obtained by performing standardization processing; the image sample set consists of several image samples. A frozen basic image encoder network model is used to extract the image semantic vector of the normalized image; the image semantic vector is then multiplied and quantized to obtain the reference bitstream. The distorted image corresponding to the image sample and the reference bitstream are sent synchronously to the receiving end. The receiving end obtains the reconstructed semantic vector corresponding to the image sample based on the reference bitstream. Based on the reconstructed semantic vector and the distorted image, a pseudo reference image is generated. Perform cross-image difference alignment between the pseudo-reference image and the distorted image within a preset salient region to obtain the difference feature tensor; Based on the differential feature tensor, the inputs to the teacher network's quality regression subheader and distortion classification subheader are obtained. Using the differential feature tensor, the teacher network obtains the quality prediction confidence index and distortion classification prediction confidence index for each image sample in the image sample set, thereby determining the type of each image sample. The types are divided into: low uncertainty, medium uncertainty, and high uncertainty. Based on the differential feature tensor and the type of each sample, the teacher network parameters are frozen to obtain the trained student network. Obtain the differential feature tensor of the test data, and using the trained student network, output subjective quality scores and several preset distortion probability vectors. ,Pick This is the dominant distortion type for the test data.

2. The low bit-rate semi-reference image quality inspection method based on self-distillation according to claim 1, characterized in that, Obtain a sample set from intelligent vision terminals, perform standardization processing to obtain normalized images, specifically including: Terminal camera collects intelligent vision terminal sample set The resolution is fixed at 1280*720 pixels, and the frame format is YUV-420. right A 3×3 Gaussian filter kernel is applied to each of the three chromaticity components; Histogram equalization is applied to adjust the brightness distribution, making the pixel grayscale uniform. The processed image is converted to RGB and normalized to the [-1, 1] interval using the following formula.

3. The low bit-rate semi-reference image quality inspection method based on self-distillation according to claim 1, characterized in that, Using a frozen basic image encoder network model, the image semantic vector of the normalized image is extracted, specifically including: Normalized image Feed into the basic image encoder; via the formula: Read the CLS Token from the last layer of the basic image encoder network to obtain the image semantic vector. ; in, This is the frozen base image encoder network model; This represents the i-th semantic dimension.

4. The low bitrate semi-reference image quality inspection method based on self-distillation according to claim 1, characterized in that, The image semantic vector is multiplied and quantized to obtain a reference bitstream, specifically including: According to the preset division formula: The image semantic vector is divided into m=16 32-dimensional sub-vectors. ; According to the preset formula: For each In the corresponding codebook Calculate the Euclidean distance to obtain the first-level quantization index. ; Where K represents the number of codewords in each codebook; =[ ] , representing the m-th codebook; {1,...,256} represents the first-level quantization index; The 16 first-level quantization indices are stored sequentially as a 16-byte unsigned integer array to obtain the first-level quantization reconstruction vector. ; Perform the following operation on each sub-residual: ;in, This indicates a secondary residual index. This represents the first-order quantization residual of the m-th subvector. This represents the jm-th codeword in the m-th residual codebook; The first-level quantization index of each sub-vector Sum of secondary residual indexes The index pairs are combined and then compressed into a reference bitstream b using RangeCoder entropy encoding.

5. The low bit-rate semi-reference image quality inspection method based on self-distillation according to claim 1, characterized in that, The receiving end obtains the reconstructed semantic vector based on the reference bitstream, specifically including: The receiving end obtains reference bitstream b: Through the formula: Decode the reference bitstream into a reconstructed semantic vector. ; This represents the decoding function.

6. The low bit-rate semi-reference image quality inspection method based on self-distillation according to claim 1, characterized in that, Based on the reconstructed semantic vectors and the distorted image, a pseudo-reference image is generated, specifically including: Reconstruct semantic vectors Input mapping network Output text prompt t; With distorted images As initial noise, load the Latent Diffusion model; Perform 5 steps of DPM-Solver sampling to obtain a pseudo-reference image. ;in, This represents the 5th step of the Latent Diffusion model, the DPM-Solver backsampler. ∈ This represents a pseudo-reference image.

7. The low bit-rate semi-reference image quality inspection method based on self-distillation according to claim 1, characterized in that, Perform cross-image difference alignment between the pseudo-reference image and the distorted image within a predefined salient region to obtain the difference feature tensor, specifically including: The pseudo-reference image and the distorted image are divided into 16×16 patches, and the token sequence is as follows: ; Through the formula: Calculate self-attention weights ; And select the Top-20% Patch to form set β; in, This represents the average attention level for the i-th patch; , This represents the head query / key weight matrix for the h-th term; Input the token in β into a 6-layer Cross-ViT and output the differential feature tensor D.

8. The low bit-rate semi-reference image quality inspection method based on self-distillation according to claim 1, characterized in that, Using the differential feature tensor, the teacher network obtains the confidence indices for quality prediction and distortion classification prediction for each sample in the sample set, thereby determining the type of each sample, specifically including: For the same differential feature tensor D, Dropout is enabled in the teacher network and forward propagation is performed M=3 times to simulate the prediction volatility of the network output, and three sets of quality prediction results {Q(1),Q(2),Q(3)} are obtained, as well as the corresponding distortion type classification probability vector {c(1),c(2),c(3)}. Calculate the variance of the three quality prediction scores, and obtain... , as a confidence index for quality prediction; Through the formula: Calculate the entropy of the three classification probability vectors. It will be used as a confidence index for distorted classification prediction; Based on the confidence indices for quality prediction and distortion classification prediction of all samples, the distributions of the confidence indices for quality prediction and distortion classification prediction are obtained. Based on the preset quantiles of the distribution, the quality prediction volatility threshold and the classification prediction confidence threshold are obtained; When the confidence index of a sample's quality prediction is less than or equal to the quality prediction volatility threshold, and the confidence index of the distorted classification prediction is less than or equal to the classification prediction confidence threshold, the sample is classified as low uncertainty. When the confidence index of a sample's quality prediction is greater than the quality prediction volatility threshold or the confidence index of a distorted classification prediction is greater than the classification prediction confidence threshold, the sample is classified as moderately uncertain. When the confidence index of a sample's quality prediction is greater than the quality prediction volatility threshold and the confidence index of its distorted classification prediction is greater than the classification prediction confidence threshold, the sample is classified as high uncertainty.

9. The low bit-rate semi-reference image quality inspection method based on self-distillation according to claim 1, characterized in that, Based on the difference feature tensor and the type of each sample, and utilizing a student network, the system outputs a subjective quality score and several preset distortion probability vectors. ,Pick The dominant distortion types include: Input the difference feature tensor, and after mean pooling and max pooling, we get: ;in, () is the mean pooling function. () is the max pooling function; based on Perform a two-layer fully connected layer on the quality subheader, with the dimension changing from 384 to 64 to 1, and output the subjective quality prediction. ; based on The classification subheader is processed by performing a two-layer fully connected layer, resulting in a dimensionality change from 384 to 64 to 8. The distortion probability is then obtained using Softmax. ;in, ; According to the formula: Calculate the uncertainty-weighted distillation loss; in, , Subjective quality prediction and distortion probability of network output for teachers; in , The weights are set as follows: ， ， in, Indicates the preset adjustment factor; Backpropagation only updates the student network and the quantization codebook; the teacher network remains frozen. Obtain a well-trained student network; Input the difference feature tensor into the trained student network, and the network outputs the subjective quality score and the distortion probability vector. ,Pick The dominant distortion type.

10. A low bit-rate semi-reference image quality inspection system based on self-distillation, characterized in that, The system includes: The image acquisition module is used to acquire a sample set of intelligent vision terminals and perform standardization processing to obtain normalized images; The vector extraction module is used to extract normalized image semantic vectors using a frozen base image encoder network model. The bitstream acquisition module is used to perform product quantization on the image semantic vector to obtain a reference bitstream; The vector reconstruction module is used to synchronously send the distorted image corresponding to the sample and the reference bitstream to the receiving end. The receiving end obtains the reconstructed semantic vector corresponding to the sample based on the reference bitstream. The image generation module is used to generate a pseudo-reference image based on the reconstructed semantic vector and the distorted image; The vector acquisition module is used to perform cross-image difference alignment between the pseudo-reference image and the distorted image within a preset salient region to obtain the difference feature tensor. The teacher network module is used to obtain the input of the teacher network quality regression subheader and distortion classification subheader based on the differential feature tensor, and to obtain the quality prediction confidence index and distortion classification prediction confidence index of each image sample in the image sample set using the differential feature tensor, thereby determining the type of each image sample; among which, the types are divided into: low uncertainty, medium uncertainty and high uncertainty. The student network module is used to freeze the teacher network parameters based on the differential feature tensor and the type of each sample to obtain a trained student network; it also obtains the differential feature tensor of the test data and uses the trained student network to output subjective quality scores and several preset distortion probability vectors. ,Pick This is the dominant distortion type for the test data.

Images