Code decoupling and pulse driving just recognizable distortion prediction method

Abstract

The invention discloses a coding decoupling and pulse driving just-recognizable distortion prediction method. The method comprises the following steps: establishing a multi-coding standard cross-model just-recognizable distortion data set; inputting an original image in the cross-model just recognizable distortion data set into the constructed reference image generation network to generate a reference image with a coding distortion form; and inputting the original image and the reference image into a pulse-driven JRD prediction network comprising a feature extraction module, a feature transformation module, a feature fusion module and a classification prediction module, generating distortion perception features, and generating a multi-classification prediction result according to the distortion perception features. In order to solve the problems that just recognizable distortion and model coupling of current machine vision coding and a prediction model need to refer to a coding image, distortion estimation with high generalization ability can be provided for a machine vision task by establishing a cross-model JRD data set and designing a coding decoupling and pulse driving prediction model; the method is suitable for guiding coding optimization and improving machine vision coding efficiency.

Description

Technical Field

[0001] This invention relates to the field of machine perception coding optimization, and in particular to a coding decoupling and pulse-driven method for predicting just-identifiable distortion, providing a technical option for machine vision coding schemes. Background Technology

[0002] With the rapid development of artificial intelligence and machine vision technologies, applications such as intelligent transportation, industrial inspection, autonomous driving, and security monitoring are becoming increasingly widespread, generating massive amounts of image and video data. In recent years, the field of machine vision has introduced the concept of "Just Recognizable Distortion" (JRD), which refers to the maximum degree of distortion that a machine vision model can tolerate during image or video compression without significantly affecting its task performance. If the quantization parameter level of compression distortion is less than the JRD value, the performance degradation of the machine vision model (such as object detection and image classification accuracy) will not be significant. Therefore, applying JRD to guide encoding can achieve lower bitrate allocation while maintaining task accuracy, which aligns perfectly with the optimization goals of VCM (Visual Character Module).

[0003] Existing technologies, based on the HEVC / H.265 encoding standard, construct the JRD dataset for object detection and image classification tasks using 11 discrete quantization parameters (QP) as distortion levels, and propose an ensemble learning-based JRD prediction framework. This framework uses multiple binary classifiers based on VGG-19 for multiple inference predictions. However, this framework can only predict a limited number of pre-defined QP levels, and misclassification by the binary classifiers further increases the prediction error. Inspired by the concept of SUR (Satisfied User Ratio), researchers have also proposed a probabilistic quality assessment metric for machines, the "Satisfied Machine Ratio (SMR)," and proposed a deep learning model to predict SMR values. This quantifies the machine's consistent perception of compression distortion, and finally determines the critical point of just identifiable distortion based on the SMR threshold. Existing technologies have further constructed target-level datasets based on the VVC / H.266 standard and 64 QP levels, proposing a framework for training a binary predictor and combining it with a fault-tolerant strategy. This approach performs binary classification prediction by inputting the original image and an encoded compressed reference image, and uses a sliding window strategy to correct the classification results to improve the model's prediction accuracy. However, this method requires encoding multiple compressed original images, and predicting the JRD of an image requires multiple binary classification inferences. In extreme cases, the original image needs to be compressed 64 times, making it difficult to apply in practice.

[0004] Current research on just-identifiable distortion (JRD) mainly covers the introduction of related concepts, the construction of datasets for machine vision tasks, and the design of prediction models. Academic research on JRD is still in its early stages. Although preliminary JRD datasets and prediction models have been constructed, existing solutions are not yet mature. Specifically, the definition of JRD is coupled with machine vision models, lacking generalization ability between models; and existing research is limited to a single encoding standard, with the characteristics and variation patterns of JRD under different distortion types yet to be explored. Furthermore, existing prediction methods require multiple inference predictions based on the encoded and compressed image, resulting in deep coupling between the prediction method and the encoding process, making practical applications difficult. Summary of the Invention

[0005] This invention addresses the problems of just-identifiable distortion and model coupling in current machine vision coding, as well as the need for the prediction model to reference the coded image. It proposes a just-identifiable distortion prediction method based on coding decoupling and impulse-driven approach. By establishing a cross-model JRD dataset and designing a coding decoupling and impulse-driven prediction model, it can provide distortion estimation with strong generalization ability for machine vision tasks, which is suitable for guiding coding optimization and improving machine vision coding efficiency.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A coded decoupling and pulse-driven identifiable distortion prediction method includes the following steps: S1. Establish a cross-model identifiable distortion dataset with multiple coding standards; S2. Input the original images from the cross-model identifiable distortion dataset into the reference image generation network to generate reference images with encoded distortion forms; S3. Input the original image and reference image into a pulse-driven JRD prediction network that includes a feature extraction module, a feature transformation module, a feature fusion module, and a classification prediction module, generate distortion-aware features, and generate multi-classification prediction results based on the distortion-aware features.

[0007] Preferably, when dealing with image classification tasks, S1 includes: S11. Select machine vision models and construct a set of machine vision models; record the top-1 category of each machine vision model for the original image; S12. The machine vision model sequentially identifies compressed images with different quantization parameters under a coding standard. When the TOP-1 category on a certain compressed image changes, the maximum quantization parameter level that can maintain the correct classification result in the original image is taken as its just identifiable distortion value under the current machine vision model. S13. All machine vision models sequentially execute the recognition process of S12, and record the cross-model just-recognizable distortion value of the original image as the just-recognizable distortion value among the just-recognizable distortions labeled by all machine vision models. S14. Obtain the cross-model just-identifiable distortion values ​​for each coding standard and construct a just-identifiable distortion dataset under multiple coding standards.

[0008] Preferably, S2 includes: S21. Construct a reference image generation network that includes a downsampling module, a multi-scale grouped dilated convolution module, and an upsampling module; After edge reflection filling, the original image is processed by convolution kernels combined with normalization layers and activation functions to map it into shallow features. The shallow features are input into downsampling blocks for feature encoding to generate deep features containing semantic information. The deep features are then fused into multi-scale information by a multi-scale grouped dilated convolution module to generate multi-scale semantic features. The multi-scale semantic features are then decoded by an upsampling module to obtain the reference image.

[0009] As a preferred embodiment, in S21, the shallow features of the input downsampling module are processed in two paths. The main path uses 3×3 convolution for feature extraction, and the side path uses 1×1 convolution for channel transformation. The two outputs are added element by element and deep features are obtained through activation functions. After grouping deep features along the channel dimension, they are fed into the three branches of the multi-scale grouped dilated convolution module in parallel. The three branches are set with different dilation rates to capture different quantization distortions. The output features of each branch are nonlinearly mapped by a relocalizable activation function and then added element by element to output multi-scale semantic features. The upsampling module uses a residual structure; the decoded feature map is filled by reflection and the high-dimensional features are mapped to the RGB image space through a convolutional layer. Finally, the Tanh activation function is used to normalize the output pixel values ​​to generate a reference image.

[0010] Preferably, the relocalizable activation function of the multi-scale grouped dilated convolution module allows the inflection point and output distribution of the activation function to be adaptively shifted in the feature space, and its mathematical expression is as follows: in, As input features, and These are the displacement parameters that the network learns automatically during training.

[0011] As a preferred embodiment, S2 also includes: S22. Input the reference image and its corresponding real image into the discriminant network to construct a loss function to train the reference image generation network; the real image is the original image corresponding to the reference image in the cross-model just-recognizable distortion dataset after encoding and compression.

[0012] Preferably, S22 includes: Both the reference image and the real image are downsampled twice to form two sets of multi-resolution hierarchical images with multiple scales; each scale corresponds to a structurally independent discriminator; the discriminator at each resolution level extracts features from the reference image and the real image respectively, and outputs two-dimensional discriminant feature matrices of the reference image and the real image; each element in the two-dimensional discriminant feature matrix corresponds to a local receptive field region, and its value represents the confidence that the region belongs to the real image; During training, the loss function constrains all elements of the two-dimensional discriminant feature matrix of the reference image to tend to 0, representing a forgery, while all elements of the two-dimensional discriminant feature matrix of the real image tend to 1, representing authenticity.

[0013] Preferably, a hybrid loss function that includes adversarial loss, feature matching loss, and perceptual reconstruction loss is used. Optimize the reference image generation network: in, To combat the losses, For feature matching loss, To perceive the reconstruction loss, The balancing weights for each loss; The adversarial loss employs least-squares generative adversarial loss, which provides a more stable gradient signal by minimizing the mean square error between the discriminator output and the real image; the feature matching loss is used to measure the difference between the reference image and the real image in the intermediate layer feature space of the multi-scale discriminator. Distance; Perceptual reconstruction loss is derived from the feature space extracted by the VGG-19 network. Distance and pixel space Distance-weighted composition.

[0014] Preferably, S3 includes: S31. Input the original image and the reference image in pairs into the feature extraction module of two weight-shared branch networks to generate multi-scale original image features and reference image features; S32. Input the original image features and reference image features into the feature transformation module. After being unified to a fixed size and spliced ​​in the channel dimension, high-dimensional pulse features are generated. After performing channel compression, integrating cross-channel semantics, and residual connection on the high-dimensional pulse features, the features are mapped to the preset target dimension to generate transformed original image features and transformed reference image features. S33. The feature fusion module fuses the features of the original image after feature transformation and the features of the transformed reference image to obtain distortion-perceived features for final classification. S34. The classification prediction module generates multi-classification prediction results based on the distortion perception features.

[0015] Preferably, in S31, the original image and the reference image are processed by a shallow feature extraction structure to output a shallow feature map; the shallow feature map is pulse-coded by a leaky integrated firing neuron; the encoded pulse sequence is fed into three stages of the SpikingResformer network, with each stage connected by a downsampling layer; each stage consists of several stacked residual blocks; the residual blocks include a dual-pulse self-attention module with residual connections for capturing long-range global dependencies and a grouped pulse feedforward network for extracting local spatial detail features.

[0016] Compared with the prior art, the beneficial effects of the present invention are reflected in: 1. Unlike traditional techniques that define Just-Recognizable Distortion (JRD) only under a single encoding standard and a single machine vision model, making it difficult for the defined JRD to guide encoding in other models, this invention effectively solves the problem of coupling JRD with a single machine vision model by using multiple classic machine vision models under multiple encoding standards (H.264, H.265, H.266) to jointly define and construct a large-scale JRD dataset. This allows JRD to more efficiently guide encoding optimization in machine vision. Furthermore, the fine annotation based on multiple encoding standards provides crucial data support for in-depth analysis of the impact factors of distortion introduced by different encoding compression forms on machine vision, and for exploring the core influencing factors of machine vision perception.

[0017] 2. Unlike traditional techniques that require pre-encoding and compression of the original image using an encoder to obtain a distorted reference image, this invention employs a deep learning-based reference image generation network. This approach leverages the efficient inference capabilities of deep neural networks to quickly simulate the encoding process and generate a reference image with similar encoding distortion characteristics from the input original image. This effectively avoids the cumbersome and time-consuming pre-encoding step in traditional methods, thereby significantly optimizing the JRD prediction process and improving the immediacy of prediction.

[0018] 3. Unlike traditional technologies that use high-power convolutional neural networks (CNNs) for inference and prediction, which are difficult to meet the deployment requirements of edge devices, this invention designs a pulse-driven JRD prediction network. It utilizes the event-driven characteristics of spiking neural networks (SNNs) for low-power feature extraction and combines a windowed soft-labeling strategy to model the ordered correlation between JRD labels. This model not only achieves higher accuracy in JRD prediction on public datasets but also significantly reduces the computational complexity and power consumption of the model, thereby improving the feasibility of deploying the model on resource-constrained edge computing devices. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the encoding decoupling and pulse-driven just-identifiable distortion prediction model of Embodiment 1 of the present invention; Figure 2 This is a reference image generation network block diagram of Embodiment 1 of the present invention; Figure 3 This is a block diagram of the discriminator model of the discriminator network in Embodiment 1 of the present invention; Figure 4 This is a distribution diagram of the JRD dataset of the cross-model under the AVC / H.264 encoding standard in Embodiment 1 of the present invention; Figure 5 This is a distribution diagram of the JRD dataset of the cross-model under the HEVC / H.265 encoding standard in Embodiment 1 of the present invention; Figure 6 This is a distribution diagram of the JRD dataset for the cross-model under the VVC / H.266 encoding standard in Embodiment 1 of the present invention. Detailed Implementation

[0020] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0021] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0022] The technical problem to be solved by this invention: JRD (Just Recognizable Distortion) is limited to a single encoding standard, failing to cover the diverse distortion types introduced by different encoding standards. The ability of a JRD with distortion types caused by a single encoding standard to generalize to JRDs with other similar distortion types remains to be explored.

[0023] The problem of deep coupling between JRD and models: JRD is only based on a single machine vision model, resulting in severely insufficient generalization ability. Due to the significant differences in architectural design and feature extraction mechanisms among different models, the JRD threshold based on a specific model often cannot be transferred to other models, thus limiting the applicability of JRD in general machine vision tasks.

[0024] The problem of deep coupling between JRD prediction and encoder: JRD prediction model requires the distorted image after encoding and compression as a reference for prediction. This pre-encoding method is difficult to provide real-time quantization parameter guidance in the actual encoding process, which greatly limits the application of JRD in machine vision encoding.

[0025] To address the aforementioned problem of just-identifiable distortion in machine vision, this invention proposes a code-decoupled and impulse-driven just-identifiable distortion prediction method. By establishing a cross-model JRD dataset and designing a code-decoupled and impulse-driven prediction model, it can provide distortion estimates with strong generalization capabilities for machine vision tasks. This method is suitable for guiding code optimization and improving machine vision coding efficiency. Example 1: A code-decoupled and pulse-driven just-identifiable distortion prediction method is proposed. This method analyzes the characteristics of a cross-model JRD dataset, constructs a reference image generation network based on the commonalities of JRDs to achieve code decoupling, and finally uses a pulse-driven JRD prediction network to achieve efficient classification prediction. The method includes the following steps: S1. Establish a cross-model identifiable distortion dataset with multiple coding standards; This invention constructs a cross-model JRD dataset using image classification as an example, aiming to analyze the current problem of JRD definitions relying on a single model and a single coding standard. This dataset consists of original images... Multiple QP (quantization parameter) levels after full intra-frame coding according to multiple coding standards Distorted images composition, This is a preset set of quantization parameter ranges under different encoding standards.

[0026] In terms of dataset selection, this invention is based on the ImageNet dataset, which contains 1,000 categories and is highly representative in the field of image classification.

[0027] Regarding video coding standards, this invention selected three coding standards—AVC / H.264 Advanced Video Coding (using its official reference software JM 19.0), HEVC / H.265 High Efficiency Video Coding (using its official reference software HM 18.0), and VVC / H.266 Multi-Functional Video Coding (using its official reference software VTM 14.0)—for intra-frame coding of the dataset. For H.264 and H.265 standards, the QP range 0-51 was selected for image compression; for the H.266 coding standard, the QP range 0-63 was selected for image compression.

[0028] In terms of machine vision model architecture, classic image classification models VGG-19-bn, ResNet-101, ResNeXt-101-32x8d, and DenseNet-201 were selected for JRD annotation. When an image of a certain QP level causes the machine task performance to degrade to an unacceptable level (defined in this invention as a change in the TOP-1 category), the QP value of its next higher level is used as the JRD of that image.

[0029] The JRD dataset is designed for two tasks: object detection and image classification. The main difference between the two lies in the evaluation metrics used during construction, while the subsequent annotation process is the same. For image classification, the evaluation metric is that the top-1 classification category remains unchanged. For object detection, three evaluation metrics are used: category invariance, Intersection over Union (IOU) not less than a set threshold (e.g., 0.5 or 0.75), and confidence score not less than a set threshold (e.g., 0.5 or 0.75).

[0030] Taking image classification as an example, the specific construction method of the just-identifiable distortion dataset across models is as follows: S11. By selecting a set of machine vision models (This example uses N=4) Record each model (1≤k≤N) for the original image The TOP-1 category, i.e. .

[0031] S12, The machine vision model sequentially compresses the original image under different quantization parameters q. To perform identification, when the TOP-1 category on a compressed image changes (i.e. The highest QP level of the original image that maintains the correct classification result is used as its value in the current machine vision model. The discernible distortion is defined as follows: .

[0032] in, This represents the set of all optional quantization parameters. The model represents the compressed image The prediction category.

[0033] S13. For the selected machine vision model All models undergo this process sequentially, and the model with the lowest QP level among all machine vision models' labeled JRDs is recorded as its JRD value, which is the cross-model just-recognizable distortion of the image. Defined as: S14. Based on the above method, the present invention constructs a just-identifiable distortion dataset for image classification tasks, defined by multiple machine vision models, under multiple coding standards.

[0034] Specifically, in this embodiment, each target image in the cross-model just-identifiable distortion dataset consists of an original image and multiple QP-level compressed images corresponding to it after full intra-frame encoding using H.264, H.265, and H.266. Each target image is labeled with JRD under the three encoding standards, which is suitable for training models to predict JRD under different encoding standards.

[0035] To verify the effectiveness of the cross-model just-identifiable distortion (JRD) definition method proposed in this invention, and to further explore the impact of distortion types introduced by different coding standards on machine vision perception, this embodiment conducts a detailed statistical distribution analysis of a cross-model JRD dataset constructed based on multiple coding standards. The overall distribution of the dataset is shown in Table 1. Figure 4 , Figure 5 , Figure 6 A more detailed bar chart distribution is shown.

[0036] Table 1: Cross-model Exactly Identifiable Distortion Datasets with Multiple Coding Standards

[0037] As shown in Table 1, the statistical results reveal the distribution pattern of machine vision models' tolerance to compression distortion in image classification tasks under the three mainstream coding standards: AVC / H.264, HEVC / H.265, and VVC / H.266. The statistics show that for the three coding standards, the number of samples is not evenly distributed across each QP interval, but rather increases with the increase of the QP value. Therefore, machine vision models are more robust than the human visual system; most images can still be correctly recognized by the machine model after high-intensity compression distortion. This further verifies the significant potential of JRD-guided coding schemes in reducing bitrate.

[0038] To analyze the JRD relationship among different machine vision models, the median and mean values ​​of the JRD dataset were statistically analyzed, as shown in Table 2. Table 2: Statistics of median and mean JRD for each model

[0039] The data in Table 2 shows that models with higher classification accuracy on ImageNet are generally more tolerant of higher levels of encoding compression distortion and have stronger robustness.

[0040] Furthermore, the proportion of models with the same JRD in the same image was statistically analyzed, and the detailed distribution is as follows (specific models are omitted in the model description). Taking the VGG model as an example, the proportion of identical JRDs with ResNet, ResNeXt, and DenseNet under the H.264 standard is 23.60%, 23.32%, and 23.77%, respectively; under the H.265 standard, it is 21.24%, 20.94%, and 21.87%, respectively; and under the H.266 standard, it is only 8.01%, 6.95%, and 10.26%, respectively. In addition, since H.264 and H.265 have the same encoding range, the proportion of identical JRDs of the same model in the same image under the two encoding standards was also compared. The proportions for VGG, ResNet, ResNeXt, and DenseNet are 25.79%, 42.61%, 44.66%, and 34.50%, respectively.

[0041] The statistical analysis above shows that JRDs labeled based on a specific model are difficult to apply to other models. Even for the same model, JRDs under different encoding standards are affected by variations in distortion type. Therefore, it is necessary to establish cross-model datasets with multiple encoding standards to improve the generalization ability of JRDs and more effectively guide machine vision encoding.

[0042] Combination Figure 4 , Figure 5 , Figure 6 Further analysis revealed that for the H.264 and H.265 encoding standards, a large number of images are still concentrated at JRD 51, indicating that they can still be compressed to a higher degree. This data distribution under the H.266 encoding standard shows that even with extremely poor image quality, machine vision models pre-trained on large-scale datasets can still perform correct image classification.

[0043] Meanwhile, this invention provides a more detailed analysis of the cross-model dataset constructed using the H.264 and H.265 standards. The proportion of images with identical JRD values ​​is only 22.70%, further verifying that there are significant differences in the JRD distribution under different distortion types. The JRD defined under the H.264 encoding standard cannot be used to guide H.265 encoding.

[0044] Furthermore, at JRD value 51, the number of images in H.264 is slightly higher than that in H.265, indicating that different forms of distortion have different impacts on the analysis of machine models. The JRD dataset is labeled with machine vision models based on a CNN architecture. Research by Geirhos et al. shows that CNN models exhibit a clear texture preference when recognizing images, while the work of Subramanian et al. further demonstrates that neural networks possess spatial frequency channels that are 2-4 times wider than human vision, representing their greater sensitivity to high-frequency signals.

[0045] Based on the above analysis, one possible reason is that the distortion produced by the two coding standards affects the texture and high-frequency features of the image in different ways. H.264 coding, under high compression rates, produces block effects due to its block transform mechanism, resulting in significant spatial discontinuities in its distortion distribution. This structured distortion allows for the preservation of local textures and high-frequency details within macroblocks to some extent. In contrast, H.265, in order to optimize subjective quality for the human eye, tends to suppress discontinuities in the image. Its distortion tends towards global smoothing and blurring, manifesting more as a loss of high-frequency details and texture information. The loss of feature information caused by the distortion form of H.265 may be more significant than that of H.264, resulting in a slightly higher number of images with JRD of 51 in H.264 compared to H.265.

[0046] The dataset constructed in this invention provides data support and reference for further exploration and research on the characteristics of JRD. The conclusions show that machine vision models have high robustness, and applying JRD to guide encoding has enormous compression potential. However, JRDs for a single model are difficult to apply to guide other machine models, and there are significant differences between JRDs under different encoding standards. Therefore, the prerequisite for effectively guiding machine vision encoding is to establish a generalizable JRD under different encoding standards.

[0047] S2. Input the original images from the cross-model identifiable distortion dataset into the reference image generation network to generate reference images; like Figure 2 , Figure 3As shown, this invention solves the problem of JRD prediction coupling with encoder by constructing a reference image generation network and designing a deep neural network to learn the mapping relationship between the original image and the image at a preset QP level, which is used to generate a distorted image with coded compression.

[0048] S21. Construct a reference image generation network that includes a downsampling module, a multi-scale grouped dilated convolution module, and an upsampling module; After edge reflection filling, the original image is processed by convolutional kernels combined with normalization layers and activation functions to map shallow features. These shallow features are then input into downsampling blocks for feature encoding to generate deep features containing semantic information. The deep features are then fused using a multi-scale grouped dilated convolution module to generate multi-scale semantic features. Finally, the multi-scale semantic features are upsampled to obtain the reference image. Specifically, this includes: After edge reflection filling, the original image is processed by a 7×7 convolution kernel combined with a normalization layer and activation function to map the image from pixel space to shallow features.

[0049] Subsequently, the shallow features are input into four downsampled blocks for feature encoding. Inspired by deep residual networks, to alleviate the gradient vanishing problem as network depth increases, residual convolution is used instead of standard convolution in this stage. Within each downsampled block, the input features are processed in two paths: the main path uses 3×3 convolution for feature extraction, and the side path uses 1×1 convolution for channel transformation. The outputs of the two paths are then element-wise summed and passed through an activation function. After four consecutive downsampling processes, the spatial resolution of the feature map is progressively reduced, thereby obtaining deep features containing rich semantic information.

[0050] The deep features are then fed into nine stacked dilated convolutional blocks. The core of each module is multi-scale grouped dilated convolution: the deep features are grouped along the channel dimension and then fed into three branches in parallel. Each branch uses a 3×3 convolutional kernel with dilation rates of 1, 3, and 5, respectively. The branch with a dilation rate of 1 is used to capture local pixel-level quantization distortion, while the branches with dilation rates of 3 and 5 are used to capture long-range block artifacts and structural distortions. To enhance the network's ability to fit coding artifacts, the output features of each branch are non-linearly mapped using a relocalizable activation function (RPReLU). The features processed by the three branches are then element-wise summed to achieve multi-scale information fusion, outputting the fused multi-scale semantic features.

[0051] Unlike traditional ReLU, which is truncated at zero, RPReLU allows the inflection points of the activation function and the output distribution to be adaptively shifted in the feature space. Its mathematical expression is as follows: in, As input features, and These are the displacement parameters that the network automatically learns during training. This design significantly improves the flexibility of feature representation, enabling the network to more accurately simulate complex nonlinear distortion mappings.

[0052] Finally, the multi-scale semantic features are decoded through four upsampling blocks. This stage also employs a residual structure, combined with transposed convolutions to progressively restore the spatial dimensions of the feature maps, reducing abstract semantic features to high-resolution detailed features. Subsequently, the decoded feature maps are filled with reflections and then mapped to the RGB image space through a 7×7 convolutional layer. Finally, the Tanh activation function is used to normalize the output pixel values, generating the final reference image.

[0053] S22. Generate a real image encoded and compressed by the encoder using the original image; input the reference image and its corresponding real image into the discriminant network to construct a loss function to train the reference image generation network.

[0054] Discriminant Network: To further enhance the realism of the generated reference image in terms of texture details, this invention introduces a multi-scale discriminant network to perform adversarial training with the reference image generation network. The discriminant network adopts a fully convolutional PatchGAN architecture, aiming to supervise the generation quality of the reference image from different resolution levels and discriminate the local consistency of encoded distortion features (such as block artifacts and ringing artifacts).

[0055] When training a reference image generation network, it requires real images encoded and compressed by an encoder as labels, and then trains by inputting the original image. Once the reference image generation network is trained, it only needs to input one original image to quickly generate a reference image. The distortion of this reference image is very close to that of the real image used in the previous training, thus avoiding the need to use encoder-encoded compressed images to guide JRD prediction.

[0056] Because the JRD itself is defined as a record of the lowest QP level that each model can tolerate, it is intended to guide which QP level to use to achieve a higher degree of image compression without affecting the performance of the machine model. Some existing JRD prediction models require multiple compressions by the encoder to obtain images with multiple QP levels for JRD prediction. This multiple encoding process actually contradicts the original purpose of JRD. If multiple encodings are still needed to predict which QP level to use, it would be better to use the traditional method of directly compressing the image at a fixed QP level. Therefore, a reference image generation network can be viewed as a simulated encoder encoding process. Once pre-trained, it can predict an image—the reference image—that closely approximates the actual encoding process in terms of distortion from the original input image in a very short time.

[0057] Specifically, the discriminator network receives both the generated reference image and the corresponding encoder-compressed real image as input. During the discrimination process, both are downsampled twice, forming two sets of three multi-resolution hierarchical images, including the original scale, half-scale, and quarter-scale. Each scale corresponds to a structurally independent discriminator.

[0058] At each resolution level, the discriminator extracts features from the reference image and the real image through multiple convolutional operations, and outputs a corresponding two-dimensional discriminative feature matrix. Each element in the matrix corresponds to a local receptive field region of the input image, and its value represents the confidence that the region belongs to the real image.

[0059] During training, the loss function constrains all elements in the reference image feature matrix to tend towards 0, representing forgery; while all elements in the real image feature matrix are constrained to tend towards 1, representing authenticity. Through this constraint, the discriminator continuously enhances its ability to distinguish between real and fake images, thereby enabling the generator network to generate more realistic reference images to deceive the discriminator, forming a dynamic adversarial relationship between the generator and the discriminator.

[0060] Within the multi-scale framework, the original scale discriminator has the smallest receptive field, focusing on capturing subtle textures in the image; while the 1 / 2 scale and 1 / 4 scale discriminators have larger receptive fields, paying more attention to the overall structural consistency of the image. Finally, the three discriminators calculate their respective discriminative losses (Loss1, Loss2, Loss3), and the sum of the three is used as the overall adversarial loss output.

[0061] Loss Function: In the reference image generation network, this invention introduces a hybrid loss function that includes adversarial loss, feature matching loss, and perceptual reconstruction loss. Optimize the network: in, To combat the losses, For feature matching loss, To perceive the reconstruction loss, The balancing weights for each loss.

[0062] In the formula of this embodiment, it is uniformly stipulated that: Input the original image; For the generated reference image; For the corresponding real image; The number of multi-scale discriminators (in this embodiment) =3); They represent the first The output of the scale discriminator.

[0063] Combat loss ( The least squares generative adversarial loss (LSGAN) is employed. By minimizing the mean squared error between the discriminator output and the target label, a more stable gradient signal is provided to the generator. Its calculation formula is as follows: Feature matching loss: used to measure the reference image With real images Multiscale discriminant In the intermediate layer feature space distance.

[0064] in, Indicates the first The first scale Each feature layer This represents the total number of feature layers in the discriminator. This represents the total number of elements in the feature map.

[0065] Perceptual reconstruction loss ( Feature space extracted from VGG-19 network Distance and pixel space Distance-weighted composition: in, This refers to the total number of convolutional layers used in the VGG network for feature extraction. For the first The total number of elements in the layer feature map. This represents the feature extraction operation at the i-th layer. Describing the L1 norm, For the first Layer weight coefficients, This is the balance coefficient.

[0066] Furthermore, in the discriminant network, the total loss of the discriminator... Defined as the sum of the discriminative losses across the three scales, its mathematical expression is as follows: S3. Input the original image and reference image into the JRD prediction network, which includes a feature extraction module, a feature transformation module, a feature fusion module, and a classification prediction module, generate distortion-aware features, and generate multi-class prediction results based on the distortion-aware features.

[0067] JRD prediction is the core issue in applying it to encoding guidance. Existing binary classifier prediction methods require pre-encoding the original image to generate multiple QP-level distorted images and performing repeated inferences, resulting in significant computational redundancy and potential conflicts between the independent decisions of the binary classifiers. Therefore, this invention defines the JRD prediction problem as a multi-classification problem and designs an impulse-driven JRD prediction network to achieve efficient prediction, outputting classification results with only a single inference.

[0068] like Figure 1 The JRD prediction network shown uses the original image and the reference image generated in S2 as dual inputs. Statistical analysis of the dataset constructed in S1 shows that the overall distribution trend of JRD under different encoding standards is similar, and the median is concentrated around a quantization parameter (QP) of 42 (the proportions of H.264, H.265, and H.266 are 52.38%, 52.55%, and 53.12%, respectively). Finally, the image with a QP of 42 was determined as the generation target, and the features of this distorted image were used to assist in JRD prediction.

[0069] The prediction network mainly consists of a feature extraction module, a feature transformation module, a feature fusion module, and a classification prediction module. The specific implementation is as follows: S31. Input the original image and the reference image in pairs into the feature extraction module of two weight-shared branch networks to generate multi-scale original image features and reference image features; In the feature extraction stage, a feature extraction module based on SpikingResformer (a spiking neural network that combines the multi-stage architecture of ResNet residual network with the global modeling capability of Transformer) is constructed. Specifically, this module consists of two branch networks with shared weights, and the original image and the reference image are fed into it as paired inputs for parallel processing.

[0070] First, the input image is processed through a shallow feature extraction structure consisting of a 7×7 convolutional layer and a 3×3 max pooling layer. This reduces the resolution while preserving key structural information of the image, and outputs a shallow feature map.

[0071] Subsequently, the shallow feature map is pulse-coded by leaky integrated discharge (LIF) neurons. In this process, the LIF neurons treat the element values ​​of the feature map as a constant input current, and through repeated accumulation of membrane potential and threshold excitation within a time step of T=4, the continuous analog signal is transformed into a discrete binary pulse sequence, completing the conversion from the real number domain to the pulse domain.

[0072] Next, the encoded pulse sequence is fed into the three stages of the SpikingResformer network, with each stage connected by a downsampling layer. Each stage consists of several stacked residual blocks (the number of residual blocks in stages 1 to 3 is 1, 2, and 3, respectively). Each residual block contains a residual-connected dual-pulse self-attention module and a grouped pulse feedforward network. The former is used to capture long-range global dependencies, while the latter is used to extract local spatial detail features, thereby achieving the stepwise extraction of high-level semantic features of the image within the pulse domain.

[0073] S32. Input the original image features and reference image features into the feature transformation module. After being unified to a fixed size and spliced ​​in the channel dimension, high-dimensional pulse features are generated. After performing channel compression, integrating cross-channel semantics, and residual connection on the high-dimensional pulse features, the features are mapped to the preset target dimension to generate transformed original image features and transformed reference image features. The feature transformation module uses an adaptive average pooling layer to unify the multi-scale feature maps with different resolutions output from each stage of the feature extraction module to a fixed size (7×7 in this embodiment), and concatenates them along the channel dimension to construct a high-dimensional pulse feature containing multi-scale information. Subsequently, this high-dimensional feature passes sequentially through a 1×1 convolutional layer for compressing channels and integrating cross-channel semantics, a 3×3 grouped convolutional layer that introduces residual connections and can efficiently capture local spatial features, and a 1×1 convolutional layer for mapping features to a preset target dimension, thereby outputting a compact deep multi-scale semantic feature.

[0074] S33. The feature fusion module fuses the features of the original image after feature transformation and the features of the transformed reference image to obtain distortion-perceived features for final classification. The feature fusion module is responsible for fusing the original image features after feature transformation with the reference image features to obtain distortion-perceived features for final classification. , .

[0075] in and These are the original image features and the reference image features after feature transformation, respectively. This indicates that an absolute difference calculation is performed to calculate the feature differences between the original image and the reference image. This is a splicing operation at the channel dimension.

[0076] S34. The classification prediction module generates multi-classification prediction results based on the distortion perception features.

[0077] The classification prediction module will integrate the fused distortion perception features. The input is fed into a fully connected layer, which maps it from the feature space to the class prediction space, generating an impulse response sequence that includes a time dimension. This impulse sequence is then averaged over time to integrate prediction information from different time steps and obtain a stable output value, thus yielding the final multi-class prediction result.

[0078] The classification categories are the JRDs labeled for the target images in the cross-model JRD dataset. For H.264 and H.265, the encoding range is 0-51, so there are 52 classification categories: 0, 1, ..., 50, 51. For H.266, the encoding range is 0-63, so there are 64 classification categories: 0, 1, ..., 62, 63.

[0079] Unlike conventional discrete classification tasks, the labels in the JRD prediction task exhibit a significant progressive relationship and strong ordered correlation, with adjacent labels showing highly similar levels of image distortion. To effectively model this ordinal relationship and improve the model's prediction accuracy, this invention introduces a windowed linear soft labeling strategy. This strategy sets the application window around the JRD ground truth value, assigns probability values ​​to labels within the window, and the probability decreases linearly from the center outwards until it reaches zero at the window edge.

[0080] The soft label strategy for windowing is defined as follows: in, Indicates the current output category index (0≤ ≤63), The category index representing the actual JRD. The radius of the window.

[0081] Taking a window radius R=3 as an example, if the JRD ground truth value of the target image is 48, in a conventional hard-labeling classification task, only the probability of class 48 is assigned a value of 1, while the probabilities of neighboring classes (such as 47, 49) are assigned a value of 0. This approach ignores the distortion similarity between adjacent QP levels. However, under the windowed linear soft-labeling strategy, neighboring classes 47 and 49 are assigned a probability of 0.67, and 46 and 50 are assigned a probability of 0.33. This probability distribution simulates the gradual process of distortion, effectively improving the model's prediction accuracy while allowing for small deviations.

[0082] This embodiment trains the reference image generation network on a single GTX 3090 graphics card with a batch size of 4, a resolution of 512×512, an initial learning rate of 0.0002, and an initial training epoch of 40. The training set used by the model consists of 100,000 images, all taken from other images in the ImageNet dataset, without involving any images from datasets related to JRD. This ensures that the generated reference images are based on the model's generalization ability, guaranteeing the rigor and fairness of subsequent JRD prediction experiments. Since the publicly available OW-JRD dataset is constructed based on the H.266 standard, images with a QP level of 42 after H.266 encoding are selected as real image labels to train the network. After training, simply inputting the original OW-JRD images into the pre-trained generation network is sufficient to generate the reference images required for the subsequent JRD prediction model.

[0083] In this embodiment, the JRD prediction network was trained on a GTX4080 with a batch size of 8, a time step of 4, an initial learning rate of 0.00005, and a cosine annealing strategy was used to dynamically adjust the model's learning rate. The window size was 7 (which showed optimal performance in testing). Experimental results were compared on the OW-JRD public dataset proposed by ZhangTMM23.

[0084] Table 3 shows a preliminary comparison of the performance of the proposed pulse-driven JRD prediction model with that of existing state-of-the-art prediction models, where E A Let E be the mean absolute error (MAE) across all images in the dataset. A[27-51] This refers to samples in the dataset whose JRD values ​​fall within the QP range of [27, 51].

[0085] Table 3: Performance Comparison of the JRD Prediction Network of this Invention with Existing Technology Models

[0086] This embodiment effectively decouples the JRD prediction model from the encoding process by constructing a reference image generation network, under the condition of only inputting a single original image. Experimental results show that the average difference in peak signal-to-noise ratio (PSNR) between the generated reference image and the encoded real image is 1.1661 dB, and the average difference in structural similarity (SSIM) is 0.0276, which can provide reliable feature references for the prediction task.

[0087] In terms of energy consumption, the proposed spiking-driven JRD prediction model demonstrates significant advantages in sparse computation. Based on the event-driven nature of spiking neural networks, the average spiking rate during model inference is approximately 25.86%, effectively reducing redundant computation. Specifically, the prediction model has 0.967 G synaptic operands (SOPs), significantly lower than the 3.74 G multiply-accumulate operations (MACs) required by an artificial neural network (ANN) of equivalent architecture. Energy consumption evaluation results show that the energy consumption per inference cycle of this method is approximately 0.87 mJ, compared to 17.20 mJ for the traditional ANN version, representing an energy efficiency improvement of approximately 19.76 times, indicating good application potential for low-power hardware deployment.

Claims

1. A method for predicting exactly identifiable distortion based on encoding decoupling and pulse driving, characterized in that, Includes the following steps: S1. Establish a cross-model identifiable distortion dataset with multiple coding standards; S2. Input the original images from the cross-model identifiable distortion dataset into the reference image generation network to generate reference images with encoded distortion forms; S3. Input the original image and reference image into a pulse-driven JRD prediction network that includes a feature extraction module, a feature transformation module, a feature fusion module, and a classification prediction module, generate distortion-aware features, and generate multi-classification prediction results based on the distortion-aware features.

2. The identifiable distortion prediction method based on encoding decoupling and pulse driving according to claim 1, characterized in that, For image classification tasks, S1 includes: S11. Select machine vision models and construct a set of machine vision models; record the top-1 category of each machine vision model for the original image; S12. The machine vision model sequentially identifies compressed images with different quantization parameters under a coding standard. When the TOP-1 category on a certain compressed image changes, the maximum quantization parameter level that can maintain the correct classification result in the original image is taken as its just identifiable distortion value under the current machine vision model. S13. All machine vision models sequentially execute the recognition process of S12, and record the cross-model just-recognizable distortion value of the original image as the just-recognizable distortion value among the just-recognizable distortions labeled by all machine vision models. S14. Obtain the cross-model just-identifiable distortion values ​​for each coding standard and construct a just-identifiable distortion dataset under multiple coding standards.

3. The encoding decoupling and pulse-driven identifiable distortion prediction method according to claim 1, characterized in that, S2 include: S21. Construct a reference image generation network that includes a downsampling module, a multi-scale grouped dilated convolution module, and an upsampling module; After edge reflection filling, the original image is processed by convolution kernels combined with normalization layers and activation functions to map it into shallow features. The shallow features are input into downsampling blocks for feature encoding to generate deep features containing semantic information. The deep features are then fused into multi-scale information by a multi-scale grouped dilated convolution module to generate multi-scale semantic features. The multi-scale semantic features are then decoded by an upsampling module to obtain the reference image.

4. The identifiable distortion prediction method based on encoding decoupling and pulse driving according to claim 3, characterized in that, In S21, the shallow features of the input downsampling module are processed in two paths. The main path uses 3×3 convolution for feature extraction, and the side path uses 1×1 convolution for channel transformation. The two outputs are added element by element and the deep features are obtained through the activation function. After grouping deep features along the channel dimension, they are fed into the three branches of the multi-scale grouped dilated convolution module in parallel. The three branches are set with different dilation rates to capture different quantization distortions. The output features of each branch are nonlinearly mapped by a relocalizable activation function and then added element by element to output multi-scale semantic features. The upsampling module uses a residual structure; the decoded feature map is filled by reflection and the high-dimensional features are mapped to the RGB image space through a convolutional layer. Finally, the Tanh activation function is used to normalize the output pixel values ​​to generate a reference image.

5. The identifiable distortion prediction method for encoding decoupling and pulse-driven methods according to claim 4, characterized in that, The relocalizable activation function of the multi-scale grouped dilated convolution module allows the inflection point and output distribution of the activation function to be adaptively shifted in the feature space. Its mathematical expression is as follows: in, As input features, and These are the displacement parameters that the network learns automatically during training.

6. The identifiable distortion prediction method based on encoding decoupling and pulse driving according to claim 3, characterized in that, S2 also includes: S22. Input the reference image and its corresponding real image into the discriminant network to construct a loss function to train the reference image generation network; the real image is the original image corresponding to the reference image in the cross-model just-recognizable distortion dataset after encoding and compression.

7. The identifiable distortion prediction method for encoding decoupling and pulse-driven methods according to claim 6, characterized in that, S22 includes: Both the reference image and the real image are downsampled twice to form two sets of multi-resolution hierarchical images with multiple scales; each scale corresponds to a structurally independent discriminator; the discriminator at each resolution level extracts features from the reference image and the real image respectively, and outputs two-dimensional discriminant feature matrices of the reference image and the real image; each element in the two-dimensional discriminant feature matrix corresponds to a local receptive field region, and its value represents the confidence that the region belongs to the real image; During training, the loss function constrains all elements of the two-dimensional discriminant feature matrix of the reference image to tend to 0, representing a forgery, and constrains all elements of the two-dimensional discriminant feature matrix of the real image to tend to 1, representing a real image.

8. The identifiable distortion prediction method for encoding decoupling and pulse-driven methods according to claim 7, characterized in that, Use a hybrid loss function that includes adversarial loss, feature matching loss, and perceptual reconstruction loss. Optimize the reference image generation network: in, To combat the losses, For feature matching loss, To perceive the reconstruction loss, The balancing weights for each loss; The adversarial loss employs least-squares generative adversarial loss, which provides a more stable gradient signal by minimizing the mean square error between the discriminator output and the real image; the feature matching loss is used to measure the difference between the reference image and the real image in the intermediate layer feature space of the multi-scale discriminator. Distance; Perceptual reconstruction loss is derived from the feature space extracted by the VGG-19 network. Distance and pixel space Distance-weighted composition.

9. The identifiable distortion prediction method for encoding decoupling and pulse-driven methods according to claim 1, characterized in that, S3 include: S31. Input the original image and the reference image in pairs into the feature extraction module of two weight-shared branch networks to generate multi-scale original image features and reference image features; S32. Input the original image features and reference image features into the feature transformation module. After being unified to a fixed size and spliced ​​in the channel dimension, high-dimensional pulse features are generated. After performing channel compression, integrating cross-channel semantics, and residual connection on the high-dimensional pulse features, the features are mapped to the preset target dimension to generate transformed original image features and transformed reference image features. S33. The feature fusion module fuses the features of the original image after feature transformation and the features of the transformed reference image to obtain distortion-perceived features for final classification. S34. The classification prediction module generates multi-classification prediction results based on the distortion perception features.

10. The identifiable distortion prediction method for encoding decoupling and pulse-driven methods according to claim 9, characterized in that, In S31, the original image and the reference image are processed by a shallow feature extraction structure to output a shallow feature map. The shallow feature map is pulse-coded by a leaky integrated firing neuron. The encoded pulse sequence is fed into three stages of the SpikingResformer network, which are connected by downsampling layers. Each stage consists of several stacked residual blocks. The residual blocks include a dual-pulse self-attention module with residual connections for capturing long-range global dependencies and a grouped pulse feedforward network for extracting local spatial detail features.

Images