Image processing method and apparatus

Abstract

Embodiments of the present application provide a method for image processing, which comprises: obtaining a reconstructed image and a standard image of the reconstructed image, the reconstructed image being an image reconstructed according to a first image, the first image being an image obtained by compressing the standard image; inputting the reconstructed image and the standard image into a single-layer convolutional neural network to obtain a feature map of the reconstructed image and a feature map of the standard image, the single-layer convolutional neural network being a first layer convolutional layer of a pre-trained model, the pre-trained model being a pre-trained convolutional neural network; and obtaining a distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image. The method for image processing according to the embodiments of the present application is used to obtain the distortion degree of the reconstructed image and the original image, so as to evaluate the quality of the reconstructed image, thereby guiding the optimization of the encoder and / or the decoder.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method and apparatus. Background Technology

[0002] Cameras, characterized by high resolution, non-contact operation, ease of use, and low cost, have wide applications in environmental perception. For example, in the field of autonomous driving, an increasing number of cameras are being installed on vehicles to achieve blind-spot-free coverage and artificial intelligence (AI) vehicles with machine vision. As camera resolution, frame rate, and sampling depth continue to improve, the video output from cameras demands ever-increasing transmission bandwidth. To alleviate the pressure on transmission networks… Figure 1 A schematic block diagram of an existing solution is shown, such as Figure 1 As shown, Bayer raw images or videos output by cameras often have high precision and require high transmission bandwidth. For example, for ultra-high definition (UHD) video with a frame rate of 30fps, a sampling depth of 16bit depth, and a resolution of 4K, the bandwidth requirement is as high as 4Gbps (4K*2K*30*16). Therefore, to alleviate the pressure on the transmission network, the Bayer raw images or videos need to be compressed before being transmitted to the mobile data computing platform (MDC). Simultaneously, image signal processing (ISP) is required to reduce bandwidth demand, enabling high-definition video services without upgrading existing network bandwidth. To meet existing bandwidth requirements, the Bayer raw images or videos output by cameras need to be compressed at a high rate. This often employs lossy image or video compression techniques, inevitably leading to image or video quality degradation. Furthermore, ISP processing also results in information loss in the images or videos. The RGB image obtained after encoder compression and ISP processing is transmitted to the MDC for image reconstruction. Image reconstruction is performed by the decoder. The quality of the reconstructed image is crucial for subsequent tasks such as object detection and semantic segmentation. Therefore, it is particularly important that the image reconstructed by the decoder has high quality. However, since the RGB image obtained after compression and ISP processing is degraded, the quality of the reconstructed image cannot be guaranteed. Therefore, there is an urgent need for a method to determine the distortion between the reconstructed image and the original image, thereby evaluating the quality of the reconstructed image and optimizing the decoder based on the evaluation results to ensure that the reconstructed image output by the decoder has high quality in practical applications. Summary of the Invention

[0003] This application provides an image processing method for obtaining the distortion of a reconstructed image and the original image to evaluate the quality of the reconstructed image, thereby guiding the optimization of the encoder and / or decoder.

[0004] In a first aspect, an image processing method is provided, comprising: acquiring a reconstructed image and a standard image of the reconstructed image, wherein the reconstructed image is an image reconstructed from a first image, and the first image is an image obtained by compressing the standard image; inputting the reconstructed image and the standard image into a single-layer convolutional neural network to obtain feature maps of the reconstructed image and the standard image, wherein the parameters of the single-layer convolutional neural network are derived from the parameters of the first convolutional layer of a pre-trained model, and the pre-trained model is a pre-trained convolutional neural network; and acquiring the distortion of the feature map of the reconstructed image relative to the feature map of the standard image.

[0005] In this embodiment, since the pre-trained model is designed for machine tasks, using a single-layer convolutional neural network from the pre-trained model to extract image features can better adapt to machine vision tasks. The initial convolutional layer often extracts the low-level general features of the image. In this embodiment, the single-layer convolutional neural network is the first convolutional layer of the pre-trained model, and the image features extracted using this single-layer convolutional neural network are also low-level general features. The encoder-decoder optimized for low-level general features can adapt to various task scenarios and improve multi-task generalization ability. In addition, compared with the prior art, which uses a complete neural network to process images, this embodiment only requires single-layer convolution calculation, which reduces computational complexity and reduces the computing power requirements of the hardware.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, a single-layer convolutional neural network includes multiple convolutional kernels, the first feature map of the reconstructed image and the first feature map of the standard image have first weights, the first feature map of the reconstructed image and the first feature map of the standard image are obtained by the first convolutional kernel, the second feature map of the reconstructed image and the second feature map of the standard image have second weights, the first feature map of the reconstructed image and the second feature map of the standard image are obtained by the second convolutional kernel, and the first convolutional kernel and the second convolutional kernel belong to multiple convolutional kernels.

[0007] In this application, different weights are assigned to different feature maps to achieve different effects. For example, the weight of feature maps with detailed features can be appropriately increased to increase the importance of the detail effect. This has a beneficial effect on both human visual perception and the subsequent execution of machine vision tasks.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the weights are determined by normalization parameters, which are the normalization parameters of the pre-trained model.

[0009] The weighting coefficients used for weighting can be determined manually or based on the normalization parameters.

[0010] In conjunction with the first aspect, in certain implementations of the first aspect, obtaining the distortion of the feature map of the reconstructed image relative to the feature map of the standard image includes: calculating the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0011]

[0012] wfMSE represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, where x is the standard image, y is the reconstructed image, and f is the standard image. i () represents the convolution operation, i is the i-th convolution kernel, and w i , where C is the number of feature map channels in the reconstructed image or the standard image, H is the height of the standard image or the reconstructed image, and W is the width of the standard image or the reconstructed image.

[0013] The above method is based on the existing computation method MSE, which performs single-layer convolution operation on the image. It is simple to compute and has a small computational cost. The convolution kernel parameters and weight parameters are derived from the pre-trained model for machine tasks. Based on this, the encoder and decoder are optimized so that the output reconstructed image is more suitable for machine vision tasks. Different weights are assigned to different feature maps, and the weighting coefficients can be freely adjusted. For feature maps related to details, the weights can be increased to enhance the image's detail and texture features.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, w i =1, or or γ i σ is the scaling factor for the pre-trained model. i These are the normalization coefficients of the pre-trained model.

[0015] Since the pre-trained model is trained for machine vision tasks, when the weight coefficients come from the pre-trained model rather than being designed manually, it can be ensured that the encoder and / or decoder optimized accordingly will perform better when facing the same machine vision task.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: calculating the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0017]

[0018] wfSSIM represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, μ is the mean, σ is the covariance, C1 is a constant, and C2 is a constant.

[0019] The embodiments of this application can also obtain the distortion of the reconstructed image based on other indicators, such as the calculation method of structural similarity between images. Based on structural similarity calculation, because statistical measures such as mean and variance are used, it is easier to avoid the influence of noise (such as ringing noise), thereby obtaining more stable results and effectively improving accuracy in machine tasks such as semantic segmentation.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: calculating the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0021]

[0022] wfMSSSIM represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, and C3 is a constant.

[0023] The SSIM-based structural similarity method can also be adapted, such as the MSSSIM-based method. Compared to the SSIM method, it enhances image details while preserving noise resistance. The resulting optimized encoder-decoder outputs higher-quality reconstructed images, which is significant for subsequent machine tasks such as object detection.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, a single-layer convolutional neural network includes multiple convolutional kernels. A first convolutional kernel is used to obtain a first feature of the reconstructed image and a first feature of the standard image, and the coefficients of the first convolutional kernel have a first weight. A second convolutional kernel is used to obtain a second feature of the reconstructed image and a second feature of the standard image, and the coefficients of the second convolutional kernel have a second weight. The first and second convolutional kernels belong to multiple convolutional kernels.

[0025] This method uses weighted convolution kernels to extract features from the reconstructed and standard images separately, eliminating the need for separate weighting of the feature maps and saving computation. Furthermore, the coefficients of the convolution kernel are generally much smaller than the number of pixels in the image (image pixels are related to the specific resolution), and the computational cost of weighting the convolution kernel coefficients is much less than that of weighting the feature maps, especially at high image resolutions.

[0026] In conjunction with the first aspect, in certain implementations of the first aspect, obtaining the distortion of the feature map of the reconstructed image relative to the feature map of the standard image includes: calculating the distortion of the feature map of the reconstructed image according to the following formula:

[0027]

[0028] wfMSE represents the distortion of the feature map of the reconstructed image, x represents the standard image, y represents the reconstructed image, and g represents the distortion of the feature map. i ()=w i ×f i ()=fw i (), f i () represents the convolution operation, i is the i-th convolution kernel, and w i , where C is the number of feature map channels in the reconstructed image or the standard image, H is the height of the standard image or the reconstructed image, and W is the width of the standard image or the reconstructed image.

[0029] This method is a fast implementation of the previous MSE method. The previous method performs convolution operations on the standard image and the reconstructed image separately and then calculates the residual. This requires two convolution operations for a single convolution kernel. This method, however, calculates the residual first and then convolves the residual. This requires only one convolution operation for a single convolution kernel, thus saving computation.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: evaluating the quality of the reconstructed image based on the distortion of the feature map of the reconstructed image relative to the feature map of the standard image to obtain an evaluation result; and optimizing the encoder and / or decoder based on the evaluation result, wherein the encoder and / or decoder are used to output the reconstructed image.

[0031] During the training of the encoder and / or decoder, it is desirable for the reconstructed image output by the encoder and / or decoder to be as close as possible to the standard image. The distortion of the reconstructed image represents the difference between the reconstructed image and the standard image. Therefore, the parameters of the encoder and / or decoder can be updated according to the distortion to minimize the distortion of the output reconstructed image, thereby optimizing the encoder and / or decoder.

[0032] In conjunction with the first aspect, in some implementations of the first aspect, the feature map of the reconstructed image and the feature map of the standard image are full-resolution images.

[0033] The single-layer convolutional neural network used in this embodiment does not include pooling layers, and the convolutional layers do not perform downsampling operations on the reconstructed image and the standard image. Without downsampling, the image is not compressed, thus allowing the output of a full-resolution image. Optimizing the encoder-decoder based on the evaluation results of the full-resolution image ensures the quality of the reconstructed image output by the optimized encoder-decoder, making it more visually friendly to the human eye.

[0034] Secondly, an image processing apparatus is provided, comprising: an acquisition unit for acquiring a reconstructed image and a standard image of the reconstructed image, wherein the reconstructed image is an image reconstructed from a first image, and the first image is an image obtained by compressing the standard image; a processing unit for inputting the reconstructed image and the standard image into a single-layer convolutional neural network to obtain feature maps of the reconstructed image and the standard image, wherein the parameters of the single-layer convolutional neural network are derived from the parameters of the first convolutional layer of a pre-trained model, and the pre-trained model is a pre-trained convolutional neural network; the processing unit is further configured to acquire the distortion of the feature map of the reconstructed image relative to the feature map of the standard image.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, a single-layer convolutional neural network includes multiple convolutional kernels, the first feature map of the reconstructed image and the first feature map of the standard image have first weights, the first feature map of the reconstructed image and the first feature map of the standard image are obtained by the first convolutional kernel, the second feature map of the reconstructed image and the second feature map of the standard image have second weights, the first feature map of the reconstructed image and the second feature map of the standard image are obtained by the second convolutional kernel, and the first convolutional kernel and the second convolutional kernel belong to multiple convolutional kernels.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, the weights are determined by normalization parameters, which are the normalization parameters of the pre-trained model.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to: calculate the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0038]

[0039] wfMSE represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, where x is the standard image, y is the reconstructed image, and f is the standard image. i () represents the convolution operation, i is the i-th convolution kernel, and w i , where C is the number of feature map channels in the reconstructed image or the standard image, H is the height of the standard image or the reconstructed image, and W is the width of the standard image or the reconstructed image.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, w i =1, or or γ i σ is the scaling factor for the pre-trained model. i These are the normalization coefficients of the pre-trained model.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is further configured to: calculate the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0042]

[0043] wfSSIM represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, μ is the mean, σ is the covariance, C1 is a constant, and C2 is a constant.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is further configured to: calculate the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0045]

[0046] wfMSSSIM represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, and C3 is a constant.

[0047] In conjunction with the second aspect, in some implementations of the second aspect, a single-layer convolutional neural network includes multiple convolutional kernels. The first convolutional kernel is used to obtain the first feature of the reconstructed image and the first feature of the standard image, and the coefficients of the first convolutional kernel have a first weight. The second convolutional kernel is used to obtain the second feature of the reconstructed image and the second feature of the standard image, and the coefficients of the second convolutional kernel have a second weight. The first and second convolutional kernels belong to multiple convolutional kernels.

[0048] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to: calculate the distortion of the feature map of the reconstructed image according to the following formula:

[0049]

[0050] wfMSE represents the distortion of the feature map of the reconstructed image, x represents the standard image, y represents the reconstructed image, and g represents the distortion of the feature map. i ()=w i ×f i ()=fw i (), f i () represents the convolution operation, i is the i-th convolution kernel, and w i , where C is the number of feature map channels in the reconstructed image or the standard image, H is the height of the standard image or the reconstructed image, and W is the width of the standard image or the reconstructed image.

[0051] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is further configured to: evaluate the quality of the reconstructed image based on the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, to obtain an evaluation result; and optimize the encoder and / or decoder based on the evaluation result, wherein the encoder and / or decoder are used to output the reconstructed image.

[0052] In conjunction with the second aspect, in some implementations of the second aspect, the feature maps of the reconstructed image and the standard image are full-resolution images.

[0053] Thirdly, a computer-readable medium is provided, characterized in that the computer-readable medium stores program code for execution by a device, the program code including a method for performing any of the implementations of the first aspect described above.

[0054] Fourthly, a chip is provided, which includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes any of the implementation methods of the first aspect described above. Attached Figure Description

[0055] Figure 1 This is a schematic block diagram illustrating the compression and transmission of images output from a camera according to an embodiment of this application;

[0056] Figure 2 This is a schematic block diagram illustrating the use of images acquired by the camera in various tasks according to an embodiment of this application;

[0057] Figure 3 This is a schematic structural diagram of a convolutional neural network according to an embodiment of this application;

[0058] Figure 4 This is a schematic flowchart of an image processing method according to an embodiment of this application;

[0059] Figure 5 This is a schematic diagram of the convolution operation on the standard image and the reconstructed image according to an embodiment of this application;

[0060] Figure 6 This is a schematic diagram illustrating the process of obtaining the distortion of a reconstructed image according to an embodiment of this application;

[0061] Figure 7 This is a schematic diagram illustrating the application scenario of the image processing method of this application in the optimization of the encoder and / or decoder;

[0062] Figure 8 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application;

[0063] Figure 9 This is a schematic diagram of the structure of the image processing apparatus 900 according to an embodiment of this application. Detailed Implementation

[0064] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0065] Figure 2 This illustration shows a schematic block diagram of images acquired by a camera according to an embodiment of this application, used for various tasks. Taking an autonomous driving scenario as an example, images of the vehicle's surroundings acquired by a camera sensor mounted on the vehicle are compressed by an encoder or processed by an ISP to obtain RGB images. The accuracy of RGB images is much lower than that of the original images output by the camera, thereby reducing the bandwidth requirements for network transmission. The RGB images are processed by an encoder and decoder (CODEC) (including compression and reconstruction of the original image) to obtain reconstructed images. These reconstructed images are used for tasks such as object detection, semantic segmentation, and traffic light and lane line detection. Obviously, the quality of the reconstructed image is crucial to the performance of tasks such as object detection, semantic segmentation, and traffic light and lane line detection. For example, if the quality of the reconstructed image is poor, it will be difficult to accurately identify objects in object detection tasks, which may pose a significant danger to autonomous driving. Therefore, it is necessary to ensure that the reconstructed images output by the CODEC have high quality. Consequently, a method is needed to determine the distortion between the reconstructed image output by the CODEC and the original image, thereby evaluating the quality of the reconstructed image. Based on the evaluation results, the CODEC can be optimized so that the optimized CODEC can output reconstructed images with higher quality.

[0066] To some extent, the aforementioned tasks such as object detection, semantic segmentation, and traffic light / lane detection can be considered machine vision tasks. This means that the reconstructed images are directly processed by the machine system, and therefore, the reconstructed images primarily meet the requirement of being quickly recognized and detected by the machine system. To address the needs of these machine vision tasks, existing technologies have proposed various machine vision-oriented evaluation metrics to evaluate the quality of reconstructed images, including Top-1 Acc for image classification, mAP for object detection, mIoU for semantic segmentation, and Acc for lane detection. Optimizing the CODEC using these evaluation metrics for a specific machine vision task can achieve good results. However, such optimized CODECs become coupled with the specific machine vision task and cannot adapt to multiple task scenarios. For example, using the Top-1 Acc image classification metric to optimize the CODEC results in reconstructed images that perform well for image classification tasks, but still perform poorly for tasks such as object detection, semantic segmentation, and lane detection, failing to achieve task-generalization evaluation. In addition to the aforementioned machine vision-oriented evaluation metrics, existing technologies also include human vision-oriented evaluation metrics, such as peak signal-to-noise ratio (PSNR), multiscale structural similarity index (MSSSIM), and learned perceptual image patch similarity (LPIPS). The reconstructed image output by the CODEC optimized according to human vision-oriented evaluation metrics can better match the subjective perception of the human eye. For example, if the reconstructed image is directly displayed on a display screen inside a vehicle for the driver to view, the reconstructed image needs to have higher clarity and be easy for the human eye to view. However, in reality, the complexity of the human eye cannot be satisfied by a single evaluation metric, and existing evaluation metrics for human vision all have their own shortcomings. Taking LPIPS as an example, evaluating the quality of reconstructed images using the LPIPS evaluation metric requires calculating all convolutional layers of the network, resulting in high computational complexity. In addition, because the network involves pooling and downsampling, image information is lost during the evaluation process, and only low-resolution feature maps are used in the evaluation process, making it difficult to produce accurate evaluation results. Inaccurate evaluation results also make it difficult to guarantee the optimization results of CODEC, thus making it difficult to guarantee the quality of the reconstructed image.

[0067] Since images acquired by cameras are primarily used by machine systems to perform related tasks—for example, in autonomous driving scenarios, images acquired by camera sensors are mainly used for tasks such as object detection, semantic segmentation, and detection of traffic lights and lane lines—to achieve perception of the environment surrounding the vehicle—methods for evaluating the quality of reconstructed images should first address machine vision. Simultaneously, to meet the application scenarios of various machine vision tasks, methods for evaluating the quality of reconstructed images should be decoupled from specific tasks. Furthermore, in some cases, it is necessary to consider human vision as well; for example, images acquired by camera sensors may also need to be displayed on an in-vehicle screen for the driver to view.

[0068] Therefore, this application proposes an image processing method to obtain the distortion of a reconstructed image and the original image, in order to evaluate the quality of the reconstructed image and thus guide the optimization of the CODEC. The image processing method of this application is geared towards machine vision tasks, can meet the needs of various tasks, and also takes into account human vision.

[0069] To better understand the solutions of the embodiments of this application, the convolutional neural network (CNN) that may be involved in the embodiments of this application will be briefly introduced below.

[0070] Figure 3 A schematic diagram of a convolutional neural network according to an embodiment of this application is shown. The CNN 300 may include an input layer 310, a convolutional / pooling layer 320 (where the pooling layer is optional), and a fully connected layer 330. The details of these layers are described below.

[0071] Convolutional / pooling layers 320:

[0072] Convolutional layers:

[0073] like Figure 3 The convolutional / pooling layer 320 shown may include layers as in Examples 321-326. For instance, in one implementation, layer 321 is a convolutional layer, layer 322 is a pooling layer, layer 323 is a convolutional layer, layer 324 is a pooling layer, layer 325 is a convolutional layer, and layer 326 is a pooling layer; in another implementation, layers 321 and 322 are convolutional layers, layer 323 is a pooling layer, layers 324 and 325 are convolutional layers, and layer 326 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0074] The following section will use convolutional layer 321 as an example to introduce the internal working principle of a single convolutional layer.

[0075] Convolutional layer 321 can include multiple convolution operators, also known as convolution kernels. In image processing, these operators act as filters, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, typically predefined. During convolution, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) horizontally, extracting specific features. The size of this weight matrix should be related to the image size. Importantly, the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same size (rows × columns) are used instead. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" dimensions mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract edge information, another to extract specific colors, and yet another to blur unwanted noise. These multiple weight matrices have the same size (rows × columns), and the resulting convolutional feature maps are also of the same size. These extracted convolutional feature maps are then merged to form the output of the convolution operation.

[0076] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 300 to make correct predictions.

[0077] When a convolutional neural network 300 has multiple convolutional layers, the initial convolutional layers (e.g., 321) tend to extract more general features, which can also be called low-level general features or low-level features. As the depth of the convolutional neural network 300 increases, the features extracted by later convolutional layers (e.g., 326) become more and more complex, such as high-level semantic features.

[0078] Pooling layer:

[0079] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 3Layers 321-326 in example 320 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator selects the pixel with the largest value within a specific range as the result of max pooling. Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input image of the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0080] It should be noted that, in addition to the pooling layers mentioned above, convolutional layers themselves can also compress images. For example, when a convolutional kernel performs a convolution operation on an image, if its stride is greater than 1, image compression can be achieved; this type of image compression is called downsampling.

[0081] Fully connected layer 330:

[0082] After processing by the convolutional / pooling layers 320, the convolutional neural network 300 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 320 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 300 needs to utilize fully connected layers 330 to generate one or a set of the required number of classes in the output. Therefore, the fully connected layers 330 can include multiple hidden layers (such as...). Figure 3 As shown in 331, 332 to 33n) and output layer 340, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0083] After the multiple hidden layers in the fully connected layer 330, the final layer of the entire convolutional neural network 300 is the output layer 340. This output layer 340 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 300 has propagated forward (e.g., ... Figure 3 Propagation from 310 to 340 degrees is forward propagation, and backward propagation (such as...) is completed. Figure 3The propagation from 340 to 310 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 300 and the error between the output of the convolutional neural network 300 through the output layer and the ideal result.

[0084] It should be noted that, as Figure 3 The convolutional neural network 300 shown is only an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models.

[0085] Existing technologies use [methods] when obtaining the distortion of the reconstructed image and the original image. Figure 3 The convolutional neural network shown processes the reconstructed and original images. However, due to pooling and downsampling operations during image processing, information is lost in the processed image, resulting in low-resolution feature maps and inaccurate distortion levels in both the reconstructed and original images. Consequently, the evaluation of the reconstructed image quality is also inaccurate. Optimizing the CODEC based on these inaccurate evaluations leads to poor optimization results, and the reconstructed image output by the CODEC may contain defects. Furthermore, using… Figure 3 The entire convolutional layer shown processes the image, resulting in high computational complexity.

[0086] Figure 4 A schematic flowchart illustrating an image processing method according to an embodiment of this application is shown, using... Figure 4 The image processing method shown can obtain the distortion of the reconstructed image and the original image to evaluate the quality of the reconstructed image, thereby guiding the optimization of encoders, decoders, and ISP image processing algorithms. Specific application scenarios include processing images acquired by cameras in assisted / autonomous vehicles or processing images acquired by cameras in smart city systems / video surveillance systems. Figure 4 The method includes steps 401 to 403, which will be described below.

[0087] S401, acquire the reconstructed image and the standard image of the reconstructed image. The reconstructed image is the image reconstructed from the first image, and the first image is the image obtained by compressing the standard image.

[0088] Figure 4The image processing method shown can be applied to the training process of a codec. In one possible training method, a standard image and its corresponding compressed image need to be obtained. These can be from any training set such as ImageNet, Kitti, Cocoa, or Cityscapes. The compressed image is then input into the codec, which outputs a reconstructed image that corresponds to the aforementioned compressed and standard images. The distortion between the reconstructed and standard images determines the optimization direction of the codec. Generally, it is desirable for the reconstructed image output by the codec to be as close as possible to the standard image; therefore, it is necessary to obtain the distortion between the reconstructed and standard images.

[0089] S402, input the reconstructed image and the standard image into a single-layer convolutional neural network to obtain the feature maps of the reconstructed image and the standard image. The parameters of the single-layer convolutional neural network are derived from the parameters of the first convolutional layer of the pre-trained model, which is a pre-trained convolutional neural network.

[0090] To obtain the distortion of the reconstructed image and the standard image, it is first necessary to obtain the feature maps of the reconstructed image and the standard image. This embodiment uses a convolutional neural network (CNN) to extract features from the reconstructed image and the standard image. Unlike existing technologies that use complete neural networks, this embodiment uses a single-layer CNN for feature extraction. The parameters of this single-layer CNN are derived from the parameters of the first convolutional layer of a pre-trained model, including the parameters of the convolutional kernels and normalization parameters. The pre-trained model is a pre-trained CNN model, such as ResNet, AlexNet, VGGNet, RegNet, etc., classification models trained on the large-scale ImageNet training set. The single-layer CNN includes multiple convolutional kernels, each used to extract different features. Each kernel has a clear physical meaning; for example, the first kernel is used to extract rapidly changing texture details, and the second kernel is used to extract image edge features and color information. In this embodiment, since the pre-trained model is designed for machine vision tasks, using a single-layer CNN with parameters derived from the pre-trained model to extract image features can better adapt to machine vision tasks. Figure 3 As described above, the initial convolutional layer often extracts the low-level general features of the image. The single-layer convolutional neural network in this embodiment is the first convolutional layer of the pre-trained model. The image features extracted by using this single-layer convolutional neural network are also low-level general features. The encoder and decoder optimized for low-level general features can adapt to various task scenarios and improve multi-task generalization ability. In addition, compared with the prior art of using a complete neural network to process images, the single-layer convolutional neural network in this embodiment only needs to perform single-layer convolution calculation, which reduces the computational complexity and the computing power requirements of the hardware.

[0091] Furthermore, the single-layer convolutional neural network used in this embodiment does not include pooling layers, and the convolutional layers do not perform downsampling operations on the reconstructed image and the standard image. Without downsampling, the image is not compressed, thus allowing the output of a full-resolution image. Optimizing the encoder-decoder based on the evaluation results of the full-resolution image ensures the quality of the reconstructed image output by the optimized encoder-decoder, making it more visually friendly to the human eye.

[0092] In one possible implementation, a first convolutional kernel is used to extract a first feature to obtain a first feature map of the reconstructed image and a first feature map of the standard image, and a second convolutional kernel is used to extract a second feature to obtain a second feature map of the reconstructed image and a second feature map of the standard image. Embodiments of this application weight the first and second feature maps respectively, giving the first feature map a first weight and the second feature map a second weight. Different weights are assigned to different feature maps to achieve different effects. For example, the weight of feature maps related to detailed features can be appropriately increased to enhance the importance of the detail effect, thus having a beneficial effect on both human visual perception and the subsequent execution of machine vision tasks.

[0093] In another possible implementation, the coefficients of the first and second convolutional kernels can be directly weighted separately, giving the coefficients of the first kernel a third weight and the coefficients of the second kernel a fourth weight. The weighted convolutional kernels then extract features from the reconstructed and standard images separately, eliminating the need for further weighting of the feature maps and saving computation. Furthermore, generally, the coefficients of the convolutional kernels are far fewer than the number of pixels in the image (image pixels are related to the specific resolution), and the computational cost of weighting the convolutional kernel coefficients is far less than the computational cost of weighting the feature maps, especially at high image resolutions.

[0094] In another possible implementation, the weighting can be combined with the specific application scenario, by weighting both the convolution kernel coefficients and the feature maps.

[0095] The weighting coefficients used for weighting can be determined manually or based on the normalization parameters, which are derived from the normalization parameters of the pre-trained model.

[0096] S403, obtain the distortion of the feature map of the reconstructed image relative to the feature map of the standard image.

[0097] Algorithms for obtaining the distortion of the feature map of a reconstructed image relative to the feature map of a standard image can be based on existing evaluation metrics, such as MSE or MSSSIM. Based on the above description, four methods for calculating the distortion of the feature map of a reconstructed image relative to the feature map of a standard image are given below. However, it should be understood that the following four methods are merely examples of methods for calculating the distortion of the feature map of a reconstructed image relative to the feature map of a standard image and do not constitute a limitation of this application. In addition to the following four methods, other possible calculation methods may be used in the embodiments of this application.

[0098] Method 1: The distortion of the feature map of the reconstructed image relative to the feature map of the standard image can be calculated using the following formula:

[0099]

[0100] Where wfMSE represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, x is the standard image, y is the reconstructed image, and f is the standard image. i () represents the convolution operation, i is the i-th convolution kernel, and w i , where C is the number of feature map channels in the reconstructed image or the standard image, H is the height of the standard image or the reconstructed image, and W is the width of the standard image or the reconstructed image.

[0101] The convolution kernel parameters and weight coefficients are derived from the pre-trained model. The convolution kernel parameters include 64 7x7 convolution kernels. The weight coefficients can be manually set or derived from the normalized parameters of the pre-trained model. The normalized parameters include the scaling factor γ. i and normalization coefficient σ i .

[0102] Specifically, the convolution operation on the standard image and the reconstructed image is as follows: Figure 5 As shown, 64 7*7 convolutional kernels are used to extract features from the standard image x to obtain the feature map f(x) of the standard image. The feature map obtained by the i-th convolutional kernel is f(x). i (x); 64 7*7 convolutional kernels are used to extract features from the reconstructed image y to obtain the feature map f(y) of the reconstructed image. The feature map obtained by the i-th convolutional kernel is f(y). i (y). stride=1 means that the image is not downsampled, which can maintain the image with high resolution and help to recover more details of the reconstructed image.

[0103] When the weight coefficients are derived from the normalized parameters of the pre-trained model, their values ​​can specifically be w. i =1, or or w iDifferent values ​​of correspond to different enhancement effects. Increasing the weight of feature maps related to details can achieve detail enhancement.

[0104] The first method described above is based on the existing computational method MSE. It performs a single-layer convolution operation on the image, which is simple and has a small computational cost. The convolution kernel parameters and weight parameters are derived from a pre-trained model for machine tasks. Based on this, the encoder and decoder are optimized, making the output reconstructed image more suitable for machine vision tasks. Different weights are assigned to different feature maps, and the weighting coefficients can be freely adjusted. For feature maps related to details, the weights can be increased to enhance the image's detail and texture features.

[0105] Method 2: In response to Method 1, this application proposes a fast implementation method based on Method 1. Specifically, the distortion of the reconstructed image's feature map relative to the standard image's feature map can be calculated using the following formula:

[0106]

[0107] Where wfMSE represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, x is the standard image, y is the reconstructed image, and g is the standard image. i ()=w i ×f i ()=fw i (), f i () represents the convolution operation, i is the i-th convolution kernel, and w i , where C is the number of feature map channels in the reconstructed image or the standard image, H is the height of the standard image or the reconstructed image, and W is the width of the standard image or the reconstructed image.

[0108] Method 1 involves convolving the standard image and the reconstructed image separately before calculating the residual. This requires two convolution operations for a single convolution kernel. Method 2, on the other hand, calculates the residual first and then convolves the residual. This requires only one convolution operation for a single convolution kernel, thus saving computation.

[0109] Method 1 weights the feature maps, while Method 2 weights the convolution kernel coefficients to obtain a new convolution kernel g. i ()=w i ×f i ()=fw i Since the convolution kernel has only 7*7 coefficients, which are independent of the image resolution, while the image has W×H pixels (generally much larger than 7*7), and the higher the image resolution, the more pixels there are, the more computationally intensive the convolution kernel coefficients can be compared with the feature map, especially for high-resolution images.

[0110] The process of obtaining the distortion of the reconstructed image according to Method 2 is as follows: Figure 6 As shown, firstly, the encoder and decoder output the reconstructed image y corresponding to the standard image x. Then, the residual z = xy between the standard image and the reconstructed image is calculated, and then a weighted convolution kernel g is used. i () Perform a convolution operation on the residuals to obtain the convolution result. Finally, calculate the convolution result. The variance of is wfMSE.

[0111] Method 3: Methods 1 and 2 use the MSE calculation method to obtain the distortion of the reconstructed image, which is a calculation method based on the pixel distortion between images. In this application embodiment, the distortion of the reconstructed image can also be obtained based on other indicators such as the structure similarity index (SSSIM) between images.

[0112]

[0113] Where wfSSIM is the distortion of the reconstructed image relative to the standard image, μ is the mean, σ is the covariance, C1 is a constant, and C2 is a constant.

[0114] Method 3 calculates the distortion of the reconstructed image relative to the standard image based on structural similarity metrics. It performs SSIM calculation on the convolutional feature map and then calculates a weighted average of the results to obtain wfSSIM. Compared to methods 1 and 2, which are based on pixel-level calculations, method 3, based on structural similarity, uses statistical measures such as mean and variance, making it easier to avoid the influence of noise (e.g., ringing noise), thus achieving more stable results and effectively improving accuracy in machine tasks such as semantic segmentation.

[0115] Method 4: The SSIM-based calculation method in Method 3 can also have other variations, such as the MSSSIM-based calculation method.

[0116]

[0117]

[0118] Where wfMSSSIM represents the distortion of the reconstructed image's feature map relative to the standard image's feature map, and C3 is a constant. α M β j γ j The value can be:

[0119] β1=γ1=0.0448

[0120] β² = γ² = 0.2856

[0121] β3=γ3=0.3001

[0122] β4=γ4=0.2363

[0123] α5=β5=γ5=0.1333

[0124] Method 4 measures the structural similarity between images at multiple scales. Compared with Method 3, it enhances image details while preserving noise resistance. The reconstructed image output by the encoder-decoder optimized in this way has higher quality, which is of great significance for subsequent machine tasks such as object detection.

[0125] Based on the above description, the distortion of the feature map of the reconstructed image relative to the feature map of the standard image can be obtained. After obtaining the distortion, the method of this embodiment further includes evaluating the quality of the reconstructed image based on the distortion to obtain an evaluation result; and then optimizing the encoder and / or decoder based on the evaluation result, wherein the encoder and / or decoder are used to output the reconstructed image. For example, during the training process of the encoder and / or decoder, it is naturally desirable for the reconstructed image output by the encoder and / or decoder to be as close as possible to the standard image. The distortion of the reconstructed image represents the distortion between the reconstructed image and the standard image. Therefore, the parameters of the encoder and / or decoder can be updated according to the distortion to minimize the distortion of the output reconstructed image, thereby achieving the purpose of optimizing the encoder and / or decoder.

[0126] Figure 7 This illustration shows an application scenario of the image processing method according to an embodiment of this application in the optimization of the encoder and / or decoder. For example... Figure 7 As shown in Figure (a), the image processing method of this application embodiment can optimize the encoder and decoder separately, or optimize the encoder and decoder when they are designed as a single unit. Since the single-layer convolutional neural network used in the image processing method of this application embodiment comes from an existing pre-trained model rather than being manually designed, it is compatible with existing AI models. Furthermore, the single-layer convolutional neural network is the first convolutional layer of the pre-trained model, extracting the low-level common features of the image, enabling the optimized encoder and decoder to simultaneously adapt to various machine vision tasks, such as… Figure 7 (a) includes tasks such as object detection, semantic segmentation, and traffic light / lane detection. For example... Figure 7As shown in Figure (b), the image processing method of this application embodiment can optimize the encoder and decoder. Then, with the encoder fixed, the decoder is jointly optimized using existing machine vision task-oriented evaluation metrics in conjunction with the specific machine vision task. The resulting encoder can adapt to various machine vision tasks, improving task generalization ability. Furthermore, by combining the decoder with a specific machine vision task, the output reconstructed image can better match the application scenario of the specific machine vision task. In addition, as... Figure 7 As shown in Figure (c), the image processing method of this application embodiment can optimize the encoder and decoder, then fix the encoder, and then use existing evaluation metrics for machine vision tasks in conjunction with the specific machine vision task to optimize the backbone network of the decoder without optimizing the head network. The backbone network is used to perform feature extraction and is part of the decoder network. The head network makes further predictions based on the features extracted by the backbone network and is also part of the decoder network. In this way, self-supervised learning can be used in the training process of the decoder without the need for labeled data, because labeled data is needed in the process of making predictions in the head network.

[0127] Furthermore, the image processing method of this application embodiment can be used to guide the optimization of the ISP processing method. ISP processing includes a series of steps such as depigmentation, color transformation, white balance, denoising, tone mapping, and gamma correction. Due to the lack of image quality evaluation indicators for machine vision, parameter tuning generally requires an end-to-end approach that combines ISP and specific tasks. That is, the output accuracy of the specific task guides ISP parameter tuning. However, the specific task requires labeled data, which limits the application of this parameter tuning method. In contrast, the image processing method of this application embodiment is decoupled from the specific task and can directly guide ISP parameter tuning without executing the specific task, thus simplifying the ISP parameter tuning process.

[0128] Table 1 below shows the ranking of the performance of the reconstructed images output by the optimized encoder and / or decoder in machine vision tasks and human vision, based on the image processing method proposed in the embodiments of this application and the methods of the prior art.

[0129] Table 1

[0130] Object detection ranking Semantic segmentation ranking Ranking of human vision Overall Ranking wfMSSSIM 2 2 2 1 wfMSE-w1 1 4 1 2 wfMSE-w0 3 3 3 3 MSSSIM 4 1 4 4 MSE 5 6 5 5 DISTS 6 5 6 6

[0131] Where wfMSE-w0 indicates no weighting, and wfMSE-w1 indicates a weighting coefficient of 1. As can be seen from Table 1, the encoder and / or decoder optimized by the image processing method according to the embodiments of this application have better performance than the prior art in both machine vision tasks and human vision.

[0132] The image processing method of the present application embodiments has been described in detail above with reference to the accompanying drawings. The image processing apparatus of the present application embodiments will now be described with reference to the accompanying drawings. It should be understood that the image processing apparatus described below can perform the various steps of the image processing method of the present application embodiments, and repeated descriptions will be appropriately omitted in the following description of the image processing apparatus of the present application embodiments.

[0133] Figure 8 This is a schematic block diagram of an image processing apparatus according to an embodiment of this application. The image processing apparatus can be a terminal or a chip within the terminal, such as... Figure 8 As shown, it includes an acquisition unit 801 and a processing unit 802, which will be briefly introduced below.

[0134] The acquisition unit 801 is used to acquire the reconstructed image and a standard image of the reconstructed image.

[0135] The processing unit 802 is used to input the reconstructed image and the standard image into a single-layer convolutional neural network to obtain the feature map of the reconstructed image and the feature map of the standard image. The single-layer convolutional neural network is the first convolutional layer of the pre-trained model.

[0136] The processing unit 802 is also used to obtain the distortion of the feature map of the reconstructed image relative to the feature map of the standard image.

[0137] In some implementations, a single-layer convolutional neural network includes multiple convolutional kernels. The first feature map of the reconstructed image and the first feature map of the standard image have a first weight, and the first feature map is obtained by the first convolutional kernel. The second feature map of the reconstructed image and the second feature map of the standard image have a second weight, and the second feature map is obtained by the second convolutional kernel. The first and second convolutional kernels belong to multiple convolutional kernels.

[0138] In some implementations, the weights are determined by normalization parameters, which are the normalization parameters of the pre-trained model.

[0139] In some implementations, the processing unit 802 is specifically used to: calculate the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0140]

[0141] wfMSE represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, where x is the standard image, y is the reconstructed image, and f is the standard image. i () represents the convolution operation, i is the i-th convolution kernel, and w i , where C is the number of feature map channels in the reconstructed image or the standard image, H is the height of the standard image or the reconstructed image, and W is the width of the standard image or the reconstructed image.

[0142] In some implementations, w i =1, or or γ i σ is the scaling factor for the pre-trained model. i These are the normalization coefficients of the pre-trained model.

[0143] In some implementations, the processing unit 802 is further configured to: calculate the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0144]

[0145] wfSSIM represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, μ is the mean, σ is the covariance, C1 is a constant, and C2 is a constant.

[0146] In some implementations, the processing unit 802 is further configured to: calculate the distortion of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula:

[0147]

[0148] wfMSSSIM represents the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, and C3 is a constant.

[0149] In some implementations, a single-layer convolutional neural network includes multiple convolutional kernels. A first convolutional kernel is used to obtain a first feature of the reconstructed image and a first feature of the standard image. The coefficients of the first convolutional kernel have a first weight. A second convolutional kernel is used to obtain a second feature of the reconstructed image and a second feature of the standard image. The coefficients of the second convolutional kernel have a second weight. The first and second convolutional kernels belong to multiple convolutional kernels.

[0150] In some implementations, the processing unit 802 is specifically used to: calculate the distortion of the feature map of the reconstructed image according to the following formula:

[0151]

[0152] wfMSE represents the distortion of the feature map of the reconstructed image, x represents the standard image, y represents the reconstructed image, and g represents the distortion of the feature map. i ()=w i ×f i ()=fw i (), f i () represents the convolution operation, i is the i-th convolution kernel, and w i, where C is the number of feature map channels in the reconstructed image or the standard image, H is the height of the standard image or the reconstructed image, and W is the width of the standard image or the reconstructed image.

[0153] In some implementations, the processing unit 802 is further configured to: evaluate the quality of the reconstructed image based on the distortion of the feature map of the reconstructed image relative to the feature map of the standard image, to obtain an evaluation result; and optimize the encoder and / or decoder based on the evaluation result, wherein the encoder and / or decoder are used to output the reconstructed image.

[0154] In some implementations, single-layer convolutional neural networks do not perform pooling and downsampling operations on the reconstructed and standard images.

[0155] It should be understood that Figure 8 The image processing apparatus shown can be used to implement the image processing method 400 described above, wherein the acquisition unit 801 is used to implement step 401, and the processing unit 802 is used to implement steps 402 and 403. Figure 8 The image processing apparatus shown can also be used to achieve Figures 5 to 7 The image processing method described above, specifically the steps, can be referred to the above. Figures 5 to 7 For the sake of brevity, the embodiments of this application will not be described in detail here.

[0156] It should be understood that the image processing device 800 in this embodiment can be implemented by software, for example, by a computer program or instructions having the above-mentioned functions. The corresponding computer program or instructions can be stored in the internal memory of the terminal, and the processor reads the corresponding computer program or instructions from the memory to implement the above functions. Alternatively, the image processing device 800 in this embodiment can also be implemented by hardware. The processing unit 802 is a processor (such as an NPU, GPU, or processor in a system-on-a-chip), and the acquisition unit 801 is a data interface. Alternatively, the image processing device 800 in this embodiment can also be implemented by a combination of a processor and software units. Specifically, the acquisition unit 801 can be an interface circuit of the processor, or an encoder and / or decoder, etc. For example, the encoder and / or decoder sends the output reconstructed image to the processor interface circuit.

[0157] Figure 9 This is a schematic diagram of the structure of the image processing apparatus 900 according to an embodiment of this application. Figure 9 The illustrated device 900 includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, processor 902, and communication interface 903 are interconnected via the bus 904.

[0158] It should be understood that Figure 8The acquisition unit 801 in the device 900 can be equivalent to the communication interface 903 in the device 900, and the processing unit 1002 and the processing unit 802 can be equivalent to the processor 902 in the device 900. The following is a detailed description of each unit in the device 900.

[0159] The memory 901 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 901 may store a program, and when the program stored in the memory 901 is executed by the processor 902, the processor 902 is used to execute the various steps of the image processing method of the embodiments of this application.

[0160] Specifically, processor 902 can be used to execute Figure 4 Steps 402 and 403 in the method shown. Additionally, processor 902 can also execute... Figures 5 to 7 The process is shown.

[0161] When the processor 902 executes steps 402 and 403, the processor 902 can obtain the reconstructed image and its corresponding standard image output by the encoder and / or decoder through the communication interface 903, and process the obtained reconstructed image and its corresponding standard image.

[0162] The processor 902 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the image processing method of the embodiments of this application.

[0163] The processor 902 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the method of this application can be accomplished through integrated logic circuits in the processor 902 or through software instructions.

[0164] The processor 902 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 901. The processor 902 reads the information in memory 901 and, in conjunction with its hardware, performs the functions required by the units included in the device, or executes the image processing method of the method embodiments of this application.

[0165] The communication interface 903 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 900 and other devices or communication networks. For example, the reconstructed image and its corresponding standard image can be acquired through the communication interface 903.

[0166] Bus 904 may include a pathway for transmitting information between various components of device 900 (e.g., memory 901, processor 902, communication interface 903).

[0167] This application also provides a computer-readable medium storing program code, which, when run on a computer, causes the computer to perform the above-described actions. Figures 4 to 7 The method described.

[0168] This application also provides a chip, including: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the above-mentioned... Figures 4 to 7 The method described.

[0169] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0174] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of image processing, characterized by, The method comprises: obtaining a reconstructed image and a standard image of the reconstructed image, the reconstructed image being an image reconstructed according to a first image, the first image being an image obtained by compressing the standard image; inputting the reconstructed image and the standard image into a single-layer convolutional neural network to obtain a feature map of the reconstructed image and a feature map of the standard image, parameters of the single-layer convolutional neural network being obtained from parameters of a first convolutional layer of a pre-trained model, the pre-trained model being a pre-trained convolutional neural network; obtaining a distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image.

2. The method of claim 1, wherein, The single-layer convolutional neural network comprises a plurality of convolutional kernels, a first feature map of the reconstructed image and a first feature map of the standard image have a first weight, the first feature map of the reconstructed image and the first feature map of the standard image are obtained by a first convolutional kernel, a second feature map of the reconstructed image and a second feature map of the standard image have a second weight, the second feature map of the reconstructed image and the second feature map of the standard image are obtained by a second convolutional kernel, the first convolutional kernel and the second convolutional kernel belong to the plurality of convolutional kernels.

3. The method of claim 2, wherein, The first weight and the second weight are determined by a normalization parameter, and the normalization parameter is a normalization parameter of the pre-trained model.

4. The method of any one of claims 1 to 3, wherein, The method further comprises: The distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image is calculated according to the following formula: The wfMSE is a distortion degree of a feature map of the reconstructed image relative to a feature map of the standard image, the x is the standard image, the y is the reconstructed image, the f i () is a convolution operation, the i is an i-th convolution kernel, the w i is a weight coefficient, the C is a feature map channel number of the reconstructed image or a feature map channel number of the standard image, the H is a height of the standard image or the reconstructed image, and the W is a width of the standard image or the reconstructed image.

5. The method of claim 4, wherein, w i = 1, or the or the The γ i is a scaling coefficient of the pre-trained model, and the σ i is a normalization coefficient of the pre-trained model.

6. The method of any one of claims 1 to 3, wherein, The method further comprises: The distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image is calculated according to the following formula: The wfSSIM is a distortion degree of a feature map of the reconstructed image relative to a feature map of the standard image, the x is a standard image, the y is a reconstructed image, and the f i () is a convolution operation, the i is an i-th convolution kernel, and the w i is a weight coefficient, the C is a feature map channel number of the reconstructed image or a feature map channel number of the standard image, the μ is a mean value, the σ is a covariance, the C1 is a constant, and the C2 is a constant.

7. The method of any one of claims 1 to 3, wherein, The method further comprises: The distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image is calculated according to the following formula: The wfMSSSIM is a distortion degree of a feature map of the reconstructed image relative to a feature map of the standard image, the x is a standard image, the y is a reconstructed image, and the f i () is a convolution operation, the i is an i-th convolution kernel, and the w i is a weight coefficient, the C is a feature map channel number of the reconstructed image or a feature map channel number of the standard image, the μ is a mean value, the σ is a covariance, the C1 is a constant, the C2 is a constant, and the C3 is a constant.

8. The method of claim 1, wherein, The single-layer convolutional neural network comprises a plurality of convolutional kernels, a first convolutional kernel is used to obtain a first feature of the reconstructed image and a first feature of the standard image, coefficients of the first convolutional kernel have a first weight, a second convolutional kernel is used to obtain a second feature of the reconstructed image and a second feature of the standard image, coefficients of the second convolutional kernel have a second weight, the first convolutional kernel and the second convolutional kernel belong to the plurality of convolutional kernels.

9. The method of claim 8, wherein, The method further comprises: The distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image is calculated according to the following formula: The wfMSE is a distortion degree of a feature map of the reconstructed image, the x is the standard image, the y is the reconstructed image, the g i ()=w i ×f i ()=fw i (), the f i () is a convolution operation, the i is an i-th convolution kernel, the w i is a weight coefficient, the C is a feature map channel number of the reconstructed image or a feature map channel number of the standard image, the H is a height of the standard image or the reconstructed image, and the W is a width of the standard image or the reconstructed image.

10. The method of claim 1, 2, 3, 8 or 9, wherein, The method further comprises: evaluating the quality of the reconstructed image according to the distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image to obtain an evaluation result; optimizing an encoder and / or a decoder according to the evaluation result, the encoder and / or the decoder being used to output the reconstructed image.

11. The method of claim 1, 2, 3, 8 or 9, wherein, The feature map of the reconstructed image and the feature map of the standard image are full-resolution images.

12. An apparatus for image processing, characterized by The method comprises: an obtaining unit, configured to obtain a reconstructed image and a standard image of the reconstructed image, the reconstructed image being an image reconstructed according to a first image, the first image being an image obtained by compressing the standard image; The processing unit is configured to input the reconstructed image and the standard image into a single-layer convolutional neural network to obtain a feature map of the reconstructed image and a feature map of the standard image, wherein parameters of the single-layer convolutional neural network are derived from parameters of a first convolutional layer of a pre-trained model. The processing unit is further configured to obtain a distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image.

13. The apparatus of claim 12, wherein, The single-layer convolutional neural network comprises a plurality of convolution kernels, the first feature map of the reconstructed image and the first feature map of the standard image are obtained by a first convolution kernel, the second feature map of the reconstructed image and the second feature map of the standard image are obtained by a second convolution kernel, and the first convolution kernel and the second convolution kernel belong to the plurality of convolution kernels.

14. The apparatus of claim 13, wherein, The first weight and the second weight are determined by a normalization parameter, and the normalization parameter is a normalization parameter of the pre-trained model.

15. The apparatus of any one of claims 12 to 14, wherein, The processing unit is specifically configured to: calculate the distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula: The wfMSE is a distortion degree of a feature map of the reconstructed image relative to a feature map of the standard image, the x is the standard image, the y is the reconstructed image, and the f i () is a convolution operation, the i is an i-th convolution kernel, and the w i is a weight coefficient, the C is a feature map channel number of the reconstructed image or a feature map channel number of the standard image, the H is a height of the standard image or the reconstructed image, and the W is a width of the standard image or the reconstructed image.

16. The apparatus of claim 15, wherein, w i = 1, or the or the The γ i is a scaling coefficient of the pre-trained model, and the σ i is a normalization coefficient of the pre-trained model.

17. The apparatus of any one of claims 12 to 14, wherein, The processing unit is further configured to: calculate the distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula: The wfSSIM is a distortion degree of a feature map of the reconstructed image relative to a feature map of the standard image, the x is a standard image, the y is a reconstructed image, and the f i () is a convolution operation, the i is an i-th convolution kernel, and the w i is a weight coefficient, the C is a feature map channel number of the reconstructed image or a feature map channel number of the standard image, the μ is a mean value, the σ is a covariance, the C1 is a constant, and the C2 is a constant.

18. The apparatus of any one of claims 12 to 14, wherein, The processing unit is further configured to: calculate the distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image according to the following formula: The wfMSSSIM is a distortion degree of a feature map of the reconstructed image relative to a feature map of the standard image, the x is a standard image, the y is a reconstructed image, and the f i () is a convolution operation, the i is an i-th convolution kernel, and the w i is a weight coefficient, the C is a feature map channel number of the reconstructed image or a feature map channel number of the standard image, the μ is a mean value, the σ is a covariance, the C1 is a constant, the C2 is a constant, and the C3 is a constant.

19. The apparatus of claim 12, wherein, The single-layer convolutional neural network comprises a plurality of convolution kernels, a first convolution kernel is used to obtain a first feature of the reconstructed image and a first feature of the standard image, a coefficient of the first convolution kernel has a first weight, a second convolution kernel is used to obtain a second feature of the reconstructed image and a second feature of the standard image, a coefficient of the second convolution kernel has a second weight, and the first convolution kernel and the second convolution kernel belong to the plurality of convolution kernels.

20. The apparatus of claim 19, wherein, The processing unit is specifically configured to: calculate the distortion degree of the feature map of the reconstructed image according to the following formula: The wfMSE is a distortion degree of a feature map of the reconstructed image, the x is the standard image, the y is the reconstructed image, the g i ()=w i ×f i ()=fw i (), the f i () is a convolution operation, the i is an i-th convolution kernel, the w i is a weight coefficient, the C is a feature map channel number of the reconstructed image or a feature map channel number of the standard image, the H is a height of the standard image or the reconstructed image, and the W is a width of the standard image or the reconstructed image.

21. The apparatus of claims 12, 13, 14, 19, or 20, wherein, The processing unit is further configured to: evaluate a quality of the reconstructed image according to the distortion degree of the feature map of the reconstructed image relative to the feature map of the standard image to obtain an evaluation result; optimize an encoder and / or a decoder according to the evaluation result, wherein the encoder and / or the decoder are used to output the reconstructed image.

22. The apparatus of claims 12, 13, 14, 19, or 20, wherein, The feature map of the reconstructed image and the feature map of the standard image are full-resolution images.

23. A computer readable medium characterized by The computer readable medium stores program codes for execution by a device, and the program codes comprise codes for executing the method in any one of claims 1 to 11.

24. A chip, characterized by The chip comprises a processor and a data interface, the processor reads instructions stored on a memory through the data interface, and executes the method in any one of claims 1 to 11.

Images