Image processing method and image processing apparatus

Abstract

The application provides an image processing method and an image processing device for obtaining the similarity of two images to evaluate the quality of the images, thereby guiding the optimization of an encoder and / or a decoder. The method comprises: obtaining a first image and a second image; inputting the first image and the second image into a single-layer convolutional neural network respectively to obtain feature maps of a first scale of the first image and the second image; performing M-1 times of down-sampling on the feature maps of the first scale of the first image and the feature maps of the second scale of the second image respectively to obtain feature maps of M-1 scales of the first image and feature maps of M-1 scales of the second image; and determining the similarity of the first image relative to the second image according to the feature maps of the first scale of the first image, the feature maps of the first scale of the second image, the feature maps of M-1 scales of the first image, the feature maps of M-1 scales of the second image, and a weight matrix related to M scales and the number C of feature maps.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image processing method and an image processing apparatus. Background Technology

[0002] Cameras, characterized by high resolution, non-contact operation, ease of use, and low cost, have wide applications in environmental perception. For example, in the field of autonomous driving, an increasing number of cameras are being installed on vehicles, enabling blind-spot-free coverage—allowing users to observe any location within the vehicle—and facilitating machine vision artificial intelligence (AI) processing, such as object recognition, semantic segmentation, traffic light detection, and lane detection, thus contributing to the safety of autonomous driving. However, as camera resolution, frame rate, and sampling depth continue to improve, the output video demands increasingly higher transmission bandwidth.

[0003] To alleviate pressure on transmission networks, compression methods are generally used to reduce bandwidth requirements, enabling the transmission of high-definition video without upgrading existing network bandwidth. To meet current bandwidth demands, encoders can compress images or videos output from cameras at high rates. This often employs lossy image or video compression techniques, inevitably leading to image or video quality degradation. Severe image or video degradation can affect the accuracy of subsequent tasks such as object detection and semantic segmentation. The compressed image can then be reconstructed by a decoder to obtain a reconstructed image. The quality of the reconstructed image is crucial for subsequent tasks; therefore, high-quality reconstructed images are paramount. However, due to the degradation inherent in compressed images, the quality of the reconstructed image cannot be guaranteed.

[0004] Therefore, there is an urgent need for a method to evaluate the quality of reconstructed images and provide assurance for subsequent task processing. Summary of the Invention

[0005] This application provides an image processing method and an image processing apparatus for obtaining the similarity between a reconstructed image and the original image to evaluate the quality of the reconstructed image.

[0006] In a first aspect, an image processing method is provided, comprising: acquiring a first image and a second image; inputting the first image and the second image into a single-layer convolutional neural network respectively to obtain feature maps of the first image at a first scale and feature maps of the second image at a first scale, wherein the single-layer convolutional neural network includes C convolutional kernels, and the number of feature maps of the first image at a first scale and the number of feature maps of the second image at a first scale are both C; performing M-1 downsampling operations on the feature maps of the first image at a first scale and the feature maps of the second image at a second scale respectively to obtain feature maps of the first image at M-1 scales and feature maps of the second image at M-1 scales, wherein C and M are both positive integers greater than 1; determining the similarity between the first image and the second image based on the feature maps of the first image at a first scale, the feature maps of the second image at a first scale, the feature maps of the first image at M-1 scales, the feature maps of the second image at M-1 scales, and a weight matrix related to the M scales and the number of feature maps C, wherein the weight matrix includes weight coefficients of different feature maps at different scales.

[0007] The first image and the second image are two different images. When the first image is the original image, the second image can be the reconstructed image; when the first image is the reconstructed image, the second image can be the original image.

[0008] The parameters of a single-layer convolutional neural network (CNN) are derived from the parameters of the first convolutional layer of a pre-trained model, which is a pre-trained CNN model. A single-layer CNN can include C convolutional kernels, with different kernels used to extract different features, resulting in different feature maps. These different feature maps can be called first-scale feature maps. When an image processing device inputs a first image into a single-layer CNN, it obtains the first-scale feature map of the first image. When the image processing device inputs a second image into a single-layer CNN, it obtains the first-scale feature map of the second image.

[0009] The image processing device can perform the first downsampling of the first scale feature map of the first image in M-1 downsampling steps to obtain the second scale feature map of the first image, perform the second downsampling of the second scale feature map of the first image in M-1 downsampling steps to obtain the third scale feature map of the first image, and so on, to obtain the Mth scale feature map of the first image.

[0010] It should be understood that the image processing device downsamples the feature map of the first image at the first scale M-1 times to obtain feature maps of the first image at M-1 scales. Similarly, the image processing device can obtain feature maps of the second image at M-1 scales.

[0011] It should also be understood that the number of feature maps at the first scale of the first image is C. After the first downsampling of the feature maps at the first scale of the first image, the number of feature maps at the second scale of the first image is obtained, which is also C. Similarly, the number of feature maps at other scales of the first image is also C. That is, the number of feature maps at each scale in the M-1 scales of the first image is C. Similarly, the number of feature maps at each scale in the M-1 scales of the second image is also C.

[0012] It should be noted that downsampling the first feature map in the first scale feature map of the first image can yield the first feature map in the second scale feature map of the first image, and so on. This process will not be repeated here.

[0013] The image processing method provided in this application utilizes a single-layer convolutional neural network to obtain feature maps at the first scale. This requires only a single-layer convolution calculation, making it computationally simple. Extracting general features at the lower level through a single-layer convolution helps ensure the image's generalization ability across multiple tasks. The high resolution of the first-scale feature map ensures image quality. Furthermore, multiple downsampling operations on the high-resolution feature map yield more image feature information. Additionally, a weight matrix is ​​used to assign weights to different feature maps at different scales, flexibly adjusting the weight values ​​in the image feature and scale dimensions to optimize machine vision performance. This achieves efficient image encoding and compression performance for machine vision, and facilitates evaluating the quality of the first image using the similarity between the first image and the second image.

[0014] In conjunction with the first aspect, in certain implementations of the first aspect, before determining the similarity between the first image and the second image based on the feature map at a first scale of the first image, the feature map at a first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and a weight matrix related to the number of M scales and feature maps C, the method further includes: determining the weight coefficients of the feature maps at the first scale based on the peak signal-to-noise ratio of the feature maps at the first scale, wherein the feature maps at the first scale include the feature maps at the first scale of the first image and the feature maps at the first scale of the second image; determining the similarity between the first image and the second image based on the feature maps at the first scale of the first image, the feature maps at the first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and a weight matrix related to the number of M scales and feature maps C, includes: determining the similarity based on the weight coefficients of the feature maps at the first scale, the feature maps at the first scale of the first image, the feature maps at the first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and the weight matrix.

[0015] If the number of feature maps at the first scale of both the first and second images is C, then the number of peak signal-to-noise ratios (PSNRs) of the feature maps at the first scale is also C. The image processing device can obtain the weight coefficient of each feature map at the first scale based on the PSNR. The image processing device can determine the similarity based on the weight coefficients of the feature maps at the first scale, the feature maps at the first scale of the first image, the feature maps at the first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and the weight matrix.

[0016] The image processing method provided in this application can make the peak signal-to-noise ratio of the feature map at the first scale consistent with the color, brightness and other features of the first and second images. Based on the peak signal-to-noise ratio of the feature map at the first scale, the weight coefficient of each feature map in the feature map at the first scale is obtained. By using the weight coefficient to weight each feature map, the detail effect of the feature map can be increased, the performance of human vision and machine vision can be improved, and the needs of both human vision and machine vision can be taken into account.

[0017] In conjunction with the first aspect, in certain implementations of the first aspect, the aforementioned similarity is obtained using the following formula:

[0018]

[0019]

[0020]

[0021]

[0022] in, For the aforementioned similarity, For the first image, For the second image, For convolution operations in a single-layer convolutional neural network, The first layer in a single-layer convolutional neural network One convolutional kernel, , For the first One scale, , These are the weight coefficients of the feature map at the first scale. For the first The first scale Structural similarity of feature maps The weight matrix is ​​the first... Line number The weight coefficients corresponding to the columns, The first feature map of the first scale of the first image Each feature map The first feature map of the second image at the first scale Each feature map For the first scale Peak signal-to-noise ratio of each feature map, It is a constant. For the first image The first scale The variance of each feature map For the second image The first scale The variance of each feature map For the first image The first scale The first feature map and the second image's first feature map The first scale The covariance of each feature map The maximum pixel value in the feature map of the first and second images at the first scale. The minimum pixel value in the feature maps of the first and second images at the first scale. The height of the first or second image. The width of the first or second image.

[0023] The first layer in a single-layer convolutional neural network The nth convolutional kernel, also known as the nth convolutional kernel. This application does not limit the number of channels.

[0024] In conjunction with the first aspect, in some implementations of the first aspect, the weight matrix is ​​a C-row, M-column matrix, where the weight coefficients of each row in the weight matrix are used to represent the weight coefficients of the feature map at different scales, and the sum of the weight coefficients of each row in the weight matrix is ​​1.

[0025] The weight coefficients in each row of the weight matrix are the weight coefficients of the feature map corresponding to the same channel (or the feature map corresponding to the same convolution kernel) at different scales, and the sum of the weight coefficients of the feature map corresponding to the same channel at different scales is 1.

[0026] In conjunction with the first aspect, in some implementations of the first aspect, each weight coefficient in each row of the above weight matrix is ​​the same.

[0027] The image processing method provided in this application, with the weight coefficients of each row in the weight matrix, is a very simple implementation method. It is easy to calculate and helps to improve the efficiency of similarity calculation.

[0028] In conjunction with the first aspect, in some implementations of the first aspect, M=5, and the weight coefficients of each row in the weight matrix are the same, and are [0.2 0.2 0.2 0.2 0.2].

[0029] In conjunction with the first aspect, in some implementations of the first aspect, M=5, and the weight coefficients of each row in the weight matrix are the same, and are [0.0448 0.2856 0.3001 0.2363 0.1333].

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the aforementioned C convolutional kernels are divided into high-frequency feature groups, mid-frequency feature groups, and low-frequency feature groups according to their feature frequencies. The M feature maps at different scales are divided into low-scale feature maps, mid-scale feature maps, and high-scale feature maps according to their resolutions. The weight coefficient distribution of the feature maps corresponding to the high-frequency feature groups, mid-frequency feature groups, and low-frequency feature groups at different scales in the weight matrix satisfies the following conditions: the non-zero weight coefficients of the feature maps corresponding to the high-frequency feature groups are distributed in the low-scale feature maps of the first and second images, and the weight coefficients of the high-scale feature maps are 0; the non-zero weight coefficients of the feature maps corresponding to the low-frequency feature groups are distributed in the low-scale, mid-scale, and high-scale feature maps; the distribution of non-zero weight coefficients of the feature maps corresponding to the mid-frequency feature groups is more than that of the feature maps corresponding to the high-frequency feature groups, but less than that of the feature maps corresponding to the low-frequency feature groups.

[0031] Image processing equipment can sort C convolutional kernels according to their feature frequencies from high to low or from low to high, and then group the sorted convolutional kernels into high-frequency feature groups, mid-frequency feature groups, and low-frequency feature groups.

[0032] The feature maps corresponding to high-frequency feature groups can be understood as the feature maps corresponding to each convolutional kernel in the high-frequency feature group, with each kernel's feature map including M scales. Similarly, the feature maps corresponding to mid-frequency feature groups can be understood as the feature maps corresponding to each convolutional kernel in the mid-frequency feature group, with each kernel's feature map including M scales.

[0033] Image processing equipment can sort M feature maps at different scales according to their resolution from high to low or from low to high, and then divide the sorted M feature maps into low-scale feature maps, medium-scale feature maps, and high-scale feature maps.

[0034] The image processing method provided in this application allows the weight matrix to assign weight coefficients to different features at different scales, thereby leveraging the different roles of each feature. For features corresponding to high-frequency feature groups, higher weights are assigned to low-scale feature maps. For features corresponding to low-frequency feature groups, the ability to extract contour information at high scales is considered, and weights are evenly distributed across different scales. For features corresponding to mid-frequency feature groups, the weight matrix is ​​positioned between high-frequency and low-frequency feature groups, taking into account both high-frequency details and contour information, which helps improve machine vision performance.

[0035] In conjunction with the first aspect, in some implementations of the first aspect, each non-zero weight coefficient of the feature map corresponding to the high-frequency feature group is the same, each non-zero weight coefficient of the feature map corresponding to the low-frequency feature group is the same, and each non-zero weight coefficient of the feature map corresponding to the mid-frequency feature group is the same.

[0036] Each convolutional kernel in the high-frequency, low-frequency, and mid-frequency feature groups corresponds to a feature map with M scales. The non-zero weight coefficients of the feature map corresponding to the high-frequency feature group are distributed across the low-scale feature maps within the M scales, and each weight coefficient in the low-scale feature map is identical. The non-zero weight coefficients of the feature map corresponding to the low-frequency feature group are distributed across the low-scale, mid-scale, and high-scale feature maps (i.e., the M scales), and each weight coefficient in the M scale feature maps is identical. The same applies to the mid-frequency feature group, and will not be elaborated further here.

[0037] In conjunction with the first aspect, in some implementations of the first aspect, M=5, the weight coefficient of the feature map corresponding to the high-frequency feature group is [1 0 0 0 0], the weight coefficient of the feature map corresponding to the low-frequency feature group is [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5], and the weight coefficient of the feature map corresponding to the mid-frequency feature is [1 / 3 1 / 3 1 / 3 0 0].

[0038] In [1 0 0 0 0], [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5] and [1 / 3 1 / 3 1 / 3 0 0], the weight coefficients in the first column are used to represent the weight coefficients of the low-scale feature map, the weight coefficients in the second and third columns are used to represent the weight coefficients of the medium-scale feature map, and the weight coefficients in the fourth and fifth columns are used to represent the weight coefficients of the high-scale feature map.

[0039] The convolution kernels corresponding to the high-frequency feature group are used to extract the texture features of the image, the convolution kernels corresponding to the low-frequency feature group can be used to extract the smoothness features of the image, and the convolution kernels corresponding to the mid-frequency feature group can be used to extract the edge features of the image.

[0040] The image processing method provided in this application allows the weight matrix to assign weight coefficients to different features at different scales, thereby leveraging the role of different features. For detail features, a weight of 1 is assigned to the high-resolution first-scale feature map. For smooth features, the ability to extract contour information at high scales is considered, and weights are evenly distributed across different scales. For edge features, the weight matrix is ​​positioned between detail features and smooth features, taking into account both high-frequency details and contour information, which helps to improve machine vision performance.

[0041] In conjunction with the first aspect, in some implementations of the first aspect, the aforementioned feature frequencies are determined by the magnitude of the convolutional kernel of the single-layer convolutional neural network and / or the scaling factor and normalization factor of the normalization layer of the single-layer convolutional neural network.

[0042] In conjunction with the first aspect, in some implementations of the first aspect, the first image is the input of the codec, the second image is the output of the codec, the codec is used to compress and reconstruct the first image to output the second image, and the similarity is used to optimize the codec.

[0043] The first image is the input to the codec, also known as the original image; the second image is the output of the codec, also known as the reconstructed image. The image processing device can use the above method to obtain the similarity between the original image and the reconstructed image, so as to optimize the codec.

[0044] The image processing method provided in this application can be used to obtain the similarity between the reconstructed image and the original image to evaluate the quality of the reconstructed image, thereby guiding the optimization of the codec.

[0045] In conjunction with the first aspect, in some implementations of the first aspect, the first image is the image of the third image after image signal processing, the third image is the input of the codec, the second image is the output of the codec, the codec is used to perform image signal processing, compression and reconstruction on the third image to output the second image, and the similarity is used to optimize the codec.

[0046] The first image is the image after image signal processing of the third image, also known as the original image; the third image is the input of the codec, and the second image is the output of the codec, also known as the reconstructed image; the codec is used to perform image signal processing, compression, and reconstruction on the third image to output the second image. The image processing device can use the above method to obtain the similarity between the original image and the reconstructed image, so as to optimize the codec.

[0047] Secondly, an image processing apparatus is provided, comprising: an acquisition module and a processing module. The acquisition module is used to: acquire a first image and a second image; the processing module is used to: input the first image and the second image into a single-layer convolutional neural network respectively to obtain feature maps of the first image at a first scale and feature maps of the second image at a first scale, wherein the single-layer convolutional neural network includes C convolutional kernels, and the number of feature maps of the first image at a first scale and the number of feature maps of the second image at a first scale are both C; perform M-1 downsampling operations on the feature maps of the first image at a first scale and the feature maps of the second image at a second scale respectively to obtain feature maps of the first image at M-1 scales and feature maps of the second image at M-1 scales, wherein C and M are both positive integers greater than 1; determine the similarity between the first image and the second image based on the feature maps of the first image at a first scale, the feature maps of the second image at a first scale, the feature maps of the first image at M-1 scales, the feature maps of the second image at M-1 scales, and a weight matrix related to the M scales and the number of feature maps C, wherein the weight matrix includes weight coefficients for different feature maps at different scales.

[0048] In conjunction with the second aspect, in some implementations of the second aspect, the above processing module is further configured to: determine the weight coefficients of the feature map at the first scale based on the peak signal-to-noise ratio of the feature map at the first scale, wherein the feature map at the first scale includes the feature map at the first scale of the first image and the feature map at the first scale of the second image; and determine the similarity based on the weight coefficients of the feature map at the first scale, the feature map at the first scale of the first image, the feature map at the first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and the weight matrix.

[0049] In conjunction with the second aspect, in some implementations of the second aspect, the above similarity is obtained by the following formula:

[0050]

[0051]

[0052]

[0053]

[0054] in, For the aforementioned similarity, For the first image, For the second image, For convolution operations in a single-layer convolutional neural network, The first layer in a single-layer convolutional neural network One convolutional kernel, , For the first One scale, , These are the weight coefficients of the feature map at the first scale. For the first The first scale Structural similarity of feature maps The weight matrix is ​​the first... Line number The weight coefficients corresponding to the columns, The first feature map of the first scale of the first image Each feature map The first feature map of the second image at the first scale Each feature map For the first scale Peak signal-to-noise ratio of each feature map, It is a constant. For the first image The first scale The variance of each feature map For the second image The first scale The variance of each feature map For the first image The first scale The first feature map and the second image's first feature map The first scale The covariance of each feature map The maximum pixel value in the feature map of the first and second images at the first scale. The minimum pixel value in the feature maps of the first and second images at the first scale. The height of the first or second image. The width of the first or second image.

[0055] In conjunction with the second aspect, in some implementations of the second aspect, the aforementioned weight matrix is ​​a C-row, M-column matrix, where the weight coefficients of each row in the weight matrix are used to represent the weight coefficients of the feature map at different scales, and the sum of the weight coefficients of each row in the weight matrix is ​​1.

[0056] In conjunction with the second aspect, in some implementations of the second aspect, each weight coefficient in each row of the above weight matrix is ​​the same.

[0057] In conjunction with the second aspect, in some implementations of the second aspect, M=5, and the weight coefficients of each row in the weight matrix are the same, and are [0.2 0.2 0.2 0.2 0.2].

[0058] In conjunction with the second aspect, in some implementations of the second aspect, M=5, and the weight coefficients in each row of the weight matrix are the same, and are [0.0448 0.2856 0.3001 0.2363 0.1333].

[0059] In conjunction with the second aspect, in some implementations of the second aspect, the aforementioned C convolutional kernels are divided into high-frequency feature groups, mid-frequency feature groups, and low-frequency feature groups according to their feature frequencies. The M feature maps at different scales are divided into low-scale feature maps, mid-scale feature maps, and high-scale feature maps according to their resolutions. The weight coefficient distribution of the feature maps corresponding to the high-frequency, mid-frequency, and low-frequency feature groups in the weight matrix at different scales satisfies the following conditions: the non-zero weight coefficients of the feature maps corresponding to the high-frequency feature groups are distributed in the low-scale feature maps of the first and second images, and the weight coefficients of the high-scale feature maps are 0; the non-zero weight coefficients of the feature maps corresponding to the low-frequency feature groups are distributed in the low-scale, mid-scale, and high-scale feature maps; the distribution of non-zero weight coefficients of the feature maps corresponding to the mid-frequency feature groups is more than that of the feature maps corresponding to the high-frequency feature groups, but less than that of the feature maps corresponding to the low-frequency feature groups.

[0060] In conjunction with the second aspect, in some implementations of the second aspect, each non-zero weight coefficient of the feature map corresponding to the high-frequency feature group is the same, each non-zero weight coefficient of the feature map corresponding to the low-frequency feature group is the same, and each non-zero weight coefficient of the feature map corresponding to the mid-frequency feature group is the same.

[0061] In conjunction with the second aspect, in some implementations of the second aspect, M=5, the weight coefficient of the feature map corresponding to the high-frequency feature group is [1 0 0 0 0], the weight coefficient of the feature map corresponding to the low-frequency feature group is [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5], and the weight coefficient of the feature map corresponding to the mid-frequency feature is [1 / 3 1 / 3 1 / 3 0 0].

[0062] In [1 0 0 0 0], [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5] and [1 / 3 1 / 3 1 / 3 0 0], the weight coefficients in the first column are used to represent the weight coefficients of the low-scale feature map, the weight coefficients in the second and third columns are used to represent the weight coefficients of the medium-scale feature map, and the weight coefficients in the fourth and fifth columns are used to represent the weight coefficients of the high-scale feature map.

[0063] In conjunction with the second aspect, in some implementations of the second aspect, the aforementioned feature frequencies are determined by the magnitude of the convolutional kernel of the single-layer convolutional neural network and / or the scaling factor and normalization factor of the normalization layer of the single-layer convolutional neural network.

[0064] In conjunction with the second aspect, in some implementations of the second aspect, the first image is the input of the codec, the second image is the output of the codec, the codec is used to compress and reconstruct the first image to output the second image, and the similarity is used to optimize the codec.

[0065] In conjunction with the second aspect, in some implementations of the second aspect, the first image is the image of the third image after image signal processing, the third image is the input of the codec, the second image is the output of the codec, the codec is used to perform image signal processing, compression and reconstruction on the third image to output the second image, and the similarity is used to optimize the codec.

[0066] Thirdly, an image processing apparatus is provided, comprising a processor and a memory for storing a computer program, the processor for calling and running the computer program from the memory, such that the apparatus performs the method in any of the possible implementations of the first aspect described above.

[0067] Optionally, the processor may be one or more, and the memory may be one or more.

[0068] Optionally, the memory may be integrated with the processor, or the memory may be separated from the processor.

[0069] Optionally, the device may also include a transmitter and a receiver, which may be separate or integrated together and referred to as a transceiver.

[0070] Fourthly, this application provides a processor, including: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive signals through the input circuit and transmit signals through the output circuit, causing the processor to execute the method in any of the possible implementations of the first aspect described above.

[0071] In specific implementation, the processor can be a chip, the input circuit can be input pins, the output circuit can be output pins, and the processing circuit can be transistors, gate circuits, flip-flops, and various logic circuits. The input signal received by the input circuit can be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit can be, for example, but not limited to, output to and transmitted by a transmitter. Furthermore, the input circuit and the output circuit can be the same circuit, which is used as both the input circuit and the output circuit at different times. This application does not limit the specific implementation of the processor and various circuits.

[0072] Fifthly, a computer-readable storage medium is provided that stores a computer program (also referred to as code or instructions) that, when run on a computer, causes the computer to perform the methods in any of the possible implementations of the first aspect described above.

[0073] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program (also referred to as code or instructions) that, when the computer program is run, causes a computer to perform the method in any of the possible implementations of the first aspect described above. Attached Figure Description

[0074] Figure 1 This is a schematic diagram illustrating the application scenarios applicable to the embodiments of this application;

[0075] Figure 2 A schematic block diagram of an image compression method;

[0076] Figure 3 A schematic block diagram illustrating another image compression method provided in an embodiment of this application;

[0077] Figure 4 A schematic flowchart illustrating an image processing method provided in an embodiment of this application;

[0078] Figure 5 A schematic block diagram illustrating downsampling provided for an embodiment of this application;

[0079] Figure 6 A schematic block diagram illustrating the calculation of similarity in an embodiment of this application;

[0080] Figure 7 This is a schematic diagram of a feature map grouping provided in an embodiment of this application;

[0081] Figure 8 A simulation diagram provided for an embodiment of this application;

[0082] Figure 9 Another simulation diagram provided for an embodiment of this application;

[0083] Figure 10 This is yet another simulation diagram provided for an embodiment of this application;

[0084] Figure 11 Another simulation diagram provided for an embodiment of this application;

[0085] Figure 12 This is a schematic block diagram of an image processing apparatus provided in an embodiment of this application;

[0086] Figure 13 This is a schematic block diagram of another image processing apparatus provided in the embodiments of this application. Detailed Implementation

[0087] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0088] To better understand the embodiments of this application, the following descriptions are provided:

[0089] First, the term "and / or" in the embodiments of this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. A and B can be singular or plural. Additionally, the character " / " in this document generally indicates an "or" relationship between the preceding and following related objects, but it may also indicate an "and / or" relationship. Please refer to the preceding and following text for a more detailed understanding.

[0090] Second, in the embodiments of this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0091] For example, Figure 1 A schematic diagram of a vehicle 100 is shown. (As shown) Figure 1 As shown, vehicle 100 includes camera 101, display device 102, and mobile data center (MDC) 103. Camera 101 and display device 102, as well as camera 101 and MDC 103, can communicate via a network. Vehicle 100 may include multiple cameras, including camera 101. This embodiment uses camera 101 as an example for illustration only, but does not limit the scope of the application.

[0092] It should be understood that some or all of the functions of vehicle 100 may be controlled by MDC 103. MDC 103 may include at least one processor capable of executing instructions stored in a computer-readable medium such as memory. In some embodiments, MDC 103 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed manner. The processor may be any conventional processor, such as a central processing unit (CPU). Alternatively, the processor may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0093] In addition to instructions, the memory can also store data such as road maps, route information, vehicle position, direction, speed, and other vehicle data, as well as other information. This information can be used by vehicle 100 and MDC 103 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0094] It should be understood that Figure 1 The structure of the vehicle should not be construed as a limitation on the embodiments of this application. Optionally, the vehicle 100 described above can be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, etc., and the embodiments of this application do not impose any particular limitation.

[0095] The aforementioned camera 101 can capture images or videos and send them to the display device 102 and MDC 103 via the network. After receiving the image or video, the display device 102 can display the image or video for user viewing. After receiving the image or video, the MDC 103 can perform processing such as target recognition, semantic segmentation, traffic light detection, and lane line detection on the image or video.

[0096] However, as camera resolution, frame rate, and sampling depth continue to improve, the output video demands increasingly higher transmission bandwidth. To alleviate the pressure on the transmission network, compression methods are generally used to reduce bandwidth requirements.

[0097] For example, a camera can output Bayer raw images or videos. Bayer raw images or videos have high precision and require high transmission bandwidth. For instance, a camera can output ultra-high definition (UHD) video with a frame rate of 30 frames per second (fps), a sampling depth of 16 bits, and a resolution of 4K (4096×2160). Transmitting this video requires up to 4 Gbps of bandwidth. .

[0098] Therefore, in order to alleviate the pressure on the transmission network, the camera can compress the images or videos before transmitting the original Bayer images or videos.

[0099] For example, Figure 2 A schematic block diagram of an image compression method is shown, such as... Figure 2 As shown, camera 101 outputs a Bayer raw image. This Bayer raw image is processed by image signal processing (ISP) module 201 to obtain a red-green-blue (RGB) image. The precision of the RGB image is much lower than that of the raw image output by the camera, thus reducing the bandwidth requirements for network transmission. This RGB image is then processed by encoder-decoder 202 to obtain a reconstructed image; that is, after encoding and compression, the reconstructed image is obtained through decoding. If this reconstructed image is transmitted to MDC 103, it can be used for tasks such as object detection, semantic segmentation, traffic light detection, and lane line detection. If this reconstructed image is transmitted to display device 102, it can be displayed for user viewing.

[0100] While compression technology can reduce the bandwidth requirements for image or video transmission, it inevitably leads to degradation in image or video quality. Furthermore, ISP processing also results in information loss in images or videos. This degradation in image or video quality reduces the accuracy of artificial intelligence tasks and lowers image or video clarity and resolution, affecting user viewing and ultimately impacting user experience. Therefore, the above... Figure 2 In this process, the quality of the reconstructed image obtained after decoding is crucial for subsequent tasks, so a high-quality reconstructed image is particularly important.

[0101] The quality requirements for reconstructed images in autonomous driving visual perception are to meet the needs of both human vision and machine vision.

[0102] For example, the quality requirements of autonomous driving visual perception for reconstructed images may include: 1) machine vision-oriented: autonomous driving identifies targets in images; 2) adapting to various machine vision tasks, including target detection, semantic segmentation, traffic light / lane line detection, etc.; 3) human vision: images acquired by autonomous driving are provided for human viewing.

[0103] However, the objective evaluation metrics commonly used in machine vision do not take into account the needs of human vision, and the objective evaluation metrics commonly used in human vision do not take into account the needs of machine vision. In other words, the existing evaluation metrics cannot take into account the needs of both human vision and machine vision.

[0104] For example, to some extent, the tasks mentioned above, such as object detection, semantic segmentation, and traffic light / lane detection, can be called machine vision tasks. This means that the reconstructed image is directly processed by the machine system, and therefore, the reconstructed image primarily meets the requirement of being quickly recognized and detected by the machine system. Various machine vision-oriented evaluation metrics are used to evaluate the quality of reconstructed images, including image classification metrics such as accuracy (Acc), object detection metrics such as mean average precision (mAP), semantic segmentation metrics such as mean intersection over union (mIoU), and lane detection metrics such as Acc.

[0105] Optimizing the CODEC using the aforementioned evaluation metrics for a single machine vision task can achieve good results. However, the optimized CODEC will be coupled with the specific machine vision task and cannot adapt to multiple task scenarios. For example, using image classification evaluation metrics such as Acc to optimize the CODEC, the reconstructed image output by the optimized CODEC has good results for image classification tasks, but still performs poorly for tasks such as object detection, semantic segmentation, and lane detection. It cannot achieve task generalization evaluation, and at the same time, it does not consider the impact on human vision.

[0106] In addition to the aforementioned machine vision-oriented evaluation metrics, there are also human vision-oriented evaluation metrics, including peak signal-to-noise ratio (PSNR), multiscale structural similarity (MSSSIM), learned perceptual image patch similarity (LPIPS), and depth image structural texture similarity (DISTS). The reconstructed image output by the CODEC optimized according to human vision-oriented evaluation metrics can better match the subjective perception of the human eye. For example, if the reconstructed image is directly displayed on a display screen inside a vehicle for the driver to view, the reconstructed image needs to have higher clarity and be easy for the human eye to view.

[0107] However, in reality, the complexity of the human eye cannot be satisfied by a single evaluation metric, and existing evaluation metrics for human vision all have their own shortcomings. Taking LPIPS as an example, evaluating the quality of reconstructed images using the LPIPS evaluation metric requires calculating all convolutional layers of the network, resulting in high computational complexity. In addition, because the network involves pooling and downsampling, image information is lost during the evaluation process, and only low-resolution feature maps are used in the evaluation process, making it difficult to produce accurate evaluation results. Inaccurate evaluation results also make it difficult to guarantee the optimization results of CODEC, thus making it difficult to guarantee the quality of the reconstructed image.

[0108] In summary, there are three problems with image reconstruction:

[0109] 1) ISP processing can cause damage to image or video information.

[0110] 2) The quality of the reconstructed image obtained by the decoder is crucial for subsequent tasks, so it is especially important that the reconstructed image has high quality.

[0111] 3) Existing image quality evaluation metrics only optimize CODEC for a single machine vision task. The optimized CODEC will be coupled with the corresponding single machine vision task and cannot be adapted to multiple machine vision tasks. At the same time, the impact on human vision is not considered.

[0112] In view of this, embodiments of this application provide an image processing method and an image processing apparatus for obtaining the similarity between a reconstructed image and the original image to evaluate the quality of the reconstructed image, thereby guiding the optimization of the encoder and / or decoder. At the same time, the reconstructed image can meet the needs of various machine vision tasks as well as the needs of human vision. In addition, it can also avoid information damage caused by ISP processing.

[0113] It should be noted that the technical solution proposed in this application provides an evaluation index that is "machine-oriented, task-decoupled, and takes into account human vision" to calculate the similarity between the reconstructed image and the original image, so as to evaluate the quality of the reconstructed image and thus guide the optimization of the encoder and / or decoder.

[0114] It should also be noted that the technical solutions proposed in this application can be applied to the Internet of Vehicles, such as vehicle-to-everything (V2X), long-term evolution-vehicle (LTE-V), and vehicle-to-vehicle (V2V). For example, they can be applied to vehicles with driving mobility functions, or other devices with driving mobility functions in vehicles. These other devices include, but are not limited to, vehicle terminals, vehicle controllers, vehicle modules, vehicle components, vehicle chips, vehicle units, vehicle radar, or vehicle cameras, and other sensors. Vehicles can implement the vehicle control method provided in the embodiments of this application through these vehicle terminals, vehicle controllers, vehicle modules, vehicle components, vehicle chips, vehicle units, vehicle radar, or vehicle cameras. Of course, the solutions in the embodiments of this application can also be used in other intelligent terminals with mobility control functions besides vehicles, or installed in other intelligent terminals with mobility control functions besides vehicles, or installed in components of such intelligent terminals. These intelligent terminals can be intelligent transportation equipment, smart home devices, robots, etc. Examples include, but are not limited to, smart terminals or controllers, chips, radar or cameras, and other sensors and components within smart terminals.

[0115] The technical solutions provided in this application can be applied to the above-mentioned... Figure 2 The image compression scene shown Figure 2 In the image compression scenario shown, the ISP module 201 and CODEC 202 are deployed independently, which can be geared towards machine vision to improve the detection accuracy of machine vision tasks. Meanwhile, the machine vision task model does not need to be modified (i.e., MDC103 does not need to be changed). Furthermore, the image reconstructed by the decoder can also be adapted to human vision, making it more visually friendly. The technical solutions provided in this application embodiment can be applied to the above-mentioned... Figure 2In addition to the image compression scenarios shown, it can also be applied to image compression scenarios where ISP and CODEC are deployed in a unified manner. Among them, the image compression scenario where ISP and CODEC are deployed in a unified manner can solve the above-mentioned problem 1).

[0116] For example, Figure 3 A schematic block diagram of another image compression method is shown. (e.g.) Figure 3 As shown, camera 101 outputs a Bayer original image, which is then processed by CODEC 301 to obtain a reconstructed image. CODEC 301 can perform image signal processing, compression, and reconstruction on the Bayer original image to obtain the reconstructed image. If the reconstructed image is transmitted to MDC 103, it can be used for tasks such as object detection, semantic segmentation, traffic light and lane line detection. If the reconstructed image is transmitted to display device 102, it can be displayed for user viewing.

[0117] The technical solutions provided in the embodiments of this application are applied to... Figure 3 The image compression scenario shown can be applied to machine vision, improving the detection accuracy of machine vision tasks. Simultaneously, the reconstructed image can also be optimized for human vision, making it more visually friendly. Furthermore, it avoids information loss caused by ISP processing. Moreover, this image compression scenario does not require a separate ISP module, simplifying deployment.

[0118] It should be noted that the image compression scenario involving the integrated deployment of ISP and CODEC is proposed in this application.

[0119] It should also be noted that the technical solutions provided in the embodiments of this application are applied to Figure 3 In the image compression scenario shown, an additional ISP is needed to process the Bayer original image to obtain the original image, and to calculate the similarity between the reconstructed image and the original image to evaluate the quality of the reconstructed image, thereby guiding the optimization of the encoder and / or decoder.

[0120] Figure 4 This is a schematic flowchart illustrating an image processing method 400 provided in an embodiment of this application. The method 400 can be executed by an image processing device. The method 400 can be applied to the above-mentioned... Figure 2 or Figure 3 The scenario shown is not limited to this one.

[0121] like Figure 4 As shown, the method 400 may include the following steps:

[0122] S401, Obtain the first image and the second image.

[0123] The first image and the second image are two different images. When the first image is the original image, the second image can be the reconstructed image; when the first image is the reconstructed image, the second image can be the original image.

[0124] For example, the first image can be the one described above. Figure 2 The second image can be an RGB image in the image. Figure 2 The reconstructed image in the image.

[0125] For example, the first image can be the one described above. Figure 2 The reconstructed image in the image, the second image can be Figure 2 RGB image in the image.

[0126] There are many possible ways for an image processing device to acquire the first image and the second image, and this application embodiment does not limit this.

[0127] In one possible implementation, as described above Figure 2 In the scenario shown, when the first image is the original image and the second image is the reconstructed image, the image processing device can obtain the first image from the ISP and the second image from the codec.

[0128] In another possible implementation, in the scenario shown in 3 above, when the first image is the original image and the second image is the reconstructed image, the image processing device needs to process the Bayer original image with the help of an additional ISP to obtain the first image. The image processing device can then obtain the second image from the codec.

[0129] S402. Input the first image and the second image into a single-layer convolutional neural network to obtain the first-scale feature map of the first image and the first-scale feature map of the second image. The single-layer convolutional neural network includes C convolutional kernels, and the number of the first-scale feature maps of the first image and the first-scale feature maps of the second image are both C.

[0130] The parameters of a single-layer convolutional neural network come from the parameters of the first convolutional layer of a pre-trained model. The pre-trained model is a convolutional neural network model that has been trained in advance, such as classification models like ResNet, AlexNet, VGGNet, and RegNet trained on the large-scale training set ImageNet.

[0131] A single-layer convolutional neural network can include C convolutional kernels, with different kernels used to extract different features, resulting in different feature maps. These different feature maps can be called first-scale feature maps.

[0132] It should be noted that the first-scale feature map here refers to the image obtained by processing the first and second images once (i.e., by processing with a single-layer convolutional neural network).

[0133] When an image processing device inputs a first image into a single-layer convolutional neural network, it obtains a feature map of the first image at a first scale. The number of feature maps at the first scale of the first image is C. When the image processing device inputs a second image into a single-layer convolutional neural network, it obtains a feature map of the second image at a first scale. The number of feature maps at the first scale of the second image is also C. There is a one-to-one correspondence between the C convolutional kernels and the C feature maps at the first scale of the first image, and a one-to-one correspondence between the C convolutional kernels and the C feature maps at the first scale of the second image.

[0134] In addition, each of the above C convolutional kernels can be... The convolutional kernel is set with a stride of 1, meaning that the single-layer convolutional neural network does not perform downsampling. This results in a high-resolution feature map at the first scale.

[0135] S403. Perform M-1 downsampling operations on the feature map of the first image at the first scale and the feature map of the second image at the second scale, respectively, to obtain feature maps of the first image at M-1 scales and feature maps of the second image at M-1 scales, where C and M are both positive integers greater than 1.

[0136] The specific implementation of downsampling can be average pooling downsampling, which can reduce the loss of image information compared to other downsampling methods (such as random downsampling).

[0137] The number of feature maps at each scale in the first image (M-1 scales) and the second image (M-1 scales) is C.

[0138] The image processing device can perform the first downsampling of the first scale feature map of the first image in M-1 downsampling steps to obtain the second scale feature map of the first image, perform the second downsampling of the second scale feature map of the first image in M-1 downsampling steps to obtain the third scale feature map of the first image, and so on, to obtain the Mth scale feature map of the first image.

[0139] It should be understood that the image processing device downsamples the feature map of the first image at the first scale M-1 times to obtain feature maps of the first image at M-1 scales. Similarly, the image processing device can obtain feature maps of the second image at M-1 scales.

[0140] It should also be understood that the number of feature maps at the first scale of the first image is C. After the first downsampling of the feature maps at the first scale of the first image, the number of feature maps at the second scale of the first image is obtained, which is also C. Similarly, the number of feature maps at other scales of the first image is also C. That is, the number of feature maps at each scale in the M-1 scales of the first image is C. Similarly, the number of feature maps at each scale in the M-1 scales of the second image is also C.

[0141] It should be noted that downsampling the first feature map in the first scale feature map of the first image can yield the first feature map in the second scale feature map of the first image, and so on. This process will not be repeated here.

[0142] For example, Figure 5 A schematic block diagram of a downsampling method is shown. (e.g.) Figure 5 As shown, the first image and the second image are processed by a single-layer convolutional neural network to obtain feature maps of the first image at the first scale and feature maps of the second image at the first scale. The feature maps of the first image at the first scale and feature maps of the second image at the first scale are downsampled for the first time to obtain feature maps of the first image at the second scale and feature maps of the second image at the second scale. The feature maps of the first image at the second scale and feature maps of the second image at the second scale are downsampled for the second time to obtain feature maps of the first image at the third scale and feature maps of the second image at the third scale. And so on, after the (M-1)th downsampling, the feature maps of the first image at the Mth scale and feature maps of the second image at the Mth scale can be obtained. If M=5 and C=64, the image processing device can obtain feature maps of the first image at four scales through S403, namely the feature maps of the second, third, fourth, and fifth scales of the first image, and can also obtain feature maps of the second image at four scales, namely the feature maps of the second, third, fourth, and fifth scales of the second image. At the same time, each feature map at the four scales of the first and second images includes 64 feature maps.

[0143] S404. Based on the feature map of the first image at the first scale, the feature map of the second image at the first scale, the feature maps of the first image at M-1 scales, the feature maps of the second image at M-1 scales, and the weight matrix related to the number of M scales and feature maps C, determine the similarity between the first image and the second image. The weight matrix includes the weight coefficients of different feature maps at different scales.

[0144] The dimension of the weight matrix is ​​C. M, the weight matrix, includes the weight coefficients of different feature maps at different scales. For example, the weight coefficient of the fourth feature map at the first scale is the value in the 4th row and 1st column of the weight matrix. It should be understood that the values ​​in the weight matrix can be interpreted as weight coefficients.

[0145] It should be noted that the weight matrix can be understood as a matrix composed of the weight coefficients of different feature maps at different scales. The weight matrix is ​​only one form of representation, and the weight coefficients of different feature maps at different scales can also be represented in other forms, such as charts, text, etc. This application does not limit this.

[0146] The image processing method provided in this application uses a single-layer convolutional neural network to obtain a feature map at the first scale. It only requires a single-layer convolution calculation, which is simple. By extracting low-level general features through a single-layer convolution, it is beneficial to ensure the generalization ability of the image for multiple tasks. The feature map at the first scale has a high resolution, which can ensure the image quality. At the same time, multiple downsampling of the high-resolution feature map can obtain more image feature information. In addition, weights are assigned to different feature maps at different scales using a weight matrix, which flexibly adjusts the weight values ​​in the image feature and scale dimensions, optimizes machine vision performance, and achieves efficient image encoding and compression performance for machine vision. This is beneficial to evaluate the quality of the first image by using the similarity between the first image and the second image.

[0147] It should be noted that when the embodiments of this application are applied to the above... Figure 2 In the scenario shown, the first image is the input to the codec, and the second image is the output of the codec. The codec is used to compress and reconstruct the first image to output the second image, and similarity is used to optimize the codec.

[0148] When the embodiments of this application are applied to the above... Figure 3 In the scenario shown, the first image is the image after image signal processing of the third image, also known as the original image; the third image is the input of the codec, and the second image is the output of the codec, also known as the reconstructed image; the codec is used to perform image signal processing, compression, and reconstruction on the third image to output the second image. The image processing device can use the above method 400 to obtain the similarity between the original image and the reconstructed image for optimizing the codec.

[0149] The image processing method provided in this application can be used to obtain the similarity between the reconstructed image and the original image to evaluate the quality of the reconstructed image, thereby guiding the optimization of the codec.

[0150] As an optional embodiment, before determining the similarity between the first image and the second image based on the first-scale feature map of the first image, the first-scale feature map of the second image, the M-1 scale feature maps of the first image, the M-1 scale feature maps of the second image, and a weight matrix related to the number of M scales and feature maps C, the method 400 further includes: determining the weight coefficients of the first-scale feature map based on the peak signal-to-noise ratio of the first-scale feature map, wherein the first-scale feature map includes the first-scale feature map of the first image and the first-scale feature map of the second image; determining the similarity between the first image and the second image based on the first-scale feature map of the first image, the first-scale feature map of the second image, the M-1 scale feature maps of the first image, the M-1 scale feature maps of the second image, and a weight matrix related to the number of M scales and feature maps C includes: determining the similarity based on the weight coefficients of the first-scale feature map, the first-scale feature map of the first image, the first-scale feature map of the second image, the M-1 scale feature maps of the first image, the M-1 scale feature maps of the second image, and the weight matrix.

[0151] If the number of feature maps at the first scale of both the first and second images is C, then the number of peak signal-to-noise ratios (PSNRs) of the feature maps at the first scale is also C. The image processing device can obtain the weight coefficient of each feature map at the first scale based on the PSNR. The image processing device can determine the similarity based on the weight coefficients of the feature maps at the first scale, the feature maps at the first scale of the first image, the feature maps at the first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and the weight matrix.

[0152] It should be understood that the feature map at the first scale has two weighting coefficients: one is determined based on the peak signal-to-noise ratio, and the other is the value of the first column in the weight matrix.

[0153] The image processing method provided in this application embodiment can make the peak signal-to-noise ratio of the feature map at the first scale consistent with the color, brightness and other features of the first and second images. Based on the peak signal-to-noise ratio of the feature map at the first scale, the weight coefficient of each feature map in the feature map at the first scale is obtained. By using the weight coefficient to weight each feature map, the detail effect of the feature map can be increased, the performance of human vision and machine vision can be improved, and the needs of human vision and machine vision can be taken into account, thus solving the above-mentioned problem 3).

[0154] As an optional embodiment, the above similarity can be obtained by the following formula:

[0155]

[0156]

[0157]

[0158]

[0159] in, For similarity, For the first image, For the second image, For convolution operations in a single-layer convolutional neural network, The first layer in a single-layer convolutional neural network One convolutional kernel, , For the first One scale, , These are the weight coefficients of the feature map at the first scale. For the first The first scale Structural similarity of feature maps The weight matrix is ​​the first... Line number The weight coefficients corresponding to the columns, The first feature map of the first scale of the first image Each feature map The first feature map of the second image at the first scale Each feature map For the first scale Peak signal-to-noise ratio of each feature map, It is a constant. For the first image The first scale The variance of each feature map For the second image The first scale The variance of each feature map For the first image The first scale The first feature map and the second image's first feature map The first scale The covariance of each feature map The maximum pixel value in the feature map of the first and second images at the first scale. The minimum pixel value in the feature maps of the first and second images at the first scale. The height of the first or second image. The width of the first or second image.

[0160] The first layer in a single-layer convolutional neural network The nth convolutional kernel, also known as the nth convolutional kernel. The embodiments of this application do not limit the number of channels.

[0161] It can also be called the adaptive structure similarity (ASSIM) index. It should be understood that this name is just an example and is not limited to the embodiments of this application.

[0162] Alternatively, C can be 64 and M can be 5. The value can be 0.1, and a single-layer convolutional neural network can include 64 layers. The convolutional kernel has a stride of 1, meaning that the single-layer convolutional neural network does not perform downsampling. This results in a high-resolution feature map at the first scale.

[0163] For example, Figure 6 A schematic diagram of similarity calculation is shown, such as... Figure 6 As shown, a single-layer convolutional neural network includes C=64 layers. The convolution kernel has a stride of 1 and M = 5. The feature map of the first scale of the first image is... The feature map of the second scale of the first image is The feature map at the third scale of the first image is The feature map at the fourth scale of the first image is The feature map at the fifth scale of the first image is The feature map of the first scale of the second image is The feature map of the second scale of the second image is The feature map at the third scale of the second image is The feature map of the fourth scale of the second image is The feature map at the fifth scale of the second image is Image processing equipment can be based on and Calculate the weight coefficients of the feature map at the first scale. Structural similarity with the first scale According to and Calculate structural similarity at the second scale According to and Calculate structural similarity at the third scale According to and Calculate structural similarity at the fourth scale According to and Calculate structural similarity at the fifth scale and according to , , , , as well as The similarity score ASSIM is obtained using the above formula. Wherein, , .

[0164] As an optional embodiment, the weight matrix is ​​a C-row, M-column matrix, where the weight coefficients in each row of the weight matrix represent the weight coefficients of the feature map at different scales, and the sum of the weight coefficients in each row of the weight matrix is ​​1.

[0165] The weight coefficients in each row of the weight matrix are the weight coefficients of the feature map corresponding to the same channel (or the feature map corresponding to the same convolution kernel) at different scales, and the sum of the weight coefficients of the feature map corresponding to the same channel at different scales is 1.

[0166] For example, in the above Figure 6 In the example shown, the single-layer convolutional neural network includes C=64 layers. The convolutional kernel has 64 channels, and M is 5. Each of these 64 channels corresponds to a feature map with 5 scales. For example, the feature map corresponding to the first channel can include a feature map of the first scale of the first image. The feature map of the second scale of the first image The feature map at the third scale of the first image The feature map at the fourth scale of the first image The feature map at the fifth scale of the first image .

[0167] In this case, there are multiple possible implementations for the weight coefficients in each row of the weight matrix.

[0168] In one possible implementation, every weight coefficient in each row of the weight matrix is ​​the same.

[0169] For example, if M=5, then each row of the weight matrix has 5 weight coefficients, all of which are the same, and the sum of the weight coefficients in each row is 1. Then each weight coefficient in each column is 1 / 5=0.2, and the weight coefficients in each row of the weight matrix are [0.2 0.2 0.2 0.2 0.2].

[0170] If C=64, then the weight matrix is ​​a 64-row, 5-column matrix, and the weight coefficient of each row in these 64 rows is [0.2 0.2 0.2 0.2 0.2].

[0171] The weight coefficients in the weight matrix can also be displayed in the form of charts or text, but the embodiments of this application are not limited thereto.

[0172] In this implementation, all weight coefficients in each row of the weight matrix are the same, which simplifies the calculation and improves the efficiency of similarity calculation.

[0173] In another possible implementation, the weight coefficients in the weight matrix can reuse the weight coefficients in the traditional multi-scale structure similarity index (MSSSIM), which are determined through human visual experiments.

[0174] The weighting coefficients in the traditional MSSSIM index include 0.0448, 0.2856, 0.3001, 0.2363, and 0.1333. In this embodiment, if M=5, the weighting coefficients in each row of the weight matrix are the same and can be [0.0448 0.2856 0.3001 0.2363 0.1333].

[0175] If C=64, then the weight matrix is ​​a 64-row, 5-column matrix, and the weight coefficient of each row in these 64 rows is [0.0448 0.2856 0.3001 0.2363 0.1333].

[0176] In this implementation, the weight coefficients in the weight matrix can reuse the weight coefficients in the traditional MSSSIM index, which helps to meet the visual needs of the human eye.

[0177] In another possible implementation, the C convolutional kernels are divided into high-frequency, mid-frequency, and low-frequency feature groups according to their feature frequencies. The M feature maps at different scales are divided into low-scale, mid-scale, and high-scale feature maps according to their resolution. The weight coefficient distribution of the feature maps corresponding to the high-frequency, mid-frequency, and low-frequency feature groups in the weight matrix at different scales satisfies the following conditions: the non-zero weight coefficients of the feature maps corresponding to the high-frequency feature groups are distributed in the low-scale feature maps of the first and second images, and the weight coefficients of the high-scale feature maps are 0; the non-zero weight coefficients of the feature maps corresponding to the low-frequency feature groups are distributed in the low-scale, mid-scale, and high-scale feature maps; the distribution of non-zero weight coefficients of the feature maps corresponding to the mid-frequency feature groups is more than that of the feature maps corresponding to the high-frequency feature groups, but less than that of the feature maps corresponding to the low-frequency feature groups.

[0178] Image processing equipment can sort C convolutional kernels according to their feature frequencies from high to low or from low to high, and then group the sorted convolutional kernels into high-frequency feature groups, mid-frequency feature groups, and low-frequency feature groups.

[0179] For example, C=64, the image processing device can sort the 64 convolutional kernels from high to low according to their feature frequencies, and then group the sorted convolutional kernels into high-frequency feature groups, mid-frequency feature groups, and low-frequency feature groups. Figure 7 A schematic diagram of convolution kernel grouping is shown, such as... Figure 7 As shown, the convolution kernels of the high-frequency feature group can include 8 convolution kernels, namely convolution kernel 1 to convolution kernel 8; the convolution kernels of the mid-frequency feature group can include 24 convolution kernels, namely convolution kernel 9 to convolution kernel 32; and the convolution kernels of the low-frequency feature group can include 32 convolution kernels, namely convolution kernel 33 to convolution kernel 64.

[0180] The feature maps corresponding to high-frequency feature groups can be understood as the feature maps corresponding to each convolutional kernel in the high-frequency feature group, with each kernel's feature map including M scales. Similarly, the feature maps corresponding to mid-frequency feature groups can be understood as the feature maps corresponding to each convolutional kernel in the mid-frequency feature group, with each kernel's feature map including M scales.

[0181] Image processing equipment can sort M feature maps at different scales according to their resolution from high to low or from low to high, and then divide the sorted M feature maps into low-scale feature maps, medium-scale feature maps, and high-scale feature maps.

[0182] The image processing method provided in this application allows the weight matrix to assign weight coefficients to different features at different scales, thereby leveraging the different roles of each feature. For features corresponding to high-frequency feature groups, higher weights are assigned to low-scale feature maps. For features corresponding to low-frequency feature groups, the ability to extract contour information at high scales is considered, and weights are evenly distributed across different scales. For features corresponding to mid-frequency feature groups, the weights are positioned between high-frequency and low-frequency feature groups, taking into account both high-frequency details and contour information, which helps improve machine vision performance.

[0183] Optionally, all non-zero weight coefficients in the feature map corresponding to the high-frequency feature group are the same, all non-zero weight coefficients in the feature map corresponding to the low-frequency feature group are the same, and all non-zero weight coefficients in the feature map corresponding to the mid-frequency feature group are the same.

[0184] Optionally, if M=5, the weight coefficient of the feature map corresponding to the high-frequency feature group is [1 0 0 0 0], the weight coefficient of the feature map corresponding to the low-frequency feature group is [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5], and the weight coefficient of the feature map corresponding to the mid-frequency feature group is [1 / 3 1 / 3 1 / 3 0 0].

[0185] In [1 0 0 0 0], [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5] and [1 / 3 1 / 3 1 / 3 0 0], the weight coefficients in the first column are used to represent the weight coefficients of the low-scale feature map, the weight coefficients in the second and third columns are used to represent the weight coefficients of the medium-scale feature map, and the weight coefficients in the fourth and fifth columns are used to represent the weight coefficients of the high-scale feature map.

[0186] If C=64, the weight matrix is ​​a 64-row, 5-column matrix. If the high-frequency feature group corresponds to 8 convolutional kernels, then there are 8 rows of weight coefficients in the weight matrix corresponding to the high-frequency feature group. Each of these 8 rows of weight coefficients is the same and is [1 0 0 0 0]. If the mid-frequency feature group corresponds to 24 convolutional kernels, then there are 24 rows of weight coefficients in the weight matrix corresponding to the mid-frequency feature group. Each of these 24 rows of weight coefficients is the same and is [1 / 3 1 / 3 1 / 30 0]. If the low-frequency feature group corresponds to 32 convolutional kernels, then there are 32 rows of weight coefficients in the weight matrix corresponding to the mid-frequency feature group. Each of these 32 rows of weight coefficients is the same and is [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5].

[0187] It should be noted that the weight coefficients in the first column of the weight matrix above represent the weight coefficients of the low-scale feature map, the weight coefficients in the second and third columns represent the weight coefficients of the medium-scale feature map, and the weight coefficients in the fourth and fifth columns represent the weight coefficients of the high-scale feature map. This is merely an example, and the embodiments of this application do not limit it. The image processing method provided in the embodiments of this application can assign weight coefficients to different features at different scales using the weight matrix, thereby leveraging the different roles of the features. For detail features, a weight of 1 is assigned to the high-resolution first-scale feature map. For smooth features, considering the ability to extract contour information at high scales, the weights are evenly distributed across different scales. For edge features, the weight distribution is between that of detail features and smooth features, taking into account both high-frequency details and contour information, which helps to improve machine vision performance.

[0188] Optionally, the feature frequencies are determined by the magnitude of the convolutional kernel of the single-layer convolutional neural network and / or the scaling factor and normalization factor of the normalization layer of the single-layer convolutional neural network.

[0189] In one possible implementation, the magnitude of the i-th convolutional kernel in a single-layer convolutional neural network can be represented by the symbol... This indicates that the feature frequency can be determined by the magnitude of the convolution kernel in a single-layer convolutional neural network.

[0190] In another possible implementation, the scaling factor and normalization factor of the normalization layer can be parameters of the batch normalization (BN) layer of a pre-trained model, which is a pre-trained convolutional neural network model, such as the ResNet-50 pre-trained model. The scaling factor of the normalization layer can be represented by a symbol... It can be indicated that the normalization coefficient can be represented by the symbol. This indicates that the feature frequency can be determined by the scaling factor and normalization factor of the normalization layer of a single-layer convolutional neural network.

[0191] In another possible implementation, the feature frequency can be determined by the magnitude of the convolutional kernel of the single-layer convolutional neural network, the scaling factor of the normalization layer, and the normalization factor. For example, the feature frequency is determined by... Sure.

[0192] The above combination Figures 1 to 7 The image processing method provided in the embodiments of this application has been described in detail below, in conjunction with... Figures 8 to 11 The simulation results of the embodiments of this application are described.

[0193] The image processing method provided in this application calculates the similarity ASSIM, which, compared to calculating the mean square error (MSE) based on the pixel domain, the mean square error (wfMSE) based on the feature domain, and the MSSSIM based on the feature domain (which can be simply referred to as wfMSSSIM), can improve the performance of machine vision (such as object detection mAP and semantic segmentation mIoU) and human vision (such as LPIPS and DISTS).

[0194] Compared to wfMSSSIM, wfMSE, and MSE, LPIPS has superior visual performance in the human eye.

[0195] For example, Figure 8 A simulation diagram is shown. For example... Figure 8 As shown, Figure 8 The curves in the figure represent the change in LPIPS as bits per pixel (BPP) varies. The figure includes four curves: one filled with black circles, one filled with white circles, one filled with black triangles, and one filled with white triangles. The curve filled with black circles represents ASSIM, the curve filled with white circles represents wfMSSSIM, the curve filled with black triangles represents wfMSE, and the curve filled with white triangles represents MSE. As can be seen from the figure, ASSIM is superior to the other curves in terms of LPIPS.

[0196] Compared to wfMSSSIM, wfMSE, and MSE, DISTS has better visual performance in the human eye.

[0197] For example, Figure 9 A simulation diagram is shown. For example... Figure 9 As shown, Figure 9 The curves in the figure represent the change in DISTS as BPP changes. The figure includes four curves: one filled with black circles, one filled with white circles, one filled with black triangles, and one filled with white triangles. The curve filled with black circles represents ASSIM, the curve filled with white circles represents wfMSSSIM, the curve filled with black triangles represents wfMSE, and the curve filled with white triangles represents MSE. As can be seen from the figure, ASSIM, when above the other curves, yields a better DISTS.

[0198] Compared to wfMSSSIM, wfMSE, and MSE, ASSIM has better mAP performance in machine vision.

[0199] For example, Figure 10 A simulation diagram is shown. For example... Figure 10 As shown, Figure 10 The curves in the figure represent the change in mAP as BPP changes. The figure includes five curves: a curve filled with black circles, a curve filled with white circles, a curve filled with black triangles, a curve filled with white triangles, and a straight line. The curve filled with black circles represents ASSIM, the curve filled with white circles represents wfMSSSIM, the curve filled with black triangles represents wfMSE, the curve filled with white triangles represents MSE, and the straight line represents no compression. As can be seen from the figure, ASSIM is above the curves for wfMSSSIM, wfMSE, and MSE. Furthermore, when BPP is 1.2 or higher, ASSIM's mAP is above the straight line for no compression, indicating that ASSIM has a superior mAP. Additionally, as indicated by the arrows in the figure, ASSIM, compared to MSE, saves approximately 50% of the encoding bits for object detection, such as Faster R-CNN, at the same machine vision performance.

[0200] ASSIM offers superior mIoU performance in machine vision compared to wfMSE and MSE.

[0201] For example, Figure 11 A simulation diagram is shown. For example... Figure 11 As shown, Figure 11 The curves in the figure represent the change in mIoU with varying BPP. The figure includes five curves: one filled with black circles, one filled with white circles, one filled with black triangles, one filled with white triangles, and a straight line. The curve filled with black circles represents ASSIM, the curve filled with white circles represents wfMSSSIM, the curve filled with black triangles represents wfMSE, the curve filled with white triangles represents MSE, and the straight line represents no compression. As can be seen from the figure, ASSIM is above the wfMSE and MSE curves but below the wfMSSSIM curve, indicating that ASSIM has a better mIoU than wfMSE and MSE. Furthermore, at the arrows shown in the figure, compared to MSE, ASSIM saves approximately 40% of the coding bits for semantic segmentation, such as the Pyramid Scene Parsing Network (PSPnet), at the same machine vision performance.

[0202] The sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0203] The above text combines Figures 1 to 11 The image processing method of the embodiments of this application is described in detail below, and will be combined with Figure 12 and Figure 13 The image processing apparatus of the embodiments of this application is described in detail below.

[0204] Figure 12 An image processing apparatus 1200 according to an embodiment of this application is shown. The image processing apparatus 1200 includes an acquisition module 1210 and a processing module 1220. The acquisition module 1210 is used to: acquire a first image and a second image; the processing module 1220 is used to: input the first image and the second image into a single-layer convolutional neural network respectively to obtain feature maps of the first image at a first scale and feature maps of the second image at a first scale, wherein the number of feature maps of the first image at a first scale and the number of feature maps of the second image at a first scale are both C; perform M-1 downsampling operations on the feature maps of the first image at a first scale and the feature maps of the second image at a second scale respectively to obtain feature maps of the first image at M-1 scales and feature maps of the second image at M-1 scales, wherein C and M are both positive integers greater than 1; determine the similarity between the first image and the second image based on the feature maps of the first image at a first scale, the feature maps of the second image at a first scale, the feature maps of the first image at M-1 scales, the feature maps of the second image at M-1 scales, and a weight matrix related to the M scales and the number of feature maps C, wherein the weight matrix includes weight coefficients for different feature maps at different scales.

[0205] Optionally, the processing module 1220 is further configured to: determine the weight coefficients of the feature map at the first scale based on the peak signal-to-noise ratio of the feature map at the first scale, wherein the feature map at the first scale includes the feature map at the first scale of the first image and the feature map at the first scale of the second image; and determine the similarity based on the weight coefficients of the feature map at the first scale, the feature map at the first scale of the first image, the feature map at the first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and the weight matrix.

[0206] Alternatively, the above similarity can be obtained using the following formula:

[0207]

[0208]

[0209]

[0210]

[0211] in, For the aforementioned similarity, For the first image, For the second image, For convolution operations in a single-layer convolutional neural network, The first layer in a single-layer convolutional neural network One convolutional kernel, , For the first One scale, , These are the weight coefficients of the feature map at the first scale. For the first The first scale Structural similarity of feature maps The weight matrix is ​​the first... Line number The weight coefficients corresponding to the columns, The first feature map of the first scale of the first image Each feature map The first feature map of the second image at the first scale Each feature map For the first scale Peak signal-to-noise ratio of each feature map, It is a constant. For the first image The first scale The variance of each feature map For the second image The first scale The variance of each feature map For the first image The first scale The first feature map and the second image's first feature map The first scale The covariance of each feature map The maximum pixel value in the feature map of the first and second images at the first scale. The minimum pixel value in the feature maps of the first and second images at the first scale. The height of the first or second image. The width of the first or second image.

[0212] Optionally, the weight matrix is ​​a C-row, M-column matrix, where the weight coefficients in each row of the weight matrix represent the weight coefficients of the feature map at different scales, and the sum of the weight coefficients in each row of the weight matrix is ​​1.

[0213] Optionally, each weight coefficient in each row of the above weight matrix is ​​the same.

[0214] Optionally, with M=5, the weight coefficients in each row of the weight matrix are the same and are [0.2 0.2 0.2 0.2 0.2].

[0215] Optionally, with M=5, the weight coefficients in each row of the weight matrix are the same, and are [0.0448 0.2856 0.3001 0.2363 0.1233].

[0216] Optionally, the C convolutional kernels are divided into high-frequency, mid-frequency, and low-frequency feature groups according to their feature frequencies, and the M feature maps at different scales are divided into low-scale, mid-scale, and high-scale feature maps according to their resolutions. The weight coefficient distribution of the feature maps corresponding to the high-frequency, mid-frequency, and low-frequency feature groups in the weight matrix at different scales satisfies the following conditions: the non-zero weight coefficients of the feature maps corresponding to the high-frequency feature groups are distributed in the low-scale feature maps of the first and second images, and the weight coefficients of the high-scale feature maps are 0; the non-zero weight coefficients of the feature maps corresponding to the low-frequency feature groups are distributed in the low-scale, mid-scale, and high-scale feature maps; the distribution of non-zero weight coefficients of the feature maps corresponding to the mid-frequency feature groups is more than that of the feature maps corresponding to the high-frequency feature groups, but less than that of the feature maps corresponding to the low-frequency feature groups.

[0217] Optionally, all non-zero weight coefficients in the feature map corresponding to the high-frequency feature group are the same, all non-zero weight coefficients in the feature map corresponding to the low-frequency feature group are the same, and all non-zero weight coefficients in the feature map corresponding to the mid-frequency feature group are the same.

[0218] Optionally, with M=5, the weight coefficient of the feature map corresponding to the high-frequency feature group is [1 0 0 0 0], the weight coefficient of the feature map corresponding to the low-frequency feature group is [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5], and the weight coefficient of the feature map corresponding to the mid-frequency feature group is [1 / 3 1 / 3 1 / 3 0 0].

[0219] In [1 0 0 0 0], [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5] and [1 / 3 1 / 3 1 / 3 0 0], the weight coefficients in the first column are used to represent the weight coefficients of the low-scale feature map, the weight coefficients in the second and third columns are used to represent the weight coefficients of the medium-scale feature map, and the weight coefficients in the fourth and fifth columns are used to represent the weight coefficients of the high-scale feature map.

[0220] Optionally, the aforementioned feature frequencies are determined by the magnitude of the convolution kernel of the single-layer convolutional neural network and / or the scaling factor and normalization factor of the normalization layer of the single-layer convolutional neural network.

[0221] Optionally, the first image is the input of the codec, and the second image is the output of the codec. The codec is used to compress and reconstruct the first image to output the second image, and the similarity is used to optimize the codec.

[0222] Optionally, the first image is the image after image signal processing of the third image, the third image is the input of the codec, the second image is the output of the codec, the codec is used to perform image signal processing, compression and reconstruction on the third image to output the second image, and the similarity is used to optimize the codec.

[0223] It should be understood that the image processing apparatus 1200 here is embodied in the form of a functional module. The term "module" here may refer to application-specific integrated circuits (ASICs), electronic circuits, processors (e.g., shared processors, proprietary processors, or group processors, etc.) and memory for executing one or more software or firmware programs, integrated logic circuits, and / or other suitable components that support the described functions.

[0224] The image processing apparatus 1200 described above has the function of implementing the corresponding steps performed by the image processing device in the above method embodiments; the above functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0225] In the embodiments of this application, Figure 12 The image processing device 1200 in the image processing device 1200 can also be a chip or a chip system, such as a system on chip (SoC).

[0226] Figure 13 This is a schematic block diagram of another image processing apparatus 1300 provided in an embodiment of this application. The image processing apparatus 1300 includes a processor 1310, a communication interface 1320, and a memory 1330. The processor 1310, the communication interface 1320, and the memory 1330 communicate with each other via internal connection paths. The memory 1330 is used to store instructions, and the processor 1310 is used to execute the instructions stored in the memory 1330 to control the communication interface 1320 to send and / or receive signals.

[0227] It should be understood that the image processing apparatus 1300 may specifically be the image processing device in the above method embodiments, or the functions of the image processing device in the above method embodiments may be integrated into the image processing apparatus 1300. The image processing apparatus 1300 may be used to execute the various steps and / or processes corresponding to the image processing device in the above method embodiments. Optionally, the memory 1330 may include read-only memory and random access memory, and provide instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor 1310 may be used to execute instructions stored in the memory, and when the processor executes the instructions, the processor may execute the various steps and / or processes corresponding to the image processing device in the above method embodiments.

[0228] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or as a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor executes the instructions in the memory, combining them with its hardware to complete the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0229] This application also provides a computer-readable storage medium for storing a computer program that implements the method corresponding to the image processing device in the above embodiments.

[0230] This application also provides a chip system for supporting the image processing device in the above method embodiments to implement the functions shown in this application embodiment.

[0231] This application provides a computer program product, which includes a computer program (also called code or instructions). When the computer program is run on a computer, the computer can execute the method corresponding to the image processing device in the above embodiments.

[0232] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0233] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0234] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0235] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Acquire the first image and the second image; The first image and the second image are respectively input into a single-layer convolutional neural network to obtain a first-scale feature map of the first image and a first-scale feature map of the second image. The single-layer convolutional neural network includes C convolutional kernels, and the number of first-scale feature maps of the first image and the number of first-scale feature maps of the second image are both C. The feature map at the first scale of the first image and the feature map at the second scale of the second image are downsampled M-1 times respectively to obtain feature maps at M-1 scales of the first image and the feature maps at M-1 scales of the second image, where C and M are both positive integers greater than 1; The similarity between the first image and the second image is determined based on the feature map of the first image at a first scale, the feature map of the second image at a first scale, the feature maps of the first image at M-1 scales, the feature maps of the second image at M-1 scales, and a weight matrix related to the M scales and the number C of feature maps. The weight matrix includes weight coefficients for different feature maps at different scales.

2. The method according to claim 1, characterized in that, Before determining the similarity between the first image and the second image based on the feature map at a first scale of the first image, the feature map at a first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and a weight matrix related to the number of M scales and feature maps C, the method further includes: The weight coefficients of the feature map at the first scale are determined based on the peak signal-to-noise ratio of the feature map at the first scale. The feature map at the first scale includes the feature map at the first scale of the first image and the feature map at the first scale of the second image. The step of determining the similarity between the first image and the second image based on the feature map at a first scale of the first image, the feature map at a first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and a weight matrix related to the number of M scales and feature maps C, includes: The similarity is determined based on the weight coefficients of the feature map at the first scale, the feature map at the first scale of the first image, the feature map at the first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and the weight matrix.

3. The method according to claim 2, characterized in that, The similarity is obtained using the following formula: in, For the aforementioned similarity, For the first image, For the second image, the For the convolution operation of the single-layer convolutional neural network, The first layer in the single-layer convolutional neural network One convolutional kernel, , For the first One scale, , These are the weight coefficients of the feature map at the first scale. For the first The first scale Structural similarity of feature maps The weight matrix is ​​the first... Line number The weight coefficients corresponding to the columns, The first feature map of the first image at the first scale Each feature map The first feature map of the second image at the first scale Each feature map For the first scale Peak signal-to-noise ratio of each feature map, It is a constant. For the first image The first scale The variance of each feature map The second image The first scale The variance of each feature map For the first image The first scale The first feature map and the second image of the second image The first scale The covariance of each feature map The maximum pixel value in the feature map of the first and second images at the first scale. The minimum pixel value in the feature map of the first image and the second image at the first scale. The height of the first image or the second image. The width of the first image or the second image.

4. The method according to any one of claims 1 to 3, characterized in that, The weight matrix is ​​a C-row, M-column matrix. The weight coefficients in each row of the weight matrix represent the weight coefficients of the feature map at different scales. The sum of the weight coefficients in each row of the weight matrix is ​​1.

5. The method according to claim 4, characterized in that, In the weight matrix, each weight coefficient in each row is the same.

6. The method according to claim 5, characterized in that, M=5, and the weight coefficients in each row of the weight matrix are the same, and are [0.2 0.2 0.2 0.2 0.2].

7. The method according to claim 4, characterized in that, M=5, and the weight coefficients in each row of the weight matrix are the same, and are [0.0448 0.2856 0.3001 0.2363 0.1333].

8. The method according to claim 4, characterized in that, The C convolutional kernels are divided into high-frequency feature groups, mid-frequency feature groups, and low-frequency feature groups according to their feature frequencies. The M feature maps at different scales are divided into low-scale feature maps, mid-scale feature maps, and high-scale feature maps according to their resolution. The weight coefficient distributions of the feature maps corresponding to the high-frequency feature group, the mid-frequency feature group, and the low-frequency feature group at different scales satisfy the following conditions: The non-zero weight coefficients of the feature map corresponding to the high-frequency feature group are distributed in the low-scale feature map, and the weight coefficients of the high-scale feature map are 0. The non-zero weight coefficients of the feature maps corresponding to the low-frequency feature groups are distributed in the low-scale feature maps, the medium-scale feature maps, and the high-scale feature maps; The distribution of non-zero weight coefficients in the feature map corresponding to the mid-frequency feature group is greater than that in the feature map corresponding to the high-frequency feature group, but less than that in the feature map corresponding to the low-frequency feature group.

9. The method according to claim 8, characterized in that, All non-zero weight coefficients in the feature map corresponding to the high-frequency feature group are the same, all non-zero weight coefficients in the feature map corresponding to the low-frequency feature group are the same, and all non-zero weight coefficients in the feature map corresponding to the mid-frequency feature group are the same.

10. The method according to claim 9, characterized in that, M=5, the weight coefficient of the feature map corresponding to the high frequency feature group is [1 0 0 0 0], the weight coefficient of the feature map corresponding to the low frequency feature group is [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5], and the weight coefficient of the feature map corresponding to the mid frequency feature is [1 / 3 1 / 3 1 / 3 0 0]. In [1 0 0 0 0], [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5] and [1 / 3 1 / 3 1 / 3 0 0], the weight coefficients in the first column are used to represent the weight coefficients of the low-scale feature map, the weight coefficients in the second column and the third column are used to represent the weight coefficients of the medium-scale feature map, and the weight coefficients in the fourth column and the fifth column are used to represent the weight coefficients of the high-scale feature map.

11. The method according to any one of claims 8 to 10, characterized in that, The characteristic frequency is determined by the magnitude of the convolution kernel of the single-layer convolutional neural network and / or the scaling factor and normalization factor of the normalization layer of the single-layer convolutional neural network.

12. The method according to any one of claims 1 to 3, 5-10, characterized in that, The first image is the input of the codec, and the second image is the output of the codec. The codec is used to compress and reconstruct the first image to output the second image, and the similarity is used to optimize the codec.

13. The method according to any one of claims 1 to 3, 5-10, characterized in that, The first image is the image of the third image after image signal processing. The third image is the input of the codec, and the second image is the output of the codec. The codec is used to perform image signal processing, compression, and reconstruction on the third image to output the second image. The similarity is used to optimize the codec.

14. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the first image and the second image; The processing module is used to input the first image and the second image into a single-layer convolutional neural network to obtain a first-scale feature map of the first image and a first-scale feature map of the second image. The single-layer convolutional neural network includes C convolutional kernels, and the number of first-scale feature maps of the first image and the number of first-scale feature maps of the second image are both C. The feature map at the first scale of the first image and the feature map at the second scale of the second image are downsampled M-1 times respectively to obtain feature maps at M-1 scales of the first image and the second image, where C and M are both positive integers greater than 1; and, The similarity between the first image and the second image is determined based on the feature map of the first image at a first scale, the feature map of the second image at a first scale, the feature maps of the first image at M-1 scales, the feature maps of the second image at M-1 scales, and a weight matrix related to the M scales and the number C of feature maps. The weight matrix includes weight coefficients for different feature maps at different scales.

15. The apparatus according to claim 14, characterized in that, The processing module is also used for: The weight coefficients of the feature map at the first scale are determined based on the peak signal-to-noise ratio of the feature map at the first scale. The feature map at the first scale includes the feature map at the first scale of the first image and the feature map at the first scale of the second image. The similarity is determined based on the weight coefficients of the feature map at the first scale, the feature map at the first scale of the first image, the feature map at the first scale of the second image, the feature maps at M-1 scales of the first image, the feature maps at M-1 scales of the second image, and the weight matrix.

16. The apparatus according to claim 15, characterized in that, The similarity is obtained using the following formula: in, For the aforementioned similarity, For the first image, For the second image, the For the convolution operation of the single-layer convolutional neural network, The first layer in the single-layer convolutional neural network One convolutional kernel, , For the first One scale, , These are the weight coefficients of the feature map at the first scale. For the first The first scale Structural similarity of feature maps The weight matrix is ​​the first... Line number The weight coefficients corresponding to the columns, The first feature map of the first image at the first scale Each feature map The first feature map of the second image at the first scale Each feature map For the first scale Peak signal-to-noise ratio of each feature map, It is a constant. For the first image The first scale The variance of each feature map The second image The first scale The variance of each feature map For the first image The first scale The first feature map and the second image of the second image The first scale The covariance of each feature map The maximum pixel value in the feature map of the first and second images at the first scale. The minimum pixel value in the feature map of the first image and the second image at the first scale. The height of the first image or the second image. The width of the first image or the second image.

17. The apparatus according to any one of claims 14 to 16, characterized in that, The weight matrix is ​​a C-row, M-column matrix. The weight coefficients in each row of the weight matrix represent the weight coefficients of the feature map at different scales. The sum of the weight coefficients in each row of the weight matrix is ​​1.

18. The apparatus according to claim 17, characterized in that, In the weight matrix, each weight coefficient in each row is the same.

19. The apparatus according to claim 18, characterized in that, M=5, and the weight coefficients in each row of the weight matrix are the same, and are [0.2 0.2 0.2 0.2 0.2].

20. The apparatus according to claim 17, characterized in that, M=5, and the weight coefficients in each row of the weight matrix are the same, and are [0.0448 0.2856 0.3001 0.2363 0.1333].

21. The apparatus according to claim 17, characterized in that, The C convolutional kernels are divided into high-frequency feature groups, mid-frequency feature groups, and low-frequency feature groups according to their feature frequencies. The M feature maps at different scales are divided into low-scale feature maps, mid-scale feature maps, and high-scale feature maps according to their resolution. The weight coefficient distributions of the feature maps corresponding to the high-frequency feature group, the mid-frequency feature group, and the low-frequency feature group in the weight matrix at different scales satisfy the following conditions: The non-zero weight coefficients of the feature maps corresponding to the high-frequency feature groups are distributed in the low-scale feature maps of the first image and the second image, while the weight coefficients of the high-scale feature maps are 0. The non-zero weight coefficients of the feature maps corresponding to the low-frequency feature groups are distributed in the low-scale feature maps, the medium-scale feature maps, and the high-scale feature maps; The distribution of non-zero weight coefficients in the feature map corresponding to the mid-frequency feature group is greater than that in the feature map corresponding to the high-frequency feature group, but less than that in the feature map corresponding to the low-frequency feature group.

22. The apparatus according to claim 21, characterized in that, All non-zero weight coefficients in the feature map corresponding to the high-frequency feature group are the same, all non-zero weight coefficients in the feature map corresponding to the low-frequency feature group are the same, and all non-zero weight coefficients in the feature map corresponding to the mid-frequency feature group are the same.

23. The apparatus according to claim 22, characterized in that, M=5, the weight coefficient of the feature map corresponding to the high frequency feature group is [1 0 0 0 0], the weight coefficient of the feature map corresponding to the low frequency feature group is [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5], and the weight coefficient of the feature map corresponding to the mid frequency feature is [1 / 3 1 / 3 1 / 3 0 0]. In [1 0 0 0 0], [1 / 5 1 / 5 1 / 5 1 / 5 1 / 5] and [1 / 3 1 / 3 1 / 3 0 0], the weight coefficients in the first column are used to represent the weight coefficients of the low-scale feature map, the weight coefficients in the second column and the third column are used to represent the weight coefficients of the medium-scale feature map, and the weight coefficients in the fourth column and the fifth column are used to represent the weight coefficients of the high-scale feature map.

24. The apparatus according to any one of claims 21 to 23, characterized in that, The characteristic frequency is determined by the magnitude of the convolution kernel of the single-layer convolutional neural network and / or the scaling factor and normalization factor of the normalization layer of the single-layer convolutional neural network.

25. The apparatus as described in any one of claims 14 to 16, 18-23, characterized in that, The first image is the input of the codec, and the second image is the output of the codec. The codec is used to compress and reconstruct the first image to output the second image, and the similarity is used to optimize the codec.

26. The apparatus according to any one of claims 14 to 16, 18-23, characterized in that, The first image is the image of the third image after image signal processing. The third image is the input of the codec, and the second image is the output of the codec. The codec is used to perform image signal processing, compression, and reconstruction on the third image to output the second image. The similarity is used to optimize the codec.

27. An image processing apparatus, characterized in that, It includes a processor and a memory, the memory being used to store code instructions, and the processor being used to execute the code instructions to perform the method as described in any one of claims 1 to 13.

28. A chip system, characterized in that, Includes: a processor for retrieving and running a computer program from memory, causing a communication device equipped with the chip system to perform the method of any one of claims 1 to 13.

29. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the method of any one of claims 1 to 13 to be performed.

30. A computer program product, characterized in that, The computer program product includes instructions that, when executed, cause the method of any one of claims 1 to 13 to be performed.

Images