Image processing model, training method and device of image processing model

Abstract

The application provides an image processing model, a training method of the image processing model, an image processing method and device, relates to the field of artificial intelligence, and particularly to the field of computer vision. The image processing model comprises a frequency separation module, a low-frequency processing module and a frequency fusion module. The frequency separation module performs frequency separation processing on input multiple frames of images, extracts different frequency components. The low-frequency processing module fuses low-frequency components of the multiple frames of images. The frequency fusion module fuses the fused low-frequency components and high-frequency components of the multiple frames of images, and obtains enhanced images. The scheme of the embodiment of the application can reduce the calculation overhead and hardware power consumption, and improve the operation speed of the image processing model.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to an image processing model, a training method for an image processing model, and an image processing method and device. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application areas, such as manufacturing, inspection, document analysis, medical diagnosis, and the military. It involves the use of cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's like equipping computers with eyes (cameras) and brains (algorithms) to replace the human eye in identifying, tracking, and measuring objects, thus enabling computers to perceive their environment. Because perception can be viewed as extracting information from sensory signals, computer vision can also be considered the study of how artificial systems can "perceive" from images or multidimensional data. In short, computer vision uses various imaging systems to replace the visual organs to acquire input information, and then computers replace the brain to process and interpret this input information. The ultimate goal of computer vision research is to enable computers to observe and understand the world through vision like humans do, and to have the ability to autonomously adapt to their environments.

[0003] Image quality significantly impacts image processing. With advances in imaging technology, the quality of images and videos captured by devices like smartphones has improved significantly. However, due to the limitations of the optical sensor hardware, image quality still suffers from issues such as high noise, low resolution, loss of detail, and color cast. Image enhancement can improve image quality and enhance the accuracy of subsequent computer vision processing.

[0004] Deep learning has achieved remarkable results in various computer vision tasks. Compared to traditional image enhancement algorithms, deep learning-based image enhancement algorithms can significantly improve image quality. Generally speaking, the more complex the neural network model, the better the image enhancement effect. However, more complex neural network models require more hardware resources, which means greater computational overhead and runtime memory usage. This seriously affects the real-time processing performance of neural network models, limiting their application in scenarios with high real-time requirements. Moreover, overly large neural network models cannot be deployed on devices with less computing power, such as mobile phones and other terminal devices. Summary of the Invention

[0005] The present application provides an image processing model, an image processing model training method, an image processing method and an image processing device, which can reduce computing overhead and hardware power consumption and improve the computing speed of the image processing model.

[0006] In a first aspect, an image processing model is provided, which includes: a frequency separation module, a low-frequency processing module and a frequency fusion module; the frequency separation module is used to perform frequency separation processing on multiple frames of images respectively, and output the high-frequency component and low-frequency component of each frame of the multiple frames; the low-frequency processing module is used to perform fusion processing on the low-frequency components of the multiple frames of images, and output the fused low-frequency components; the frequency fusion module is used to perform frequency fusion processing on the fused low-frequency components and the high-frequency components of at least one frame of the multiple frames, and output an enhanced image corresponding to the at least one frame of the image.

[0007] According to the solution of the embodiment of the present application, the different frequency components of the multi-frame images are extracted, and the high-frequency components are relatively separated, ensuring that the high-frequency information in the image is not affected by the inter-frame misalignment problem, retaining good high-frequency details, avoiding the problem of blurring of the enhanced image, and fusing the low-frequency components of the multi-frame images to obtain the low-frequency information in the multi-frame images, achieving inter-frame complementarity, solving problems such as color cast and general texture blur, improving the denoising effect, ensuring the authenticity of the picture color, enhancing the texture details of the image, and improving the imaging quality. Moreover, the solution in the embodiment of the present application separates the high-frequency components so that the high-frequency components are not affected by the inter-frame misalignment problem, and thus there is no need to use a deeper convolutional layer to align the multi-frame images, reducing the computational overhead and memory overhead during model operation, and improving the model's running speed.

[0008] The “high frequency” and “low frequency” in the embodiments of the present application are relative concepts, as long as the frequency of the high frequency component is higher than the frequency of the low frequency component.

[0009] Optionally, the frequency separation module may perform frequency separation processing on multiple frames of images respectively by frequency conversion.

[0010] The number of multiple frames of images and the number of enhanced images corresponding to at least one frame of image can be the same or different. The enhanced image is the output image of the model. In other words, the number of images input to the model and the number of images output from the model can be the same or different.

[0011] The enhanced image corresponding to the output at least one frame of image may be one frame or multiple frames.

[0012] In combination with the first aspect, in some implementations of the first aspect, the number of enhanced images corresponding to at least one frame of image is greater than or equal to 2.

[0013] According to the solution of the embodiment of the present application, when at least one frame of image is two or more frames of image, the image processing model is a multi-input multi-output model, which can simultaneously output enhanced images corresponding to two or more frames of image. Compared with the multi-input single-output model, the multi-input multi-output model can process multiple frames of image simultaneously in one reasoning process, which can appropriately reduce the speed of model reasoning, and thus appropriately increase the network capacity, such as using deeper convolutional layers to obtain better imaging quality, or increase the effective information required to restore details, such as inputting more images to obtain better imaging quality, allowing devices with weaker computing power to use larger capacity convolutional neural networks to further improve imaging quality.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the model also includes: a high-frequency processing module, used to perform convolution processing on the high-frequency components of at least one frame of images in multiple frames; and a frequency fusion module, used to perform frequency fusion processing on the fused low-frequency components and the high-frequency components after convolution processing of at least one frame of images, and output an enhanced image corresponding to at least one frame of images.

[0015] The convolution process may be performed once or multiple times.

[0016] According to the solution of the embodiment of the present application, by performing convolution processing on the high-frequency components, it is beneficial to further restore the high-frequency details to improve the image quality in the high-frequency area.

[0017] In combination with the first aspect, in some implementations of the first aspect, the frequency separation module is specifically used to: perform frequency separation processing on multiple frames of images through wavelet transform, and output high-frequency components and low-frequency components of each frame of the multiple frames.

[0018] Exemplarily, the wavelet transform may be discrete wavelet transform (DWT).

[0019] In combination with the first aspect, in certain implementations of the first aspect, the low-frequency processing module is specifically used to: serially merge the low-frequency components of multiple frames of images; perform convolution processing on the result of the serial merging, and output the fused low-frequency components.

[0020] In combination with the first aspect, in certain implementations of the first aspect, the frequency fusion module is specifically used to: serially merge the fused low-frequency component and the high-frequency component of at least one frame of image respectively; perform convolution processing on the result of the serial merging, and output an enhanced image corresponding to at least one frame of image.

[0021] In combination with the first aspect, in certain implementations of the first aspect, the frequency fusion module is specifically used to: serially merge the fused low-frequency component and the high-frequency component after convolution processing of at least one frame of image; perform convolution processing on the result of the serial merging, and output an enhanced image corresponding to at least one frame of image.

[0022] In combination with the first aspect, in some implementations of the first aspect, the image processing model further includes an input processing module, which is used to preprocess multiple frames of initial images and output multiple frames of images.

[0023] For example, the multiple frames of initial images may be multiple adjacent frames of images in a video. For another example, the multiple frames of initial images may be multiple images captured when a shutter button is pressed.

[0024] In combination with the first aspect, in certain implementations of the first aspect, the input processing module is specifically used to: perform feature extraction on the multiple frames of initial images, and use the extracted feature maps of the multiple frames of initial images as the multiple frames of images.

[0025] Exemplarily, feature extraction is performed on multiple frames of initial images through one or more convolution processes.

[0026] In combination with the first aspect, in some implementations of the first aspect, the input processing module is specifically used to: downsample the multiple frames of initial images, and use the downsampled images as the multiple frames of images.

[0027] Exemplarily, downsampling processing is performed on multiple frames of initial images through pixel reorganization to obtain downsampled images.

[0028] This will produce an image with the required resolution.

[0029] In combination with the first aspect, in certain implementations of the first aspect, the input processing module is specifically used to: downsample multiple frames of initial images, perform feature extraction on the downsampled images, and use the extracted feature maps as multiple frames of images.

[0030] In a second aspect, a training method for an image processing model is provided, the method comprising: obtaining multiple frames of sample images and a target image corresponding to at least one frame of sample images in the multiple frames of sample images, the signal-to-noise ratio of the target image corresponding to at least one frame of sample images being higher than that of at least one frame of sample images; performing forward propagation on a first image processing model based on the multiple frames of sample images to obtain a first enhanced image corresponding to at least one frame of sample images; adjusting the parameters of the first image processing model with the goal of reducing the value of a target loss function to obtain a trained first image processing model, the target loss function being determined based on the loss function of the first image processing model, the loss function of the first image processing model being used to calculate the difference between the first enhanced image corresponding to at least one frame of sample images and the target image corresponding to at least one frame of sample images; the loss function comprising a weight value, and the weight value of the loss function corresponding to part or all of the areas in the first enhanced image corresponding to at least one frame of sample images being indicated by a weight map.

[0031] According to the solution of the embodiment of the present application, different weight values ​​of the loss function are set for different areas of the enhanced image output by the model, and higher weight values ​​are set for areas that require the model to pay more attention. This can guide the model to pay more attention to the specified areas and further improve the performance of the model in processing the specified areas.

[0032] The target image corresponding to the sample image can also be understood as the true value of the supervisory label (groundtruth, GT) corresponding to the sample image.

[0033] It should be understood that the “first” in “first image processing model” is only used to distinguish different image processing models and does not have any other limiting effect.

[0034] Exemplarily, the parameters of the first image processing model may be adjusted using a gradient descent method.

[0035] In combination with the second aspect, in certain implementations of the second aspect, the first image processing model includes a first frequency separation module, a first low-frequency processing module and a first frequency fusion module, wherein the first frequency separation module is used to perform frequency separation processing on multiple frames of sample images respectively, and output the first high-frequency component and the first low-frequency component of each frame of the sample image in the multiple frames of sample images; the first low-frequency processing module is used to perform fusion processing on the first low-frequency components of the multiple frames of sample images, and output the fused first low-frequency component; the first frequency fusion module is used to perform frequency fusion processing on the fused first low-frequency component and the first high-frequency component of at least one frame of sample image respectively, and output the first enhanced image corresponding to at least one frame of sample image.

[0036] In combination with the second aspect, in certain implementations of the second aspect, the weight value of the loss function corresponding to the high-frequency area in the first enhanced image corresponding to at least one frame of sample image is greater than the weight value of the loss function corresponding to the low-frequency area in the first enhanced image corresponding to at least one frame of sample image.

[0037] According to the solution of the embodiment of the present application, by setting a higher weight value of the loss function for the high-frequency area, the model pays more attention to high-frequency difficult samples, or in other words, the model is guided to pay more attention to high-frequency details, so as to improve the performance of the trained model in processing high-frequency details.

[0038] In combination with the second aspect, in certain implementations of the second aspect, the weight map is determined based on the high-frequency weight map, and the high-frequency weight map is used to indicate the difference between the weight value of the loss function corresponding to the high-frequency area and the weight value of the loss function corresponding to the low-frequency area.

[0039] For example, the weight graph w satisfies the following formula:

[0040] w=1+w hf ;

[0041] w hf Represents the high-frequency weight map.

[0042] That is to say, the weight value of the loss function corresponding to the low-frequency area is 1, and the weight value of the loss function corresponding to the high-frequency area is the sum of 1 and the value in the high-frequency weight map.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the high-frequency components of the target image are extracted by a high-pass filter, and the obtained high-frequency components are used as the high-frequency weight map w hf .

[0044] For example, w hf Satisfies the following formula:

[0045] w hf =ifft(HP(fft(GT)));

[0046] Among them, fft() represents fast Fourier transform, ifft() represents inverse fast Fourier transform, and HP() represents high-pass filter.

[0047] In combination with the second aspect, in certain implementations of the second aspect, the difference between the first enhanced image corresponding to at least one frame of sample image and the target image corresponding to at least one frame of sample image includes: the difference between some pixels in the first enhanced image corresponding to at least one frame of sample image and some pixels in the target image corresponding to at least one frame of sample image.

[0048] According to the solution of the embodiment of the present application, the difference between some pixels in the first enhanced image and the target image is used as the difference between the first enhanced image and the target image, which can reduce the amount of calculation; at the same time, some pixels can be selected as needed to calculate the value of the loss function, which can enable the model to focus only on the required part of the pixels, thereby better meeting different needs.

[0049] In combination with the second aspect, in some implementations of the second aspect, some pixels include pixels whose pixel values ​​are less than or equal to a first threshold.

[0050] According to the solution of the embodiment of the present application, when calculating the value of the loss function of the first image processing model, only pixels with pixel values ​​less than or equal to the first threshold are considered, and pixels with higher pixel values ​​are not considered, that is, the image is highlighted and suppressed to reduce the focus on the highlight area, which is conducive to restoring the details of the dark area.

[0051] Pixels with values ​​higher than 4096 collected by the sensor are all white and appear as bright pixels in the image.

[0052] For example, the first threshold may be 4096.

[0053] Exemplarily, the portion of pixels may be obtained through a clipping operation.

[0054] In combination with the second aspect, in some implementations of the second aspect, the loss function is determined based on at least two of the following: content loss, structural similarity loss, contextual bilateral loss, or color loss.

[0055] That is, the loss function of the first image processing model can be determined based on at least two of the following: content loss between the first enhanced image and the target image, SSIM loss between the first enhanced image and the target image, cobi loss between the first enhanced image and the target image, and color loss between the first enhanced image and the target image.

[0056] In an embodiment of the present application, constructing a content loss can perform pixel-level constraints on the enhanced image output by the model to ensure the fidelity of the content, that is, to ensure the accuracy of the content in the enhanced image output by the model; constructing a structural similarity loss can improve the general texture of the output image; constructing a contextual bilateral loss can increase the weak contrast texture details of the enhanced image output by the model; constructing a color loss can constrain the color of the output enhanced image and correct the color cast problem of the output enhanced image; and using a multi-index fusion loss function can further enhance the details of dark areas and reduce purple fringing artifacts, that is, reduce the artificial distortion generated by processing algorithms such as cameras or software, and avoid high-frequency texture blurring and color distortion.

[0057] In conjunction with the second aspect, in some implementations of the second aspect, the loss function L satisfies the formula:

[0058] L=w*(α*L content +β*L SSIM +γ*L cobi +ρ*L color );

[0059] Among them, L content Indicates content loss, L SSIM represents the structural similarity loss, L cobi represents the contextual bilateral loss, L color represents color loss, α, β, γ and ρ are parameters, and w represents the weight map.

[0060] For example, the value range of α may be [0, 1], the value range of β may be [0, 1], the value range of γ may be [0, 1], and the value range of ρ may be [0, 1]. The specific values ​​may be adjusted according to actual conditions.

[0061] In combination with the second aspect, in certain implementations of the second aspect, the method also includes: the second image processing model performs forward propagation based on multiple frames of sample images to obtain a second enhanced image corresponding to at least one frame of sample image, and the second image processing model and the first image processing model share part of the network structure; and the target loss function is determined based on the loss function of the first image processing model, including: the target loss function is determined based on the loss function of the first image processing model, the loss function of the second image processing model, or the distillation loss between the second image processing model and the first image processing model, and the loss function of the second image processing model is used to calculate the difference between the second enhanced image corresponding to at least one frame of sample image and the target image corresponding to at least one frame of sample image.

[0062] According to the solution of the embodiments of the present application, a self-distillation method is used to enable the first image processing model with a low computational load to learn the knowledge learned by the second image processing model with a high computational load, further improving the imaging quality of the enhanced image and reducing model artifacts. Furthermore, compared to the training method based on knowledge distillation, the self-distillation method can train the first and second image processing models simultaneously, significantly reducing the training time required.

[0063] It should be understood that the “second” in “second image processing model” is only used to distinguish different image processing models and does not have any other limiting effect.

[0064] The first image processing model can be understood as a student model, and the second image processing model can be understood as a teacher model.

[0065] A second image processing model is constructed based on the first image processing model. The network structure of the second image processing model is more complex than that of the first image processing model, or in other words, the network capacity is larger.

[0066] In combination with the second aspect, in certain implementations of the second aspect, the distillation loss between the second image processing model and the first image processing model is determined based on at least one of the following: feature distillation loss or output distillation loss; wherein, the feature distillation loss is used to calculate the difference between the features of the intermediate layer of the first image processing model and the features of the intermediate layer of the second image processing model; the output distillation loss is used to calculate the difference between the first enhanced image and the second enhanced image.

[0067] In conjunction with the second aspect, in certain implementations of the second aspect, the distillation loss L between the second image processing model and the first image processing model is KD Satisfies the following formula:

[0068] L KD =a*L feature +b*L output ;

[0069] Among them, L feature represents the feature distillation loss, L output represents the output distillation loss, a represents L feature The weight coefficient, b represents L output The weight coefficient of .

[0070] According to the solution of the embodiment of the present application, the value of the distillation loss is jointly determined based on the feature distillation loss and the output distillation loss, so that the first image processing model can learn both the output of the second image processing model and the hidden features of the second image processing model, and better utilize the second image processing model, so that the first image processing model with a small amount of computation can achieve the effect of the second image processing model with a large amount of computation.

[0071] In conjunction with the second aspect, in some implementations of the second aspect, the target loss function satisfies the following formula:

[0072] L target =L KD +L1+L2;

[0073] Among them, L target represents the target loss function, L KD represents the distillation loss between the first image processing model and the second image processing model, L1 represents the loss function of the first image processing model, and L2 represents the loss function of the second image processing model.

[0074] In combination with the second aspect, in some implementations of the second aspect, the second image processing model includes a second frequency separation module, a second low-frequency processing module and a second frequency fusion module, wherein: the second frequency separation module is used to perform frequency separation processing on multiple frames of sample images respectively, and output the second high-frequency component and the second low-frequency component of each frame of the sample image in the multiple frames of sample images; the second low-frequency processing module is used to perform fusion processing on the second low-frequency components of the multiple frames of sample images, and output the fused second low-frequency component; the second frequency fusion module is used to perform frequency fusion processing on the fused second low-frequency component and the second high-frequency component of at least one frame of sample image respectively, and output a second enhanced image corresponding to at least one frame of sample image; the first frequency separation module and the second frequency separation module share a network structure, and the first low-frequency processing module and the second low-frequency processing module share part of the network structure.

[0075] Furthermore, if the first image processing model includes a first high-frequency processing module, and the second image processing model includes a second high-frequency processing module, the first high-frequency processing module and the second high-frequency processing module share part of the network structure.

[0076] Furthermore, if the first image processing model includes a first input processing module and the second image processing model includes a second input processing module, the first input processing module and the second input processing module may also share a network structure.

[0077] The modules in the first image processing model and the modules in the second image processing model share a network structure, which can be understood as the modules in the second image processing model share the modules in the first image processing model.

[0078] For example, the first frequency separation module and the second frequency separation module share a network structure, that is, the first frequency separation module is the second frequency separation module.

[0079] For another example, the first input processing module and the second input processing module share a network structure, that is, the first input processing module is the second input processing module.

[0080] That the modules in the first image processing model and the modules in the second image processing model share part of the network structure can be understood as follows: the modules in the second image processing model share the modules in the first image processing model, and the modules in the second image processing model have other network structures.

[0081] For example, the second low-frequency processing module shares part of its network structure with the first low-frequency processing module. The second low-frequency processing module shares the first low-frequency processing module, and one or more convolutional layers are added after the first low-frequency processing module. In other words, the first low-frequency processing module and the one or more convolutional layers added after the first low-frequency processing module together constitute the second low-frequency processing module.

[0082] For another example, the second high-frequency processing module shares part of the network structure with the first high-frequency processing module, the second high-frequency processing module shares the first high-frequency processing module, and one or more convolutional layers are added after the first high-frequency processing module. That is, the first high-frequency processing module and the one or more convolutional layers added after the first high-frequency processing module jointly constitute the second high-frequency processing module.

[0083] According to the scheme of the embodiments of the present application, the convolutional layer is added in the feature processing process of the low-frequency processing module or the high-frequency processing module of the second image processing model, which can improve the performance of the second image processing model, so that the first image processing model can learn more optimal performance.

[0084] In combination with the second aspect, in some implementations of the second aspect, the features of the intermediate layer of the first image processing model include features of the first frequency fusion processing module, and the features of the intermediate layer of the second image processing model include features of the second frequency fusion processing module.

[0085] According to the scheme of the embodiments of the present application, the knowledge distillation is performed based on the features of the frequency fusion module, which is beneficial to improve the distillation effect and enable the first image processing model to better learn the features of the second image processing model.

[0086] In a third aspect, a method for training an image processing model is provided. The method includes: obtaining a training sample, the training sample including a plurality of sample images and a target image corresponding to at least one sample image in the plurality of sample images, a signal-to-noise ratio of the target image being higher than that of the at least one sample image; and training a teacher network and a student network based on the training sample, and taking the trained student network as a target image processing model. The teacher network and the student network share part of the network structure, and the teacher network and the student network include a frequency separation module, a low-frequency processing module, and a frequency fusion module. The frequency separation module is configured to perform frequency separation processing on the plurality of sample images respectively, and output high-frequency components and low-frequency components of each sample image in the plurality of sample images. The low-frequency processing module is configured to perform fusion processing on the low-frequency components of the plurality of sample images, and output fused low-frequency components. The frequency fusion module is configured to perform frequency fusion processing on the fused low-frequency components and the high-frequency components of the at least one sample image respectively, and output an enhanced image corresponding to the at least one sample image.

[0087] According to the scheme of the embodiments of the present application, the first image processing model with small calculation amount learns the knowledge learned by the second image processing model with large calculation amount in a self-distillation manner, which further improves the imaging quality of the enhanced image and reduces the artifacts of the model. At the same time, compared with the training method of knowledge distillation, the self-distillation manner can train the first image processing model and the second image processing model at the same time, which greatly reduces the time required for training.

[0088] With reference to the third aspect, in some implementations of the third aspect, the training of the teacher network and the student network based on the training sample comprises: training the teacher network and the student network based on a value of a target loss function calculated based on the training sample, the target loss function being determined according to a loss function of the teacher network, a loss function of the student network and a distillation loss between the teacher network and the student network.

[0089] With reference to the third aspect, in some implementations of the third aspect, the distillation loss is determined according to at least one of: a feature distillation loss or an output distillation loss; wherein the feature distillation loss is used to calculate a difference between features of an intermediate layer of the teacher network and features of an intermediate layer of the student network; and the output distillation loss is used to calculate a difference between an output of the teacher network and an output of the student network.

[0090] With reference to the third aspect, in some implementations of the third aspect, the features of the intermediate layer of the student network comprise features of a frequency fusion processing module of the student network, and the features of the intermediate layer of the teacher network comprise features of a frequency fusion processing module of the teacher network.

[0091] According to the scheme of the embodiments of the present application, knowledge distillation is performed based on the features of the frequency fusion module, which is beneficial to improve the distillation effect and enable the student network to better learn the features of the teacher network.

[0092] With reference to the third aspect, in some implementations of the third aspect, the distillation loss L KD satisfies the following formula:

[0093] L KD = a * L feature + b * L output ;

[0094] wherein L feature denotes the feature distillation loss, L output denotes the output distillation loss, a denotes a weight coefficient of L feature , and b denotes a weight coefficient of L output .

[0095] According to the scheme of the embodiments of the present application, the value of the distillation loss is determined according to the feature distillation loss and the output distillation loss, so that the student network can learn both the output of the teacher network and the hidden features of the teacher network, better utilize the teacher network, and enable the student network with small calculation amount to achieve the effect of the teacher network with large calculation amount.

[0096] In combination with the third aspect, in certain implementations of the third aspect, the frequency separation module of the teacher network and the frequency separation module of the student network share a network structure, and the low-frequency processing module of the teacher network and the low-frequency processing module of the student network share part of the network structure.

[0097] In a fourth aspect, an image processing method is provided, which includes: performing frequency separation processing on multiple frames of images, outputting high-frequency components and low-frequency components of each frame of the multiple frames; performing fusion processing on the low-frequency components of the multiple frames of images, and outputting the fused low-frequency components; performing frequency fusion processing on the fused low-frequency components and the high-frequency components of at least one frame of the multiple frames, and outputting an enhanced image corresponding to the at least one frame of the images.

[0098] According to the solution of the embodiment of the present application, the different frequency components of the multi-frame images are extracted, and the high-frequency components are relatively separated, ensuring that the high-frequency information in the image is not affected by the inter-frame misalignment problem, retaining good high-frequency details, avoiding the problem of blurring of the enhanced image, and fusing the low-frequency components of the multi-frame images to obtain the low-frequency information in the multi-frame images, achieving inter-frame complementarity, solving problems such as color cast and general texture blur, improving the denoising effect, ensuring the authenticity of the picture color, enhancing the texture details of the image, and improving the imaging quality. Moreover, the solution in the embodiment of the present application separates the high-frequency components so that the high-frequency components are not affected by the inter-frame misalignment problem, and thus there is no need to use a deeper convolutional layer to align the multi-frame images, reducing the computational overhead and memory overhead during model operation, and improving the model's running speed.

[0099] In combination with the fourth aspect, in certain implementations of the fourth aspect, the method also includes: performing convolution processing on the high-frequency components of at least one frame of images in the multiple frames respectively; and performing frequency fusion processing on the fused low-frequency components and the high-frequency components of at least one frame of images in the multiple frames, and outputting an enhanced image corresponding to the at least one frame of images, including: performing frequency fusion processing on the fused low-frequency components and the high-frequency components after convolution processing of at least one frame of images respectively, and outputting an enhanced image corresponding to the at least one frame of images.

[0100] In combination with the fourth aspect, in certain implementations of the fourth aspect, frequency separation processing is performed on multiple frames of images respectively, and the high-frequency components and low-frequency components of each frame of the multiple frames are output, including: frequency separation processing is performed on multiple frames of images respectively through wavelet transform, and the high-frequency components and low-frequency components of each frame of the multiple frames are output.

[0101] In combination with the fourth aspect, in certain implementations of the fourth aspect, the low-frequency components of multiple frames of images are fused and the fused low-frequency components are output, including: serially merging the low-frequency components of multiple frames of images; performing convolution processing on the result of the serial merging and outputting the fused low-frequency components.

[0102] In combination with the fourth aspect, in certain implementations of the fourth aspect, frequency fusion processing is performed on the fused low-frequency component and the high-frequency component of at least one frame of the multiple frames, and an enhanced image corresponding to the at least one frame of the image is output, including: serially merging the fused low-frequency component and the high-frequency component of at least one frame of the image, respectively; performing convolution processing on the result of the serial merging, and outputting an enhanced image corresponding to the at least one frame of the image.

[0103] In combination with the fourth aspect, in certain implementations of the fourth aspect, frequency fusion processing is performed on the fused low-frequency component and the high-frequency component after convolution processing of at least one frame of images in multiple frames, and an enhanced image corresponding to at least one frame of image is output, including: serially merging the fused low-frequency component and the high-frequency component after convolution processing of at least one frame of image; convolution processing is performed on the result of the serial merging, and an enhanced image corresponding to at least one frame of image is output.

[0104] In combination with the fourth aspect, in certain implementations of the fourth aspect, the method further includes: preprocessing the multiple frames of initial images and outputting the multiple frames of images.

[0105] In combination with the fourth aspect, in certain implementations of the fourth aspect, multiple frames of initial images are preprocessed and multiple frames of images are output, including: feature extraction of the multiple frames of initial images, and using the extracted feature maps of the multiple frames of initial images as the multiple frames of images.

[0106] In a fifth aspect, a training device for an image processing model is provided, which includes a module / unit for executing the method in any one of the implementation modes of the second aspect or the third aspect.

[0107] In a sixth aspect, an image processing device is provided, which includes a module / unit for executing the method in any one of the implementation modes in the fourth aspect.

[0108] It should be understood that the expansion, limitation, explanation and description of the relevant content in the above-mentioned first aspect also apply to the same content in the second aspect, third aspect, fourth aspect, fifth aspect and sixth aspect.

[0109] In the seventh aspect, a training device for an image processing model is provided, which includes: a memory for storing programs; a processor for executing the programs stored in the memory, and when the program stored in the memory is executed, the processor is used to execute the method in any one of the implementation methods of the second aspect or the third aspect.

[0110] The processor in the seventh aspect can be a central processing unit (CPU) or a combination of a CPU and a neural network operation processor. The neural network operation processor can include a graphics processing unit (GPU), a neural-network processing unit (NPU), a tensor processing unit (TPU), and the like. The TPU is a fully customized artificial intelligence accelerator application-specific integrated circuit (ASIC) for machine learning by Google.

[0111] In an eighth aspect, an image processing apparatus is provided. The apparatus includes a memory configured to store a program, and a processor configured to execute the program stored in the memory. When the program stored in the memory is executed, the processor is configured to perform the method in any one of the implementation manners of the fourth aspect.

[0112] The processor in the eighth aspect can be a central processing unit (CPU) or a combination of a CPU and a neural network operation processor. The neural network operation processor can include a graphics processing unit (GPU), a neural-network processing unit (NPU), a tensor processing unit (TPU), and the like. The TPU is a fully customized artificial intelligence accelerator application-specific integrated circuit (ASIC) for machine learning by Google.

[0113] In a ninth aspect, a computer readable medium is provided. The computer readable medium stores program codes for execution by an apparatus. The program codes include codes for performing the method in any one of the implementation manners of the second aspect to the fourth aspect.

[0114] In a tenth aspect, a computer program product containing instructions which, when the computer program product is run on a computer, cause the computer to perform the method in any one of the implementation manners of the second aspect to the fourth aspect.

[0115] In an eleventh aspect, a chip is provided. The chip includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface and performs the method in any one of the implementation manners of the second aspect to the fourth aspect.

[0116] Optionally, as an implementation manner, the chip can further include a memory. The memory stores instructions. The processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform the method in any one of the implementation manners of the second aspect to the fourth aspect.

[0117] The chip may specifically be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0118] In the twelfth aspect, an electronic device is provided, which includes the device in any one of the implementation methods of the above-mentioned fifth to eighth aspects.

[0119] In the thirteenth aspect, a mobile terminal is provided, which includes the device in any one of the implementation methods of the fifth aspect or the eighth aspect.

[0120] In the fourteenth aspect, a cloud is provided, which includes the device in any one of the implementation methods of the fifth aspect or the eighth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 This is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0122] Figure 2 is a schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0123] Figure 3 This is a schematic diagram of a chip hardware structure provided by an embodiment of the present application;

[0124] Figure 4 This is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0125] Figure 5 is a schematic block diagram of an image processing model provided in an embodiment of the present application;

[0126] Figure 6 is a schematic block diagram of another image processing model provided in an embodiment of the present application;

[0127] Figure 7 is a schematic flow chart of a training method for an image processing model provided in an embodiment of the present application;

[0128] Figure 8 is a schematic block diagram of a self-distillation network provided in an embodiment of the present application;

[0129] Figure 9 is a schematic block diagram of an image processing model provided in an embodiment of the present application;

[0130] Figure 10 is a schematic block diagram of a self-distillation network provided in an embodiment of the present application;

[0131] Figure 11 is a schematic diagram of processing effects of different models provided by an embodiment of the present application;

[0132] Figure 12 is a schematic diagram of processing effects of models obtained by different training methods provided by an embodiment of the present application;

[0133] Figure 13 is a schematic diagram of processing effects of different models provided by an embodiment of the present application;

[0134] Figure 14 is a schematic diagram of processing effects of different models provided by an embodiment of the present application;

[0135] Figure 15 is a schematic diagram of processing effects of an image processing model provided by an embodiment of the present application;

[0136] Figure 16 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

[0137] Figure 17 is a schematic block diagram of a training device of an image processing model provided by an embodiment of the present application;

[0138] Figure 18 is a schematic block diagram of an image processing device provided by an embodiment of the present application;

[0139] Figure 19 is a schematic block diagram of a training device of an image processing model provided by an embodiment of the present application;

[0140] Figure 20 is a schematic block diagram of an image processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0141] The technical solutions in the present application will be described below with reference to the drawings.

[0142] The solutions of the embodiments of the present application can be applied to fields requiring improved video quality, such as monitoring and terminal playing. The intelligent city and terminal playing scenarios will be briefly introduced respectively.

[0143] Monitoring:

[0144] The monitoring scenarios include intelligent city, field monitoring, indoor monitoring, outdoor monitoring, in-vehicle monitoring, etc. In the intelligent city scenario, multiple attribute recognitions are required, such as pedestrian attribute recognition and cycling attribute recognition, and the imaging quality of the monitoring picture has a significant impact on the accuracy of attribute recognition. If the collected monitoring picture has problems such as low resolution, poor contrast, and loss of image details, the subsequent recognition effect will be affected.

[0145] Processing surveillance videos using the image processing model of the present application significantly improves the video's imaging quality, better meeting the clarity requirements of subsequent video processing operations, and thus improving the accuracy of attribute recognition. Furthermore, the quality of the resulting surveillance videos processed using the image processing model of the present application better meets the requirements of surveillance personnel, making it easier for them to view and obtain valuable information.

[0146] Terminal playback:

[0147] When users play videos on terminal devices (for example, mobile phones, smart cars, projection screens) or electronic devices such as TVs, enhancing the video can significantly improve the image quality of the video and enhance the user experience.

[0148] Electronic devices (e.g., mobile phones, smart cars, and projection screens) use the image processing model of the embodiments of the present application to enhance the video to be processed, for example, perform video noise reduction processing, output the enhanced video, and improve the user experience.

[0149] The method and apparatus provided in the embodiments of the present application can also be used to expand the training database, such as Figure 1 The I / O interface 112 of the execution device 110 shown can send the image frames processed by the execution device and the image frames to be processed input by the user as training data pairs to the database 130, so that the training data maintained by the database 130 is richer, thereby providing richer training data for the training work of the training device 120.

[0150] The following describes the method provided by this application from the perspectives of model training and model application:

[0151] The training method of the image processing model provided in the embodiment of the present application relates to the processing of computer vision, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, and the training data (such as the multi-frame sample images in the present application and the target image corresponding to at least one frame of the multi-frame sample images) is subjected to symbolic and formal intelligent information modeling, extraction, preprocessing, training, etc., and finally a trained image processing model is obtained; and the image processing method provided in the embodiment of the present application can use the above-mentioned trained image processing model to input input data (such as the initial video or multi-frame initial images in the present application) into the trained image processing model to obtain output data (such as the enhanced video or enhanced image in the present application). It should be noted that the training method of the image processing model and the image processing method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0152] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiments of the present application will be introduced first.

[0153] (1) Neural network

[0154] The neural network can be composed of neural units, and the neural unit can refer to an operation unit with x s and intercept 1 as input. The output of the operation unit can be:

[0155]

[0156] wherein s = 1, 2, … n, n is a natural number greater than 1, W s is the weight of x s , b is the bias of the neural unit. f is the activation function of the neural unit, which is used to perform nonlinear transformation on the features obtained in the neural network, and convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. The neural network is a network formed by connecting many single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0157] (2) Deep neural network

[0158] Deep neural network (DNN), also known as multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer.

[0159] Although DNN looks very complex, in terms of the work of each layer, it is actually not complex. In simple terms, it is expressed as the following linear relationship expression: wherein is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and a() is the activation function. Each layer only performs linear transformation on the input vector After such a simple operation, the output vector is obtained. Due to the large number of DNN layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0160] In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as

[0161] It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0162] (3) Convolutional Neural Network

[0163] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which is composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as position-independent feature extraction. Convolution kernels can be formalized as matrices of random size, and during CNN training, the kernels can be learned to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0164] (4) Loss function

[0165] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually a process of optimization, which is to pre-configure the parameters of each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss as much as possible.

[0166] (5) Backpropagation algorithm

[0167] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial network model during training, reducing the model's reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters in the initial model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain optimal model parameters, such as the weight matrix.

[0168] (6) Image / Video Enhancement

[0169] Image / video enhancement refers to actions performed on images / videos to improve image quality. For example, enhancement processing includes super-resolution, noise reduction, sharpening, or demosaicing.

[0170] The following describes the system architecture provided by the embodiments of the present application.

[0171] See also Figure 1 , an embodiment of the present application provides a system architecture 100. As shown in system architecture 100, a data acquisition device 160 is used to acquire training data. For example, in the embodiment of the present application, the training data may include: multiple frames of sample images and a target image corresponding to at least one of the multiple frames of sample images; after acquiring the training data, the data acquisition device 160 stores the training data in a database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0172] The training device 120 is described below based on the training data to obtain the target model / rule 101. Illustratively, the training device 120 processes the multi-frame sample image, compares the output enhanced image with the target image, until the difference between the output enhanced image and the target image of the training device 120 is less than a certain threshold, thereby completing the training of the target model / rule 101. The specific description is described in detail in the training method below.

[0173] The target model / rule 101 can be used to implement the image processing method provided by the embodiments of the present application, that is, the video or image to be processed is input into the target model / rule 101 after relevant preprocessing, and the enhanced video or image can be obtained. The target model / rule 101 in the embodiments of the present application can be a neural network. It should be noted that in actual application, the training data maintained in the database 130 may not all come from the collection of the data collection device 160, but may also be received from other devices. In addition, it should be noted that the training device 120 may not completely train the target model / rule 101 based on the training data maintained in the database 130, but may also obtain training data from the cloud or other places for model training. The above description should not be regarded as a limitation of the embodiments of the present application.

[0174] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in the figure. The execution device 110 can be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR), a vehicle-mounted terminal, a television, etc., and can also be a server or a cloud, etc. Figure 1 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. The user can input data to the I / O interface 112 through the client device 140, and the input data in the embodiments of the present application can include video or image to be processed. Figure 1 During the preprocessing of the input data by the execution device 110, or during the calculation and other related processing of the calculation module 111 of the execution device 110, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0175]

[0176] ​Finally, the I / O interface 112 returns the processing result, such as the enhanced video obtained above, to the client device 140 for providing to the user.

[0177] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0178] In the attached Figure 1 In the case shown in FIG, the user can manually input data, which can be operated through the interface provided by I / O interface 112. In another case, client device 140 can automatically send input data to I / O interface 112. If the automatic transmission of input data by client device 140 requires user authorization, the user can set the corresponding permissions in client device 140. The user can view the results output by execution device 110 on client device 140, which can be presented in the form of display, sound, action, etc. Client device 140 can also serve as a data acquisition terminal, collecting input data input into I / O interface 112 and output results from I / O interface 112 as new sample data and storing them in database 130. Of course, the collection can also be performed without client device 140, and instead the input data input into I / O interface 112 and output results from I / O interface 112 as new sample data can be directly stored in database 130 by I / O interface 112.

[0179] It is worth noting that the Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, Figure 1 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0180] like Figure 1 As shown, the target model / rule 101 is obtained through training by the training device 120. In the embodiment of the present application, the target model / rule 101 may be the image processing model in the present application.

[0181] CNN is a very common neural network. Figure 2This section focuses on a detailed introduction to the structure of CNNs. As mentioned in the previous basic concepts, a convolutional neural network is a deep neural network with a convolutional structure and a deep learning architecture. Deep learning architectures use machine learning algorithms to perform multiple levels of learning at different levels of abstraction. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images.

[0182] like Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210 , a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a fully connected layer 230 .

[0183] Convolutional layer / pooling layer 220:

[0184] Convolutional layer:

[0185] like Figure 2 The convolutional layer / pooling layer 220 shown may include layers 221-226. For example, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer. In another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0186] The following will take convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0187] Convolution layer 221 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from an input image matrix. A convolution operator is essentially a weight matrix, which is typically predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), thereby extracting specific features from the image. The size of the weight matrix should be related to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolution output with a single depth dimension. However, in most cases, a single weight matrix is ​​not used. Instead, multiple weight matrices of the same size (rows × columns) are applied, i.e., multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolved image, where the dimension is determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract edge information, another weight matrix can be used to extract specific colors, and yet another weight matrix can be used to blur unwanted noise in the image. The weight matrices have the same size (rows × columns), and the feature maps extracted by these weight matrices are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0188] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0189] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0190] Pooling layer:

[0191] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 2Each layer 221-226 illustrated in Figure 220 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In the image processing process, the sole purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain a smaller image. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolutional layer should be related to the image size, the operators in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0192] Fully connected layer 230:

[0193] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is still not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the fully connected layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the fully connected layer 230 may include multiple hidden layers (such as Figure 2 As shown in 231, 232 to 23n), the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0194] After the multiple hidden layers in the fully connected layer 230, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically for calculating the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 2 The propagation from 210 to 240 is forward propagation) and the reverse propagation (such as Figure 2 The propagation from 240 to 210 is called back propagation) and then the weight values ​​and biases of the aforementioned layers will be updated to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0195] It should be noted that if Figure 2The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, only including Figure 2 For example, the convolutional neural network used in the embodiment of the present application may only include the input layer 210, the convolution layer / pooling layer 220 and the output layer 240.

[0196] The following describes a chip hardware structure provided by an embodiment of the present application.

[0197] Figure 3 A chip hardware structure provided in an embodiment of the present application includes a neural network processor 50. The chip can be set as follows Figure 1 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 1 The training device 120 shown in FIG. 1 is used to complete the training work of the training device 120 and output the target model / rule 101. Figure 2 The algorithms of each layer in the convolutional neural network shown in the figure can be used in Figure 3 The image processing method and the training method of the image processing model in the embodiment of the present application can be implemented in the chip shown. Figure 3 is implemented in the chip shown.

[0198] The neural network processor 50 can be a neural network processing unit (NPU), a tensor processing unit (TPU), or a graphics processing unit (GPU), etc., any processor suitable for large-scale XOR operation processing. Take NPU as an example: the neural network processor NPU50 is mounted on the main central processing unit (CPU) (hostCPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU is the operation circuit 503, and the controller 504 controls the operation circuit 503 to extract data from the memory (weight memory or input memory) and perform operations. Among them, TPU is a dedicated integrated circuit for artificial intelligence accelerator fully customized by Google for machine learning.

[0199] In some implementations, the arithmetic circuit 503 includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 503 is a two-dimensional systolic array. The arithmetic circuit 503 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 503 is a general-purpose matrix processor.

[0200] For example, assume there are input matrix A, weight matrix B, and output matrix C. Operation circuit 503 retrieves the weight data of matrix B from weight memory 502 and caches it on each PE in operation circuit 503. Operation circuit 503 retrieves the input data of matrix A from input memory 501, performs matrix operations based on the input data of matrix A and the weight data of matrix B, and stores the partial or final matrix results in accumulator 508.

[0201] The vector calculation unit 507 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 507 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0202] In some implementations, the vector calculation unit 507 can store the processed output vector to the unified buffer 506. For example, the vector calculation unit 507 can apply a nonlinear function to the output of the operation circuit 503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 507 generates a normalized value, a merged value, or both. In some implementations, the vector calculation unit 507 stores the processed vector to the unified memory 506. In some implementations, the vector processed by the vector calculation unit 507 can be used as an activation input to the operation circuit 503, for example, for use in a subsequent layer in a neural network, such as Figure 2 As shown, if the current processing layer is hidden layer 1 (231), the vector processed by the vector calculation unit 507 can also be used in the calculation of hidden layer 2 (232).

[0203] The unified memory 506 is used to store input data and output data.

[0204] The weight data is directly stored in the weight memory 502 via the direct memory access controller (DMAC) 305. The input data is also stored in the unified memory 506 via the DMAC.

[0205] The bus interface unit (BIU) 310 is used for interaction between the DMAC and the instruction fetch buffer 509. The bus interface unit 501 is also used for the instruction fetch buffer 509 to obtain instructions from the external memory. The bus interface unit 501 is also used for the storage unit access controller 505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0206] DMAC is mainly used to store input data in the external memory DDR into the unified memory 506, or store weight data into the weight memory 502, or store input data into the input memory 501.

[0207] An instruction fetch buffer 509 connected to the controller 504 and used to store instructions used by the controller 504;

[0208] The controller 504 is used to call the instructions cached in the memory 509 to control the working process of the computing accelerator.

[0209] Generally, the unified memory 506, the input memory 501, the weight memory 502 and the instruction fetch memory 509 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0210] in, Figure 2 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 503 or the vector calculation unit 507. For example, the training method of the image processing model and the image processing method in the embodiment of the present application can be performed by the operation circuit 503 or the vector calculation unit 507.

[0211] like Figure 4As shown, the embodiments of the present application provide a system architecture 300. The system architecture includes a local device 301, a local device 302, an execution device 310 and a data storage system 350, wherein the local device 301 and the local device 302 are connected with the execution device 310 through a communication network.

[0212] The execution device 310 can be implemented by one or more servers. Optionally, the execution device 310 can be used in cooperation with other computing devices, such as data storage, routers, load balancers and the like. The execution device 310 can be arranged on one physical site or distributed on multiple physical sites. The execution device 310 can use data in the data storage system 350 or call program code in the data storage system 350 to implement the training method of the image processing model of the embodiments of the present application.

[0213] Users can operate respective user devices (such as the local device 301 and the local device 302) to interact with the execution device 310. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other types of cellular phones, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0214] Each user's local device can interact with the execution device 310 through a communication network of any communication mechanism / communication standard, which can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0215] In one implementation, the local device 301 and the local device 302 obtain the image processing model from the execution device 310, deploy the image processing model on the local device 301 and the local device 302, and use the image processing model for image enhancement.

[0216] In another implementation, the image processing model can be directly deployed on the execution device 310, and the execution device 310 obtains the video to be processed from the local device 301 and the local device 302 and uses the image processing model to perform image enhancement on the video to be processed.

[0217] The above execution device 310 can also be a cloud device, in which case the execution device 310 can be deployed in the cloud; or the above execution device 310 can also be a terminal device, in which case the execution device 310 can be deployed on the user terminal side, and the embodiments of the present application do not limit this.

[0218] Generally, the more complex a neural network is, the better the enhancement processing results. However, more complex neural network models also require more hardware computing resources, meaning greater computational overhead and runtime memory usage. Such neural network models place high demands on device hardware, resulting in slower execution speeds and difficulty in real-time applications. Overly large neural network models cannot even be deployed on devices with limited computing power. For example, the denoising convolutional neural network (DnCNN) model improves noise reduction accuracy through the use of 20 residual blocks, but also increases model complexity. When processing 4K video at a resolution of 4096x2304 at 30 frames per second, the peak computing power of existing mobile phones is typically 20 to 60 GMACs, with an available cache memory size of approximately 8MB. The computational power required for DnCNN is 1.6 TMACs, far exceeding the peak computing power of current mobile phones. For example, RDN+ consists of residual dense blocks (RDBs) with complex connections. For a 4K video with a resolution of 4096x2304, each RDB will occupy 3.6GB of cache, far exceeding the maximum cache capacity of current mobile phones. Moreover, even on a high-performance platform, DnCNN and RDN+ both require over 300ms to infer a single 4K image frame, far exceeding the time limit for real-time video inference (30 frames per second).

[0219] In the field of image enhancement, the adjacent frames of the current frame can be used to provide more effective information for the enhancement processing of the current frame. That is, the current frame and the adjacent frames are used as the input of the image processing model to improve the imaging quality of the current frame. However, there are problems such as pixel misalignment between the current frame and the adjacent frames. Deeper convolutional layers are still needed in the image processing model to solve the pixel misalignment problem and obtain better image enhancement effects. If the image processing model lacks the depth of the convolutional layer required for pixel alignment, the enhanced image may have problems such as ghosting, blurred high-frequency details, and color cast. Deeper convolutional layers require greater computing power and cannot be deployed on devices with weaker computing power.

[0220] Therefore, how to reduce the hardware computing resources required to run image processing models has become an urgent problem to be solved.

[0221] This application provides an image processing model that can be used to process images or videos, while improving the imaging quality, reducing the hardware computing resources required for model operation and increasing the model's operating speed.

[0222] The image processing model in the embodiments of the present application can be applied to various enhancement tasks such as image / video denoising, super-resolution, or high dynamic range, and can also be used in image / video restoration tasks or reconstruction tasks to improve the imaging quality of images / videos.

[0223] Specifically, the image processing model in the embodiments of the present application performs frequency separation processing on the input multiple frames of images, extracts different frequency components, fuses the low-frequency components of the multiple frames of images, and fuses the fused low-frequency components and the high-frequency components of the multiple frames of images to obtain the enhanced images.

[0224] Figure 5 A structural schematic diagram of an image processing model provided in the embodiments of the present application is shown in FIG. 5. Figure 5 As shown in the figure, the image processing model 500 includes an input processing module 510, a frequency separation module 520, a low-frequency processing module 530, a high-frequency processing module 540, and a frequency fusion module 550.

[0225] The image processing model can be deployed on an image processing device, that is, the image processing device can perform operations in the image processing model. The image processing device can be a cloud service device or a terminal device, for example, a computer, a server, a mobile phone, a television, or other devices with sufficient computing power to perform operations of the image processing model, or an image signal processor (ISP), or a system composed of a cloud service device and a terminal device. For example, the image processing model can be deployed on the execution device 110 in FIG. 1, Figure 1 the neural network processor 50 in FIG. 2, or the execution device 310 in FIG. 3, or a local device. Figure 3 Figure 4 For example, the image processing model can be deployed on the execution device 110 as shown in FIG. 1, and the image processing model is the target model / rule 101 in FIG. 1.

[0226] For example, the image processing model can be deployed on the execution device 110 as shown in FIG. 1, and the image processing model is the target model / rule 101 in FIG. 1. Figure 1 Figure 1 The input processing module 510 is configured to receive multiple frames of initial images, pre-process the multiple frames of initial images, and output multiple frames of images. The multiple frames of initial images are the multiple frames of input images in FIG. 1.

[0227] Optionally, the image processing model can be processed by a CPU, or jointly processed by a CPU and a GPU, or not processed by a GPU, but processed by other processors suitable for neural network calculation, which is not limited in the present application.

[0228] The input processing module 510 is configured to receive multiple frames of initial images, pre-process the multiple frames of initial images, and output multiple frames of images. The multiple frames of initial images are the multiple frames of input images in FIG. 1. Figure 5

[0229] ​​​The input processing module 510 is an optional module. In the case that the image processing model 500 does not include the input processing module 510 , multiple frames of initial images can be directly input into the frequency separation module 520 .

[0230] The frequency separation module 520 is used to perform frequency separation processing on the multiple frames of images, and output the high-frequency component and the low-frequency component of each frame of the multiple frames of images.

[0231] For example, the multi-frame image includes image A and image B. The frequency separation module 520 is used to perform frequency separation processing on image A and image B respectively to obtain high-frequency components and low-frequency components of image A and high-frequency components and low-frequency components of image B.

[0232] It should be noted that, when the image processing model 500 does not include the input processing module 510 , the multiple frames of images may be multiple frames of initial images.

[0233] The low-frequency processing module 530 is used to perform fusion processing on the low-frequency components of multiple frames of images and output the fused low-frequency components.

[0234] For example, the multiple frames of images include image A and image B. The low-frequency processing module 530 is configured to perform fusion processing on the low-frequency components of image A and the low-frequency components of image B, and output the fused low-frequency components.

[0235] The high frequency processing module 540 is configured to perform convolution processing on the high frequency components of at least one frame of the multiple frames of images, and output the high frequency components of the at least one frame of the images after the convolution processing.

[0236] For example, the multiple frames of images include image A and image B, and at least one frame of image is image A. The high-frequency processing module 540 is used to perform convolution processing on the high-frequency component of image A and output the processed high-frequency component of image A.

[0237] For another example, the multiple frames of images include image A and image B, and at least one frame of image is image A and image B. The high-frequency processing module 540 is used to perform convolution processing on the high-frequency component of image A and output the high-frequency component of the processed image A; and to perform convolution processing on the high-frequency component of image B and output the high-frequency component of the processed image B.

[0238] The high frequency processing module 540 is an optional module. In the case that the image processing model 500 does not include the high frequency processing module 540 , the high frequency components of at least one frame of image can be directly input into the frequency fusion module 550 .

[0239] By performing convolution processing on high-frequency components, it is beneficial to further restore high-frequency details to improve the image quality in high-frequency areas.

[0240] The frequency fusion module 550 is configured to perform frequency fusion processing on the fused low-frequency components and the high-frequency components of at least one frame of image, and output an enhanced image corresponding to the at least one frame of image.

[0241] When the model 500 includes a high-frequency processing module 540, the frequency fusion module 550 is used to perform frequency fusion processing on the fused low-frequency components and the high-frequency components after convolution processing of at least one frame of image, and output an enhanced image corresponding to at least one frame of image.

[0242] For example, multiple frames of images include image A and image B, and at least one frame of image is image A. The frequency fusion module 550 is used to perform frequency fusion processing on the fused low-frequency component and the high-frequency component after convolution processing of image A, and output an enhanced image corresponding to image A.

[0243] For another example, multiple frames of images include image A and image B, and at least one frame of image is image A and image B. The frequency fusion module 550 is used to perform frequency fusion processing on the fused low-frequency component and the high-frequency component of image A after convolution processing, and output an enhanced image corresponding to image A; the frequency fusion module 550 is used to perform frequency fusion processing on the fused low-frequency component and the high-frequency component of image B after convolution processing, and output an enhanced image corresponding to image B.

[0244] Figure 6 shows a structural diagram of an image processing model, Figure 6 The model in can be considered as Figure 5 A specific implementation of the model 500 in FIG.

[0245] The following combination Figure 6 Provide a detailed description of each module in the image processing model.

[0246] Input processing module: used to pre-process multiple frames of initial images and output multiple frames of images.

[0247] For example, the multiple frames of initial images may be multiple adjacent frames of images in a video. For another example, the multiple frames of initial images may be multiple images captured when a shutter button is pressed.

[0248] like Figure 6 As shown, the multiple initial images in the input image processing model are k initial images, that is, k frames of input, as shown in Figure 6 The input processing module may pre-process each of the k initial image frames and output the k frames of image.

[0249] Optionally, the preprocessing includes at least one of the following: downsampling or feature extraction.

[0250] Alternatively, it can be understood that the input processing module is specifically used to perform at least one of the following operations: downsampling processing or feature extraction.

[0251] For example, the input processing module performs downsampling processing on the multiple frames of initial images through a pixel shuffle operation to obtain downsampled images, which can be used as the multiple frames of images input to the frequency separation module.

[0252] That is to say, the structure of multiple frames of initial images is reorganized through pixel shuffle to obtain an image with the required resolution.

[0253] Images can also be represented as tensors. For example, the input image can be represented as a tensor with a resolution of M×N×C, which can be transformed into a tensor with a resolution of M / 2×N / 2×4C or a tensor with a resolution of M / 4×N / 4×16C after the pixel shuffle operation.

[0254] Exemplarily, the input processing module extracts features from the multiple frames of initial images through one or more convolution processes to obtain feature maps of the multiple frames of initial images. The feature maps of the multiple frames of initial images can be used as the multiple frames of images of the input frequency separation module.

[0255] This feature map can also be called a CNN feature map.

[0256] For example, the convolution process can be implemented by performing operations on a convolution layer. Alternatively, the convolution process can be implemented by performing operations on a convolution layer and an activation function after the convolution layer. For example, the activation function can be a rectified linear unit (ReLU) function.

[0257] like Figure 6 As shown, the multiple frames of initial images are represented as I i , that is I i Represents the i-th frame input, and the value range of i is {tk-1, tk,…, t}. After the input processing module is extracted to I i The feature map F i =Φ1(I i ), where Φ1 can be understood as a transformation function of one or more convolution processes.

[0258] Alternatively, the pre-processing operation includes down-sampling processing and feature extraction.

[0259] In this case, the input processing module performs feature extraction on multiple frames of initial images, which can be understood as: the input processing module performs feature extraction on the downsampled images to obtain multiple frames of feature maps, which are feature maps of the multiple frames of initial images.

[0260] For example, the input processing module downsamples the input tensor through the pixel shuffle operation to obtain a tensor of the corresponding resolution, and then performs feature extraction on the tensor of the corresponding resolution through one or more convolution operations.

[0261] It should be understood that the above pre-processing operations are merely illustrative, and the input processing module may also perform other pre-processing operations on multiple frames of initial images.

[0262] The input processing module is an optional module. In the case where the image processing model does not include the input processing module, multiple frames of initial images can be directly input into the frequency separation module.

[0263] Frequency separation module: used to perform frequency separation processing on multiple frames of images respectively, and output the high-frequency component and low-frequency component of each frame of the multiple frames.

[0264] The “high frequency” and “low frequency” in the embodiments of the present application are relative concepts, as long as the frequency of the high frequency component is higher than the frequency of the low frequency component.

[0265] The high-frequency component can also be called the high-frequency feature, and the low-frequency component can also be called the low-frequency feature.

[0266] The frequency separation module can perform frequency separation on multiple frames of images by frequency conversion.

[0267] Optionally, the frequency separation module is specifically configured to perform frequency separation processing on multiple frames of images using wavelet transform, outputting high-frequency components and low-frequency components of each frame in the multiple frames. These high-frequency components and low-frequency components provide basic features for subsequent targeted processing of different frequencies.

[0268] Exemplarily, the wavelet transform may be discrete wavelet transform (DWT).

[0269] For example, DWT is used to separate each frame image in multiple frames into a high-high (hh) component, a high-low (hl) component, a low-high (lh) component, and a low-low (ll) component, among which the hh component is used as the high-frequency component, and the hl component, lh component, and ll component are used as the low-frequency components.

[0270] In the case where the image processing model comprises the input processing module, the data input into the frequency separation module is the result output by the input processing module, for example, the feature map of the multi-frame initial image. That is, the high-frequency component H i and the low-frequency component L i of the image I i may be obtained by performing frequency separation processing on F i by the frequency separation module.

[0271] In the case where the image processing model does not comprise the input processing module, the data input into the frequency separation module can be the multi-frame initial image. That is, the high-frequency component H i and the low-frequency component L i of the image I i may be obtained by performing frequency separation processing on I i itself by the frequency separation module.

[0272] The low-frequency processing module: performs fusion processing on the low-frequency components of the multi-frame image, and outputs the fused low-frequency component.

[0273] Optionally, the low-frequency processing module is specifically configured to: concatenate the low-frequency components of the multi-frame image, perform convolution processing on the concatenated result, and output the fused low-frequency component.

[0274] In order to more clearly describe the image processing model of the present application, Figure 6 the concatenation operation is separately shown before the low-frequency processing module. It should be understood that the concatenation operation can be completed by the low-frequency processing module.

[0275] The concatenated result can be represented as [L t-k-1 , L t-k ,... L t ], and the fused low-frequency component can be represented as M = Φ2([L t-k-1 , L t-k ,... L t ]). Wherein, Φ2 can be understood as a transformation function of one or more convolution processing. Φ1 and Φ2 can be the same or different.

[0276] It should be noted that the convolution processing in the embodiments of the present application can be one time or multiple times.

[0277] The fusion processing on the low-frequency components of the multi-frame image includes fusion processing on the low-frequency components of part or all of the images in the multi-frame image. In order to facilitate description, only the fusion processing on the low-frequency components of all the images in the multi-frame image is described in the embodiments of the present application, which does not constitute a limitation on the solutions of the embodiments of the present application.

[0278] High-frequency processing module: used to perform convolution processing on the high-frequency components of at least one frame of images among multiple frames of images, and output the high-frequency components of at least one frame of images after convolution processing.

[0279] That is to say, the high-frequency processing module continues to extract features from the high-frequency components of at least one frame of image using one or more convolution processes to obtain the high-frequency components of at least one frame of image after convolution processing.

[0280] The number of multiple frames of images and the number of at least one frame of image may be the same or different. Figure 6 As shown in Figure 1, the input of the model is the k-frame initial image, and the output of the model is the corresponding k-frame enhanced image, that is, the output of the tk-1 frame to the t-frame output. Figure 6 The number of images output by the model can be other values. For example, the number of images output by the model can be 1, and the output image can be the t-th frame output. In other words, the number of input images and the number of output images of the model can be set as needed, and this embodiment of the application does not limit this.

[0281] The enhanced image corresponding to the output at least one frame of image may be one frame or multiple frames.

[0282] Optionally, the number of enhanced images corresponding to at least one frame of image is greater than or equal to 2.

[0283] Image I i The high frequency component H i The result after processing by the high frequency processing module can be expressed as H i '=Φ3(H i ). Φ3 can be understood as a transformation function of one or more convolution operations. Φ1, Φ2, and Φ3 can be the same or different.

[0284] The high frequency processing module is an optional module.

[0285] Frequency fusion module: performs frequency fusion processing on the fused low-frequency components and the high-frequency components of at least one frame of the multi-frame image, and outputs an enhanced image corresponding to the at least one frame of the image.

[0286] The enhanced image is the output image of the model.

[0287] Optionally, the frequency fusion module is specifically used to: respectively combine the fused low-frequency component and the high-frequency component of at least one frame of image in series; perform convolution processing on the result of the series combination, and output an enhanced image corresponding to at least one frame of image.

[0288] When the model includes a high-frequency processing module, the frequency fusion module is used to perform frequency fusion processing on the fused low-frequency component and the high-frequency component after convolution processing of at least one frame of image, and output an enhanced image corresponding to at least one frame of image.

[0289] Optionally, the frequency fusion module is specifically used to: respectively perform serial merging on the fused low-frequency component and the high-frequency component after convolution processing of at least one frame of image; perform convolution processing on the result of the serial merging, and output an enhanced image corresponding to at least one frame of image.

[0290] In order to more clearly describe the image processing model of this application, Figure 6 The serial merging operation is shown separately before the frequency fusion module. It should be understood that the serial merging operation can be completed by the frequency fusion module.

[0291] The fused low-frequency component M and image I i The high frequency component H i The result of the series combination can be expressed as [M,H i ], Image I i The corresponding enhanced image, i.e. image I i The corresponding output can be expressed as output i =Φ4([M,H i ]). Wherein, Φ4 can be understood as a transformation function of one or more convolution processes. Φ1, Φ2, Φ3, and Φ4 can be the same or different.

[0292] In the case where the model includes a high-frequency processing module, the high-frequency component of the at least one frame of image is the high-frequency component of the at least one frame of image processed by the high-frequency processing module.

[0293] The fused low-frequency component M and the processed image I i The high frequency component H i The result of the series combination can be expressed as [M,H i '], image I i The corresponding enhanced image, i.e. image I i The corresponding output can be expressed as output i =Φ4([M,H i ']).

[0294] The frequency fusion model can also be used to achieve output integration so that the resolution of the output image is consistent with the resolution of the input image, that is, the resolution of the enhanced image corresponding to at least one frame of image is consistent with the resolution of at least one frame of image.

[0295] According to the solution of the embodiment of the present application, the different frequency components of the multi-frame images are extracted, and the high-frequency components are relatively separated, ensuring that the high-frequency information in the image is not affected by the inter-frame misalignment problem, retaining good high-frequency details, avoiding the problem of blurring of the enhanced image, and fusing the low-frequency components of the multi-frame images to obtain the low-frequency information in the multi-frame images, achieving inter-frame complementarity, solving problems such as color cast and general texture blur, improving the denoising effect, ensuring the authenticity of the picture color, enhancing the texture details of the image, and improving the imaging quality. Moreover, the solution in the embodiment of the present application separates the high-frequency components so that the high-frequency components are not affected by the inter-frame misalignment problem, and thus there is no need to use a deeper convolutional layer to align the multi-frame images, reducing the computational overhead and memory overhead during model operation, and improving the model's running speed.

[0296] Furthermore, when at least one image frame consists of two or more frames, the image processing model is a multi-input, multi-output model that can simultaneously output enhanced images corresponding to the two or more frames. For example, in a video ISP, when processing a 30-fps video, if the model's output frame rate is set to 2, the model only needs to perform inference at a rate of 15 times per second to achieve the actual 30-fps effect. Similarly, when the model's output frame rate is set to 3, the model only needs to perform inference at a rate of 10 times per second to achieve the actual 30-fps effect. In other words, compared to a multi-input, single-output model, a multi-input, multi-output model can simultaneously process multiple frames in a single inference pass. This can moderately reduce the model's inference speed, thereby increasing network capacity, such as by using deeper convolutional layers to achieve better image quality. Alternatively, it can increase the effective information required for detail restoration, such as by inputting more images to achieve better image quality. This allows devices with less computational power to use larger convolutional neural networks to further improve image quality.

[0297] The image processing model in the embodiment of the present application can be trained by an existing training method, that is, a training sample is input into the image processing model for training to obtain a trained image processing model. The trained image processing model can be used to enhance the video or image to be processed.

[0298] Specifically, in the process of training the above-mentioned image processing model, a set of initial model parameters can be set for the image processing model, and then the parameters of the image processing model can be adjusted according to the difference between the sample image in the training sample and the GT corresponding to the sample image until the training termination condition is met. The parameters of the image processing model at this time are used as the final parameters of the image processing model. In this way, the training of the image processing model is completed, and a trained image processing model is obtained.

[0299] In order to improve the performance of the image processing model, an embodiment of the present application provides a training method for the image processing model.

[0300] Figure 7 The training method 700 of the image processing model provided in the embodiment of the present application is shown. The training method 700 can be executed by a training device for the image processing model. The training device for the image processing model can be a cloud service device or a terminal device, for example, a device with sufficient computing power such as a computer or a server to execute the training method for the image processing model, or a system composed of a cloud service device and a terminal device. For example, the training method 700 can be executed by Figure 1 Training equipment 120, Figure 3 Neural network processor 50 or Figure 4 The execution device 310 or local device executes.

[0301] For example, the training method 700 may be specifically performed as follows: Figure 1 The training device 120 shown in FIG. 1 is executed, and the training samples in the training process can be as follows: Figure 1 The training data maintained in the database 130 is shown.

[0302] Optionally, the training method 700 can be processed by a CPU, or by a CPU and a GPU together, or by using other processors suitable for neural network calculations without using a GPU, and this application does not impose any restrictions.

[0303] The training method 700 includes steps S710 to S730.

[0304] S710: Obtain training samples.

[0305] The training samples include multiple sample images and a target image corresponding to at least one of the multiple sample images. The signal-to-noise ratio of the target image corresponding to the at least one sample image is higher than that of the at least one sample image.

[0306] The target image corresponding to the sample image can also be understood as the true value of the supervisory label (groundtruth, GT) corresponding to the sample image.

[0307] The signal-to-noise ratio refers to the ratio of signal to noise, for example, the ratio of the power spectrum of the signal to the noise, or the ratio of the variance of the signal to the noise. The higher the signal-to-noise ratio, the better the image quality and the clearer the image.

[0308] In other words, GT can be understood as the high-definition image corresponding to the sample image.

[0309] S720: The first image processing model performs forward propagation based on multiple frames of sample images to obtain a first enhanced image corresponding to at least one frame of sample image.

[0310] The image processing model is used to enhance the image in the input model and output the enhanced image, that is, the enhanced image, or also called the output image.

[0311] It should be understood that the “first” in “first image processing model” is only used to distinguish different image processing models and does not have any other limiting function. The first image processing model can be used as follows: Figure 5 Image processing model 500 is shown.

[0312] Optionally, the first image processing model includes a first frequency separation module, a first low-frequency processing module and a first frequency fusion module, wherein the first frequency separation module is used to perform frequency separation processing on multiple frames of sample images respectively, and output the first high-frequency component and the first low-frequency component of each frame of the sample image in the multiple frames; the first low-frequency processing module is used to perform fusion processing on the first low-frequency components of multiple frames of sample images, and output the fused first low-frequency component; the first frequency fusion module is used to perform frequency fusion processing on the fused first low-frequency component and the first high-frequency component of at least one frame of sample image respectively, and output the first enhanced image corresponding to at least one frame of sample image.

[0313] Optionally, the first image processing model further includes a first high-frequency processing module. The first high-frequency processing module is configured to perform convolution processing on the first high-frequency component of at least one sample image among the multiple sample image frames, and the first frequency fusion model is configured to perform frequency fusion processing on the fused first low-frequency component and the convolution-processed first high-frequency component of the at least one image frame, and output a first enhanced image corresponding to the at least one sample image frame.

[0314] For other descriptions of the first image processing model, please refer to the model 500 mentioned above. That is, the input image in the model 500 is replaced with the sample image, and no further details are given here.

[0315] Specifically, multiple frames of sample images are used as input to the image processing model, and after forward propagation, an enhanced image corresponding to at least one frame of sample image is obtained.

[0316] Accordingly, multiple frames of sample images are used as input to the first image processing model, and after forward propagation, a first enhanced image corresponding to at least one frame of sample image can be obtained.

[0317] It should be understood that the term "first" is used only to define the result obtained by the first image processing model and does not have any other limiting effect. For example, the term "first" in "first enhanced image" only defines that the enhanced image is output by the first image processing model and does not have any other limiting effect. In other words, any image output by the first image processing model can be referred to as the first enhanced image.

[0318] At S730, parameters of the first image processing model are adjusted with the goal of reducing a target loss function to obtain a trained first image processing model. The target loss function is determined based on a loss function of the first image processing model. The loss function of the first image processing model is used to calculate a difference between a first enhanced image corresponding to at least one sample image frame and a target image corresponding to at least one sample image frame.

[0319] That is, in step S730, the loss function value of the first image processing model can be calculated based on the first enhanced image corresponding to the at least one sample image frame and the target image corresponding to the at least one sample image frame, thereby determining the value of the target loss function. The parameters of the first image processing model are adjusted with the goal of reducing the value of the target loss function until a trained first image processing model is obtained.

[0320] For example, the parameters of the first image processing model can be adjusted using a gradient descent method, that is, backpropagating the value of the target loss function to calculate the parameter gradient of the model, and adjusting the parameters in the first image processing model based on the parameter gradient until the training is completed.

[0321] Optionally, the difference between the first enhanced image corresponding to at least one frame of sample image and the target image corresponding to at least one frame of sample image includes: the difference between all pixels in the first enhanced image corresponding to at least one frame of sample image and all pixels in the target image corresponding to at least one frame of sample image.

[0322] In other words, the value of the loss function of the first image processing model can be calculated based on all pixels in the first enhanced image corresponding to the at least one frame of sample image and all pixels in the target image corresponding to the at least one frame of sample image.

[0323] Optionally, the difference between the first enhanced image corresponding to at least one frame of sample image and the target image corresponding to at least one frame of sample image includes: the difference between some pixels in the first enhanced image corresponding to at least one frame of sample image and some pixels in the target image corresponding to at least one frame of sample image.

[0324] In other words, the value of the loss function of the first image processing model can be calculated based on some pixels in the first enhanced image corresponding to the at least one frame of sample image and some pixels in the target image corresponding to the at least one frame of sample image.

[0325] Using the difference between some pixels in the first enhanced image and the target image as the difference between the first enhanced image and the target image can reduce the amount of calculation; at the same time, some pixels can be selected as needed to calculate the value of the loss function, which can enable the model to only focus on the required pixels and better meet different needs.

[0326] Optionally, some pixels include pixels whose pixel values ​​are less than or equal to a first threshold.

[0327] That is to say, when calculating the value of the loss function of the first image processing model, the image is highlighted, that is, pixels with higher pixel values ​​are not considered to reduce attention to the highlight area, which is conducive to restoring the details of the dark area.

[0328] Pixels with values ​​higher than 4096 collected by the sensor are all white and appear as bright pixels in the image.

[0329] For example, the first threshold may be 4096.

[0330] Exemplarily, the portion of pixels may be obtained through a clipping operation.

[0331] Specifically, a clip operation is performed on the first enhanced image corresponding to the at least one frame of sample image to obtain part of the pixels in the first enhanced image corresponding to the at least one frame of sample image; and a clip operation is performed on the target image corresponding to the at least one frame of sample image to obtain part of the pixels in the target image corresponding to the at least one frame of sample image.

[0332] For the convenience of description, the “first enhanced image corresponding to at least one frame of sample image” will be referred to as the “first enhanced image”, and the “target image corresponding to at least one frame of sample image” will be referred to as the “target image”.

[0333] For example, the result of the clipping operation can satisfy the following formula:

[0334] output'=clip(output,0,4096);

[0335] GT'=clip(GT,0,4096);

[0336] Wherein, output represents the enhanced image, output' represents the enhanced image after cropping, i.e., some pixels in the aforementioned enhanced image; GT' represents the GT after cropping, i.e., some pixels in the aforementioned target image, and clip() represents the cropping operation.

[0337] In this case, the value of the loss function can be calculated based on output' and GT'.

[0338] In one implementation, the loss function may be determined according to at least two of the following: content loss, structural similarity loss (SSIM loss), contextual bilateral loss (cobi loss), or color loss.

[0339] This loss function can also be called a multi-index loss function.

[0340] That is, the loss function of the first image processing model can be determined based on at least two of the following: content loss between the first enhanced image and the target image, SSIM loss between the first enhanced image and the target image, cobi loss between the first enhanced image and the target image, and color loss between the first enhanced image and the target image.

[0341] For the sake of brevity, the following describes the loss function of the image processing model. It should be understood that the following description of the loss function applies to the first image processing model. This can be accomplished by replacing "image processing model" with "first image processing model" and "enhanced image" with "first enhanced image."

[0342] Content loss uses GT to constrain the enhanced image output by the model at the pixel level to ensure the fidelity of the content, that is, to ensure the accuracy of the content in the enhanced image output by the model.

[0343] For example, the content loss can be determined by the L1 norm.

[0344] For example, the content loss value L content Satisfies the following formula:

[0345] L content =|output-GT|;

[0346] Alternatively, the content loss L content Satisfies the following formula:

[0347] L content =|output'-GT'|;

[0348] SSIM loss is used to improve the general texture of the output image.

[0349] For example, the structural similarity loss L SSIM Satisfies the following formula:

[0350] L SSIM=SSIM(output-GT);

[0351] Alternatively, the structural similarity loss L SSIM Satisfies the following formula:

[0352] L SSIM =SSIM(output'-GT');

[0353] Cobi loss can adopt the visually friendly VGG loss function to align VGG features to increase the weak contrast texture details of the enhanced image output by the model.

[0354] For example, the contextual bilateral loss L cobi Satisfies the following formula:

[0355] L cobi =cobi(output-GT);

[0356] Alternatively, the contextual bilateral loss L cobi Satisfies the following formula:

[0357] L cobi =cobi(output'-GT');

[0358] Color loss is used to constrain the color of the output image and correct the color cast problem of the output image.

[0359] For example, the color loss L color Satisfies the following formula:

[0360] L color =∠ RGB (output-GT);

[0361] Among them, ∠ RGB Indicates calculating the pixel's RGB vector angle.

[0362] Alternatively, the color loss L color Satisfies the following formula:

[0363] L color =∠ RGB (output'-GT');

[0364] Optionally, the loss function L satisfies the following formula:

[0365] L=α*L content +β*L SSIM +γ*L cobi +ρ*L color ;

[0366] Among them, α, β, γ, and ρ are parameters, representing the corresponding weight coefficients of content loss, SSIM loss, cobi loss, and color loss in the loss function, respectively. For example, the value range of α can be [0, 1], the value range of β can be [0, 1], the value range of γ can be [0, 1], and the value range of ρ can be [0, 1]. The specific values ​​can be adjusted according to actual conditions.

[0367] The loss function includes a weight value, and the weight value of the loss function corresponding to a portion or all of the regions in the first enhanced image corresponding to at least one frame of the sample image is indicated by a weight map.

[0368] That is to say, the weight values ​​corresponding to different regions of the enhanced image output by the model can be the same or different.

[0369] According to the solution of the embodiment of the present application, different weight values ​​of the loss function are set for different areas of the enhanced image output by the model, and higher weight values ​​are set for areas that require the model to pay more attention. This can guide the model to pay more attention to the specified areas and further improve the performance of the model in processing the specified areas.

[0370] Optionally, a weight value of the loss function corresponding to a high-frequency region in the first enhanced image corresponding to at least one frame of sample image is greater than a weight value of the loss function corresponding to a low-frequency region in the first enhanced image corresponding to at least one frame of sample image.

[0371] In other words, the weight value of the loss function corresponding to the high-frequency area in the enhanced image output by the model can be increased, that is, a higher weight value of the loss function is set for the high-frequency area in the enhanced image output by the model.

[0372] In this way, the model can pay more attention to high-frequency difficult samples, or in other words, guide the model to pay attention to high-frequency details, so as to improve the performance of the trained model in processing high-frequency details.

[0373] Optionally, the weight map is determined based on a high-frequency weight map, which is used to indicate the difference between the weight value of the loss function corresponding to the high-frequency area and the weight value of the loss function corresponding to the low-frequency area.

[0374] For example, the weight graph w satisfies the following formula:

[0375] w=1+w hf ;

[0376] w hf Represents the high-frequency weight map, and the "1" in the formula is an all-1 matrix.

[0377] That is to say, the weight value of the loss function corresponding to the low-frequency area is 1, and the weight value of the loss function corresponding to the high-frequency area is the sum of 1 and the value in the high-frequency weight map.

[0378] For example, the high-frequency components of GT are extracted by a high-pass filter, and the obtained high-frequency components are used as the high-frequency weight map w hf .

[0379] For example, w hf Satisfies the following formula:

[0380] w hf =ifft(HP(fft(GT)));

[0381] Wherein, fft() represents fast Fourier transform, ifft() represents inverse fast Fourier transform, and HP() represents high-pass filter. Optionally, the loss function L satisfies the following formula:

[0382] L=w*(α*L content +β*L SSIM +γ*L cobi +ρ*L color );

[0383] Furthermore, the loss function L satisfies the following formula:

[0384] L=(1+w hf )*(α*L content +β*L SSIM +γ*L cobi +ρ*L color );

[0385] w hf Represents a high-frequency weight map, where "1" in the formula is an all-1 matrix.

[0386] According to the solution of the embodiment of the present application, the use of a multi-index fusion loss function can further enhance the details of dark areas and reduce purple fringing artifacts, that is, reduce the artificial distortion generated by processing algorithms such as cameras or software, and avoid high-frequency texture blurring and color distortion.

[0387] In order to further improve the imaging quality of the video, the embodiment of the present application also provides a method for training an image processing model using self-distillation. The training process can also be understood as a model compression process.

[0388] The student network is distilled using self-distillation to enable it to learn the knowledge learned by the teacher network. The trained student network is used as the target image processing model to improve the imaging quality of the video.

[0389] In order to better illustrate the training method of the image processing model in the embodiment of the present application, Figure 8 Describe the self-distillation compression network used in the training process.

[0390] Figure 8 A structural schematic diagram of a self-distillation compression network 800 in an embodiment of the present application is shown.

[0391] The self-distillation compression network 800 includes a student network and a teacher network. The teacher network is constructed based on the student network and shares some of its structure. In other words, the teacher network and the student network share some of their structure.

[0392] like Figure 8 As shown, the student network includes a network sharing portion 811 and a student network non-sharing portion 812. The teacher network includes a network sharing portion 811 and a teacher network non-sharing portion 822.

[0393] The structure of the teacher network is more complex than that of the student network, and its image processing effect is better than that of the student network.

[0394] According to the value of the loss function of the student network 810 (such as Figure 8 The student loss in ), the value of the loss function of the teacher network (such as Figure 8 The student network and the teacher network are trained using the teacher loss in

[15] and the distillation loss between them.

[0395] like Figure 8 As shown in , the distillation loss is determined by feature distillation or output distillation. The distillation point of feature distillation is the middle layer of the network, that is, based on the features of the middle layer of the teacher network (such as Figure 8 The teacher features in the student network) have an impact on the features of the middle layer of the student network (such as Figure 8 The distillation point of output distillation is the output of the network, that is, based on the output of the teacher network (such as Figure 8 The output of the teacher in the student network (such as Figure 8 knowledge distillation based on the student output in

[15] .

[0396] The student loss is used to calculate the difference between the student output and the supervision label, and the teacher loss is used to calculate the difference between the teacher output and the supervision label.

[0397] The following combination Figure 8 The training method in the embodiment of the present application is described.

[0398] Optionally, method 700 also includes: a second image processing model performs forward propagation based on multiple frames of sample images to obtain a second enhanced image corresponding to at least one frame of sample image, and the second image processing model and the first image processing model share part of the network structure.

[0399] The target loss function is determined based on the loss function of the first image processing model, including: the target loss function is determined based on the loss function of the first image processing model, the loss function of the second image processing model, or the distillation loss between the second image processing model and the first image processing model, and the loss function of the second image processing model is used to calculate the difference between the second enhanced image corresponding to at least one frame of sample image and the target image corresponding to at least one frame of sample image.

[0400] That is, the first image processing model is used as the student network, and the second image processing model is built on the basis of the first image processing model, and the second image processing model is used as the teacher network. The network structure of the second image processing model is more complex than that of the first image processing model, or in other words, the network capacity is larger.

[0401] It should be understood that the “second” in “second image processing model” is only used to distinguish different image processing models and does not have any other limiting function. The second image processing model can be used as follows Figure 5 Image processing model 500 is shown.

[0402] That is, the structures of the first image processing model and the second image processing model may be the same.

[0403] Optionally, the second image processing model includes a second frequency separation module, a second low-frequency processing module and a second frequency fusion module.

[0404] Among them, the second frequency separation module is used to perform frequency separation processing on multiple frames of sample images respectively, and output the second high-frequency component and the second low-frequency component of each frame of sample images in the multiple frames of sample images; the second low-frequency processing module is used to perform fusion processing on the second low-frequency components of multiple frames of sample images, and output the fused second low-frequency component; the second frequency fusion module is used to perform frequency fusion processing on the fused second low-frequency component and the second high-frequency component of at least one frame of sample image respectively, and output the second enhanced image corresponding to at least one frame of sample image.

[0405] Optionally, the second image processing model further includes a second high-frequency processing module. The second high-frequency processing module is configured to perform convolution processing on the second high-frequency component of at least one sample image among the multiple sample image frames, and the second frequency fusion module is configured to perform frequency fusion processing on the fused second low-frequency component and the convolution-processed second high-frequency component of the at least one sample image frame, respectively, to output a second enhanced image corresponding to the at least one sample image frame.

[0406] For other descriptions of the second image processing model, please refer to the model 500 mentioned above. That is, the input image in the model 500 is replaced with the sample image, and no further details are given here.

[0407] For example, the multiple frames of sample images are input into the image processing model, and after forward propagation, at least one frame of sample image corresponding enhanced image is obtained.

[0408] Correspondingly, the multiple frames of sample images are input into the second image processing model, and after forward propagation, at least one frame of sample image corresponding second enhanced image is obtained.

[0409] It should be understood that "second" is only used to define the result processed by the second image processing model, and has no other limiting effect. For example, "second" in "second enhanced image" is only used to define that the enhanced image is output by the second image processing model, and has no other limiting effect. That is, the image output by the second image processing model can be called second enhanced image.

[0410] For ease of description, hereinafter, "at least one frame of sample image corresponding second enhanced image" is referred to as "second enhanced image".

[0411] As described above, the first image processing model and the second image processing model share part of the network structure. Specifically, part of the structure can be shared in any one or more of the frequency separation module, the low-frequency processing module, the frequency fusion module, the high-frequency processing module, or the input processing module.

[0412] Optionally, the first frequency separation module and the second frequency separation module share network structure, and the first low-frequency processing module and the second low-frequency processing module share part of the network structure.

[0413] Further, if the first image processing model includes a first high-frequency processing module and the second image processing model includes a second high-frequency processing module, the first high-frequency processing module and the second high-frequency processing module share part of the network structure.

[0414] Further, if the first image processing model includes a first input processing module and the second image processing model includes a second input processing module, the first input processing module and the second input processing module can also share network structure.

[0415] The modules in the first image processing model and the modules in the second image processing model share network structure can be understood as the modules in the second image processing model share the modules in the first image processing model.

[0416] For example, the first frequency separation module and the second frequency separation module share network structure, that is, the first frequency separation module is the second frequency separation module.

[0417] For another example, the first input processing module and the second input processing module share network structure, that is, the first input processing module is the second input processing module.

[0418] That the modules in the first image processing model and the modules in the second image processing model share part of the network structure can be understood as follows: the modules in the second image processing model share the modules in the first image processing model, and the modules in the second image processing model have other network structures.

[0419] For example, the second low-frequency processing module shares part of its network structure with the first low-frequency processing module. The second low-frequency processing module shares the first low-frequency processing module, and one or more convolutional layers are added after the first low-frequency processing module. In other words, the first low-frequency processing module and the one or more convolutional layers added after the first low-frequency processing module together constitute the second low-frequency processing module.

[0420] For another example, the second high-frequency processing module shares part of its network structure with the first high-frequency processing module. The second high-frequency processing module shares the first high-frequency processing module and adds one or more convolutional layers after the first high-frequency processing module. In other words, the first high-frequency processing module and the one or more convolutional layers added after the first high-frequency processing module together constitute the second high-frequency processing module.

[0421] In an embodiment of the present application, adding a convolutional layer to the feature processing process of the low-frequency processing module or the high-frequency processing module of the second image processing model can improve the performance of the second image processing model so that the first image processing model can learn better performance.

[0422] The target loss function is determined based on at least one of the following: a loss function of the first image processing model, a loss function of the second image processing model, or a distillation loss between the second image processing model and the first image processing model.

[0423] For example, the loss function of the first image processing model can employ the multi-metric loss function described in step S730. That is, the difference between the first enhanced image and the target image is determined based on the formula in step S730. Alternatively, the loss function of the first image processing model can be determined using other methods, as long as the value of the loss function of the first image processing model can indicate the difference between the first enhanced image and the target image.

[0424] For example, the loss function of the second image processing model can employ the multi-metric loss function described in step S730. That is, the difference between the second enhanced image and the target image is determined based on the formula in step S730. Alternatively, the loss function of the second image processing model can be determined using other methods, as long as the value of the loss function of the second image processing model can indicate the difference between the second enhanced image and the target image.

[0425] Optionally, the distillation loss between the second image processing model and the first image processing model is determined according to at least one of the following: feature distillation loss or output distillation loss.

[0426] The feature distillation loss is used to calculate the difference between the features of the intermediate layer of the first image processing model and the features of the intermediate layer of the second image processing model. The output distillation loss is used to calculate the difference between the first enhanced image and the second enhanced image.

[0427] The first image processing model and the second image processing model perform forward propagation based on the sample image, and the feature distillation loss can be determined according to the distance between the features of the intermediate layer of the first image processing model and the features of the intermediate layer of the second image processing model.

[0428] Optionally, the features of the middle layer of the first image processing model include features of the first frequency fusion processing module, and the features of the middle layer of the second image processing model include features of the second frequency fusion processing module.

[0429] Performing knowledge distillation based on the features of the frequency fusion module is beneficial to improving the distillation effect, so that the first image processing model can better learn the features of the second image processing model.

[0430] Exemplarily, the feature distillation loss can be calculated using L2 loss.

[0431] For example, the feature distillation loss L feature The following formula can be satisfied:

[0432] L feature =|T feature -S feature | 2 ;

[0433] Among them, T feature Represents the features of the middle layer of the second image processing model, S feature Represents the features of the intermediate layer of the first image processing model.

[0434] The first image processing model and the second image processing model perform forward propagation based on the sample image, and the output distillation loss can be determined according to the distance between the first enhanced image corresponding to the sample image and the second enhanced image corresponding to the sample image.

[0435] Exemplarily, the output distillation loss may be calculated using the L2 loss.

[0436] For example, the output distillation loss L output The following formula can be satisfied:

[0437] L output =|T output -Soutput | 2 ;

[0438] wherein T output represents the output of the second image processing model, i.e., the second enhanced image, S output represents the output of the first image processing model, i.e., the first enhanced image.

[0439] Optionally, the distillation loss L KD satisfies the following formula:

[0440] L KD = a * L feature + b * L output ;

[0441] wherein L feature represents the feature distillation loss, L output represents the output distillation loss, a represents a weight coefficient of L feature , and b represents a weight coefficient of L output . For example, a is in the range of [0, 1], and b is in the range of [0, 1], for example, a is 1, and b is 1. The values of a and b can be adjusted according to actual conditions.

[0442] The value of the distillation loss is determined according to the feature distillation loss and the output distillation loss, so that the first image processing model can learn the output of the second image processing model and the hidden features of the second image processing model, and better utilize the second image processing model, so that the first image processing model with small calculation amount can achieve the effect of the second image processing model with large calculation amount.

[0443] Optionally, the target loss function satisfies the following formula:

[0444] L target = L KD + L1+ L2;

[0445] wherein L target represents the target loss function, L1 represents the loss function of the first image processing model, and L2 represents the loss function of the second image processing model.

[0446] It should be understood that the weight coefficients of L KD , L1 and L2 in the above formula are 1, which is only for illustration, and other weight coefficients can also be set for L KD , L1 and L2.

[0447] In the solution of the embodiments of this application, self-distillation is used to enable the computationally less complex first image processing model to learn the knowledge learned by the computationally more complex second image processing model, further improving the quality of the enhanced image and reducing model artifacts. Furthermore, compared to training methods based on knowledge distillation, self-distillation allows for simultaneous training of both the first and second image processing models, significantly reducing training time.

[0448] The trained first image processing model can be deployed on a computing device for performing enhanced processing on images or videos.

[0449] Figure 9 A structural diagram of an image processing model provided in an embodiment of the present application is shown. Figure 9 The image processing model shown in can be viewed as Figure 5 or Figure 6 A specific implementation of the image processing model shown in FIG. Detailed description can be found in the model 500 in the previous text. In order to avoid repetition, Figure 9 Appropriate omissions are made when showing the model.

[0450] Figure 9 The input of the image processing model in includes two frames of images, and the output of the image processing model includes two frames of images. It should be understood that the fact that the number of images input to the model and the number of images output by the model are the same is merely an example, and the number of images input to the model and the number of images output by the model may also be different. Figure 9 The number of images input to the model and the number of images output by the model are only examples and do not limit the solutions of the embodiments of the present application.

[0451] like Figure 9 As shown, the multiple frames of initial images of the input image processing model include the t-1th frame input image and the tth frame input image. The t-1th frame input image and the tth frame input image can be two adjacent frames in the video sequence.

[0452] The following combination Figure 9 The processing flow of the image processing model in the embodiment of the present application is described.

[0453] (1) Preprocessing is performed on the input multiple frames of initial images respectively to obtain preprocessing results corresponding to the multiple frames of initial images.

[0454] Step (1) is an optional step. For a detailed description, please refer to the aforementioned model 500 and will not be repeated here.

[0455] Exemplarily, downsampling processing is performed on the input multiple frames of initial images respectively, and feature extraction is performed on the results after the downsampling processing to obtain multi-frame feature maps, which are input into the frequency separation module as multi-frame images.

[0456] For example, downsampling can be accomplished using a pixel shuffle operation, which does not lose information.

[0457] For example, feature extraction can be performed through one or more convolution operations.

[0458] For example, this step can be performed by Figure 5 Alternatively, this step can be performed by the input processing module 510 in Figure 9 The input processing module in is executed.

[0459] For example, Figure 9 As shown in Figure 1, the input processing module performs pixel shuffle operations on the input image of the t-1th frame and the input image of the tth frame respectively to achieve 4x downsampling, that is, converting the two input images with a resolution of M×N×C into two images with a resolution of M / 4×N / 4×16C. Then, a 64-channel 3×3 convolution operation is performed to extract features. Figure 9 As shown, the downsampled input image of frame t-1 is input into the convolution layer and the activation function layer following the convolution layer. The features of the input image of frame t-1 are extracted through the operations of the convolution layer and the activation function layer, thereby obtaining a feature map of the input image of frame t-1. The downsampled input image of frame t is input into the convolution layer and the activation function layer following the convolution layer. The features of the input image of frame t are extracted through the operations of the convolution layer and the activation function layer, thereby obtaining a feature map of the input image of frame t. For example, the activation function may be a ReLU function.

[0460] In the embodiment of the present application, the processing of multiple frames of images can be performed in parallel. For example, in step (1), features of multiple frames of initial images can be extracted in parallel.

[0461] (2) Perform frequency separation on multiple frames of images and output the high-frequency component and low-frequency component of each frame of the multiple frames.

[0462] The multiple frames of images are the preprocessing results of the multiple frames of initial images in step (1).

[0463] In the case where step (1) is not included, the multiple frames of images may be multiple frames of initial images.

[0464] Exemplarily, the pre-processing results corresponding to the multiple frames of images can be separated into hh components, hl components, lh components and ll components respectively by using discrete wavelet transformation (DWT). The hh components are taken as high-frequency components, and the hl components, lh components and ll components are taken as low-frequency components. That is, the hh components are input into the high-frequency processing module, and the other components are input into the low-frequency processing module. If the model does not include the high-frequency processing module, the hh components can also be directly input into the frequency fusion module.

[0465] wherein the wavelet basis of the DWT is Haar wavelet.

[0466] After the DWT separation, the resolution of each component is reduced to 1 / 2 of the resolution before the separation. For example, after the DWT separation, the resolution of each component is 1 / 2 of the resolution of the multiple initial images. Alternatively, after the DWT separation, the resolution of each component is 1 / 2 of the resolution of the pre-processing results of the multiple initial images.

[0467] Exemplarily, this step can be performed by the frequency separation module 520 in Figure 5 Alternatively, this step can be performed by the frequency separation module in Figure 9

[0468] As shown in Figure 9 , the frequency separation module performs frequency separation processing on the feature map of the t-1th input image obtained in step (1) by using DWT, to obtain hh components, hl components, lh components and ll components corresponding to the t-1th input image. The hh components are input into the high-frequency processing module, and the other components are input into the low-frequency processing module. The frequency separation module performs frequency separation processing on the feature map of the tth input image obtained in step (1) by using DWT, to obtain hh components, hl components, lh components and ll components corresponding to the tth input image. The hh components are input into the high-frequency processing module, and the other components are input into the low-frequency processing module.

[0469] (3) serially combining the low-frequency components of the multiple frames of images, and performing convolution processing on the serially combined result to output the fused low-frequency components.

[0470] That is, the second frequency band features corresponding to the multiple frames of images are merged by the serial operation, and the merged result is subjected to feature fusion by the convolution processing.

[0471] Exemplarily, this step can be performed by the low-frequency processing module 530 in Figure 5 Alternatively, this step can be performed by the low-frequency processing module in Figure 9

[0472] For example, as shown in Figure 9 ​​As shown in Figure 1, the low-frequency processing module uses a series operation to merge the hl component, lh component, and ll component corresponding to the t-1 frame input image and the hl component, lh component, and ll component corresponding to the t frame input image, and performs feature fusion through a 3-layer 128-channel 3×3 convolution operation to obtain the fused low-frequency component.

[0473] (4) performing convolution processing on the high-frequency components of at least one frame of the multiple frames of images, and outputting the high-frequency components of the at least one frame of the images after the convolution processing.

[0474] Step (4) is an optional step. For a detailed description, please refer to model 500 in the previous text, which will not be repeated here.

[0475] That is, feature extraction is performed on the high-frequency components of at least one frame of image through one or more convolution operations.

[0476] For example, this step can be performed by Figure 5 Alternatively, this step can be performed by the high frequency processing module 540 in Figure 9 The high frequency processing module in is executed.

[0477] For example, Figure 9 As shown, the at least one image frame includes the t-1th input image frame and the tth input image frame. The high-frequency processing module processes the hh component corresponding to the t-1th input image frame through a 3×3 convolution operation with 64 channels, thereby obtaining the processed hh component corresponding to the t-1th input image frame. The high-frequency processing module processes the hh component corresponding to the tth input image frame through a 3×3 convolution operation with 64 channels, thereby obtaining the processed hh component corresponding to the tth input image frame.

[0478] (5) Performing frequency fusion processing on the fused low-frequency component and the high-frequency component of at least one frame of the multiple frames of images, and outputting an enhanced image corresponding to the at least one frame of the image.

[0479] The high-frequency component of at least one frame of image may be the high-frequency component of at least one frame of image obtained in step (4) after convolution processing.

[0480] If step (4) is not included, the high-frequency component of at least one frame of image may be the high-frequency component of at least one frame of image obtained in step (2).

[0481] Specifically, the fused low-frequency component and the high-frequency component of at least one frame of image are serially combined; a convolution process is performed on the result of the serial combination to output an enhanced image corresponding to the at least one frame of image. Alternatively, the fused low-frequency component and the high-frequency component of at least one frame of image after convolution are serially combined; a convolution process is performed on the result of the serial combination.

[0482] That is to say, the fused low-frequency component and the high-frequency component of at least one frame of image are respectively subjected to feature fusion through serial merging, and then the feature extraction of the result of serial merging is performed through one or more convolution operations, and the enhanced image corresponding to at least one frame of image is output.

[0483] The frequency fusion model can also be used to achieve output integration so that the resolution of the output image is consistent with the resolution of the input image. In other words, the enhanced image corresponding to at least one frame of image output by the frequency fusion model has the same resolution as the at least one frame of image.

[0484] For example, this step can be performed by Figure 5 Alternatively, this step can be performed by the frequency fusion module 550 in Figure 9 The frequency fusion module in is executed.

[0485] For example, Figure 9 As shown, due to the discrete wavelet transform performed in step (2), the multi-frame image is downsampled by 2 times. The frequency fusion module upsamples the fused low-frequency component by 2 times, and upsamples the hh component corresponding to the processed t-1 frame input image by 2 times. The upsampled low-frequency component and the hh component corresponding to the upsampled t-1 frame input image are merged by a series operation, and feature fusion is performed through a 16-channel 3×3 convolution operation. The result of the feature fusion is then upsampled by 4 times, and the enhanced image corresponding to the t-1 frame input image is output, that is, the t-1 frame output. The output image and the output image have the same resolution, that is, the resolution of the output image is M×N×C.

[0486] The frequency fusion module upsamples the fused low-frequency components by a factor of 2 and also upsamples the hh component corresponding to the processed input image of frame t by a factor of 2. The upsampled low-frequency components and the hh component corresponding to the upsampled input image of frame t are combined using a concatenation operation. Feature fusion is then performed through a 16-channel 3×3 convolution operation. The result of feature fusion is then upsampled by a factor of 4, and the enhanced image corresponding to the input image of frame t is output, i.e., the output of frame t. The output image and the output image have the same resolution, i.e., the output image has a resolution of M×N×C.

[0487] The upsampling process can be accomplished by using a pixel shuffle operation, which does not lose information.

[0488] Figure 9 The image processing model in can be trained using existing solutions. Alternatively, Figure 9 The image processing model in can also be obtained by training through the solution in the embodiment of the present application.

[0489] Specifically, Figure 9 The model in the example is used as the student network, i.e., the first image processing model in the embodiment of the present application. On this basis, a teacher network is constructed, i.e., the second image processing model in the embodiment of the present application. The second image processing model shares part of the network structure with the first image processing model, or in other words, part of the network structure in the first image processing model also belongs to the second graphics enhancement model.

[0490] The second image processing model and the first image processing model may include the same network structure. For example, the first image processing model and the second image processing model may each include an input processing module, a frequency separation module, a low-frequency processing module, a high-frequency processing module, and a frequency fusion module. Some modules are shared by the two models.

[0491] Figure 10 A schematic diagram of a self-distillation network structure provided in an embodiment of the present application is shown. Figure 10 The first image processing model in Figure 9 The model structure shown in the figure is the second image processing model. Figure 9 Built on the basis of Figure 9 The image processing models in

[15] share some network structures. On this basis, the second image processing model adds additional convolutional layers to achieve better denoising effect.

[0492] like Figure 10 The second image processing model shown shares the input processing module, frequency separation module, low frequency processing module and high frequency processing module of the first image processing model.

[0493] The convolution layer is stacked after the low-frequency processing module of the first image processing model to form the low-frequency processing module of the second image processing model. Figure 10 As shown in the figure, three 128-channel 3×3 convolutional layers are stacked after the three convolutional layers of the low-frequency processing module of the first image processing model. The low-frequency processing module of the first image processing model and the three stacked convolutional layers together constitute the low-frequency processing module of the second image processing model.

[0494] The convolution layer is stacked after the high-frequency processing module of the first image processing model to form the high-frequency processing module of the second image processing model. Figure 10 As shown in the figure, a 64-channel 3×3 convolution layer is superimposed after the convolution layer of the high-frequency processing module of the first image processing model. The high-frequency processing module of the first image processing model and the superimposed convolution layer together constitute the high-frequency processing module of the second image processing model.

[0495] Figure 10The first image processing model and the second image processing model in the method may have the same frequency fusion module, but the two do not share the frequency fusion module.

[0496] Input the training samples Figure 10 The network shown is trained.

[0497] Specifically, forward propagation is performed based on the sample images in the training samples to obtain the value of the loss function of the first image processing model, the value of the loss function of the second image processing model, and the value of the distillation loss, and the parameters of the network are adjusted based on the value of the loss function of the first image processing model, the value of the loss function of the second image processing model, and the value of the distillation loss.

[0498] Exemplarily, the network can be trained by a gradient descent method, and the trained first image model can be deployed in a computing device with weaker computing power to enhance images or videos.

[0499] For example, the sample image of the t-1 frame and the sample image of the t frame are input as follows Figure 10 The forward propagation is performed in the model shown in FIG, and the first enhanced image (such as FIG) corresponding to the t-1 frame sample image output by the first image processing model is obtained. Figure 10 The t-1 frame output 1#) and the first enhanced image corresponding to the t-1 frame sample image (such as Figure 10 The t-th frame output 1# in the second image processing model, and the second enhanced image corresponding to the t-1-th frame sample image output by the second image processing model (such as Figure 10 The t-1 frame output 2#) and the second enhanced image corresponding to the t-th frame sample image (such as Figure 10 Output 2# for the tth frame in the output.

[0500] The loss function of the first image processing model is calculated based on the first enhanced image and the target image. This is done by comparing the output 1# of the t-1th frame with the GT corresponding to the sample image of the t-1th frame, and then by comparing the output 1# of the tth frame with the GT corresponding to the sample image of the tth frame.

[0501] The loss function of the second image processing model is calculated based on the second enhanced image and the target image. This is done by comparing the output 2# of the t-1th frame with the GT corresponding to the sample image of the t-1th frame, and then comparing the output 2# of the tth frame with the GT corresponding to the sample image of the tth frame.

[0502] Exemplarily, the loss function of the first image processing model and the loss function of the second image processing model can adopt the multi-index loss function mentioned above. For example, the loss function of the first image processing model and the loss function of the second image processing model can both be calculated using the following formula:

[0503] L=(1+w hf )*(α*L content +β*L SSIM +γ*L cobi +ρ*L color );

[0504] Alternatively, the loss function of the first image processing model and the loss function of the second image processing model may also be calculated in other ways.

[0505] like Figure 10 As shown, the distillation loss between the first image processing model and the second image processing model can be determined according to the feature distillation loss and the output distillation loss.

[0506] Figure 10 The feature distillation loss in is used to calculate the difference between the features of the frequency fusion module of the first image processing model and the features of the frequency fusion module of the second image processing model.

[0507] Figure 10 The output distillation loss in is used to calculate the difference between the output of the first image processing model and the output of the second image processing model, i.e. Figure 10 The difference between the t-th frame output image 1# and the t-th frame output image 2#, and the difference between the t-1-th frame output image 1# and the t-1-th frame output image 2#.

[0508] For example, the distillation loss L between the second image processing model and the first image processing model is KD The following formula can be satisfied:

[0509] L KD =a*L feature +b*L output ;

[0510] Among them, L feature represents the feature distillation loss, L output represents the output distillation loss, a represents L feature The weight coefficient, b represents L output The weight coefficients are: For example, a is 1 and b is 1.

[0511] Table 1 shows the results of denoising tests on a mobile phone low-light video dataset using the model in the embodiment of the present application and other enhanced models.

[0512] Table 1

[0513]

[0514] Here, inputs / outputs represents the number of image frames input to the model divided by the number of image frames output by the model. Giga multiply-accumulation operations per second (GMACs) measures the computational complexity of the model. Peak signal-to-noise ratio (PSNR) represents the ratio of the maximum possible signal power to the destructive noise power that affects its accuracy. PSNR can be used to measure the quality of signal reconstruction in image processing. The larger the PNSR value, the higher the image quality. Time represents the time it takes for the model to infer an image with a resolution of 4096×2304 on the Kirin 990 NPU.

[0515] Table 1 uses a single-input and single-output DnCNN as a control to test the denoising effects of two model structures with different computational loads. Table 1 is explained below using a model structure with a computational load of 25GMACs as an example. The DnCNN model with a computational load of 25GMACs takes 48ms to infer once, which cannot meet the requirements of real-time processing of 4k resolution videos at 30 frames per second. The model with a computational load of 25GMACs in this application takes 48ms to infer once and outputs two frames of video at one inference, which is equivalent to processing one frame of image in only 24ms. It can be used to process 4k resolution videos at 30 frames per second in real time on the Kirin 990NPU. Moreover, compared with the DnCNN model with the same computational load, the PSNR of the model of this application is 1-2DB higher, that is, the denoising effect of the model of this application far exceeds that of the DnCNN model of the same size.

[0516] Figure 11 A comparison chart of the processing effects of different models is shown in FIG. Figure 11 (a) is a comparison diagram of the processing effects of the model with the frequency separation module and the model without the frequency separation module. Figure 11 (b) is a comparison chart of the processing effects of the model with the frequency separation module and the model without the frequency separation module.

[0517] The model with a frequency separation module, i.e., the model in the embodiment of the present application, inputs 1 / 4 of the characteristic components (i.e., hh components) obtained by DWT into the high-frequency processing module. The DWT module is replaced by the pixel shuffel downsampling operation as a control model, i.e., the control model does not have a frequency separation module. In the control model, 1 / 4 of the characteristic components of the features obtained after pixel shuffel downsampling are input into the high-frequency processing module. This makes the control model and the model of the present application have the same computational complexity to achieve fair comparison. The comparison effect diagram is as follows: Figure 11 As shown, Figure 11 In (a), the left image is the image processed by the control model without DWT, and the right image is the image processed by the model in this application. Compared with the left image, the high-frequency details in the right image are clearer and the image quality is higher. Figure 11 In (b), the left image is the image processed by the control model without DWT, the middle image is the image processed by the model in this application, and the right image is the GT. Compared with the left image, the color in the middle image is closer to the color of the GT, and the color deviation is smaller.

[0518] The solution of this application adopts a frequency division and fusion method to distinguish and process different frequency components, separate the high-frequency components to improve the high-frequency resolution of the model, which is conducive to repairing the high-frequency details of the image; by fusing the low-frequency components to reduce the image color deviation, it is conducive to restoring the true color of the image.

[0519] Figure 12 A comparison chart showing the processing effects of models obtained using different training methods is shown. Figure 12 In (a), the left image is the image processed by the model trained using the existing scheme, and the right image is the image processed by the model trained using the self-distillation method. Compared with the left image, the dark area details in the right image are clearer. Figure 12 In (b), the left image is processed by the model trained using the existing scheme, and the right image is processed by the model trained using the self-distillation method. Compared with the left image, the right image has less purple fringing.

[0520] This application's solution uses self-distillation to distill the knowledge learned by the teacher network into the less computationally intensive student network. This approach not only meets the computing power constraints of low-power terminal devices but also produces a model with improved denoising. Self-distillation significantly improves dark area detail in the model and reduces purple fringing artifacts.

[0521] Figure 13 and Figure 14 A comparison chart of the processing effects of different models is shown. Figure 13 (a) is an image taken by a mobile phone under a lighting condition of 1 lux, i.e., the image to be processed; Figure 13 (b) is the image obtained after processing by the DnCNN model with a computational cost of 25GMACs; Figure 13 (c) is the image obtained after processing by the model of this application with a computational effort of 25 GMACs; Figure 13 (d) is the image obtained after processing by the model of this application with a computational cost of 118 GMACs. Figure 13As shown, compared with the DnCNN model with a computational cost of 25 GMACs, the image processed by the model of the present application with a computational cost of 25 GMACs has less color noise and clearer high-frequency details, and the denoising visual effect of the image processed by the model of the present application with a computational cost of 25 GMACs is significantly better than that of the DnCNN model with 25 GMACs.

[0522] Figure 14 (a) is an image taken by a mobile phone under a lighting condition of 1 lux, i.e., the image to be processed; Figure 14 (b) is the image obtained after processing by the DnCNN model with a computational cost of 25GMACs; Figure 14 (c) is the image obtained after processing by the model of this application with a computational effort of 25 GMACs; Figure 14 (d) is the image obtained after processing by the model of this application with a computational cost of 118 GMACs. Figure 14 As shown, compared with the DnCNN model with a computational cost of 25 GMACs, the image processed by the model of the present application with a computational cost of 25 GMACs has less color deviation, and the denoising visual effect of the image processed by the model of the present application with a computational cost of 25 GMACs is significantly better than that of the DnCNN model with 25 GMACs.

[0523] Figure 15 Comparison of images taken with a mobile phone and images processed by the model of this application. Figure 15 (a) is a DXO test scene diagram under 5lux lighting conditions. Figure 15 In (b), the left picture is the image taken by the mobile phone, and the right picture is the image processed by the model of this application. Figure 15 As can be seen in (b), the image processed by the model of this application has more texture details. Figure 15 In (c), the left picture is the image taken by the mobile phone, and the right picture is the image processed by the model of this application. Figure 15 As can be seen in (c), the image processed by the model of the present application has less color noise. Figure 15 In (d), the left picture is the image taken by the mobile phone, and the right picture is the image processed by the model of this application. Figure 15 As can be seen from (d), the image processed by the model of the present application has more dark area details, that is, the dark area quality is higher. Figure 15 In (e), the left picture is the image taken by the mobile phone, and the right picture is the image processed by the model of this application. Figure 15 As can be seen in (e), the image processed by the model of the present application has stable high-frequency details. Figure 15 In (d), the left picture is the image taken by the mobile phone, and the right picture is the image processed by the model of this application. Figure 15As can be seen in (f), the image processed by the model of this application has stronger weak-contrast texture.

[0524] Figure 16 An image processing method 1600 is provided in an embodiment of the present application. The method 1600 can be executed by an image processing device. The image processing device can be a cloud service device or a terminal device, for example, a computer, server, mobile phone, TV, etc., which has sufficient computing power to perform the operation of the image processing model, or an image signal processor (ISP), or a system composed of a cloud service device and a terminal device. For example, the method 1600 can be performed by Figure 1 The execution device 110 in Figure 3 Neural network processor 50 or Figure 4 The execution device 310 or local device executes.

[0525] For example, the image processing method can be specifically as follows Figure 1 The execution device 110 is shown to execute.

[0526] Optionally, the image processing method can be processed by a CPU, or by a CPU and a GPU together, or other processors suitable for neural network calculations can be used without a GPU, and this application does not impose any restrictions.

[0527] Method 1600 uses the image processing model in the embodiment of the present application to process the image. In order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing method 1600 below.

[0528] Method 1600 includes steps S1610 to S1650.

[0529] S1610 , pre-processing the multiple frames of initial images and outputting the multiple frames of images.

[0530] Exemplarily, the multiple frames of initial images may be multiple frames of initial images in a video to be processed.

[0531] Step S1610 is an optional step.

[0532] Optionally, step S1610 includes: performing feature extraction on the multiple frames of initial images, and using the extracted feature maps of the multiple frames of initial images as the multiple frames of images.

[0533] S1620 , performing frequency separation processing on the multiple frames of images respectively, and outputting a high-frequency component and a low-frequency component of each frame of the multiple frames of images.

[0534] Optionally, step S1620 includes: performing frequency separation processing on the multiple frames of images respectively through wavelet transform, and outputting high-frequency components and low-frequency components of each frame of the multiple frames of images.

[0535] S1630: Fusing the low-frequency components of the multiple frames of images and outputting the fused low-frequency components.

[0536] Optionally, step S1630 includes: performing serial merging on low-frequency components of multiple frames of images; performing convolution processing on the result of the serial merging, and outputting the fused low-frequency components.

[0537] S1640 , performing convolution processing on the high-frequency components of at least one frame of the multiple frames of images, and outputting the high-frequency components of the at least one frame of the images after the convolution processing.

[0538] Step S1640 is an optional step.

[0539] S1650: Perform frequency fusion processing on the fused low-frequency component and the high-frequency component of at least one frame of the multiple frames after convolution processing, and output an enhanced image corresponding to the at least one frame of the image.

[0540] Exemplarily, when method 1600 does not include S1640, step S1650 may include: performing frequency fusion processing on the fused low-frequency component and the high-frequency component of at least one frame of image, respectively, and outputting an enhanced image corresponding to the at least one frame of image.

[0541] Optionally, step S1650 includes: serially merging the fused low-frequency component and the high-frequency component of at least one frame of image; performing convolution processing on the result of the serial merging, and outputting an enhanced image corresponding to the at least one frame of image.

[0542] According to the solution of the embodiment of the present application, the different frequency components of the multi-frame images are extracted, and the high-frequency components are relatively separated, ensuring that the high-frequency information in the image is not affected by the inter-frame misalignment problem, retaining good high-frequency details, avoiding the problem of blurring of the enhanced image, and fusing the low-frequency components of the multi-frame images to obtain the low-frequency information in the multi-frame images, achieving inter-frame complementarity, solving problems such as color cast and general texture blur, improving the denoising effect, ensuring the authenticity of the picture color, enhancing the texture details of the image, and improving the imaging quality. Moreover, the solution in the embodiment of the present application separates the high-frequency components so that the high-frequency components are not affected by the inter-frame misalignment problem, and thus there is no need to use a deeper convolutional layer to align the multi-frame images, reducing the computational overhead and memory overhead during model operation, and improving the model's running speed.

[0543] The following combination Figures 17 to 20The apparatus of the embodiment of the present application is described. It should be understood that the apparatus described below can execute the method of the embodiment of the present application, and in order to avoid unnecessary repetition, repeated descriptions are appropriately omitted when introducing the apparatus of the embodiment of the present application.

[0544] Figure 17 It is a schematic block diagram of a training device for an image processing model according to an embodiment of the present application. Figure 17 The image processing model training device 2000 shown includes an acquisition unit 2010 and a processing unit 2020.

[0545] The acquisition unit 2010 and the processing unit 2020 can be used to execute the image processing model training method 700 of the embodiment of the present application.

[0546] Exemplarily, the acquisition unit 2010 is configured to acquire multiple sample images and a target image corresponding to at least one of the multiple sample images, wherein the target image corresponding to the at least one sample image has a higher signal-to-noise ratio than the at least one sample image.

[0547] Processing unit 2020 is configured to employ a first image processing model to perform forward propagation based on multiple sample image frames to obtain a first enhanced image corresponding to at least one sample image frame; and to adjust parameters of the first image processing model with the goal of reducing a target loss function to obtain a trained first image processing model. The target loss function is determined based on a loss function of the first image processing model, where the loss function of the first image processing model is used to calculate the difference between the first enhanced image corresponding to at least one sample image frame and the target image corresponding to at least one sample image frame; the loss function includes weight values, and the weight values ​​of the loss function corresponding to some or all regions of the first enhanced image corresponding to at least one sample image frame are indicated by a weight map.

[0548] Optionally, as an embodiment, the first image processing model includes a first frequency separation module, a first low-frequency processing module and a first frequency fusion module, wherein the first frequency separation module is used to perform frequency separation processing on multiple frames of sample images respectively, and output the first high-frequency component and the first low-frequency component of each frame of the sample image in the multiple frames; the first low-frequency processing module is used to perform fusion processing on the first low-frequency components of multiple frames of sample images, and output the fused first low-frequency component; the first frequency fusion module is used to perform frequency fusion processing on the fused first low-frequency component and the first high-frequency component of at least one frame of sample image respectively, and output the first enhanced image corresponding to at least one frame of sample image.

[0549] Optionally, as an embodiment, the weight value of the loss function corresponding to the high-frequency area in the first enhanced image corresponding to at least one frame of sample image is greater than the weight value of the loss function corresponding to the low-frequency area in the first enhanced image corresponding to at least one frame of sample image.

[0550] Optionally, as an embodiment, the difference between the first enhanced image corresponding to at least one frame of sample image and the target image corresponding to at least one frame of sample image includes: the difference between some pixels in the first enhanced image corresponding to at least one frame of sample image and some pixels in the target image corresponding to at least one frame of sample image.

[0551] Optionally, as an embodiment, some pixels include pixels whose pixel values ​​are less than or equal to a first threshold.

[0552] Optionally, as an embodiment, the loss function is determined according to at least two of the following: content loss, structural similarity loss, contextual bilateral loss or color loss.

[0553] Optionally, as an embodiment, the loss function L satisfies the formula:

[0554] L=w*(α*L content +β*L SSIM +γ*L cobi +ρ*L color );

[0555] Among them, L content Indicates content loss, L SSIM represents the structural similarity loss, L cobi represents the contextual bilateral loss, L color represents color loss, α, β, γ and ρ are parameters, and w represents the weight map.

[0556] Optionally, as an embodiment, the processing unit 2020 is also used to: use a second image processing model to perform forward propagation based on multiple frames of sample images to obtain a second enhanced image corresponding to at least one frame of sample image, and the second image processing model and the first image processing model share part of the network structure; and the target loss function is determined according to the loss function of the first image processing model, including: the target loss function is determined according to the loss function of the first image processing model, the loss function of the second image processing model or the distillation loss between the second image processing model and the first image processing model, and the loss function of the second image processing model is used to calculate the difference between the second enhanced image corresponding to at least one frame of sample image and the target image corresponding to at least one frame of sample image.

[0557] Optionally, as an embodiment, the distillation loss between the second image processing model and the first image processing model is determined based on at least one of the following: feature distillation loss or output distillation loss; wherein, the feature distillation loss is used to calculate the difference between the features of the intermediate layer of the first image processing model and the features of the intermediate layer of the second image processing model; the output distillation loss is used to calculate the difference between the first enhanced image and the second enhanced image.

[0558] Optionally, as an embodiment, the distillation loss L between the second image processing model and the first image processing model is KD Satisfies the following formula:

[0559] L KD =a*L feature +b*L output ;

[0560] Among them, L feature represents the feature distillation loss, L output represents the output distillation loss, a represents L feature The weight coefficient, b represents L output The weight coefficient of .

[0561] Optionally, as an embodiment, the target loss function satisfies the following formula:

[0562] L target =L KD +L1+L2;

[0563] Among them, L target represents the target loss function, L KD represents the distillation loss between the first image processing model and the second image processing model, L1 represents the loss function of the first image processing model, and L2 represents the loss function of the second image processing model.

[0564] Optionally, as an embodiment, the second image processing model includes a second frequency separation module, a second low-frequency processing module and a second frequency fusion module, wherein: the second frequency separation module is used to perform frequency separation processing on multiple frames of sample images respectively, and output the second high-frequency component and the second low-frequency component of each frame of the sample image in the multiple frames; the second low-frequency processing module is used to perform fusion processing on the second low-frequency components of the multiple frames of sample images, and output the fused second low-frequency component; the second frequency fusion module is used to perform frequency fusion processing on the fused second low-frequency component and the second high-frequency component of at least one frame of sample image respectively, and output the second enhanced image corresponding to at least one frame of sample image; the first frequency separation module and the second frequency separation module share a network structure, and the first low-frequency processing module and the second low-frequency processing module share part of the network structure.

[0565] Optionally, as an embodiment, the characteristics of the middle layer of the first image processing model include the characteristics of the first frequency fusion processing module, and the characteristics of the middle layer of the second image processing model include the characteristics of the second frequency fusion processing module.

[0566] Figure 18 It is a schematic block diagram of an image processing device according to an embodiment of the present application. Figure 18The illustrated image processing apparatus 3000 includes an acquisition unit 3010 and a processing unit 3020.

[0567] The acquisition unit 3010 and the processing unit 3020 can be used to perform the image processing method 1600 of the embodiments of the present application.

[0568] Exemplarily, the acquisition unit 3010 is configured to acquire a plurality of images.

[0569] The processing unit 3020 is configured to separately perform frequency separation processing on the plurality of images, output high-frequency components and low-frequency components of each of the plurality of images; perform fusion processing on the low-frequency components of the plurality of images, output fused low-frequency components; and separately perform frequency fusion processing on the fused low-frequency components and high-frequency components of at least one of the plurality of images, output an enhanced image corresponding to the at least one of the plurality of images.

[0570] Optionally, as one embodiment, the processing unit 3020 is further configured to separately perform convolution processing on the high-frequency components of at least one of the plurality of images; and the processing unit 3020 is specifically configured to separately perform frequency fusion processing on the fused low-frequency components and the high-frequency components of the at least one of the plurality of images after the convolution processing, output an enhanced image corresponding to the at least one of the plurality of images.

[0571] Optionally, as one embodiment, the processing unit 3020 is specifically configured to separately perform frequency separation processing on the plurality of images by wavelet transform, output high-frequency components and low-frequency components of each of the plurality of images.

[0572] Optionally, as one embodiment, the processing unit 3020 is specifically configured to perform series combination on the low-frequency components of the plurality of images; and perform convolution processing on a result of the series combination, output the fused low-frequency components.

[0573] Optionally, as one embodiment, the processing unit 3020 is specifically configured to separately perform series combination on the fused low-frequency components and the high-frequency components of at least one of the plurality of images; and perform convolution processing on a result of the series combination, output an enhanced image corresponding to the at least one of the plurality of images.

[0574] Optionally, as one embodiment, the processing unit 3020 is specifically configured to separately perform series combination on the fused low-frequency components and the high-frequency components of at least one of the plurality of images after the convolution processing; and perform convolution processing on a result of the series combination, output an enhanced image corresponding to the at least one of the plurality of images.

[0575] Optionally, as one embodiment, the processing unit 3020 is further configured to perform preprocessing on the plurality of initial images, output the plurality of images.

[0576] Optionally, as an embodiment, the processing unit 3020 is specifically configured to: perform feature extraction on the multiple frames of initial images, and use the extracted feature maps of the multiple frames of initial images as the multiple frames of images.

[0577] It should be noted that the above-mentioned apparatus 2000 and apparatus 3000 are embodied in the form of functional units. The term "unit" herein can be implemented in the form of software and / or hardware, and is not specifically limited thereto.

[0578] For example, a "unit" may be a software program, a hardware circuit, or a combination of the two that implements the aforementioned functionality. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (e.g., a shared processor, a dedicated processor, or a group processor) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components that support the described functionality.

[0579] Therefore, the units of each example described in the embodiments of this application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0580] Figure 19 This is a schematic diagram of the hardware structure of a training device for an image processing model provided in an embodiment of the present application. Figure 19 The image processing model training apparatus 4000 shown (the apparatus 4000 may be a computer device) includes a memory 4001, a processor 4002, a communication interface 4003, and a bus 4004. The memory 4001, the processor 4002, and the communication interface 4003 are connected to each other via the bus 4004.

[0581] Memory 4001 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 4001 can store programs. When the program stored in memory 4001 is executed by processor 4002, processor 4002 and communication interface 4003 are used to perform the various steps of the image processing model training method in the embodiment of the present application. Specifically, processor 4002 can execute method 700 above.

[0582] The processor 4002 can be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit, or one or more integrated circuits that are configured to execute a program to perform the functions required by the units in the training device of the image processing model according to the embodiments of the present application, or to perform the training method of the image processing model according to the method embodiments of the present application.

[0583] The processor 4002 can also be an integrated circuit chip with a processing capability for signals. For example, it can be a Figure 3 The chip shown in the figure. In the implementation process, each step of the training method of the image processing model according to the present application can be completed by the integrated logic circuit of the hardware in the processor 4002 or the instruction in the form of software.

[0584] The processor 4002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 4001, and the processor 4002 reads the information in the storage 4001, and combines the hardware to complete the functions required by the units included in the training device of the image processing model according to the embodiments of the present application, or to perform the training method of the image processing model according to the method embodiments of the present application.

[0585] The communication interface 4003 uses a transceiver device such as but not limited to a transceiver to realize the communication between the device 4000 and other devices or communication networks. For example, the training sample (such as the multiple frame sample image and the target image corresponding to at least one frame sample image in the multiple frame sample image in the method 700) can be obtained through the communication interface 4003.

[0586] The bus 4004 can include a path for communicating information between the various components (e.g., the memory 4001, the processor 4002, the communication interface 4003) of the device 4000.

[0587] It should be understood that the acquisition unit 2010 in the image processing model training device 2000 corresponds to the communication interface 4003 in the image processing model training device 4000, and the processing unit 2020 can correspond to the processor 4002.

[0588] Figure 20 is a hardware structure schematic diagram of an image processing device provided by an embodiment of the present application. Figure 20 The image processing device 5000 shown (which can be a computer device in particular) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, the processor 5002, and the communication interface 5003 are in communication connection with each other through the bus 5004.

[0589] The memory 5001 can be a ROM, a static storage device, a dynamic storage device, or a RAM. The memory 5001 can store a program, and when the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 and the communication interface 5003 are used to perform various steps of the image processing method of the embodiments of the present application.

[0590] The processor 5002 can adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits, and is used to execute a related program to implement the functions required by the units in the image processing device of the embodiments of the present application, or to execute the image processing method of the method embodiments of the present application.

[0591] Processor 5002 may also be an integrated circuit chip with signal processing capabilities. During implementation, the various steps of the image processing method of the present application may be performed by hardware integrated logic circuits or software instructions within processor 5002. The aforementioned processor 5002 may also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present application may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules may be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in memory 5001. Processor 5002 reads information from memory 5001 and, in conjunction with its hardware, performs the functions required to be performed by the units included in the image processing device of the embodiments of the present application, or executes the image processing method of the method embodiments of the present application.

[0592] The communication interface 5003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the device 5000 and other devices or a communication network. For example, input data (such as multiple frames of images in the embodiment of the present application) can be obtained through the communication interface 5003.

[0593] The bus 5004 may include a path for transmitting information between various components of the device 5000 (eg, the memory 5001 , the processor 5002 , and the communication interface 5003 ).

[0594] It should be understood that the acquisition unit 3010 in the image processing apparatus 3000 is equivalent to the communication interface 5003 in the image processing apparatus 5000 ; and the processing unit 3020 in the image processing apparatus 3000 may be equivalent to the processor 5002 .

[0595] It should be noted that although Figure 19 and Figure 20 The devices 4000 and 5000 shown in the figure only show a memory, a processor, and a communication interface. However, in the specific implementation process, those skilled in the art should understand that the devices 4000 and 5000 also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the devices 4000 and 5000 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the devices 4000 and 5000 may also only include the devices necessary to implement the embodiments of the present application, and do not necessarily include Figure 19 or Figure 20 All devices shown in .

[0596] It can be understood that the device 4000 is equivalent to the training device 120 in 1, and the device 5000 is equivalent to Figure 1 The execution device 110 in.

[0597] An embodiment of the present application further provides a computer-readable medium, which stores a program code for execution by a device, wherein the program code includes a method for executing the method in the embodiment of the present application.

[0598] The embodiments of the present application also provide a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method in the embodiments of the present application.

[0599] An embodiment of the present application further provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to execute the method in the embodiment of the present application.

[0600] The present application also provides a mobile terminal, which includes any one of the devices in the present application, such as Figure 17 、 Figure 18 、 Figure 19 or Figure 20 The device shown.

[0601] Exemplarily, the mobile terminal may be a drone or a robot, etc.

[0602] The embodiment of the present application further provides a cloud, which includes any device of the embodiment of the present application, such as Figure 17 、 Figure 18 、 Figure 19 or Figure 20 The device shown.

[0603] The embodiment of the present application further provides a wearable device, which includes any one of the apparatuses of the embodiment of the present application, such as Figure 17 、 Figure 18 、 Figure 19 or Figure 20 The device shown.

[0604] For example, the wearable device may include glasses or smart watches.

[0605] The embodiment of the present application further provides a vehicle, which includes any one of the devices of the embodiment of the present application, such as Figure 17 、 Figure 18 、 Figure 19 or Figure 20 The device shown.

[0606] The embodiment of the present application further provides an electronic device, which includes any one of the devices in the embodiment of the present application, such as Figure 17 、 Figure 18 、 Figure 19 or Figure 20 The device shown.

[0607] Illustratively, the electronic device includes a television, a computer, or a mobile phone.

[0608] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0609] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0610] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0611] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0612] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0613] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a universal serial bus flash disk (USB flash disk, UFD), which can also be referred to as a USB flash drive or USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0614] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing model, characterized in that: include: Frequency separation module, low frequency processing module and frequency fusion module; The frequency separation module is used to perform frequency separation processing on the multiple frames of images respectively, and output the high-frequency component and the low-frequency component of each frame of the multiple frames of images; The low-frequency processing module is used to perform fusion processing on the low-frequency components of the multiple frames of images and output the fused low-frequency components; The frequency fusion module is used to use convolution processing to perform frequency fusion processing on the fused low-frequency component and the high-frequency component of at least one frame of the multiple images, and output an enhanced image corresponding to the at least one frame of the image, wherein the number of enhanced images corresponding to the at least one frame of the image is less than the number of the multiple frames of the image.

2. The image processing model according to claim 1, characterized in that The model further includes: a high-frequency processing module, configured to perform convolution processing on the high-frequency components of at least one frame of the multiple frames of images; and The frequency fusion module is used to perform frequency fusion processing on the fused low-frequency component and the high-frequency component after convolution processing of the at least one frame of image, and output an enhanced image corresponding to the at least one frame of image.

3. The image processing model according to claim 1 or 2, characterized in that The frequency separation module is specifically used for: Frequency separation processing is performed on the multiple frames of images respectively through wavelet transformation, and a high-frequency component and a low-frequency component of each frame of the multiple frames of images are output.

4. The image processing model according to claim 1 or 2, characterized in that The low-frequency processing module is specifically used for: Serially merging the low-frequency components of the multiple frames of image; The convolution process is performed on the result of the serial merging to output the fused low-frequency component.

5. The image processing model according to claim 1, wherein: The frequency fusion module is specifically used for: Combining the fused low-frequency component and the high-frequency component of the at least one frame of image in series; Perform convolution processing on the result of the serial merging and output an enhanced image corresponding to the at least one frame of image.

6. The image processing model according to claim 2, characterized in that The frequency fusion module is specifically used for: Combining the fused low-frequency component and the convolution-processed high-frequency component of the at least one frame of image in series; Perform convolution processing on the result of the serial merging and output an enhanced image corresponding to the at least one frame of image.

7. The image processing model according to claim 1 or 2, characterized in that The image processing model further includes an input processing module, which is used to pre-process multiple frames of initial images and output the multiple frames of images.

8. The image processing model according to claim 7, characterized in that The input processing module is specifically used for: Feature extraction is performed on the multiple frames of initial images, and the extracted feature maps of the multiple frames of initial images are used as the multiple frames of images.

9. A training method for an image processing model, characterized in that: The method comprises: Acquire a plurality of sample images and a target image corresponding to at least one of the plurality of sample images, wherein the target image corresponding to the at least one sample image has a higher signal-to-noise ratio than that of the at least one sample image; The first image processing model performs forward propagation based on the multiple frames of sample images to obtain a first enhanced image corresponding to the at least one frame of sample image; Adjusting parameters of the first image processing model with the goal of reducing a value of a target loss function to obtain a trained first image processing model, wherein the target loss function is determined based on a loss function of the first image processing model, and the loss function of the first image processing model is used to calculate a difference between a first enhanced image corresponding to the at least one frame of sample image and a target image corresponding to the at least one frame of sample image; The loss function includes a weight value, and the weight value of the loss function corresponding to a portion or all of the regions in the first enhanced image corresponding to the at least one frame of sample image is indicated by a weight map.

10. The method according to claim 9, characterized in that The first image processing model includes a first frequency separation module, a first low-frequency processing module and a first frequency fusion module, wherein: The first frequency separation module is configured to perform frequency separation processing on the multiple frames of sample images respectively, and output a first high-frequency component and a first low-frequency component of each frame of sample image in the multiple frames of sample images; The first low-frequency processing module is configured to perform fusion processing on the first low-frequency components of the multiple frames of sample images and output the fused first low-frequency components; The first frequency fusion module is configured to perform frequency fusion processing on the fused first low-frequency component and the first high-frequency component of the at least one frame of sample image, respectively, and output a first enhanced image corresponding to the at least one frame of sample image.

11. The method according to claim 9 or 10, characterized in that The weight value of the loss function corresponding to the high-frequency area in the first enhanced image corresponding to the at least one frame of sample image is greater than the weight value of the loss function corresponding to the low-frequency area in the first enhanced image corresponding to the at least one frame of sample image.

12. The method according to claim 9 or 10, characterized in that The difference between the first enhanced image corresponding to the at least one frame of sample image and the target image corresponding to the at least one frame of sample image includes: the difference between some pixels in the first enhanced image corresponding to the at least one frame of sample image and some pixels in the target image corresponding to the at least one frame of sample image.

13. The method according to claim 12, characterized in that The part of pixels includes pixels whose pixel values ​​are less than or equal to a first threshold.

14. The method according to claim 9 or 10, characterized in that The loss function is determined based on at least two of the following: content loss, structural similarity loss, contextual bilateral loss, or color loss.

15. The method according to claim 14, characterized in that The loss function L satisfies the formula: L=w*(α*L content +β*L SSIM +γ*L cobi +p*L color ); Among them, L content Indicates the content loss, L SSIM represents the structural similarity loss, L cobi represents the contextual bilateral loss, L color represents the color loss, α, β, γ and ρ are parameters, and w represents the weight map.

16. The method according to claim 9 or 10, characterized in that The method further comprises: A second image processing model performs forward propagation based on the multiple frames of sample images to obtain a second enhanced image corresponding to the at least one frame of sample image, wherein the second image processing model and the first image processing model share a portion of the network structure; And the target loss function is determined according to the loss function of the first image processing model, including: the target loss function is determined according to the loss function of the first image processing model, the loss function of the second image processing model or the distillation loss between the second image processing model and the first image processing model, and the loss function of the second image processing model is used to calculate the difference between the second enhanced image corresponding to the at least one frame of sample image and the target image corresponding to the at least one frame of sample image.

17. The method according to claim 16, characterized in that The distillation loss between the second image processing model and the first image processing model is determined according to at least one of the following: feature distillation loss or output distillation loss; The feature distillation loss is used to calculate the difference between the features of the intermediate layer of the first image processing model and the features of the intermediate layer of the second image processing model; The output distillation loss is used to calculate the difference between the first enhanced image and the second enhanced image.

18. The method according to claim 17, characterized in that The distillation loss L between the second image processing model and the first image processing model KD Satisfies the following formula: L KD =a*L feature +b*L output ; Among them, L feature represents the characteristic distillation loss, L output represents the output distillation loss, a represents L feature The weight coefficient, b represents L output The weight coefficient of .

19. The method according to claim 16, wherein The objective loss function satisfies the following formula: L target =L KD +L1+L2; Among them, L target Represents the target loss function, L KD Represents the distillation loss between the first image processing model and the second image processing model, L1 represents the loss function of the first image processing model, and L2 represents the loss function of the second image processing model.

20. The method according to claim 16, wherein The second image processing model includes a second frequency separation module, a second low-frequency processing module, and a second frequency fusion module, wherein: The second frequency separation module is configured to perform frequency separation processing on the multiple frames of sample images respectively, and output a second high-frequency component and a second low-frequency component of each frame of the sample image in the multiple frames of sample images; The second low-frequency processing module is configured to perform fusion processing on the second low-frequency components of the multiple frames of sample images and output the fused second low-frequency components; The second frequency fusion module is configured to perform frequency fusion processing on the fused second low-frequency component and the second high-frequency component of the at least one frame of sample image, and output a second enhanced image corresponding to the at least one frame of sample image; The first frequency separation module and the second frequency separation module share a network structure, and the first low-frequency processing module and the second low-frequency processing module share a part of the network structure.

21. The method according to claim 20, characterized in that The features of the middle layer of the first image processing model include the features of the first frequency fusion processing module, and the features of the middle layer of the second image processing model include the features of the second frequency fusion processing module.

22. An image processing method, characterized in that: include: Performing frequency separation processing on the multiple frames of images respectively, and outputting a high-frequency component and a low-frequency component of each frame of the multiple frames of images; performing fusion processing on the low-frequency components of the multiple frames of images, and outputting the fused low-frequency components; Convolution processing is used to perform frequency fusion processing on the fused low-frequency component and the high-frequency component of at least one frame of the multiple images, and an enhanced image corresponding to the at least one frame of the image is output, wherein the number of enhanced images corresponding to the at least one frame of the image is less than the number of the multiple frames of the image.

23. The image processing method according to claim 22, characterized in that: The method further comprises: performing convolution processing on high-frequency components of at least one frame of the multiple frames of images; and The performing frequency fusion processing on the fused low-frequency component and the high-frequency component of at least one frame of the multiple images, and outputting an enhanced image corresponding to the at least one frame of the image, includes: Frequency fusion processing is performed on the fused low-frequency component and the high-frequency component of the at least one frame of image after convolution processing, and an enhanced image corresponding to the at least one frame of image is output.

24. The image processing method according to claim 22 or 23, characterized in that: The performing frequency separation processing on the multiple frames of images respectively and outputting the high-frequency component and the low-frequency component of each frame of the multiple frames of images includes: Frequency separation processing is performed on the multiple frames of images respectively through wavelet transformation, and a high-frequency component and a low-frequency component of each frame of the multiple frames of images are output.

25. The image processing method according to claim 22 or 23, characterized in that: The fusing low-frequency components of the multiple frames of images and outputting the fused low-frequency components comprises: Serially merging the low-frequency components of the multiple frames of image; The convolution process is performed on the result of the serial merging to output the fused low-frequency component.

26. The image processing method according to claim 22, wherein: The performing frequency fusion processing on the fused low-frequency component and the high-frequency component of at least one frame of the multiple images, and outputting an enhanced image corresponding to the at least one frame of the image, includes: Combining the fused low-frequency component and the high-frequency component of the at least one frame of image in series; Perform convolution processing on the result of the serial merging and output an enhanced image corresponding to the at least one frame of image.

27. The image processing method according to claim 23, wherein: The performing frequency fusion processing on the fused low-frequency component and the high-frequency component after convolution processing of the at least one frame of image respectively, and outputting an enhanced image corresponding to the at least one frame of image, includes: Combining the fused low-frequency component and the convolution-processed high-frequency component of the at least one frame of image in series; Perform convolution processing on the result of the serial merging and output an enhanced image corresponding to the at least one frame of image.

28. The image processing method according to claim 22 or 23, characterized in that: The method further comprises: Preprocessing is performed on multiple frames of initial images, and the multiple frames of images are output.

29. The image processing method according to claim 28, characterized in that: The preprocessing of the multiple frames of initial images and outputting the multiple frames of images includes: Feature extraction is performed on the multiple frames of initial images, and the extracted feature maps of the multiple frames of initial images are used as the multiple frames of images.

30. A training device for an image processing model, characterized in that: include: an acquiring unit, configured to acquire a plurality of sample image frames and a target image corresponding to at least one sample image frame among the plurality of sample image frames, wherein the target image corresponding to the at least one sample image frame has a higher signal-to-noise ratio than that of the at least one sample image frame; A processing unit, configured to use a first image processing model to perform forward propagation based on the multiple frames of sample images to obtain a first enhanced image corresponding to the at least one frame of sample image; Adjusting parameters of the first image processing model with the goal of reducing a value of a target loss function to obtain a trained first image processing model, wherein the target loss function is determined based on a loss function of the first image processing model, and the loss function of the first image processing model is used to calculate a difference between a first enhanced image corresponding to the at least one frame of sample image and a target image corresponding to the at least one frame of sample image; The loss function includes a weight value, and the weight value of the loss function corresponding to a portion or all of the regions in the first enhanced image corresponding to the at least one frame of sample image is indicated by a weight map.

31. The training device according to claim 30, characterized in that The first image processing model includes a first frequency separation module, a first low-frequency processing module and a first frequency fusion module, wherein: The first frequency separation module is configured to perform frequency separation processing on the multiple frames of sample images respectively, and output a first high-frequency component and a first low-frequency component of each frame of sample image in the multiple frames of sample images; The first low-frequency processing module is configured to perform fusion processing on the first low-frequency components of the multiple frames of sample images and output the fused first low-frequency components; The first frequency fusion module is configured to perform frequency fusion processing on the fused first low-frequency component and the first high-frequency component of the at least one frame of sample image, respectively, and output a first enhanced image corresponding to the at least one frame of sample image.

32. The training device according to claim 30 or 31, characterized in that The weight value of the loss function corresponding to the high-frequency area in the first enhanced image corresponding to the at least one frame of sample image is greater than the weight value of the loss function corresponding to the low-frequency area in the first enhanced image corresponding to the at least one frame of sample image.

33. The training device according to claim 30 or 31, characterized in that The difference between the first enhanced image corresponding to the at least one frame of sample image and the target image corresponding to the at least one frame of sample image includes: the difference between some pixels in the first enhanced image corresponding to the at least one frame of sample image and some pixels in the target image corresponding to the at least one frame of sample image.

34. The training device according to claim 33, characterized in that The part of pixels includes pixels whose pixel values ​​are less than or equal to a first threshold.

35. The training device according to claim 30 or 31, characterized in that The loss function is determined based on at least two of the following: content loss, structural similarity loss, contextual bilateral loss, or color loss.

36. The training device according to claim 35, characterized in that The loss function L satisfies the formula: L=w*(α*L content +β*L SSIM +γ*L cobi +p*L color ); Among them, L content Indicates the content loss, L SSIM represents the structural similarity loss, L cobi represents the contextual bilateral loss, L color represents the color loss, α, β, γ and ρ are parameters, and w represents the weight map.

37. The training device according to claim 30 or 31, characterized in that The processing unit is further configured to: Using a second image processing model to perform forward propagation based on the multiple frames of sample images to obtain a second enhanced image corresponding to the at least one frame of sample image, the second image processing model and the first image processing model sharing a portion of the network structure; And the target loss function is determined according to the loss function of the first image processing model, including: the target loss function is determined according to the loss function of the first image processing model, the loss function of the second image processing model or the distillation loss between the second image processing model and the first image processing model, and the loss function of the second image processing model is used to calculate the difference between the second enhanced image corresponding to the at least one frame of sample image and the target image corresponding to the at least one frame of sample image.

38. The training device according to claim 37, characterized in that The distillation loss between the second image processing model and the first image processing model is determined according to at least one of the following: feature distillation loss or output distillation loss; The feature distillation loss is used to calculate the difference between the features of the intermediate layer of the first image processing model and the features of the intermediate layer of the second image processing model; The output distillation loss is used to calculate the difference between the first enhanced image and the second enhanced image.

39. The training device according to claim 38, characterized in that The distillation loss L between the second image processing model and the first image processing model KD Satisfies the following formula: L KD =a*L feature +b*L output ; Among them, L feature represents the characteristic distillation loss, L output represents the output distillation loss, a represents L feature The weight coefficient, b represents L output The weight coefficient of .

40. The training device according to claim 37, characterized in that The objective loss function satisfies the following formula: L target =L KD +L1+L2; Among them, L target Represents the target loss function, L KD Represents the distillation loss between the first image processing model and the second image processing model, L1 represents the loss function of the first image processing model, and L2 represents the loss function of the second image processing model.

41. The training device according to claim 37, characterized in that The second image processing model includes a second frequency separation module, a second low-frequency processing module, and a second frequency fusion module, wherein: The second frequency separation module is configured to perform frequency separation processing on the multiple frames of sample images respectively, and output a second high-frequency component and a second low-frequency component of each frame of the sample image in the multiple frames of sample images; The second low-frequency processing module is configured to perform fusion processing on the second low-frequency components of the multiple frames of sample images and output the fused second low-frequency components; The second frequency fusion module is configured to perform frequency fusion processing on the fused second low-frequency component and the second high-frequency component of the at least one frame of sample image, and output a second enhanced image corresponding to the at least one frame of sample image; The first frequency separation module and the second frequency separation module share a network structure, and the first low-frequency processing module and the second low-frequency processing module share a part of the network structure.

42. The training device according to claim 41, characterized in that The features of the middle layer of the first image processing model include the features of the first frequency fusion processing module, and the features of the middle layer of the second image processing model include the features of the second frequency fusion processing module.

43. An image processing device, characterized in that include: An acquisition unit, configured to acquire multiple frames of images; A processing unit for: Performing frequency separation processing on the multiple frames of images respectively, and outputting a high-frequency component and a low-frequency component of each frame of the multiple frames of images; Performing a fusion process on the low-frequency components of the multiple frames of image using a convolution process, and outputting the fused low-frequency components; Frequency fusion processing is performed on the fused low-frequency component and the high-frequency component of at least one frame of the multiple images, and an enhanced image corresponding to the at least one frame of the image is output, wherein the number of enhanced images corresponding to the at least one frame of the image is less than the number of the multiple frames of the image.

44. The device according to claim 43, characterized in that The processing unit is further configured to: performing convolution processing on the high frequency components of at least one frame of the multiple frames of images, and The processing unit is specifically configured to: Frequency fusion processing is performed on the fused low-frequency component and the high-frequency component of the at least one frame of image after convolution processing, and an enhanced image corresponding to the at least one frame of image is output.

45. The device according to claim 43 or 44, characterized in that The processing unit is specifically configured to: Frequency separation processing is performed on the multiple frames of images respectively through wavelet transformation, and a high-frequency component and a low-frequency component of each frame of the multiple frames of images are output.

46. ​​The device according to claim 43 or 44, characterized in that The processing unit is specifically configured to: Serially merging the low-frequency components of the multiple frames of image; The convolution process is performed on the result of the serial merging to output the fused low-frequency component.

47. The device according to claim 43, characterized in that The processing unit is specifically configured to: Combining the fused low-frequency component and the high-frequency component of the at least one frame of image in series; Perform convolution processing on the result of the serial merging and output an enhanced image corresponding to the at least one frame of image.

48. The device according to claim 44, characterized in that The processing unit is specifically configured to: Combining the fused low-frequency component and the convolution-processed high-frequency component of the at least one frame of image in series; Perform convolution processing on the result of the serial merging and output an enhanced image corresponding to the at least one frame of image.

49. The device according to claim 43 or 44, characterized in that The processing unit is further configured to: Preprocessing is performed on multiple frames of initial images, and the multiple frames of images are output.

50. The device according to claim 49, characterized in that The processing unit is specifically configured to: Feature extraction is performed on the multiple frames of initial images, and the extracted feature maps of the multiple frames of initial images are used as the multiple frames of images.

51. A training device for an image processing model, characterized in that The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 9 to 21.

52. An image processing device, characterized in that The method comprises a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method according to any one of claims 22 to 29.

53. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program codes for execution by a device, the program codes including codes for executing the method according to any one of claims 9 to 21 or 22 to 29.

Images