Image processing method and device, equipment and computer storage medium

Abstract

The invention discloses an image processing method, device and equipment and a computer storage medium, and the method comprises the steps: carrying out the noise reduction of an obtained first image, and obtaining a second image; fusing the first image and the second image to obtain a third image; performing detail enhancement processing on the third image to obtain a fourth image; and fusing the third image / the first image and the fourth image to obtain a fifth image.

Description

Technical Field

[0001] The embodiments of the present application relate to but are not limited to the field of computer vision, and in particular to an image processing method, apparatus, device and computer storage medium. Background Art

[0002] Image enhancement technology mainly includes dark light brightening, noise reduction, color enhancement, contrast enhancement or detail enhancement, etc. In related technologies, the image enhancement method generally uses fixed image enhancement parameters to process the image, and the quality of the enhanced image is low. Summary of the invention

[0003] Embodiments of the present application provide an image processing method, apparatus, device, and computer storage medium.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0005] Performing noise reduction processing on the acquired first image to obtain a second image;

[0006] fusing the first image and the second image to obtain a third image;

[0007] Performing detail enhancement processing on the third image to obtain a fourth image;

[0008] The third image / the first image and the fourth image are fused to obtain a fifth image.

[0009] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0010] A noise reduction unit, used to perform noise reduction processing on the acquired first image to obtain a second image;

[0011] a fusion unit, configured to fuse the first image and the second image to obtain a third image;

[0012] a detail enhancement unit, configured to perform detail enhancement processing on the third image to obtain a fourth image;

[0013] The fusion unit is further used to fuse the third image / the first image and the fourth image to obtain a fifth image.

[0014] In a third aspect, an embodiment of the present application provides an image processing device, including: a memory and a processor,

[0015] The memory is used to store computer programs.

[0016] The processor is used to call and run the computer program stored in the memory, so that the image processing device executes the method described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect.

[0018] In an embodiment of the present application, the acquired first image is subjected to noise reduction processing to obtain a second image; the first image and the second image are fused to obtain a third image; the third image is subjected to detail enhancement processing to obtain a fourth image; the third image / the first image and the fourth image are fused to obtain a fifth image. In this way, since the image is not only subjected to noise reduction processing but also subjected to detail enhancement, the noise of the obtained fifth image is low and the details are enhanced; in addition, since the first image and the second image are fused to obtain the third image, the third image not only has low noise but also does not lose the content in the first image; since the third image / the first image and the fourth image are fused to obtain the fifth image, the fifth image has enhanced details compared to the third image and does not lose the details in the third image / the first image. Therefore, the image quality of the fifth image obtained in the present application is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0020] Figure 1 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of the implementation flow of another image processing method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of an implementation flow of another image processing method provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a generative adversarial network provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a network generation result provided by an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a framework of an image processing network provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a process of S2D and D2S conversion provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0028] Fig. 9 A hardware entity schematic diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solution of the present application will be described in detail below through embodiments and in conjunction with the accompanying drawings. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0030] It should be noted that in the examples of the present application, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0031] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict. In the description of the present application, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0032] In the related art, the image enhancement technology is relatively simple, the algorithm complexity is relatively low, and it is easy to debug the effect. However, the traditional algorithm does not consider the overall information of the image, so the overall effect is not very good. Therefore, a solution based on deep learning is proposed. In the related art, the deep learning method is supervised learning. For an original image and a target image, the mapping relationship between them is learned to obtain the enhanced image. The basic steps of the deep learning-based solution include the following three steps: data set preparation, offline network training, and online network reasoning.

[0033] In related technologies, the solution for detail enhancement mainly considers denoising and sharpening at the same time when making a data set. The general idea is to add a certain proportion of noise and blur when making a training set to simulate the real situation as much as possible.

[0034] However, in the related art, the image detail enhancement scheme based on deep learning has the following main shortcomings: 1) In terms of data set preparation, most of them adopt the scheme of adding Gaussian noise and Gaussian blur, and do not take into account the camera characteristics of each terminal device. 2) The current enhancement network performs denoising or sharpening alone, and does not consider removing noise and enhancing details at the same time. 3) The current enhancement network mostly uses a single picture as the input of the network, or collects multiple pictures as input at the same time, which does not fully utilize the time domain information of the video and does not consider the execution efficiency of the network on hardware. 4) The current enhancement network does not consider how to debug for different scenarios in actual application.

[0035] In view of at least some of the problems in the related art, the image processing method of the present application is proposed.

[0036] The image processing device in this application can be called user equipment (UE), mobile station (MS), mobile terminal (MT), electronic device, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device. The image processing device in any embodiment of the present application may include one of the following or a combination of at least two of them: Internet of Things (IoT) devices, satellite terminals, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDA), handheld devices with wireless communication functions, computing devices or other processing devices connected to wireless modems, servers, mobile phones, tablet computers (Pad), computers with wireless transceiver functions, PDAs, desktop computers, personal digital assistants, portable media players, smart speakers, navigation devices, smart watches, smart glasses, smart necklaces and other wearable devices, learning machines, translation pens, translation machines, point reading machines, pedometers, digital TVs, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, and wireless terminals in smart cities. The wireless terminals in the Internet of Vehicles (IoV) system include wireless terminals in smart cities, wireless terminals in smart homes, and vehicles, on-board equipment, on-board modules, wireless modems, handheld devices, customer premises equipment (CPE), smart home appliances, etc. in the IoV system.

[0037] Figure 1 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method is applied to an image processing device, and the method includes:

[0038] S101: Perform noise reduction processing on an acquired first image to obtain a second image.

[0039] In some embodiments, the first image may be an image obtained by processing a captured original image. In other embodiments, the first image may be a captured original image. In some embodiments, the original image may be a frame of image obtained by image capture. In other embodiments, the original image may be a frame of image in an original video obtained by video capture.

[0040] In some embodiments, the first image may have one or more layers. For example, when the first image is an image obtained by processing the original captured image, and the processing increases the number of image layers (for example, S2D processing, or preprocessing and S2D processing), the first image may have multiple layers. For another example, when the first image is an original captured image, or when the first image is an image obtained by processing the original captured image, and the processing does not increase the number of image layers (for example, preprocessing), the first image may have one layer. Optionally, the preprocessing may include at least one of the following: at least one of image size change, rotation, scaling, enlargement, cropping, beauty processing, etc.

[0041] In some embodiments, the image processing device may obtain target noise reduction parameters, and perform noise reduction processing on the first image according to the target noise reduction parameters to obtain the second image. In some embodiments, the target noise reduction parameters may include parameters of one or more first network modules included in the noise reduction module. In other embodiments, the target noise reduction parameters may include fixedly set noise reduction parameters.

[0042] In any embodiment of the present application, any first network module may include at least one of the following: one or more convolutional layers, one or more upsampling layers, one or more downsampling layers, one or more activation layers, etc. In some embodiments, the layers included in different first network modules are the same or different. For example, one first network module includes an upsampling layer, and another first network module does not include an upsampling layer. In some embodiments, the parameters in the layers of the same name in different first network modules may be the same or different. In some embodiments, the parameters in the layers of the same name in a first network module may be the same or different. For example, the parameters in different convolutional layers may be the same or different.

[0043] In some embodiments, the first image and the second image have the same resolution. In some embodiments, the first image and the second image have the same number of layers. In other embodiments, the first image and the second image have different numbers of layers.

[0044] S102: Fuse the first image and the second image to obtain a third image.

[0045] In some embodiments, S102 may include: fusing the pixel value of each pixel in the first image with the pixel value of each pixel in the second image to obtain a third image.

[0046] In some embodiments, the third image has the same resolution as the first image. In some embodiments, the third image has the same number of layers as the first image. In other embodiments, the third image has a different number of layers than the first image.

[0047] S103: Perform detail enhancement processing on the third image to obtain a fourth image.

[0048] In some embodiments, the image processing device may obtain a target detail enhancement parameter, and perform detail enhancement processing on the third image according to the target detail enhancement parameter to obtain a fourth image. In some embodiments, the target detail enhancement parameter may include parameters of one or more second network modules included in the noise reduction module. In other embodiments, the target detail enhancement parameter may include a fixedly set detail enhancement parameter.

[0049] In any embodiment of the present application, any second network module may include at least one of the following: one or more convolutional layers, one or more upsampling layers, one or more downsampling layers, one or more activation layers, etc. In some embodiments, the layers included in different second network modules are the same or different. For example, one second network module includes an upsampling layer, and another second network module does not include an upsampling layer. In some embodiments, the parameters in the layers of the same name in different second network modules may be the same or different. In some embodiments, the parameters in the layers of the same name in a second network module may be the same or different. For example, the parameters in different convolutional layers may be the same or different.

[0050] In some embodiments, the third image has the same resolution as the fourth image. In some embodiments, the third image has the same number of layers as the fourth image. In other embodiments, the third image has a different number of layers than the fourth image.

[0051] S104: Fusing the third image / the first image and the fourth image to obtain a fifth image.

[0052] In some embodiments, S104 may include: fusing the pixel value of each pixel in the third image / first image with the pixel value of each pixel in the fourth image to obtain a fifth image.

[0053] In some embodiments, the fifth image has the same resolution as the third image / first image. In some embodiments, the fifth image has the same number of layers as the third image / first image. In other embodiments, the fifth image has a different number of layers than the third image / first image.

[0054] In an embodiment of the present application, the acquired first image is subjected to noise reduction processing to obtain a second image; the first image and the second image are fused to obtain a third image; the third image is subjected to detail enhancement processing to obtain a fourth image; the third image / the first image and the fourth image are fused to obtain a fifth image. In this way, since the image is not only subjected to noise reduction processing but also subjected to detail enhancement, the noise of the obtained fifth image is low and the details are enhanced; in addition, since the first image and the second image are fused to obtain the third image, the third image not only has low noise but also does not lose the content in the first image; since the third image / the first image and the fourth image are fused to obtain the fifth image, the fifth image has enhanced details compared to the third image and does not lose the details in the third image / the first image. Therefore, the image quality of the fifth image obtained in the present application is high.

[0055] Figure 2 A schematic diagram of the implementation flow of another image processing method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method is applied to an image processing device, and the method includes:

[0056] S201 . Perform space-to-depth (S2D) processing on each frame of the image to be processed to obtain the first image corresponding to each frame of the image to be processed.

[0057] In some embodiments, each frame of the image to be processed may be each frame of the image in the original video. In some embodiments, the first image corresponding to each frame of the image to be processed includes multiple layers of images.

[0058] In some embodiments, performing space-to-depth processing on each frame of the image to be processed to obtain the first image corresponding to each frame of the image to be processed may include: dividing the M×N pixels included in each frame of the image to be processed by every P pixels (for convenience, the P pixels are also referred to as the first pixel combination) to obtain (M×N) / P first pixel combinations; extracting the pixel value of the first pixel in the (M×N) / P first pixel combinations to obtain the first layer image in the first image, extracting the pixel value of the second pixel in the (M×N) / P first pixel combinations to obtain the second layer image in the first image, until extracting the pixel value of the Pth pixel in the (M×N) / P first pixel combinations to obtain the Pth layer image in the first image. For example, when P is 2, the first image includes 2 layers of images, when P is 4, the first image includes 4 layers of images, and so on.

[0059] Exemplarily, P may be any integer greater than or equal to 2. For example, P may be 2, 3, 4, 6, 9, etc. In some embodiments, the P pixels may be continuous pixels. In some embodiments, the shape corresponding to the P pixels may be a long direction or a square.

[0060] S202: Perform noise reduction processing on the acquired first image to obtain a second image.

[0061] S203: Fuse the first image and the second image to obtain a third image.

[0062] S204: Perform detail enhancement processing on the third image to obtain a fourth image.

[0063] S205: Fusing the third image / the first image and the fourth image to obtain a fifth image.

[0064] Through S202 to S205, the fifth image corresponding to each frame of the image to be processed can be obtained.

[0065] In some embodiments, when performing noise reduction processing on first images corresponding to different frames of images to be processed, the corresponding target noise reduction parameters may be the same. In other embodiments, when performing noise reduction processing on first images corresponding to different frames of images to be processed, the corresponding target noise reduction parameters may be different. In some embodiments, the target noise reduction parameters of the first image corresponding to the next frame of image to be processed may be determined based on the second image corresponding to the current frame of image to be processed.

[0066] In some embodiments, when detail enhancement processing is performed on the third images corresponding to different frames of images to be processed, the corresponding target detail enhancement parameters may be the same. In other embodiments, when detail enhancement processing is performed on the third images corresponding to different frames of images to be processed, the corresponding target detail enhancement parameters may be different. In some embodiments, the target detail enhancement parameters of the third image corresponding to the next frame of image to be processed may be determined based on the fourth image corresponding to the current frame of image to be processed.

[0067] S206 , performing depth-to-space (D2S) processing on the fifth image corresponding to each frame of the image to be processed to obtain a target image corresponding to each frame of the image to be processed.

[0068] In some embodiments, the fifth image corresponding to each frame of the to-be-processed image is a P-layer image, and the size of each layer of the P-layer image is (M×N) / P.

[0069] In some embodiments, performing depth-to-space processing on the fifth image corresponding to each frame of the image to be processed to obtain a target image corresponding to each frame of the image to be processed may include: dividing the M×N blank pixels every P pixels (for convenience, the P pixels are also referred to as second pixel combinations) to obtain (M×N) / P second pixel combinations; filling the pixel value of each pixel in the first layer image into the first pixel in the (M×N) / P second pixel combinations, and filling the pixel value of each pixel in the second layer image into the second pixel in the (M×N) / P second pixel combinations, until the pixel value of each pixel in the P layer image is filled into the Pth pixel in the (M×N) / P second pixel combinations.

[0070] Figure 3 A schematic diagram of an implementation flow of another image processing method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method is applied to an image processing device, and the method includes:

[0071] S301 , performing noise reduction processing on the first image corresponding to the current frame to be processed image to obtain a second image corresponding to the current frame to be processed image.

[0072] S302: Fusing a first image corresponding to the current frame to be processed image and a second image corresponding to the current frame to be processed image to obtain a third image corresponding to the current frame to be processed image.

[0073] S303, performing detail enhancement processing on the third image corresponding to the current frame to be processed image to obtain a fourth image corresponding to the current frame to be processed image;

[0074] S304: Fusing the third image / first image corresponding to the current frame image to be processed and the fourth image corresponding to the current frame image to be processed to obtain a fifth image corresponding to the current frame image to be processed.

[0075] S305 , extracting features of the fifth image corresponding to the current frame image to be processed, to obtain a feature image corresponding to the current frame image to be processed.

[0076] In some embodiments, S305 may include: according to the feature extraction module, extracting features of the fifth image corresponding to the current frame to be processed image, to obtain a feature image corresponding to the current frame to be processed image. In some embodiments, the feature extraction module may include at least one of the following: one or more convolutional layers, one or more upsampling layers, one or more downsampling layers, one or more activation layers, etc. In some embodiments, the parameters in the layers with the same name in the feature extraction module may be the same or different. For example, the parameters in different convolutional layers may be the same or different.

[0077] In some embodiments, the resolution of the feature image may be the same as or different from the resolution of the third image. In some embodiments, the number of layers of the feature image may be the same as or different from the number of layers of the third image.

[0078] S306. Perform detail enhancement processing on the third image corresponding to the next frame to be processed of the current frame according to the feature image corresponding to the current frame to be processed, so as to obtain the fourth image corresponding to the next frame to be processed of the current frame.

[0079] In some embodiments, S306 may include: fusing the feature image corresponding to the current frame image to be processed and the third image corresponding to the next frame image to be processed to obtain a twelfth image; performing detail enhancement processing on the twelfth image to obtain the fourth image corresponding to the next frame image to be processed of the current frame. In some embodiments, fusing the feature image corresponding to the current frame image to be processed and the third image corresponding to the next frame image to be processed includes: adding the pixel values ​​of each pixel point in the feature image corresponding to the current frame image to be processed and the third image corresponding to the next frame image to be processed to obtain the twelfth image. In other embodiments, fusing the feature image corresponding to the current frame image to be processed and the third image corresponding to the next frame image to be processed includes: superimposing the layers of the feature image corresponding to the current frame image to be processed and the third image corresponding to the next frame image to be processed to obtain the twelfth image.

[0080] In some other embodiments, S306 may include: using some network modules in the detail enhancement module to perform detail enhancement processing on the third image corresponding to the next frame of the image to be processed to obtain the thirteenth image; fusing the feature image corresponding to the current frame of the image to be processed and the thirteenth image to obtain the fourteenth image; using another part of the network modules in the detail enhancement module to perform detail enhancement processing on the fourteenth image to obtain the fourth image corresponding to the next frame of the image to be processed of the current frame. In some embodiments, fusing the feature image corresponding to the current frame of the image to be processed and the thirteenth image to obtain the fourteenth image includes: adding the pixel values ​​of each pixel point in the feature image corresponding to the current frame of the image to be processed and the thirteenth image to obtain the fourteenth image. In other embodiments, fusing the feature image corresponding to the current frame of the image to be processed and the thirteenth image to obtain the fourteenth image includes: superimposing the layers of the feature image corresponding to the current frame of the image to be processed and the thirteenth image to obtain the fourteenth image.

[0081] In an embodiment of the present application, detail enhancement processing is performed on the third image corresponding to the next frame to be processed of the current frame through the feature image corresponding to the current frame to be processed image, so that the detail enhancement of the next frame to be processed can utilize the feature information of the current frame to be processed image, thereby improving the accuracy of detail enhancement.

[0082] In some embodiments, the fusing of the first image and the second image to obtain a third image includes: multiplying the first weight by the pixel value of each pixel in the first image to obtain a sixth image; multiplying the second weight by the pixel value of each pixel in the second image to obtain a seventh image; the sum of the first weight and the second weight is 1; and fusing the sixth image and the seventh image to obtain the third image.

[0083] In some embodiments, the sixth image and the seventh image are fused to obtain the third image, including: adding the pixel values ​​of each pixel in the sixth image and the seventh image to obtain the third image, or multiplying the pixel values ​​of each pixel in the sixth image and the seventh image to obtain the third image.

[0084] In some embodiments, the fusing of the third image / the first image and the fourth image to obtain the fifth image includes: multiplying the third weight by the pixel value of each pixel in the third image / the first image to obtain the eighth image; multiplying the fourth weight by the pixel value of each pixel in the fourth image to obtain the ninth image; the sum of the third weight and the fourth weight is 1; and fusing the eighth image and the ninth image to obtain the fifth image.

[0085] In some embodiments, the eighth image and the ninth image are fused to obtain the fifth image, including: correspondingly adding the pixel values ​​of each pixel point in the eighth image and the ninth image to obtain the fifth image, or correspondingly multiplying the pixel values ​​of each pixel point in the eighth image and the ninth image to obtain the fifth image.

[0086] In some embodiments, the first weights corresponding to different frames of images to be processed may be the same. In other embodiments, the first weights corresponding to different frames of images to be processed may be different. In some embodiments, the third weights corresponding to different frames of images to be processed may be the same. In other embodiments, the fourth weights corresponding to different frames of images to be processed may be different.

[0087] In some embodiments, the first weight may be any real number between [0, 1]. For example, the first weight may be 0, 0.3, 0.5, 0.7 or 1, and so on.

[0088] In some embodiments, the third weight may be any real number between [0, 1]. For example, the third weight may be 0, 0.3, 0.5, 0.7 or 1, etc.

[0089] In other embodiments, the fusing of the first image and the second image to obtain a third image includes: adding the pixel values ​​of each pixel in the first image and the second image accordingly to obtain the third image, or adding the pixel values ​​of each pixel in the first image and the second image accordingly and then dividing by 2 to obtain the third image.

[0090] In other embodiments, the fusing of the third image / the first image and the fourth image to obtain the fifth image includes: adding the pixel values ​​of each pixel in the third image / the first image and the fourth image to obtain the fifth image, or adding the pixel values ​​of each pixel in the third image / the first image and the fourth image to obtain the fifth image and then dividing by 2.

[0091] In some further embodiments, the fusing of the first image and the second image to obtain the third image includes: correspondingly multiplying the pixel values ​​of each pixel point in the first image and the second image to obtain the third image.

[0092] In some other embodiments, the fusing of the third image / the first image and the fourth image to obtain the fifth image includes: multiplying the pixel values ​​of each pixel point in the third image / the first image and the fourth image accordingly to obtain the fifth image.

[0093] In some embodiments, at least one of the first weight, the second weight, the third weight, and the fourth weight is determined based on at least one of: sensitivity (International Organization for Standardization, ISO), dynamic range, scene color temperature, scene, and scene type.

[0094] In some embodiments, at least one of the sensitivity ISO, dynamic range, scene color temperature, scene, and scene type corresponds to each frame of the image to be processed.

[0095] Exemplarily, if the ISO sensitivity is higher, the noise is more, and the first weight can be set smaller and the second weight can be set larger. Conversely, if the ISO sensitivity is lower, the noise is less, and the first weight can be set larger and the second weight can be set smaller. Exemplarily, if the scene or scene type has more details and the noise is larger, the first weight can be set larger and the third weight can be set larger. Exemplarily, if the dynamic range is higher, the noise is larger, and the first weight can be set larger. Exemplarily, if the scene color temperature is higher, the noise is larger, and the first weight can be set larger.

[0096] In some embodiments, the denoising process is performed on the acquired first image to obtain the second image, including: obtaining a tenth image based on the trained N first network modules and the first image; the number of channels of the tenth image is the first number of channels, and the N first network modules are connected in sequence, or the N first network modules are connected in sequence and jump-connected; channel reduction is performed on the tenth image to obtain the second image; the number of channels of the second image is the second number of channels, and the second number of channels is less than the first number of channels.

[0097] In some embodiments, the trained N first network modules are used to perform denoising on the image.

[0098] In some embodiments, obtaining a tenth image based on the trained N first network modules and the first image includes: using the trained N first network modules to perform noise reduction processing on the first image to obtain the tenth image.

[0099] In some embodiments, the N first network modules are connected in a jump manner, including: the output of the first first network module is connected to the output of the second first network module; the output of the second first network module is connected to the output of the third first network module; until the output of the N-1th first network module is connected to the output of the Nth first network module. In other embodiments, the N first network modules are connected in a jump manner, including: the output of the first first network module is connected to the output of the second first network module; the output of the third first network module is connected to the output of the fourth first network module; until the output of the N-1th first network module is connected to the output of the Nth first network module.

[0100] In some embodiments, for the i-th first network module, if the input of the i-th first network module includes: the output of the i-1th first network module and the output of the i-2th first network module, the image obtained by fusing the image of the output of the i-1th first network module and the image of the output of the i-2th first network module is used as the image input to the i-th first network module. In some embodiments, fusing the image of the output of the i-1th first network module and the image of the output of the i-2th first network module may include: superimposing the image of the output of the i-1th first network module and the image of the output of the i-2th first network module, or, correspondingly adding the pixel values ​​of each pixel point in the image of the output of the i-1th first network module and the image of the output of the i-2th first network module, or, correspondingly multiplying the pixel values ​​of each pixel point in the image of the output of the i-1th first network module and the image of the output of the i-2th first network module.

[0101] In some embodiments, the detail enhancement processing of the third image to obtain the fourth image includes: obtaining an eleventh image based on the trained M second network modules and the third image; the number of channels of the eleventh image is the third number of channels, and the M second network modules are connected in sequence, or the M second network modules are connected in sequence and jump-connected; channel reduction is performed on the eleventh image to obtain the fourth image; the number of channels of the fourth image is the second number of channels, and the second number of channels is less than the third number of channels.

[0102] In some embodiments, the third channel number may be the same as or different from the first channel number. In some embodiments, M and N may be the same as or different.

[0103] In some embodiments, the trained M second network modules are used to perform detail enhancement processing on the image.

[0104] In some embodiments, an eleventh image is obtained based on the trained M second network modules and the third image, including: obtaining the eleventh image based on the feature image corresponding to the current frame of the image to be processed, the trained M second network modules and the third image corresponding to the next frame of the image to be processed.

[0105] In some embodiments, based on the feature image corresponding to the current frame image to be processed, the trained M second network modules and the third image corresponding to the next frame image to be processed, an eleventh image is obtained, including: fusing the feature image corresponding to the current frame image to be processed and the third image corresponding to the next frame image to be processed to obtain a twelfth image; using the trained M second network modules to process the twelfth image to obtain the eleventh image.

[0106] In other embodiments, according to the feature image corresponding to the current frame image to be processed, the trained M second network modules and the third image corresponding to the next frame image to be processed, an eleventh image is obtained, including: using some of the trained M second network modules to perform detail enhancement processing on the third image corresponding to the next frame image to be processed to obtain a thirteenth image; fusing the feature image corresponding to the current frame image to be processed and the thirteenth image to obtain a fourteenth image; using another part of the network modules in the detail enhancement module to process the fourteenth image to obtain the eleventh image.

[0107] In some embodiments, the trained N first network modules are obtained by training the parameter value of each first network module in the untrained N first network modules according to each image in the first set and each image in the second set, and according to an L1 loss function and / or a structural similarity (SSIM) loss function;

[0108] Each image in the second set is obtained by adding noise to each corresponding image in the first set.

[0109] In some embodiments, the first set may include a plurality of images. In some embodiments, the second set may include a plurality of images. The number of images in the first set is the same as the number of images in the second set.

[0110] In some embodiments, the first set may be referred to as a low quality (LQ) image set without noise added. In some embodiments, the second set may be referred to as a low quality image set with noise added.

[0111] In some embodiments, the image processing device may train the parameter value of each of the N untrained first network modules to obtain the trained N first network modules. In other embodiments, other devices other than the image processing device may train the parameter value of each of the N untrained first network modules to obtain the trained N first network modules; the image processing device may receive the trained N first network modules sent by other devices.

[0112] In some embodiments, the trained N first network modules are obtained by training the parameter values ​​of each first network module in the untrained N first network modules according to each image in the first set and each image in the second set, according to the L1 loss function and / or the structural similarity SSIM loss function, and according to the S2D processing unit and the first channel reduction processing unit.

[0113] In some embodiments, the parameter value of each first network module in the untrained N first network modules can be continuously updated according to each image in the first set and each image in the second set, as well as according to the L1 loss function and / or the structural similarity SSIM loss function, so that the value calculated by the L1 loss function and / or the structural similarity SSIM loss function for each image in the first set and each image obtained by the first processing unit becomes smaller and smaller until the training completion condition is met and the training is completed.

[0114] In some embodiments, the first processing unit includes: N first network modules of each updated parameter value connected in sequence and a first channel reduction processing unit.

[0115] In some other embodiments, the first processing unit includes: an S2D processing unit connected in sequence, N first network modules with parameter values ​​updated each time, and a first channel reduction processing unit.

[0116] In some further embodiments, the first processing unit includes: an S2D processing unit connected in sequence, N first network modules with parameter values ​​updated each time, a first channel reduction processing unit, and a D2S processing unit.

[0117] In some embodiments, the trained M second network modules are obtained by training the parameter value of each second network module in the untrained M second network modules according to each image in the third set and each image in the first set, and according to at least one of an L1 loss function, a perceptual loss function, and a similarity loss function;

[0118] Each image corresponding to the first set is obtained by degrading each image corresponding to the third set;

[0119] The similarity loss function is used to increase the loss weight corresponding to a specific pixel point in the target image corresponding to each image in the second set when the difference between the pixel values ​​of the specific pixel point in the target image corresponding to each image in the third set is greater than a preset difference.

[0120] In some embodiments, the third set may include a plurality of images. The number of images in the third set is the same as the number of images in the first set.

[0121] In some embodiments, the third set may be referred to as a high definition image set.

[0122] In some embodiments, the image processing device may train the parameter value of each second network module in the M untrained second network modules to obtain the trained M second network modules. In other embodiments, other devices other than the image processing device may train the parameter value of each second network module in the M untrained second network modules to obtain the trained M second network modules; the image processing device may receive the trained M second network modules sent by other devices.

[0123] In some embodiments, the trained M second network modules are trained according to each image in the third set and each image in the first set, and according to at least one of the L1 loss function, the perceptual loss function, and the similarity loss function, and according to the second channel reduction processing unit, the feature extraction unit, and the D2S processing unit (optionally), to obtain the parameter values ​​of each second network module in the untrained M second network modules.

[0124] In some embodiments, the parameter value of each second network module in the M untrained second network modules can be continuously updated according to each image in the third set and each image in the first set, and according to at least one of the L1 loss function, the perceptual loss function, and the similarity loss function, so that each image in the third set and each image obtained by passing each image in the first set through the second processing unit, the value calculated by at least one of the L1 loss function, the perceptual loss function, and the similarity loss function becomes smaller and smaller, until the training completion condition is met and the training is completed.

[0125] In some embodiments, the second processing unit includes: M second network modules of the parameter values ​​updated each time connected in sequence, a second channel reduction processing unit and an S2D processing unit, and the output of the second channel reduction processing unit is connected to the input of the feature extraction module, and the output of the feature extraction module is connected to the first second network module among the M second network modules of the parameter values ​​updated each time connected in sequence.

[0126] In other embodiments, the second processing unit includes: a D2S processing unit, M second network modules with parameter values ​​connected in sequence after each update, a second channel reduction processing unit, and an S2D processing unit, and the output of the second channel reduction processing unit is connected to the input of the feature extraction module, and the output of the feature extraction module is connected to the first second network module among the M second network modules with parameter values ​​connected in sequence after each update.

[0127] In some further embodiments, the second processing unit includes: M second network modules of the parameter values ​​connected in sequence each time after the update and a second channel reduction processing unit, and the output of the second channel reduction processing unit is connected to the input of the feature extraction module, and the output of the feature extraction module is connected to the first second network module among the M second network modules of the parameter values ​​connected in sequence each time after the update.

[0128] In some embodiments, the trained M second network modules are obtained by training the parameter values ​​of each second network module in the untrained M second network modules according to each image in the third set, each image in the second set obtained by the first processing unit, and according to at least one of the L1 loss function, the perceptual loss function, and the similarity loss function.

[0129] In some embodiments, the trained M second network modules are obtained by training the parameter values ​​of each second network module in the untrained M second network modules according to the low-frequency image of each image in the third set, the low-frequency image of each image in the first set and the L1 loss function, and / or according to the high-frequency image of each image in the third set, the high-frequency image of each image in the first set and the perceptual loss function, and / or according to each image in the third set, each image in the first set and the similarity loss function.

[0130] In some embodiments, according to the low-frequency image of each image in the third set, the low-frequency image of each image in the first set and the L1 loss function, and / or according to the high-frequency image of each image in the third set, the high-frequency image of each image in the first set and the perceptual loss function, and / or according to each image in the third set, each image in the first set and the similarity loss function, the parameter value of each second network module in the untrained M second network modules is continuously updated, so that the value calculated by the L1 loss function of the low-frequency image of each image in the third set and each image obtained by the second processing unit through the low-frequency image of each image in the first set is smaller and smaller, and / or so that the value calculated by the perceptual loss function of the high-frequency image of each image in the third set and each image obtained by the second processing unit through the high-frequency image of each image in the first set is smaller and smaller, and / or so that the value calculated by the similarity loss function of each image in the third set and each image obtained by the second processing unit through the second processing unit is smaller and smaller, until the training completion condition is met and the training is completed.

[0131] In some embodiments, the third set includes a set of video data collected by a camera device.

[0132] In some embodiments, the first set is obtained by processing each image in the third set using a degraded parameter convolution kernel.

[0133] In some embodiments, the second set is obtained by adding noise information to each image in the first set; the noise information includes noise parameters corresponding to the camera device.

[0134] In some embodiments, at least one of the first set, the second set, and the third set is obtained by an image processing device, so that the image processing device performs corresponding training. In other embodiments, at least one of the first set, the second set, and the third set is obtained by other devices, so that the other devices perform corresponding training.

[0135] In some embodiments, the degraded parameter convolution kernel is determined based on the convolution kernel of the generative network in the trained generative adversarial network;

[0136] The trained generative adversarial network is obtained by training an untrained generative adversarial network according to the fourth set;

[0137] The image quality of the images in the fourth set is lower than the image quality of the images in the third set.

[0138] In some embodiments, the degraded parameter convolution kernel is obtained by an image processing device. In other embodiments, the degraded parameter convolution kernel is obtained by other devices.

[0139] In some embodiments, the degraded parameter convolution kernel is obtained by normalizing the target convolution kernel;

[0140] The target convolution kernel is obtained by fusing the convolution kernel of the generative network in the trained generative adversarial network.

[0141] In some embodiments, the generative adversarial network includes multiple convolution kernels, so that the convolution kernels of the generative adversarial network generated after training are multiple convolution kernels, and the multiple convolution kernels of the generative network generated in the generative adversarial network after training can be fused to obtain the target convolution kernel. Then, the target convolution kernel is normalized to obtain the degraded parameter convolution kernel.

[0142] In some embodiments, the fusion of multiple convolution kernels may be performed using any of the implementations in the related art. In some embodiments, the fusion of multiple convolution kernels may be replaced by: merging multiple convolution kernels. In some embodiments, if the multiple convolution kernels are A convolution kernels, and the size of each convolution kernel is B×B, the size of the target convolution kernel is [B+(B-1)×(A-1)]×[B+(B-1)×(A-1)]. For example, if the multiple convolution kernels are 2 3×3 convolution kernels, the size of the target convolution kernel is 5×5. For another example, if the multiple convolution kernels are 3 3×3 convolution kernels, the size of the target convolution kernel is 7×7.

[0143] In some embodiments, fusing multiple convolution kernels to obtain the target convolution kernel may include fusing multiple convolution kernels to obtain a large-size convolution kernel, and performing dimensionality reduction on the large-size convolution kernel to obtain the target convolution kernel. In this case, the size of the target convolution kernel may be the same as the size of any one of the multiple convolution kernels.

[0144] In some embodiments, each frame of the image to be processed is each frame of the captured original image.

[0145] In some embodiments, each frame of the image to be processed includes multiple images, and the multiple images in each frame of the image to be processed are determined by performing image segmentation on each frame of the original image taken; different images in each frame of the image to be processed correspond to different noise reduction parameters, and / or different images in each frame of the image to be processed correspond to different detail enhancement parameters.

[0146] In some embodiments, the image processing device may obtain each frame of the original image, and then use an image segmentation model to segment each frame of the original image to obtain multiple images in each frame of the image to be processed.

[0147] In some embodiments, the image segmentation model may include a semantic segmentation network (or semantic segmentation model). In some embodiments, the semantic segmentation network may include one of the following: Fully Convolution Networks (FCN), SegNet, U-Net, DeepLab v1, DeepLab v2, DeepLab v3, DenseNet, E-Net, Link-Net, Mask Regional Convolutional Neural Network (Mask R-CNN), Pyramid Scene Parsing Network (PSPNet), RefineNet, Gated Feedback Refinement Network (G-FRNet), and evolutionary networks of these networks.

[0148] In some embodiments, the image segmentation model may include an edge segmentation network (or edge segmentation model). In some embodiments, the edge segmentation network may include one of the following: a Richer Convolutional Features for Edge Detection (RCF) network, a holistically-nested edge detection (HED) network, a Canny edge detection network, and evolutionary networks of these networks.

[0149] In some embodiments, different images in each frame of the image to be processed have different noise reduction parameters, which may include: different images in each frame of the image to be processed have different parameters in the trained N first network modules. In some embodiments, different images in each frame of the image to be processed have different detail enhancement parameters, which may include: different images in each frame of the image to be processed have different parameters in the trained M second network modules.

[0150] In some embodiments, when the image processing device obtains multiple images in each frame of the image to be processed, it can perform S2D processing on the multiple images in each frame of the image to be processed to obtain the first image corresponding to each image, and then determine the fifth image corresponding to each image based on the first image corresponding to each image, and then perform D2S processing on the fifth image corresponding to each image to obtain the target image corresponding to each image. In this way, the image processing device can obtain multiple target images corresponding to the multiple images in each frame of the image to be processed, and then the image processing device can fuse the multiple target images corresponding to each frame of the image to be processed to obtain the final image of each frame.

[0151] In some embodiments, when determining the fifth image corresponding to the different image based on the first image corresponding to the different image, the corresponding noise reduction parameters and / or detail enhancement parameters are different.

[0152] The embodiment of the present application includes two parts: data set production and network framework design and training.

[0153] The following describes the dataset generation scheme:

[0154] Step 1: Use a terminal device (which may be the same device as the image processing device or a different device) to collect low-quality image data, which may include data of the original focal length of the terminal, or may be data after digital zoom.

[0155] Step 2: Use a generative adversarial network to learn the image degradation parameters of the device.

[0156] Figure 4 A schematic diagram of a generative adversarial network provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the collected data (i.e., the low-quality image data mentioned above) is sent to the generation network, and the generation network will downsample the image at different scales (exemplarily, one or more convolutional layers, and / or, one or more downsampling layers) to obtain a low-resolution, low-quality image. Then, the low-resolution, low-quality image generated by the generation network and the original low-quality image (i.e., the low-quality image data mentioned above) are sent to the discriminant network, and the generation network and the discriminant network are trained separately in an iterative manner. For example, the discriminant network parameters are first fixed, the generation network is trained, and then the discriminant network parameters are fixed, the generation network is trained, and the training is iterated in sequence until the discriminant network cannot distinguish whether the data comes from the original low-quality data set or the low-resolution data generated by the generation network. In some embodiments, the generation network is a fully convolutional network, in which downsampling is achieved by setting the step size in the convolution.

[0157] Figure 5 A schematic diagram of a network generation result provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the generation network may include multiple convolutional layers, for example, multiple convolutional layers are conv1, conv2, and convn. The low-quality image (LowResolution, LR) passes through multiple convolutional layers in sequence to obtain a low-resolution low-quality image (Low Low Resolution, LLR). Figure 5 In a corresponding embodiment, each of the multiple convolutional layers in the generated network may include one or more convolutional kernels. In some embodiments, the number of convolutional kernels in different convolutional layers is the same or different. In some embodiments, the parameters in different convolutional kernels may be the same or different.

[0158] Step 3: extract the convolution weights of the generated network (corresponding to the convolution kernel mentioned above), superimpose and fuse the convolution weights to generate a degraded parameter convolution kernel, and normalize the convolution kernel. In this way, only its parameters are extracted. Although downsampling is performed in the training network nodes, the extracted parameters will not downsample the data.

[0159] In step 4, a high-definition video data set (corresponding to the third set mentioned above) can be collected using professional camera equipment. The obtained degraded parameter convolution kernel is used on the high-quality data set to obtain a low-quality video data set (corresponding to the first set mentioned above).

[0160] In step 5, the noise parameters of the device (i.e., the terminal device mentioned above) can be obtained by calibrating the noise, and a certain amount of noise can be added to the low-quality data set generated above, and the data with added noise (corresponding to the second set mentioned above) can be used as the input of the network.

[0161] In this way, high-quality and low-quality dataset pairs can be obtained for training video enhancement networks.

[0162] The following describes the network framework design and training:

[0163] Figure 6 A schematic diagram of a framework of an image processing network provided in an embodiment of the present application, such as Figure 6 As shown, the image processing network is used to process low-quality images to obtain high-quality images.

[0164] The image processing device may obtain each frame of the image to be processed, and use each frame of the image to be processed as a low quality image (LQ). Each frame of the image to be processed may be subjected to S2D processing to obtain the first image corresponding to each frame of the image to be processed.

[0165] Then, the first image corresponding to each frame of the image to be processed is sent to the denoising module, so that the denoising module outputs the second image corresponding to each frame of the image to be processed. The denoising module includes N trained first network modules and a first channel reduction module, and the N first network modules are connected in sequence and jump-connected.

[0166] Then, the first image corresponding to each frame of the image to be processed and the second image corresponding to each frame of the image to be processed are fused by fusion weight 1 (including the first weight and the second weight) to obtain a third image corresponding to each frame of the image to be processed.

[0167] Then, the third image corresponding to each frame of the image to be processed is input into the detail enhancement module, so that the detail enhancement module outputs the fourth image corresponding to each frame of the image to be processed. The detail enhancement module includes M trained second network modules and a second channel reduction module, and the M second network modules are connected in sequence and jump-connected.

[0168] Then, the third image corresponding to each frame of the image to be processed and the fourth image corresponding to each frame of the image to be processed are fused by fusion weight 2 (including the third weight and the fourth weight) to obtain the fifth image corresponding to each frame of the image to be processed.

[0169] Finally, the fifth image corresponding to each frame of the image to be processed is subjected to D2S processing to obtain a target image corresponding to each frame of the image to be processed, where the target image is a high-quality image (HQ).

[0170] In addition, the feature extraction module can also perform feature extraction on the fifth image corresponding to the current frame to be processed image to obtain the feature image corresponding to the current frame to be processed image. Thus, according to the feature image corresponding to the current frame to be processed image, the third image corresponding to the next frame to be processed image of the current frame is subjected to detail enhancement processing to obtain the fourth image corresponding to the next frame to be processed image of the current frame. For example, the feature image corresponding to the current frame to be processed image is fused with the third image corresponding to each frame to be processed image, and the fused image is sent to the detail enhancement module.

[0171] In the embodiment corresponding to step 6, the network framework consists of two parts. The first part is used to remove noise, and the second part is used to enhance network details. The noise reduction module uses a low-quality video dataset without added noise as the ground truth (GT) (i.e., the data generated in step 4 of data generation) to calculate the loss, and the detail enhancement module uses a high-definition video dataset as the GT to calculate the loss.

[0172] First, use the D2S module to convert the input image into multiple channels to improve the execution efficiency of the network on hardware.

[0173] Figure 7 A schematic diagram of a process of S2D and D2S conversion provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, in the S2D conversion, every 3 pixels are divided, the pixel value of the first pixel in each area is extracted as the first layer image, the pixel value of the second pixel in each area is extracted as the second layer image, and the pixel value of the third pixel in each area is extracted as the third layer image.

[0174] In the D2S conversion, the blank image is divided into every 3 pixels, and the pixel values ​​of the pixels in the first layer image are filled into the first pixel in each area, the pixel values ​​of the pixels in the second layer image are filled into the second pixel in each area, and the pixel values ​​of the pixels in the third layer image are filled into the third pixel in each area.

[0175] It should be noted that, although the division is performed every 3 pixels, in other embodiments, the division may be performed every 2 pixels, or every 4 pixels. The image corresponding to the divided area may be a rectangle or a square.

[0176] In the embodiment of the present application, a plurality of different network structural blocks (also referred to as network modules) are used, and jump connections are used between each structural block.

[0177] In the embodiment of the present application, the multi-channel image is restored to the final image through the D2S module.

[0178] In the embodiment of the present application, the enhancement result of the previous frame is transmitted to the network using a feature extraction module in a loop training manner.

[0179] In an embodiment of the present application, the input LQ image network results are fused using fusion weights. During training, fusion weight 1 is set to a random number between 0 and 1 (ie, the first weight belongs to [0, 1]), and fusion weight 2 is set to 0 (ie, the third weight is 0).

[0180] In the embodiment of the present application, the noise reduction module uses L1 Loss and SSIM Loss. The detail enhancement module first separates the high and low frequencies of the final generated HQ image, uses L1 loss for low-frequency information, and uses perceptual loss for high-frequency information. Since detail enhancement is prone to the generation of black and white edges, a new similarity loss is added, that is, if the predicted image is too different from the original image and exceeds a certain threshold, a larger loss weight is given to the current pixel.

[0181] During the network inference process, the first frame has no reference information of the previous frame, and the incoming information of the previous frame is uniformly filled with 0. That is, the feature image corresponding to the first frame is 0.

[0182] During network inference, different weights are set for fusion weight 1 and fusion weight 2 according to actual needs during forward inference to balance noise and details in different scenes. Different weights can be adjusted or scene detection modules can be added according to ISO, dynamic range, or scene color temperature to enhance the intensity of different details according to different scenes.

[0183] This application proposes a video enhancement solution based on deep learning, which has higher operating efficiency on hardware and can further improve video details and clarity by utilizing time domain information.

[0184] This application proposes a data set production scheme associated with a device to obtain more realistic data for all pairs. This application proposes a new deep learning framework that utilizes time domain information and is easy to implement in hardware, can balance noise and details, and is easy to debug according to the scene. This application designs a new Loss function (i.e., the similarity Loss mentioned above) that can further improve the effect of video enhancement.

[0185] In addition, the present application can add a segmentation network to segment the image content, and then set different noise reduction strengths and detail enhancement strengths based on different areas.

[0186] Based on the foregoing embodiments, an embodiment of the present application provides an image processing device, which includes the units included and the modules included in the units, and can be implemented by a processor in the image processing device; of course, it can also be implemented by a specific logic circuit.

[0187] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the image processing device 800 includes:

[0188] A noise reduction unit 801 is used to perform noise reduction processing on the acquired first image to obtain a second image;

[0189] A fusion unit 802, configured to fuse the first image and the second image to obtain a third image;

[0190] A detail enhancement unit 803, configured to perform detail enhancement processing on the third image to obtain a fourth image;

[0191] The fusion unit 802 is further configured to fuse the third image / the first image and the fourth image to obtain a fifth image.

[0192] In some embodiments, the image processing device 800 also includes: a conversion unit 804, which is used to: perform space-to-depth S2D processing on each frame of the image to be processed to obtain the first image corresponding to each frame of the image to be processed; perform depth-to-space D2S processing on the fifth image corresponding to each frame of the image to be processed to obtain the target image corresponding to each frame of the image to be processed.

[0193] In some embodiments, the image processing device 800 further includes: a feature extraction unit 805, configured to: extract features from the fifth image corresponding to the current frame to be processed image to obtain a feature image corresponding to the current frame to be processed image;

[0194] The detail enhancement unit 803 is also used to: perform detail enhancement processing on the third image corresponding to the next frame to be processed of the current frame according to the feature image corresponding to the current frame to be processed, so as to obtain the fourth image corresponding to the next frame to be processed of the current frame.

[0195] In some embodiments, the fusion unit 802 is further used to: multiply the first weight by the pixel value of each pixel in the first image to obtain a sixth image; multiply the second weight by the pixel value of each pixel in the second image to obtain a seventh image; the sum of the first weight and the second weight is 1; and fuse the sixth image and the seventh image to obtain the third image.

[0196] In some embodiments, the fusion unit 802 is further used to: multiply the third weight by the pixel value of each pixel in the third image / the first image to obtain an eighth image; multiply the fourth weight by the pixel value of each pixel in the fourth image to obtain a ninth image; the sum of the third weight and the fourth weight is 1; and fuse the eighth image and the ninth image to obtain the fifth image.

[0197] In some embodiments, at least one of the first weight, the second weight, the third weight, and the fourth weight is determined according to at least one of the following: ISO sensitivity, dynamic range, scene color temperature, scene, and scene type.

[0198] In some embodiments, the denoising unit 801 is further used to: obtain a tenth image based on the trained N first network modules and the first image; the number of channels of the tenth image is the first number of channels, the N first network modules are connected in sequence, or the N first network modules are connected in sequence and jump-connected; perform channel reduction on the tenth image to obtain the second image; the number of channels of the second image is the second number of channels, and the second number of channels is less than the first number of channels.

[0199] In some embodiments, the detail enhancement unit 803 is further used to: obtain an eleventh image based on the trained M second network modules and the third image; the number of channels of the eleventh image is the third number of channels, and the M second network modules are connected in sequence, or the M second network modules are connected in sequence and jump-connected; perform channel reduction on the eleventh image to obtain the fourth image; the number of channels of the fourth image is the second number of channels, and the second number of channels is less than the third number of channels.

[0200] In some embodiments, the trained N first network modules are obtained by training the parameter value of each first network module in the untrained N first network modules according to each image in the first set and each image in the second set, and according to the L1 loss function and / or the structural similarity SSIM loss function;

[0201] Each image in the second set is obtained by adding noise to each corresponding image in the first set.

[0202] In some embodiments, the trained M second network modules are obtained by training the parameter value of each second network module in the untrained M second network modules according to each image in the third set and each image in the first set, and according to at least one of an L1 loss function, a perceptual loss function, and a similarity loss function;

[0203] Each image corresponding to the first set is obtained by degrading each image corresponding to the third set;

[0204] The similarity loss function is used to increase the loss weight corresponding to a specific pixel point in the target image corresponding to each image in the second set when the difference between the pixel values ​​of the specific pixel point in the target image corresponding to each image in the third set is greater than a preset difference.

[0205] In some embodiments, the trained M second network modules are obtained by training the parameter values ​​of each second network module in the untrained M second network modules according to the low-frequency image of each image in the third set, the low-frequency image of each image in the first set and the L1 loss function, and / or according to the high-frequency image of each image in the third set, the high-frequency image of each image in the first set and the perceptual loss function, and / or according to each image in the third set, each image in the first set and the similarity loss function.

[0206] In some embodiments, the third set includes a set of video data collected by a camera device;

[0207] The first set is obtained by processing each image in the third set using a degraded parameter convolution kernel;

[0208] The second set is obtained by adding noise information to each image in the first set; the noise information includes noise parameters corresponding to the camera device.

[0209] In some embodiments, the degraded parameter convolution kernel is determined based on the convolution kernel of the generative network in the trained generative adversarial network;

[0210] The trained generative adversarial network is obtained by training an untrained generative adversarial network according to the fourth set;

[0211] The image quality of the images in the fourth set is lower than the image quality of the images in the third set.

[0212] In some embodiments, the degraded parameter convolution kernel is obtained by normalizing the target convolution kernel;

[0213] The target convolution kernel is obtained by fusing the convolution kernel of the generative network in the trained generative adversarial network.

[0214] In some embodiments, each frame of the image to be processed is each frame of the original image captured; or,

[0215] Each frame of the image to be processed includes multiple images, and the multiple images in each frame of the image to be processed are determined by performing image segmentation on each frame of the original image taken; different images in each frame of the image to be processed correspond to different noise reduction parameters, and / or different images in each frame of the image to be processed correspond to different detail enhancement parameters.

[0216] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0217] It should be noted that in the embodiments of the present application, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer storage medium. Based on this understanding, the technical solution of the embodiments of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling an image processing device to execute all or part of the methods described in each embodiment of the present application.

[0218] Fig. 9 A hardware entity diagram of an image processing device provided in an embodiment of the present application, such as Fig. 9 As shown, the hardware entity of the image processing device 900 includes: a processor 901 and a memory 902, wherein the memory 902 is used to store computer programs, and the processor 901 is used to call and run the computer program stored in the memory 902, so that the image processing device executes a method for implementing any of the above embodiments.

[0219] The memory 902 stores a computer program that can be run on the processor. The memory 902 is configured to store instructions and applications executable by the processor 901. It can also cache data to be processed or processed by the processor 901 and each module in the image processing device 900 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0220] When the processor 901 executes the program, the steps of any of the above-mentioned image processing methods are implemented. The processor 901 generally controls the overall operation of the image processing device 900.

[0221] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the image processing method of any of the above embodiments.

[0222] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0223] The above-mentioned image processing device or processor may include any one or more of the following integrations: general-purpose processor, application-specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field programmable gate array (FPGA), central processing unit (CPU), graphics processing unit (GPU), embedded neural network processor (neural-network processing units, NPU), controller, microcontroller, microprocessor, programmable logic device, discrete gate or transistor logic device, discrete hardware component. It can be understood that the electronic device that implements the above-mentioned processor function can also be other, and the embodiment of the present application is not specifically limited. The image processing device or processor can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0224] It is understood that the memory or computer storage medium in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0225] It should be understood that the "one embodiment" or "an embodiment" or "an embodiment of the present application" or "the aforementioned embodiment" or "some implementations" or "some embodiments" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" or "an embodiment of the present application" or "the aforementioned embodiment" or "some implementations" or "some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0226] Unless otherwise specified, the image processing device performs any step in the embodiments of the present application, and the processor of the image processing device may perform the step. Unless otherwise specified, the embodiments of the present application do not limit the order in which the image processing device performs the following steps. In addition, the methods used to process data in different embodiments may be the same method or different methods. It should also be noted that any step in the embodiments of the present application can be independently executed by the image processing device, that is, when the image processing device executes any step in the above embodiments, it can be independent of the execution of other steps.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0228] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0229] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0230] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0231] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0232] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0233] Those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0234] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0235] In the embodiments of the present application, the descriptions of the same steps and the same contents in different embodiments can refer to each other. In the embodiments of the present application, the term "and" does not affect the order of the steps. For example, the image processing device executes A and executes B, which means that the image processing device executes A first and then executes B, or the image processing device executes B first and then executes A, or the image processing device executes A and executes B at the same time.

[0236] As used in the embodiments of the present application and the appended claims, the singular forms "a," "an," "said," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0237] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0238] It should be noted that in each embodiment involved in the present application, all steps may be executed or part of the steps may be executed as long as a complete technical solution can be formed.

[0239] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Performing noise reduction processing on the acquired first image to obtain a second image; fusing the first image and the second image to obtain a third image; Performing detail enhancement processing on the third image to obtain a fourth image; The third image / the first image and the fourth image are fused to obtain a fifth image.

2. The method according to claim 1, characterized in that The method further comprises: Performing space-to-depth (S2D) processing on each frame of the image to be processed to obtain the first image corresponding to each frame of the image to be processed; A depth-to-space (D2S) process is performed on the fifth image corresponding to each frame of the image to be processed to obtain a target image corresponding to each frame of the image to be processed.

3. The method according to claim 1, characterized in that The method further comprises: Performing feature extraction on the fifth image corresponding to the current frame image to be processed to obtain a feature image corresponding to the current frame image to be processed; According to the feature image corresponding to the current frame image to be processed, detail enhancement processing is performed on the third image corresponding to the next frame image to be processed of the current frame to obtain the fourth image corresponding to the next frame image to be processed of the current frame.

4. The method according to claim 1, characterized in that: The fusing the first image and the second image to obtain a third image includes: Multiplying the first weight by the pixel value of each pixel in the first image to obtain a sixth image; multiplying the second weight by the pixel value of each pixel in the second image to obtain a seventh image; the sum of the first weight and the second weight is 1; fusing the sixth image and the seventh image to obtain the third image; and / or, The fusing the third image / the first image and the fourth image to obtain a fifth image includes: Multiplying the third weight by the pixel value of each pixel in the third image / the first image to obtain an eighth image; multiplying the fourth weight by the pixel value of each pixel in the fourth image to obtain a ninth image; the sum of the third weight and the fourth weight is 1; The eighth image and the ninth image are fused to obtain the fifth image.

5. The method according to claim 4, characterized in that At least one of the first weight, the second weight, the third weight, and the fourth weight is determined according to at least one of the following: ISO sensitivity, dynamic range, scene color temperature, scene, and scene type.

6. The method according to any one of claims 1 to 5, characterized in that: The step of performing noise reduction processing on the acquired first image to obtain the second image includes: According to the trained N first network modules and the first image, a tenth image is obtained; the number of channels of the tenth image is the first number of channels, and the N first network modules are connected in sequence, or the N first network modules are connected in sequence and jump-connected; Channel reduction is performed on the tenth image to obtain the second image; the number of channels of the second image is a second number of channels, and the second number of channels is less than the first number of channels.

7. The method according to any one of claims 1 to 5, characterized in that: The performing detail enhancement processing on the third image to obtain a fourth image includes: According to the trained M second network modules and the third image, an eleventh image is obtained; the number of channels of the eleventh image is the third number of channels, and the M second network modules are connected in sequence, or the M second network modules are connected in sequence and jump-connected; Channel reduction is performed on the eleventh image to obtain the fourth image; the number of channels of the fourth image is the second number of channels, and the second number of channels is less than the third number of channels.

8. The method according to claim 6, characterized in that The trained N first network modules are obtained by training the parameter value of each first network module in the untrained N first network modules according to each image in the first set and each image in the second set, and according to an L1 loss function and / or a structural similarity SSIM loss function; Each image in the second set is obtained by adding noise to each corresponding image in the first set.

9. The method according to claim 7, characterized in that: The trained M second network modules are obtained by training the parameter value of each second network module in the untrained M second network modules according to each image in the third set and each image in the first set, and according to at least one of an L1 loss function, a perceptual loss function, and a similarity loss function; Each image corresponding to the first set is obtained by degrading each image corresponding to the third set; The similarity loss function is used to increase the loss weight corresponding to a specific pixel point in the target image corresponding to each image in the second set when the difference between the pixel values ​​of the specific pixel point in the target image corresponding to each image in the third set is greater than a preset difference.

10. The method according to claim 9, characterized in that The trained M second network modules are obtained by training the parameter values ​​of each second network module in the untrained M second network modules according to the low-frequency image of each image in the third set, the low-frequency image of each image in the first set and the L1 loss function, and / or according to the high-frequency image of each image in the third set, the high-frequency image of each image in the first set and the perceptual loss function, and / or according to each image in the third set, each image in the first set and the similarity loss function.

11. The method according to any one of claims 8 to 10, characterized in that: The third set includes a set of video data collected by a camera device; The first set is obtained by processing each image in the third set using a degraded parameter convolution kernel; The second set is obtained by adding noise information to each image in the first set; the noise information includes noise parameters corresponding to the camera device.

12. The method according to claim 11, characterized in that The degraded parameter convolution kernel is determined according to the convolution kernel of the generative network in the trained generative adversarial network; The trained generative adversarial network is obtained by training an untrained generative adversarial network according to the fourth set; The image quality of the images in the fourth set is lower than the image quality of the images in the third set.

13. The method according to claim 12, characterized in that The degraded parameter convolution kernel is obtained by normalizing the target convolution kernel; The target convolution kernel is obtained by fusing the convolution kernel of the generative network in the trained generative adversarial network.

14. The method according to claim 2, characterized in that Each frame of the image to be processed is each frame of the original image taken; or, Each frame of the image to be processed includes multiple images, and the multiple images in each frame of the image to be processed are determined by performing image segmentation on each frame of the original image taken; different images in each frame of the image to be processed correspond to different noise reduction parameters, and / or different images in each frame of the image to be processed correspond to different detail enhancement parameters.

15. An image processing device, characterized in that: include: A noise reduction unit, used to perform noise reduction processing on the acquired first image to obtain a second image; a fusion unit, configured to fuse the first image and the second image to obtain a third image; a detail enhancement unit, configured to perform detail enhancement processing on the third image to obtain a fourth image; The fusion unit is further used to fuse the third image / the first image and the fourth image to obtain a fifth image.

16. An image processing device, characterized in that: include: Memory and processor, The memory is used to store computer programs. The processor is used to call and run the computer program stored in the memory, so that the image processing device executes the method according to any one of claims 1 to 14.

17. A computer storage medium, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 14.

Images