Image denoising method, device, system and electronic equipment

Abstract

The embodiment of the present application relates to an image denoising method, device and electronic device. The image denoising method is applied to electronic devices. The method comprises: obtaining a target video, wherein the target video comprises a plurality of image frames, and the plurality of image frames correspond to at least two image domains; performing 3D denoising processing on the image frames in at least two image domains at the same time to obtain the denoised image frames. By performing 3D denoising processing on the image frames in at least two image domains at the same time to obtain the denoised image frames, the embodiment of the present application can solve the image ghosting and blurring problems existing in 3D denoising, thereby improving the display quality of the image.

Description

[Technical field]

[0001] The present application relates to the field of image processing technology, and in particular to an image denoising method, device, system and electronic equipment. [Background technology]

[0002] Night scene aerial photography occupies many application scenarios in the entire aerial photography. In the past, night scene aerial photography purely relied on the size of CMOS sensors. However, there is usually a problem of too dark images in night vision scenes, and usually the image ISO needs to be increased. However, increasing the image ISO generally causes noise in the image, so the image needs to be denoised.

[0003] Traditional image denoising generally performs 3D denoising in a single image domain, and in order to eliminate large noise points, the denoising intensity needs to be increased, resulting in image artifacts and blurring. [Summary of the invention]

[0004] The embodiments of the present application provide an image denoising method, device, system and electronic device. By simultaneously performing 3D denoising processing on image frames in at least two image domains to obtain denoised image frames, the embodiments of the present application can solve the image ghosting and blurring problems existing in 3D denoising, thereby improving the display quality of the image.

[0005] The embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides an image denoising method, which is applied to an electronic device, and the method includes:

[0007] Acquire a target video, wherein the target video includes a plurality of image frames, and the plurality of image frames correspond to at least two image domains;

[0008] The image frames in at least two image domains are simultaneously subjected to 3D denoising processing to obtain denoised image frames.

[0009] In some embodiments, the at least two image domains include: a RAW domain and a YUV domain, and the image frames in the at least two image domains are simultaneously subjected to 3D denoising processing, including:

[0010] Get the 3D denoising results in the RAW domain and the 3D denoising results in the YUV domain;

[0011] According to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain, a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain are determined.

[0012] In some embodiments, determining a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain according to a 3D denoising result in the RAW domain and a 3D denoising result in the YUV domain includes:

[0013] Assume that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 ,but:

[0014] The 4D denoising results in the RAW domain are:

[0015] The 4D denoising results in the YUV domain are:

[0016] Among them, T yuv It is the delay time from the current image frame in RAW domain to the current image frame in YUV domain.

[0017] In some embodiments, obtaining a denoised image frame includes:

[0018] According to the 4D denoising result in the YUV domain, the YUV image after denoising in the YUV domain is determined;

[0019] The denoised YUV image is converted into an RGB image, and the RGB image is used as the denoised image frame.

[0020] In some embodiments, the at least two image domains include: a RAW domain and an RGB domain, and the image frames in the at least two image domains are simultaneously subjected to 3D denoising processing, including:

[0021] Get the 3D denoising results in the RAW domain and the 3D denoising results in the RGB domain;

[0022] According to the 3D denoising result in the RAW domain and the 3D denoising result in the RGB domain, a 4D denoising result in the RAW domain and a 4D denoising result in the RGB domain are determined.

[0023] In some embodiments, determining a 4D denoising result in the RAW domain and a 4D denoising result in the RGB domain according to a 3D denoising result in the RAW domain and a 3D denoising result in the RGB domain includes:

[0024] Assume that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0025] The 4D denoising results in the RAW domain are:

[0026] The 4D denoising result in the RGB domain is:

[0027] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgbIt is the delay time from the current image frame in RAW domain to the current image frame in RGB domain.

[0028] In some embodiments, the at least two image domains include: a YUV domain and an RGB domain, and the image frames in the at least two image domains are simultaneously subjected to 3D denoising processing, including:

[0029] Get the 3D denoising results in the YUV domain and the 3D denoising results in the RGB domain;

[0030] According to the 3D denoising result in the YUV domain and the 3D denoising result in the RGB domain, a 4D denoising result in the YUV domain and a 4D denoising result in the RGB domain are determined.

[0031] In some embodiments, determining a 4D denoising result in the YUV domain and a 4D denoising result in the RGB domain according to a 3D denoising result in the YUV domain and a 3D denoising result in the RGB domain includes:

[0032] Assume that the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0033] The 4D denoising results in the YUV domain are:

[0034] The 4D denoising result in the RGB domain is:

[0035] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0036] In some embodiments, the at least two image domains include: a RAW domain, a YUV domain, and an RGB domain, and the image frames in the at least two image domains are simultaneously subjected to 3D denoising processing, including:

[0037] Get the 3D denoising results in the RAW domain, the 3D denoising results in the YUV domain, and the 3D denoising results in the RGB domain;

[0038] According to the 3D denoising result in the RAW domain, the 3D denoising result in the YUV domain, and the 3D denoising result in the RGB domain, a 4D denoising result in the RAW domain, a 4D denoising result in the YUV domain, and a 4D denoising result in the RGB domain are determined.

[0039] In some embodiments, determining a 4D denoising result in the RAW domain, a 4D denoising result in the YUV domain, and a 4D denoising result in the RGB domain according to the 3D denoising result in the RAW domain, the 3D denoising result in the YUV domain, and the 3D denoising result in the RGB domain includes:

[0040] Assume that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0041] The 4D denoising results in the RAW domain are:

[0042] The 4D denoising results in the YUV domain are:

[0043] The 4D denoising result in the RGB domain is:

[0044] Among them, Y raw3 is the 3D denoising result in the RAW domain, Y yuv3 is the 3D denoising result in the YUV domain, Y rgb3 is the 3D denoising result in RGB domain, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0045] In some embodiments, the method further comprises:

[0046] Determine the delay time of each image domain;

[0047] If there is an image domain whose delay time is greater than a preset time threshold, the image domain is determined as a delayed image domain.

[0048] In some embodiments, the next image domain of the RAW domain is the YUV domain, and the next image domain of the YUV domain is the RGB domain;

[0049] The delay time of the RAW domain is the delay time of the image frame from the RAW domain to the YUV domain;

[0050] The delay time in the YUV domain is the delay time of the image frame from the YUV domain to the RGB domain;

[0051] The delay time of the RGB domain is the delay time of the image frame from the RGB domain to the mobile terminal that is communicatively connected to the electronic device.

[0052] In some embodiments, the delay time is represented by the number of image frames, and the preset time threshold is represented by the number of image frames.

[0053] In some embodiments, the method further comprises:

[0054] After the time-lapse image domain is determined, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined based on the 3D denoising result of the time-lapse image domain.

[0055] In some embodiments, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined based on the 3D denoising result of the time-lapse image domain, including:

[0056] Assuming that the time-lapse image domain is the RAW domain, the 3D denoising result of the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0057] The 4D denoising result in the RAW domain is: raw4 =Y raw3 ;

[0058] The 4D denoising results in the YUV domain are:

[0059] The 4D denoising result in the RGB domain is:

[0060] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0061] In some embodiments, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined based on the 3D denoising result of the time-lapse image domain, including:

[0062] Assuming that the time-lapse image domain is in the YUV domain, the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0063] The 4D denoising results in the RAW domain are:

[0064] The 4D denoising result in the YUV domain is: yuv4 =Y yuv3 ;

[0065] The 4D denoising result in the RGB domain is:

[0066] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in RAW domain to the current image frame in RGB domain.

[0067] In some embodiments, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined based on the 3D denoising result of the time-lapse image domain, including:

[0068] Assume that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0069] The 4D denoising results in the RAW domain are:

[0070] The 4D denoising results in the YUV domain are:

[0071] The 4D denoising result in the RGB domain is: rgb4 =Y rgb3 ;

[0072] Among them, T yuv3 is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0073] In a second aspect, an embodiment of the present application provides an image denoising device, which is applied to an electronic device, and the device includes:

[0074] A video acquisition module, used to acquire a target video, wherein the target video includes a plurality of image frames, and the plurality of image frames correspond to at least two image domains;

[0075] The image denoising module is used to perform 3D denoising processing on at least two image frames in at least two image domains simultaneously to obtain denoised image frames.

[0076] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0077] at least one processor; and

[0078] A memory communicatively connected to at least one processor; wherein the memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can be used to perform the image denoising method as described in the first aspect.

[0079] In a fourth aspect, an embodiment of the present application provides an image denoising system, comprising:

[0080] The electronic device is used to obtain a 3D denoising result in the RAW domain and a 3D denoising result in the YUV domain; and determine a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain according to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain;

[0081] The mobile terminal is used to obtain a 4D denoising result in the YUV domain sent by an electronic device, and perform 3D denoising on the 4D denoising result in the YUV domain in the RGB domain to obtain a denoised image frame.

[0082] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program enables a computer to execute instructions of some or all of the steps described in the first aspect.

[0083] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect. The computer program product may be a software installation package.

[0084] Compared with the prior art, the beneficial effect of the present application is that: an image denoising method provided in an embodiment of the present application is applied to an electronic device, the method comprising: obtaining a target video, wherein the target video comprises a plurality of image frames, and the plurality of image frames correspond to at least two image domains; performing 3D denoising processing on the image frames in at least two image domains at the same time to obtain the denoised image frames. By performing 3D denoising processing on the image frames in at least two image domains at the same time to obtain the denoised image frames, the embodiment of the present application can solve the image ghosting and blurring problems existing in 3D denoising, thereby improving the display quality of the image.

Brief Description of the Drawings

[0085] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0086] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0087] Figure 2 is a schematic diagram of an image data stream provided by an embodiment of the present application;

[0088] Figure 3 It is a flowchart of an image denoising method provided in an embodiment of the present application;

[0089] Figure 4 It is a flowchart of an image denoising method for two image domains provided in an embodiment of the present application;

[0090] Figure 5 It is a flowchart of another image denoising method in two image domains provided in an embodiment of the present application;

[0091] Figure 6 It is a flowchart of another image denoising method in two image domains provided in an embodiment of the present application;

[0092] Figure 7 It is a flowchart of an image denoising method in three image domains provided in an embodiment of the present application;

[0093] Figure 8 is a schematic diagram of a process for determining a delayed image domain provided in an embodiment of the present application;

[0094] Fig. 9 is a schematic diagram of image denoising provided by an embodiment of the present application;

[0095] Fig.10 is a timing diagram of an image denoising method provided in an embodiment of the present application;

[0096] Fig.11 is a structural schematic diagram of an image denoising device provided in an embodiment of the present application;

[0097] Fig.12 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0098] Fig.13 is a structural schematic diagram of a mobile terminal provided in an embodiment of the present application;

[0099] Fig.14 It is a structural schematic diagram of an image denoising system provided in an embodiment of the present application. [Specific implementation method]

[0100] In order to facilitate the understanding of the present application, the present application is described in more detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that when an element is described as "fixed to" another element, it can be directly on another element or there can be one or more centered elements therebetween. When an element is described as "connected" to another element, it can be directly connected to another element or there can be one or more centered elements therebetween. The orientation or positional relationship indicated by the terms "upper", "lower", "inner", "outer", "bottom" and the like used in this specification is based on the orientation or positional relationship shown in the accompanying drawings, only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" and the like are used only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0101] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification and in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0102] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0103] The following example illustrates the application environment of the image denoising method in the embodiment of the present application.

[0104] See also Figure 1 , Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0105] like Figure 1 As shown, the application scenario includes an unmanned aerial vehicle 100 and a mobile terminal 200, wherein the unmanned aerial vehicle 100 is communicatively connected to the mobile terminal 200, for example: the unmanned aerial vehicle 100 is communicatively connected to the mobile terminal 200 via a wireless network, and the pilot or user can operate the mobile terminal 200 to operate the unmanned aerial vehicle 100 via the wireless network.

[0106] In some embodiments, the unmanned aerial vehicle 100 includes: unmanned aerial vehicles such as multi-rotor unmanned aerial vehicles, fixed-wing unmanned aerial vehicles, unmanned helicopters, and hybrid-wing unmanned aerial vehicles. In some embodiments, the unmanned aerial vehicle 100 can also be an unmanned aerial vehicle driven by any type of power, including but not limited to a rotary-wing unmanned aerial vehicle, a fixed-wing unmanned aerial vehicle, a paraglider unmanned aerial vehicle, a flapping-wing unmanned aerial vehicle, and a helicopter model. In the embodiments of the present application, a hybrid-wing unmanned aerial vehicle is used as an example for description.

[0107] In some embodiments, the UAV 100 includes a gimbal camera for capturing images or videos and performing denoising on the captured images or videos, for example, 3D denoising.

[0108] Furthermore, the unmanned aerial vehicle 100 may have a corresponding volume or power according to actual needs, so as to provide a load capacity, flight speed, and flight range that can meet the use requirements. One or more sensors may also be added to the unmanned aerial vehicle 100 so that the unmanned aerial vehicle 100 can collect corresponding data.

[0109] For example, in some embodiments, the UAV 100 is provided with at least one sensor selected from the group consisting of an accelerometer, a gyroscope, a magnetometer, a GPS navigator, and a visual sensor.

[0110] The UAV 100 also includes a flight controller, which serves as the control core of the UAV flight and data transmission, etc., integrating one or more modules to execute corresponding logic control programs.

[0111] In an embodiment of the present application, the unmanned aerial vehicle includes a drone control system, which includes a state machine, a flight controller, a drone power system, and a drone sensor.

[0112] Specifically, the UAV control system includes: a state machine, a flight controller and a UAV power system. Specifically, the state machine connects the flight controller and the UAV power system. The input of the state machine is navigation data and user interaction commands, and the output is control instructions and corresponding flags. The main function of the state machine is to process user interaction commands and use navigation data to realize various functions of the UAV, such as flight mode switching, status monitoring, waypoint flight, return and other upper-level functions. Among them, the user interaction command is an interaction command issued by a ground user, such as remote control stick quantity data, button control commands and other commands. It can be understood that this application is mainly implemented in the state machine. Specifically, the control commands and corresponding flags output by the state machine include position commands, speed commands, acceleration commands, altitude commands, climb rate commands, climb acceleration commands, attitude angle commands, heading angular rate commands, attitude mode flags and position mode flags.

[0113] Specifically, the flight controller connects the state machine and the flight controller, and is used to receive the control command and corresponding flag sent by the state machine, as well as the navigation data sent by the UAV power system, and output the motor speed control command. The flight controller includes two flight modes, namely the position mode and the attitude mode. The main function of the flight controller is to use the control command and navigation data to calculate the motor speed command through a certain algorithm, so that the aircraft can achieve position and attitude control, that is, the position and attitude of the aircraft reach the desired state. Specifically, the battery speed control command, taking a common rotorcraft as an example, the data is the pulse width modulation (PWM) of the control motor.

[0114] Specifically, the UAV power system is connected to the flight controller. The UAV power system includes the UAV execution system and the state monitoring system, which are used to receive the motor speed control command sent by the flight controller, achieve the corresponding speed, thereby achieving the corresponding attitude angle and position, process the sensor data, and indirectly or directly calculate the navigation data. Specifically, the UAV power system processes the UAV sensor data by adopting a fusion algorithm to obtain navigation data. For example, the UAV power system includes GPS, gyroscope, accelerometer, and magnetometer, and the position, speed, and acceleration data of the UAV can be calculated by GPS, gyroscope, accelerometer, and magnetometer. The position, speed, and acceleration data of the UAV can be calculated by binocular vision, gyroscope, accelerometer, and magnetometer. The attitude angle and attitude angular rate of the UAV can be calculated by gyroscope, accelerometer, and magnetometer.

[0115] In some embodiments, the mobile terminal 200 includes a smart terminal, wherein the smart terminal can be any type of smart device for establishing a communication connection with the unmanned aerial vehicle 100, such as a mobile terminal such as a mobile phone, a tablet computer or a smart remote controller. The mobile terminal 200 can be equipped with one or more different user interaction devices for collecting user instructions or displaying and feeding back information to the user. Alternatively, the mobile terminal 200 includes a terminal device, wherein the terminal device includes a computer device, a PC terminal or other device for establishing a communication connection with the unmanned aerial vehicle 100, and the terminal device can be equipped with one or more different user interaction devices for collecting user instructions or displaying and feeding back information to the user.

[0116] The above-mentioned user interaction devices include but are not limited to: buttons, mice, keyboards, display screens, touch screens, speakers, remote control joysticks, etc. For example, the mobile terminal 200 may be equipped with a touch screen, through which the user's remote control instructions for the unmanned aerial vehicle 100 are received and map information, i.e., map screens, is displayed to the user through the touch screen, and image information obtained by aerial photography, i.e., image transmission screens, is displayed to the user. The user may also switch the image information currently displayed on the display screen through the remote control touch screen. The user may also control the movement of the unmanned aerial vehicle, or control the direction of the pan / tilt of the unmanned aerial vehicle, the focal length of the pan / tilt camera of the unmanned aerial vehicle, etc., through the operation of the mouse or the operation of the keys on the keyboard.

[0117] In some embodiments, the UAV 100 and the mobile terminal 200 may also integrate existing image visual processing technologies to further provide more intelligent services. For example, the UAV 100 may collect images through a dual-light camera, and the mobile terminal 200 may analyze the images, thereby enabling the user to control the UAV 100 through gestures.

[0118] In some embodiments, the wireless network can be a wireless communication network based on any type of data transmission principle for establishing a data transmission channel between two nodes, such as a Bluetooth network, a WiFi network, a wireless cellular network or a combination thereof in different signal frequency bands.

[0119] The technical solution of the present application is described below in conjunction with the accompanying drawings:

[0120] Night scene aerial photography occupies many application scenarios in the entire aerial photography. In the past, night scene aerial photography purely relied on the size of CMOS sensors. The current camera CMOS includes mosaic structures. Most of the mosaic structure CMOS are arranged in the order of RGGB. For example, the pixel of CMOS is 7952×5304, and each pixel has a filter (microprism). The first row is R, G, R, G..., and the second row is G, B, G, B... The charge accumulated by each pixel of CMOS is converted into current. The data after A / D conversion is either R, G, or B, and cannot have RGB at the same time. Arranging these pixel data into a matrix is ​​called a color filter array (CFA).

[0121] Assuming that each pixel is 14 bits wide, using TIFF to store CFA, 7952×5304×14 / 8=73.8MB. This is different from color photos in RGB TIFF format. For 16-bit RGB color photos, each pixel has RGB. If stored in TIFF without compression, it is 7952×5304×3×16 / 8=253.1MB.

[0122] A pixel of CFA is either R, G, or B, and the missing RGB is generated by interpolation of the surrounding pixels. For example, if a pixel is R, the G and B of this pixel can be generated by adding the G and B values ​​of other pixels close to this pixel and averaging them. This interpolation process of adding and averaging is called demosaic. After interpolation with the demosaic algorithm, each pixel of CFA has RGB, which is exactly the same as a photo in RGB TIFF format. Each pixel of CFA has only a 14-bit value. If the CFA image is displayed directly, it can only be displayed as a grayscale image without color. After interpolation with the demosaic algorithm, each pixel has the missing RGB supplemented, so it is not a grayscale image, but a color image.

[0123] When the illumination of a night scene is too low, the image is usually raised to the target brightness by increasing the ISO sensitivity. For example, the analog gain and / or digital gain of the camera sensor and / or the ISP gain of the image processing chip are increased, so that the image ISO is raised as a whole, and the dark image is brought to the target brightness. For example, the ISO sensitivity is raised to more than 10 times the normal ISO sensitivity.

[0124] ISO=100*Sensor analog Gain multiple*Sensor digital Gain multiple*ISP Gain multiple.

[0125] It is understandable that when the ISO sensitivity is increased, a large number of noise points will appear in the image, so image denoising is required. At present, 3D denoising is based on spatial denoising and considers the information between adjacent frames in the time domain for denoising. However, in order to eliminate large noise points, 3D denoising needs to strengthen the intensity of 3D denoising, resulting in ghosting and blurring problems in the image. Therefore, this application provides a new image denoising method to solve the ghosting and blurring problems of the image.

[0126] See also Figure 2 , Figure 2 is a schematic diagram of an image data stream provided by an embodiment of the present application;

[0127] In the embodiment of the present application, the dimension from image data acquisition to final display is called the image data stream. There are many image processing modules in this dimension. The data at the image acquisition point is generally a RAW image, called the RAW domain, with automatic exposure, autofocus, ColorBar, cropping and other processing; after the image passes through the demosaic algorithm, it is converted from a RAW image to a YUV image and becomes a YUV domain image with scaling, color, rotation and other processing; the YUV image becomes an H264 / H265 / JPG format bitstream after encoding; the bitstream becomes an RGB image after decoding, and the RGB domain has image enhancement, Gamma and other processing.

[0128] like Figure 2 As shown, the image domain includes RAW domain, YUV domain and RGB domain, wherein the image frames in each image domain are subjected to 3D denoising, and the image processing process is from RAW domain to YUV domain, and then to RGB domain. Among them, each image domain is used to perform denoising on each image frame in the target video, that is, after each image frame is acquired, it is denoised in each image domain, and the image frame in the previous image domain is transferred to the next image domain for processing.

[0129] It can be understood that 3D denoising refers to denoising by considering information between adjacent frames in the time domain on the basis of spatial domain denoising, which will not be elaborated here.

[0130] See also Figure 3 , Figure 3 It is a flowchart of an image denoising method provided in an embodiment of the present application;

[0131] The image denoising method can be applied to electronic devices, and specifically, the execution subject of the image denoising method is one or more processors of the electronic device. The electronic device includes but is not limited to devices for taking pictures or shooting videos carried by unmanned aerial vehicles such as gimbal cameras, mobile terminals, and tablet computers.

[0132] like Figure 3 As shown, the image denoising method comprises:

[0133] Step S301: Acquire a target video, wherein the target video includes a plurality of image frames, and the plurality of image frames correspond to at least two image domains;

[0134] It is understandable that the target video is composed of multiple image frames, and the electronic device, such as the camera, initially obtains RAW files. Since RAW files are raw data directly obtained from the camera's photoelectric sensor, they contain richer content in terms of color and brightness, even beyond the range that the human eye can see. Therefore, it is necessary to further convert, adjust and process the RAW files to generate image files visible to the naked eye, such as RGB images.

[0135] The image domain in the embodiment of the present application includes a RAW domain, a YUV domain and an RGB domain, wherein the image frames in each image domain are subjected to 3D denoising, and the image processing process is from the RAW domain to the YUV domain, and then to the RGB domain, that is, the image is converted from the RAW image to the YUV image, and then from the YUV image to the RGB image, thus, multiple image frames correspond to at least two image domains, that is, at least two of the RAW domain, the YUV domain and the RGB domain.

[0136] Step S302: performing 3D denoising processing on image frames in at least two image domains simultaneously to obtain denoised image frames.

[0137] Specifically, the at least two image domains include at least two of a RAW domain, a YUV domain, and an RGB domain.

[0138] The following are some specific examples:

[0139] (1) At least two image domains include: RAW domain and YUV domain

[0140] Performing 3D denoising processing on image frames in at least two image domains simultaneously includes:

[0141] Get the 3D denoising results in the RAW domain and the 3D denoising results in the YUV domain;

[0142] According to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain, a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain are determined.

[0143] In some embodiments, determining a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain according to a 3D denoising result in the RAW domain and a 3D denoising result in the YUV domain includes:

[0144] Assume that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 ,but:

[0145] The 4D denoising results in the RAW domain are:

[0146] The 4D denoising results in the YUV domain are:

[0147] Among them, T yuv It is the delay time from the current image frame in RAW domain to the current image frame in YUV domain.

[0148] Specifically, Y raw3 Refers to the grayscale image obtained by 3D denoising in the RAW domain, with a value of 0 to 255; Y yuv3 Refers to the grayscale image obtained by 3D denoising in the YUV domain; Y rgb3 Refers to the grayscale image for 3D denoising in the RGB domain.

[0149] In some embodiments, obtaining a denoised image frame includes:

[0150] According to the 4D denoising result in the YUV domain, the YUV image after denoising in the YUV domain is determined;

[0151] The denoised YUV image is converted into an RGB image, and the RGB image is used as the denoised image frame.

[0152] See also Figure 4 , Figure 4 It is a flowchart of an image denoising method for two image domains provided in an embodiment of the present application;

[0153] like Figure 4 As shown, the process of the image denoising method includes:

[0154] Step S401: obtaining 3D denoising results in the RAW domain and 3D denoising results in the YUV domain;

[0155] Specifically, the 3D denoising result in the RAW domain includes the color component of each pixel, and the 3D denoising result in the YUV domain includes the color component of each pixel.

[0156] Step S402: determining a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain according to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain;

[0157] Specifically, assuming that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 ,but:

[0158] The 4D denoising results in the RAW domain are:

[0159] The 4D denoising results in the YUV domain are:

[0160] Among them, T yuvIt is the delay time from the current image frame in RAW domain to the current image frame in YUV domain.

[0161] Step S403: determining a YUV image after denoising in the YUV domain according to the 4D denoising result in the YUV domain;

[0162] Specifically, the 4D denoising result in the YUV domain includes the Y component, U component and V component of each pixel. The Y component, U component and V component of each pixel are fused to obtain a YUV image after denoising in the YUV domain.

[0163] Step S404: converting the denoised YUV image into an RGB image, and using the RGB image as the denoised image frame.

[0164] Specifically, the processing method of converting a YUV image to an RGB image, that is, the conversion relationship between the YUV color space model and the RGB color space model, includes:

[0165]

[0166] is equivalent to:

[0167] R = 1.000Y + 0.000U + 1.140V;

[0168] G = 1.000Y-0.395U-0.581V;

[0169] B=1.000Y+2.032U+0.001V.

[0170] Among them, R represents the R color component of the RGB color space model, G represents the G color component of the RGB color space model, and B represents the B color component of the RGB color space model; Y represents the Y color component of the YUV color space model, U represents the U color component of the YUV color space model, and V represents the V color component of the YUV color space model.

[0171] (2) At least two image domains include: RAW domain and RGB domain

[0172] Performing 3D denoising processing on image frames in at least two image domains simultaneously includes:

[0173] Get the 3D denoising results in the RAW domain and the 3D denoising results in the RGB domain;

[0174] According to the 3D denoising result in the RAW domain and the 3D denoising result in the RGB domain, a 4D denoising result in the RAW domain and a 4D denoising result in the RGB domain are determined.

[0175] In some embodiments, determining a 4D denoising result in the RAW domain and a 4D denoising result in the RGB domain according to a 3D denoising result in the RAW domain and a 3D denoising result in the RGB domain includes:

[0176] Assume that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0177] The 4D denoising results in the RAW domain are:

[0178] The 4D denoising result in the RGB domain is:

[0179] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in RAW domain to the current image frame in RGB domain.

[0180] See also Figure 5 , Figure 5 It is a flowchart of another image denoising method in two image domains provided in an embodiment of the present application;

[0181] like Figure 5 As shown, the process of the image denoising method includes:

[0182] Step S501: obtaining 3D denoising results in the RAW domain and 3D denoising results in the RGB domain;

[0183] Step S502: determining a 4D denoising result in the RAW domain and a 4D denoising result in the RGB domain according to the 3D denoising result in the RAW domain and the 3D denoising result in the RGB domain;

[0184] Specifically, assuming that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0185] The 4D denoising results in the RAW domain are:

[0186] The 4D denoising result in the RGB domain is:

[0187] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in RAW domain to the current image frame in RGB domain.

[0188] Step S503: determining an RGB image according to the 4D denoising result in the RGB domain, and determining the RGB image as the image frame after denoising.

[0189] Specifically, the 4D denoising result in the RGB domain includes the R component, the G component and the B component of each pixel. According to the R component, the G component and the B component of each pixel, an RGB image is obtained by fusion, and the RGB image is determined as the image frame after denoising.

[0190] (3) At least two image domains include: YUV domain and RGB domain

[0191] Performing 3D denoising processing on image frames in at least two image domains simultaneously includes:

[0192] Get the 3D denoising results in the YUV domain and the 3D denoising results in the RGB domain;

[0193] According to the 3D denoising result in the YUV domain and the 3D denoising result in the RGB domain, a 4D denoising result in the YUV domain and a 4D denoising result in the RGB domain are determined.

[0194] In some embodiments, determining a 4D denoising result in the YUV domain and a 4D denoising result in the RGB domain according to a 3D denoising result in the YUV domain and a 3D denoising result in the RGB domain includes:

[0195] Assume that the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0196] The 4D denoising results in the YUV domain are:

[0197] The 4D denoising result in the RGB domain is:

[0198] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0199] Please refer to Figure 6 , Figure 6 It is a flowchart of another image denoising method in two image domains provided in an embodiment of the present application;

[0200] like Figure 6 As shown, the process of the image denoising method includes:

[0201] Step S601: Obtaining 3D denoising results in the YUV domain and 3D denoising results in the RGB domain;

[0202] Step S602: determining a 4D denoising result in the YUV domain and a 4D denoising result in the RGB domain according to the 3D denoising result in the YUV domain and the 3D denoising result in the RGB domain;

[0203] Step S603: determining an RGB image according to the 4D denoising result in the RGB domain, and determining the RGB image as the image frame after denoising.

[0204] Specifically, the 4D denoising result in the RGB domain includes the R component, the G component and the B component of each pixel. According to the R component, the G component and the B component of each pixel, an RGB image is obtained by fusion, and the RGB image is determined as the image frame after denoising.

[0205] (4) At least two image domains include: RAW domain, YUV domain and RGB domain

[0206] Performing 3D denoising processing on image frames in at least two image domains simultaneously includes:

[0207] Get the 3D denoising results in the RAW domain, the 3D denoising results in the YUV domain, and the 3D denoising results in the RGB domain;

[0208] According to the 3D denoising result in the RAW domain, the 3D denoising result in the YUV domain, and the 3D denoising result in the RGB domain, a 4D denoising result in the RAW domain, a 4D denoising result in the YUV domain, and a 4D denoising result in the RGB domain are determined.

[0209] See also Figure 7 , Figure 7 It is a flowchart of an image denoising method in three image domains provided in an embodiment of the present application;

[0210] like Figure 7 As shown, the process of the image denoising method includes:

[0211] Step S701: obtaining a 3D denoising result in the RAW domain, a 3D denoising result in the YUV domain, and a 3D denoising result in the RGB domain;

[0212] Step S702: Determine a 4D denoising result in the RAW domain, a 4D denoising result in the YUV domain, and a 4D denoising result in the RGB domain according to the 3D denoising result in the RAW domain, the 3D denoising result in the YUV domain, and the 3D denoising result in the RGB domain.

[0213] Specifically, according to the 3D denoising result in the RAW domain, the 3D denoising result in the YUV domain, and the 3D denoising result in the RGB domain, a 4D denoising result in the RAW domain, a 4D denoising result in the YUV domain, and a 4D denoising result in the RGB domain are determined, including:

[0214] Assume that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0215] The 4D denoising results in the RAW domain are:

[0216] The 4D denoising results in the YUV domain are:

[0217] The 4D denoising result in the RGB domain is:

[0218] Among them, Y raw3 is the 3D denoising result in the RAW domain, Y yuv3 is the 3D denoising result in the YUV domain, Y rgb3 is the 3D denoising result in RGB domain, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0219] In some embodiments, the 3D denoising result Y raw3 Including the grayscale image obtained by 3D denoising in the RAW domain, whose value range is 0 to 255; 3D denoising result Y yuv3 Including the grayscale image obtained by 3D denoising in the YUV domain, whose value range is 0 to 255; 3D denoising result Y rgb3 Including the grayscale image obtained by 3D denoising in the RGB domain, its value range is 0 to 255. It can be understood that the 4D denoising result Y raw4 , Y yuv4 and Y rgb4 , and the 3D denoising result Y raw3 , Y yuv3 and Y rgb3 The expression form of is the same. If the 3D denoising result is represented by a grayscale image, the 4D denoising result is also represented by a grayscale image.

[0220] In an embodiment of the present application, 3D denoising is performed simultaneously on image frames in at least two image domains, including: processing each color component of the 3D denoising results of the image frames in at least two image domains to obtain each color component of the 4D denoising results.

[0221] It should be noted that in order to unify each color component, the embodiment of the present application unifies the color components of the RAW domain, YUV domain and RGB domain into Y component, U component and V component. That is to say, the color model of the RAW domain is converted into the color model of the YUV domain, and the color model of the RGB domain is converted into the color model of the YUV domain, so that the 3D denoising result of the RAW domain and the 3D denoising result of the YUV domain both include the Y component, the U component and the V component.

[0222] At this time, a 4D denoising result of each image domain is obtained according to the 3D denoising results of at least two image domains. Specifically, 3D denoising is performed on each color component in the 3D denoising result of each image frame to obtain a 4D denoising result.

[0223] Specifically, assuming that the 3D denoising result Y in the RAW domain is raw3 The Y component in is Y a3 , the U component is U a3 , V component is V a3 ; 3D denoising result in YUV domain Y yuv3 The Y component in is Y b3 , the U component is U b3 , V component is V b3 ; 3D denoising result Y in RGB domain rgb3 The Y component in is Y c3 , the U component is U c3 , V component is V c3 ,but:

[0224] 4D denoising result in RAW domain Y raw4 The Y component of:

[0225] 4D denoising result in RAW domain Y raw4 The U component in:

[0226] 4D denoising result in RAW domain Y raw4 The V component in:

[0227] 4D denoising result in YUV domain yuv4 The Y component of:

[0228] 4D denoising result in YUV domain yuv4 The U component in:

[0229] 4D denoising result in YUV domain yuv4 The V component in:

[0230] 4D denoising result Y in RGB domain rgb4 The Y component of:

[0231] 4D denoising result Y in RGB domain rgb4 The U component in:

[0232] 4D denoising result Y in RGB domain rgb4 The V component in:

[0233] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0234] It should be noted that, in the three cases where at least two image domains include a RAW domain and a YUV domain, or where at least two image domains include a RAW domain and an RGB domain, or where at least two image domains include a YUV domain and an RGB domain, the method of obtaining a 4D denoising result based on a 3D denoising result can be applied to the above-mentioned color component processing method, which will not be repeated here.

[0235] In the embodiment of the present application, each image domain corresponds to a denoising module, for example: the RAW domain corresponds to the first denoising module, the first denoising module is used to denoise the image frame in the RAW domain; the YUV domain corresponds to the second denoising module, the second denoising module is used to denoise the image frame in the YUV domain; the RGB domain corresponds to the third denoising module, the third denoising module is used to denoise the image frame in the RGB domain. The first denoising module, the second denoising module, and the third denoising module work simultaneously, that is, the image frames are denoised in parallel.

[0236] Since 3D denoising is performed on each image domain, the denoising intensity of each denoising module is low. While the denoising intensity is low, a large amount of image texture details can be retained, and the image blur is low, so that the image display quality is better.

[0237] In some embodiments, the method further comprises:

[0238] Determine the delay time of each image domain;

[0239] If there is an image domain whose delay time is greater than a preset time threshold, the image domain is determined as a delayed image domain.

[0240] For details, please refer to Figure 8 , Figure 8 is a schematic diagram of a process for determining a delayed image domain provided in an embodiment of the present application;

[0241] like Figure 8 As shown, the process of determining the delayed image domain includes:

[0242] Step S801: Determine the delay time of each image domain;

[0243] Specifically, the processing order of the image domains is RAW domain-YUV domain-RGB domain, that is, the next image domain of the RAW domain is the YUV domain, and the next image domain of the YUV domain is the RGB domain.

[0244] It is understandable that a video is composed of images frame by frame. During the video shooting process, each image frame is processed continuously in the RAW domain, YUV domain and RGB domain. After an image frame is processed in the RAW domain, it will flow to the YUV domain for processing, and finally to the RGB domain for processing. Processing takes time, and there will be a time delay, that is, each image domain requires processing time, which leads to the generation of delay time.

[0245] In the embodiment of the present application, the delay time of the RAW domain is the delay time of the image frame from the RAW domain to the YUV domain; the delay time of the YUV domain is the delay time of the image frame from the YUV domain to the RGB domain; the delay time of the RGB domain is the delay time of the image frame from the RGB domain to the mobile terminal connected to the electronic device for communication.

[0246] Step S802: if there is an image domain whose delay time is greater than a preset time threshold, the image domain is determined as a delayed image domain;

[0247] Specifically, if the delay time of a certain image domain is greater than a preset time threshold, the image domain is determined as a delayed image domain, wherein the preset time threshold is set according to specific needs, for example, set to 1s, 2s, which is not limited here.

[0248] It should be noted that there is only one delayed image domain in the embodiment of the present application. It is understandable that if there are more than two delayed image domains, it is impossible to use the 3D denoising results of multiple image frames to perform 4D denoising operations at the same time.

[0249] In some embodiments, the delay time is represented by the number of image frames, and the preset time threshold is represented by the number of image frames.

[0250] Specifically, the preset time threshold is represented by the number of image frames. For example, if the preset time threshold is 4 image frames, if the delay of a certain image domain exceeds 4 image frames, the image domain is determined to be a delayed image domain, and the image domain does not participate in the 4D denoising operation. For example, the RGB domain usually has a large delay due to image transmission problems, so the RGB domain does not participate in the 4D denoising operation.

[0251] In some embodiments, the method further comprises:

[0252] After the time-lapse image domain is determined, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined based on the 3D denoising result of the time-lapse image domain.

[0253] In some embodiments, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined based on the 3D denoising result of the time-lapse image domain, including:

[0254] Assuming that the time-lapse image domain is the RAW domain, the 3D denoising result of the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0255] The 4D denoising result in the RAW domain is: raw4 =Y raw3 ;

[0256] The 4D denoising results in the YUV domain are:

[0257] The 4D denoising result in the RGB domain is:

[0258] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0259] In some embodiments, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined based on the 3D denoising result of the time-lapse image domain, including:

[0260] Assuming that the time-lapse image domain is in the YUV domain, the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0261] The 4D denoising results in the RAW domain are:

[0262] The 4D denoising result in the YUV domain is: yuv4 =Y yuv3 ;

[0263] The 4D denoising result in the RGB domain is:

[0264] Among them, T yuv is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in RAW domain to the current image frame in RGB domain.

[0265] In some embodiments, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined based on the 3D denoising result of the time-lapse image domain, including:

[0266] Assume that the 3D denoising result in the RAW domain is Y raw3 , the 3D denoising result in the YUV domain is Y yuv3 , the 3D denoising result in the RGB domain is Y rgb3 ,but:

[0267] The 4D denoising results in the RAW domain are:

[0268] The 4D denoising results in the YUV domain are:

[0269] The 4D denoising result in the RGB domain is: rgb4 =Y rgb3 ;

[0270] Among them, T yuv3 is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, T rgb It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

[0271] It should be noted that the above-mentioned method of obtaining a 4D denoising result according to a 3D denoising result can be applied to the processing method of the color component mentioned in the above embodiment, which will not be described in detail here.

[0272] In an embodiment of the present application, in order to further improve the image quality, based on 4D denoising, deep learning AI denoising technology is combined to train multiple scenes.

[0273] For details, please refer to Fig. 9 , Fig. 9 is a schematic diagram of image denoising provided by an embodiment of the present application;

[0274] The image denoising includes:

[0275] Increase ISO sensitivity, 4D denoising and AI denoising.

[0276] Specifically, 4D denoising refers to performing 3D denoising processing on image frames in at least two image domains simultaneously to obtain denoised image frames. For details, reference may be made to the contents provided in the above embodiments, which will not be described in detail here.

[0277] Among them, AI denoising is performed by deep learning of multiple training scenes, such as city, highway, and coastal scenes. Specifically, training sets corresponding to multiple training scenes are obtained, wherein each training scene corresponds to a training set, and the deep learning model is trained through each training set to obtain a trained deep learning model, and the trained deep learning model is used to denoise the image frame after 4D denoising to obtain the final denoised image frame.

[0278] In the embodiment of the present application, the ISO is greatly increased to brighten the overall dim image, and then 4D denoising and AI denoising are used to remove noise points, so that the video picture is pure and the texture details are retained, thereby improving the quality of night scene videos.

[0279] In an embodiment of the present application, since the denoising processing in the RGB domain can be completed by a display terminal, such as a mobile terminal, the present application also provides an image denoising method, which completes image denoising through the joint work of an electronic device and a mobile terminal.

[0280] For details, please refer to Fig.10 , Fig.10 is a timing diagram of an image denoising method provided in an embodiment of the present application;

[0281] like Fig.10 As shown, the timing of the image denoising method includes:

[0282] Step S101: the electronic device obtains a 3D denoising result in the RAW domain and a 3D denoising result in the YUV domain;

[0283] Step S102: the electronic device determines a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain according to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain;

[0284] Step S103: the electronic device sends the 4D denoising result in the YUV domain to the mobile terminal;

[0285] Step S104: the electronic device obtains the 4D denoising result in the YUV domain sent by the mobile terminal;

[0286] Step S105: the electronic device performs 3D denoising on the 4D denoising result in the YUV domain in the RGB domain to obtain a denoised image frame.

[0287] It should be noted that for related content, reference can be made to the content provided in the above embodiments, which will not be repeated here.

[0288] In an embodiment of the present application, an image denoising method is provided, which is applied to an electronic device. The method includes: obtaining a target video, wherein the target video includes multiple image frames, and the multiple image frames correspond to at least two image domains; performing 3D denoising processing on the image frames in at least two image domains at the same time to obtain the denoised image frames. By performing 3D denoising processing on the image frames in at least two image domains at the same time to obtain the denoised image frames, the embodiment of the present application can solve the image ghosting and blurring problems existing in 3D denoising, thereby improving the display quality of the image.

[0289] Please refer to Fig.11 , Fig.11 is a structural schematic diagram of an image denoising device provided in an embodiment of the present application;

[0290] The image denoising device can be applied to an electronic device, the electronic device is arranged on an unmanned aerial vehicle, and the electronic device is communicatively connected to a mobile terminal.

[0291] like Fig.11 As shown, the image denoising device 110 includes:

[0292] The video acquisition module 111 is used to acquire a target video, wherein the target video includes a plurality of image frames, and the plurality of image frames correspond to at least two image domains;

[0293] The image denoising module 112 is configured to perform 3D denoising processing on at least two image frames in at least two image domains simultaneously to obtain denoised image frames.

[0294] It should be noted that the above-mentioned image denoising device can execute the image denoising method provided in the embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in the embodiment of the image denoising device, please refer to the image denoising method provided in the embodiment of the present application.

[0295] In an embodiment of the present application, an image denoising device is provided, which is applied to an electronic device, and the device includes: a video acquisition module, which is used to acquire a target video, wherein the target video includes multiple image frames, and the multiple image frames correspond to at least two image domains; an image denoising module, which is used to simultaneously perform 3D denoising on at least two image frames in at least two image domains to obtain denoised image frames. By simultaneously performing 3D denoising on image frames in at least two image domains to obtain denoised image frames, the embodiment of the present application can solve the image ghosting and blurring problems existing in 3D denoising, thereby improving the display quality of the image.

[0296] See also Fig.12 , Fig.12is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0297] like Fig.12 As shown, the electronic device 120 includes: a processor 121, a memory 122 and a communication module 123. The processor 121, the memory 122 and the communication module 123 are connected to each other by a bus.

[0298] The processor 121 may be any type of processor having one or more processing cores, and may perform single-thread or multi-thread operations, and may be used to parse instructions to perform operations such as acquiring data, performing logical operations, and issuing operation results.

[0299] Specifically, the processor 121 is used to obtain a target video, wherein the target video includes multiple image frames, and the multiple image frames correspond to at least two image domains; and simultaneously perform 3D denoising processing on the image frames in the at least two image domains to obtain denoised image frames.

[0300] In an embodiment of the present application, by simultaneously performing 3D denoising processing on image frames in at least two image domains to obtain denoised image frames, the embodiment of the present application can solve the image ghosting and blurring problems existing in 3D denoising, thereby improving the display quality of the image.

[0301] The memory 122 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the image denoising method in the embodiment of the present application. The processor 121 implements the corresponding logic of the processor by running the non-transitory software programs, instructions and modules stored in the memory 122.

[0302] The memory 122 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the image denoising device, etc. In addition, the memory 122 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 122 may optionally include a memory remotely disposed relative to the processor 121, and these remote memories may be connected to the unmanned aerial vehicle via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0303] The memory 122 stores instructions that can be executed by at least one processor 121; the at least one processor 121 is used to execute the instructions to implement the image denoising method in any of the above method embodiments.

[0304] The communication module 123 is a functional module for establishing a communication connection and providing a physical channel. The communication module 123 can be any type of wireless or wired communication module, including but not limited to a WiFi module or a Bluetooth module.

[0305] An embodiment of the present application also provides an electronic device, including a processor 121, a memory 122, and a computer program stored in the memory 122 and executable on the processor 121. When the computer program is executed by the processor 121, each process of the above-mentioned image denoising method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0306] See also Fig.13 , Fig.13 is a structural schematic diagram of a mobile terminal provided in an embodiment of the present application;

[0307] like Fig.13 As shown, the mobile terminal 130 includes but is not limited to: a radio frequency unit 131, a network module 132, an audio output unit 133, an input unit 134, a sensor 135, a display unit 136, a user input unit 137, an interface unit 138, a memory 139, a processor 1310, and a power supply 1311, and the mobile terminal 130 also includes a camera. Those skilled in the art will understand that Fig.13 The structure of the mobile terminal shown in the figure does not constitute a limitation on the mobile terminal, and the mobile terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present application, the mobile terminal includes but is not limited to a device for shooting images or videos, such as a gimbal camera, a mobile phone, a tablet computer, or a smart remote control.

[0308] Processor 1310 is used to obtain the 4D denoising result in the YUV domain sent by the electronic device, and perform 3D denoising on the 4D denoising result in the YUV domain in the RGB domain to obtain a denoised image frame, and display the denoised image frame, or display the denoised video.

[0309] It should be understood that in the embodiment of the present application, the radio frequency unit 131 can be used for receiving and sending signals during information transmission or calls. Specifically, after receiving downlink data from the base station, it is sent to the processor 1310 for processing; in addition, the uplink data is sent to the base station. Generally, the radio frequency unit 131 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 131 can also communicate with the network and other devices through a wireless communication system.

[0310] The mobile terminal 130 provides users with wireless broadband Internet access through the network module 132, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0311] The audio output unit 133 can convert audio data received by the RF unit 131 or the network module 132 or stored in the memory 139 into an audio signal and output it as sound. Moreover, the audio output unit 133 can also provide audio output related to a specific function performed by the mobile terminal 130 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 133 includes a speaker, a buzzer, a receiver, and the like.

[0312] The input unit 134 is used to receive audio or video signals. The input unit 134 may include a graphics processor (GPU) 1341 and a microphone 1342, and the graphics processor 1341 processes the target image of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 136. The image frame processed by the graphics processor 1341 can be stored in the memory 139 (or other storage medium) or sent via the radio frequency unit 131 or the network module 132. The microphone 1342 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 131 in the case of a telephone call mode.

[0313] The mobile terminal 130 also includes at least one sensor 135, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 1361 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1361 and / or the backlight when the mobile terminal 130 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 135 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0314] The display unit 136 is used to display information input by the user or information provided to the user. The display unit 136 may include a display panel 1361, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0315] The user input unit 137 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile terminal. Specifically, the user input unit 137 includes a touch panel 1371 and other input devices 1372. The touch panel 1371, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1371 or near the touch panel 1371 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 1371 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the contact point coordinates, and then sends it to the processor 1310, receives the command sent by the processor 1310 and executes it. In addition, the touch panel 1371 can be implemented in various types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 1371, the user input unit 137 may also include other input devices 1372. Specifically, other input devices 1372 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail here.

[0316] Furthermore, the touch panel 1371 may be overlaid on the display panel 1361. When the touch panel 1371 detects a touch operation on or near it, it is transmitted to the processor 1310 to determine the type of touch event. Then, the processor 1310 provides corresponding visual output on the display panel 1361 according to the type of touch event. Fig.13 In the figure, the touch panel 1371 and the display panel 1361 are used as two independent components to implement the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1371 and the display panel 1361 can be integrated to implement the input and output functions of the mobile terminal, which is not limited here.

[0317] The interface unit 138 is an interface for connecting an external device to the mobile terminal 130. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 138 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 130 or may be used to transmit data between the mobile terminal 130 and an external device.

[0318] The memory 139 can be used to store software programs and various data. The memory 139 can mainly include a program storage area and a data storage area, wherein the program storage area can store at least one application 1391 required for a function (such as a sound playback function, an image playback function, etc.) and an operating system 1392, etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 139 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0319] The processor 1310 is the control center of the mobile terminal. It uses various interfaces and lines to connect various parts of the entire mobile terminal. It executes various functions of the mobile terminal and processes data by running or executing software programs and / or modules stored in the memory 139, and calling data stored in the memory 139, so as to monitor the mobile terminal as a whole. The processor 1310 may include one or more processing units; in the embodiment of the present application, the processor 1310 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1310.

[0320] The mobile terminal 130 may also include a power supply 1311 (such as a battery) for supplying power to each component. In an embodiment of the present application, the power supply 1311 may be logically connected to the processor 1310 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system.

[0321] In addition, the mobile terminal 130 includes some functional modules not shown, which will not be described in detail here.

[0322] Please refer to Fig.14 , Fig.14 is a structural schematic diagram of an image denoising system provided in an embodiment of the present application;

[0323] like Fig.14 As shown, the image denoising system 140 includes: an electronic device 141 and a mobile terminal 142 , wherein the electronic device 141 is communicatively connected to the mobile terminal 142 .

[0324] The electronic device 141 is used to obtain a 3D denoising result in the RAW domain and a 3D denoising result in the YUV domain; and determine a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain according to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain.

[0325] The mobile terminal 142 is used to obtain the 4D denoising result in the YUV domain sent by the electronic device, and perform 3D denoising on the 4D denoising result in the YUV domain in the RGB domain to obtain the denoised image frame. It is understandable that after obtaining the denoised image frame, the mobile terminal 142 can also display the denoised image frame. Since the image frame is played in a frame-by-frame form, the mobile terminal 142 is also used to display video data composed of multiple image frames, that is, to display the denoised video data.

[0326] Among them, the mobile terminal 142 in the embodiment of the present application can be multiple, and multiple mobile terminals 142 directly communicate and connect to an unmanned aerial vehicle, or some or all of the multiple mobile terminals 142 are connected to the unmanned aerial vehicle through base station communication, so as to obtain the image frame after denoising of the electronic device 141 on the unmanned aerial vehicle, or obtain the video data of the electronic device 141 on the unmanned aerial vehicle after denoising, or control the unmanned aerial vehicle by sending instructions.

[0327] For the relevant contents of the electronic device 141 of the image denoising system 140 in the embodiment of the present application, reference may be made to the electronic device mentioned in the above embodiment, and will not be repeated here.

[0328] For the relevant contents of the mobile terminal 142 of the image denoising system 140 in the embodiment of the present application, reference may be made to the mobile terminal mentioned in the above embodiment, and will not be repeated here.

[0329] In an embodiment of the present application, an image denoising system is provided, including: an electronic device, used to obtain a 3D denoising result in a RAW domain and a 3D denoising result in a YUV domain; according to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain, a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain are determined; a mobile terminal, used to obtain the 4D denoising result in the YUV domain sent by the electronic device, and perform 3D denoising on the 4D denoising result in the YUV domain in the RGB domain to obtain an image frame after denoising.

[0330] The electronic device simultaneously performs 3D denoising on image frames in at least two image domains, and the mobile terminal performs 3D denoising on the 4D denoising results in the YUV domain in the RGB domain to obtain the denoised image frames. The embodiments of the present application can solve the image ghosting and blurring problems existing in 3D denoising, thereby improving the display quality of the image.

[0331] The mobile terminal 142 of the image denoising system 140 in the embodiment of the present application includes but is not limited to mobile phones, smart watches, laptop computers, PDAs, vehicle-mounted terminals and other devices.

[0332] Furthermore, an embodiment of the present application also provides a non-transitory computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, so that the one or more processors can execute the image denoising method in any of the above method embodiments.

[0333] Furthermore, the embodiment of the present application also provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the image denoising method in any of the above method embodiments. The computer program product may be a software installation package.

[0334] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0335] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program in a computer program product. The computer program can be stored in a non-transient computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by the relevant device, the relevant device can execute the process of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0336] The above product can execute the image denoising method provided in the embodiment of the present application, and has the corresponding functional modules and beneficial effects of executing the image denoising method. For technical details not fully described in this embodiment, please refer to the image denoising method provided in the embodiment of the present application.

[0337] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image denoising method, characterized in that: Applied to electronic equipment, the method comprises: Acquire a target video, wherein the target video includes a plurality of image frames, and the plurality of image frames correspond to at least two image domains; Simultaneously performing 3D denoising processing on at least two image frames in the image domain to obtain denoised image frames; If at least two of the image domains include: a RAW domain and a YUV domain, the simultaneously performing 3D denoising processing on the image frames in the at least two image domains includes: Get the 3D denoising results in the RAW domain and the 3D denoising results in the YUV domain; According to the 3D denoising results in the RAW domain and the 3D denoising results in the YUV domain, a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain are determined; The determining of the 4D denoising result in the RAW domain and the 4D denoising result in the YUV domain according to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain includes: Assume that the 3D denoising result in the RAW domain is , the 3D denoising result in the YUV domain is ,but: The 4D denoising results in the RAW domain are: ; The 4D denoising results in the YUV domain are: ; in, It is the delay time from the current image frame in RAW domain to the current image frame in YUV domain.

2. The method according to claim 1, characterized in that The step of obtaining the denoised image frame comprises: According to the 4D denoising result in the YUV domain, the YUV image after denoising in the YUV domain is determined; The denoised YUV image is converted into an RGB image, and the RGB image is used as an image frame after denoising.

3. The method according to claim 1, characterized in that If the at least two image domains include: a RAW domain and an RGB domain, the simultaneously performing 3D denoising processing on the image frames in the at least two image domains includes: Get the 3D denoising results in the RAW domain and the 3D denoising results in the RGB domain; According to the 3D denoising result in the RAW domain and the 3D denoising result in the RGB domain, a 4D denoising result in the RAW domain and a 4D denoising result in the RGB domain are determined.

4. The method according to claim 1, characterized in that: If at least two of the image domains include: a YUV domain and an RGB domain, the simultaneously performing 3D denoising processing on the image frames in the at least two image domains includes: Get the 3D denoising results in the YUV domain and the 3D denoising results in the RGB domain; According to the 3D denoising result in the YUV domain and the 3D denoising result in the RGB domain, a 4D denoising result in the YUV domain and a 4D denoising result in the RGB domain are determined.

5. The method according to claim 1, characterized in that If the at least two image domains include: a RAW domain, a YUV domain, and an RGB domain, the simultaneously performing 3D denoising processing on image frames in at least two of the image domains includes: Get the 3D denoising results in the RAW domain, the 3D denoising results in the YUV domain, and the 3D denoising results in the RGB domain; According to the 3D denoising result in the RAW domain, the 3D denoising result in the YUV domain, and the 3D denoising result in the RGB domain, a 4D denoising result in the RAW domain, a 4D denoising result in the YUV domain, and a 4D denoising result in the RGB domain are determined.

6. The method according to claim 5, characterized in that The method further comprises: Determine the delay time of each image domain; If there is an image domain whose delay time is greater than a preset time threshold, the image domain is determined as a delayed image domain.

7. The method according to claim 6, characterized in that The next image domain after the RAW domain is the YUV domain, and the next image domain after the YUV domain is the RGB domain; The delay time of the RAW domain is the delay time of the image frame from the RAW domain to the YUV domain; The delay time in the YUV domain is the delay time of the image frame from the YUV domain to the RGB domain; The delay time of the RGB domain is the delay time of the image frame from the RGB domain to the mobile terminal that is communicatively connected to the electronic device.

8. The method according to claim 7, characterized in that The delay time is represented by the number of image frames, and the preset time threshold is represented by the number of image frames.

9. The method according to claim 6, characterized in that The method further comprises: After determining the time-lapse image domain, the 3D denoising result of the time-lapse image domain is used as the 4D denoising result of the time-lapse image domain, and the 4D denoising results of other image domains are not determined according to the 3D denoising result of the time-lapse image domain.

10. The method according to claim 9, characterized in that The method of using the 3D denoising result of the time-delay image domain as the 4D denoising result of the time-delay image domain, and not determining the 4D denoising results of other image domains based on the 3D denoising result of the time-delay image domain, includes: Assuming that the time-lapse image domain is the RAW domain, the 3D denoising result in the RAW domain is , the 3D denoising result in the YUV domain is , the 3D denoising result in the RGB domain is ,but: The 4D denoising results in the RAW domain are: ; The 4D denoising results in the YUV domain are: ; The 4D denoising result in the RGB domain is: ; in, is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

11. The method according to claim 9, characterized in that The method of using the 3D denoising result of the time-delay image domain as the 4D denoising result of the time-delay image domain, and not determining the 4D denoising results of other image domains based on the 3D denoising result of the time-delay image domain, includes: Assuming that the time-lapse image domain is in the YUV domain, the 3D denoising result in the RAW domain is , the 3D denoising result in the YUV domain is , the 3D denoising result in the RGB domain is ,but: The 4D denoising results in the RAW domain are: ; The 4D denoising results in the YUV domain are: ; The 4D denoising result in the RGB domain is: ; in, is the delay time from the current image frame in the RAW domain to the current image frame in the YUV domain, It is the delay time from the current image frame in the YUV domain to the current image frame in the RGB domain.

12. An image denoising device, characterized in that: Applied to electronic equipment, the device comprises: A video acquisition module, used to acquire a target video, wherein the target video includes a plurality of image frames, and the plurality of image frames correspond to at least two image domains; An image denoising module, used for simultaneously performing 3D denoising processing on at least two image frames in at least two of the image domains to obtain denoised image frames; If at least two of the image domains include: a RAW domain and a YUV domain, the simultaneously performing 3D denoising processing on the image frames in the at least two image domains includes: Get the 3D denoising results in the RAW domain and the 3D denoising results in the YUV domain; According to the 3D denoising results in the RAW domain and the 3D denoising results in the YUV domain, a 4D denoising result in the RAW domain and a 4D denoising result in the YUV domain are determined; The determining of the 4D denoising result in the RAW domain and the 4D denoising result in the YUV domain according to the 3D denoising result in the RAW domain and the 3D denoising result in the YUV domain includes: Assume that the 3D denoising result in the RAW domain is , the 3D denoising result in the YUV domain is ,but: The 4D denoising results in the RAW domain are: ; The 4D denoising results in the YUV domain are: ; in, It is the delay time from the current image frame in RAW domain to the current image frame in YUV domain.

13. An electronic device, characterized in that: include: at least one processor; as well as A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can be used to execute the image denoising method as described in any one of claims 1-11.

Images