Image processing method and electronic device

Abstract

The present application discloses an image processing method and electronic device, belonging to the field of image processing technology. A specific solution includes: obtaining a first image frame at a first moment and a second image frame at a second moment from a video to be processed, wherein the first moment and the second moment are adjacent moments; performing spatial displacement processing on a third image frame according to image offset parameters of the first image frame and the second image frame to obtain a fourth image frame, wherein the third image frame is an image frame after noise reduction processing of the image frame at the first moment; determining a time domain weighting coefficient according to the second image frame and the fourth image frame, and performing weighted processing on the second image frame and the fourth image frame according to the time domain weighting coefficient to obtain a fifth image frame, wherein the fifth image frame is an image frame after noise reduction processing of the image frame at the second moment.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method and electronic equipment. Background Art

[0002] With the development of image processing technology, it is difficult to achieve significant improvement in image quality by relying solely on algorithm optimization. In order to obtain better image quality, it is possible to improve image quality by obtaining better image input.

[0003] The time domain noise reduction algorithm in related technologies usually performs simple image signal processing (ISP) preprocessing on the Raw domain image, then adjusts the processed image to a grayscale distribution close to that of the YUV domain image, and then performs motion area estimation; or directly uses the image that has undergone simple ISP preprocessing to perform all time domain noise reduction.

[0004] However, on the one hand, there are differences between simple ISP preprocessing and complete ISP preprocessing, and the YUV algorithm effect cannot be fully achieved; on the other hand, ISP preprocessing is affected by ISP parameters. Video images are a dynamic process, and ISP parameters will change with scene changes. When ISP parameters change, the image after ISP preprocessing will also change accordingly, causing disturbances and affecting the algorithm effect. On the other hand, due to the large bit width and image size of Raw domain images, ISP preprocessing will also bring additional computing and cache overhead. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method and electronic device that can solve the problem in the related art that the time domain noise reduction algorithm has poor image processing effect and increases computing power.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising: obtaining a first image frame at a first moment and a second image frame at a second moment from a video to be processed, the first moment and the second moment being adjacent moments; performing spatial displacement processing on a third image frame according to image offset parameters of the first image frame and the second image frame to obtain a fourth image frame, wherein the third image frame is an image frame after noise reduction processing of the image frame at the first moment; determining a time domain weighting coefficient according to the second image frame and the fourth image frame, and performing weighted processing on the second image frame and the fourth image frame according to the time domain weighting coefficient to obtain a fifth image frame, wherein the fifth image frame is an image frame after noise reduction processing of the image frame at the second moment.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, comprising: an acquisition module, a spatial registration module and a time domain noise reduction module; the acquisition module is used to acquire a first image frame at a first moment and a second image frame at a second moment from a video to be processed, wherein the first moment and the second moment are adjacent moments; the spatial registration module is used to perform spatial displacement processing on a third image frame according to image offset parameters of the first image frame and the second image frame to obtain a fourth image frame, wherein the third image frame is the image frame after noise reduction processing of the image frame at the first moment; the time domain noise reduction module is also used to determine a time domain weighting coefficient according to the second image frame and the fourth image frame, and perform weighted processing on the second image frame and the fourth image frame according to the time domain weighting coefficient to obtain a fifth image frame, wherein the fifth image frame is the image frame after noise reduction processing of the image frame at the second moment.

[0008] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, a first image frame at a first moment and a second image frame at a second moment can be obtained from a video to be processed, where the first moment and the second moment are adjacent moments; a third image frame is spatially shifted based on image offset parameters of the first and second image frames to obtain a fourth image frame, wherein the third image frame is the image frame after noise reduction processing of the image frame at the first moment; a temporal weighting coefficient is determined based on the second and fourth image frames, and the second and fourth image frames are weighted according to the temporal weighting coefficient to obtain a fifth image frame, wherein the fifth image frame is the image frame after noise reduction processing of the image frame at the second moment. This solution, on the one hand, can directly perform noise reduction on the image frames in the video to be processed, thereby reducing the ISP preprocessing step, avoiding image deviation caused by ISP preprocessing, and reducing the computing power overhead of video noise reduction; on the other hand, since raw domain images without ISP preprocessing are used, it has higher robustness to environmental changes; and on the other hand, since noise reduction processing can be performed on the second image frame at the second moment based on the third image frame, the image quality of the image frames in the video to be processed can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flowcharts of the image processing method provided in the embodiment of the present application;

[0014] Figure 2 This is a schematic diagram of image processing for determining image offset parameters provided by an embodiment of the present application;

[0015] Figure 3 This is the second flowchart of the image processing method provided in the embodiment of the present application;

[0016] Figure 4 This is one of the structural diagrams of the image processing device provided in the embodiment of the present application;

[0017] Figure 5 This is the second structural diagram of the image processing device provided in the embodiment of the present application;

[0018] Figure 6 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0019] Figure 7 This is a hardware diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0021] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0022] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0023] The image processing method provided in the embodiments of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the image processing method. The electronic devices mentioned in the embodiments of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, wearable devices, etc. The image processing method provided in the embodiments of the present application is described below using an electronic device as an example of the execution subject.

[0024] like Figure 1 As shown, the embodiment of the present application provides an image processing method, which may include steps 101 to 103:

[0025] Step 101: Obtain a first image frame at a first moment and a second image frame at a second moment from a video to be processed.

[0026] The first moment and the second moment are adjacent moments.

[0027] It should be noted that the electronic device can perform noise reduction processing on the above-mentioned video to be processed through the image processing method provided in the embodiment of the present application. The video to be processed may include multiple image frames. The embodiment of the present application takes the electronic device performing noise reduction processing on the second image frame at the second moment as an example to explain in detail the image processing method provided in the embodiment of the present application.

[0028] Specifically, the electronic device can obtain the first image frame at the first moment t0 from the video to be processed. and the second image frame at the second time t1 The second image frame It is a Raw image without ISP preprocessing.

[0029] Optionally, the electronic device may obtain a first image frame from the video to be processed through an image sensor. and the second image frame The Bayer pattern of the image may include at least one of the following: an RGGB sensor array, an RYYB sensor array, an RGBW sensor array, and an RGBIR sensor array.

[0030] Step 102: Perform spatial displacement processing on the third image frame according to the image offset parameters of the first image frame and the second image frame to obtain a fourth image frame.

[0031] The third image frame is the image frame after noise reduction processing of the image frame at the first moment.

[0032] Optionally, the electronic device can determine the image offset parameters between the first image frame and the second image frame through a grayscale-based image matching algorithm; wherein the image matching algorithm is one of a block matching algorithm, a mean absolute difference algorithm, a sum of squared errors algorithm, a mean sum of squared errors algorithm, an optical flow method, a normalized product correlation algorithm, and a sequential similarity algorithm.

[0033] Optionally, the electronic device may also determine the image offset parameter through other modules capable of detecting motion vectors, such as a mobile phone gyroscope.

[0034] The following describes in detail a process in which an electronic device determines an image offset parameter between a first image frame and a second image frame, taking a block matching algorithm as an example of an image matching algorithm.

[0035] First, the electronic device can and the second image frame They are divided into (m×n) sub-image blocks of the same size, where 1≤m≤H, 1≤n≤W, and H is the first image frame. and the second image frame The height of W is the first image frame and the second image frame Afterwards, the electronic device can use a block matching algorithm to calculate the width of the first image frame. With the second image frame Image offset parameters of the sub-image block at the corresponding position in the x and y directions.

[0036] For example, Figure 2 As shown, if the electronic device sends the first image frame Split into sub-image blocks and The second image frame Split into sub-image blocks and The electronic device calculates the sub-image block and sub-image blocks Taking the image offset parameters in the x and y directions as an example, the electronic device can use formula (1) to calculate the sub-image block Crop the top, bottom, left and right edges of the image;

[0037]

[0038] Among them, r is the number of cropping rows or columns, M is the height of the sub-image block, and N is the width of the sub-image block. After that, the electronic device can use For the window, Perform a traversal movement with a step size of 1, and calculate the Sum of Absolute Differences (SAD) value at the corresponding pixel position at each movement according to Formula (2) and Formula (3), and then determine the minimum value of all SAD values ​​as the image offset parameter according to Formula (4), Formula (5) and Formula (6).

[0039]

[0040]

[0041] Image offset parameters

[0042] Image offset parameters

[0043] Δ(x min ,y min )=min(Δ) (6)

[0044] Based on the above scheme, since the image offset parameters between the first image frame and the second image frame can be determined by a grayscale-based image matching algorithm, the offset between the first image frame and the second image frame can be determined, thereby providing a basis for obtaining the fourth image frame based on the third image frame.

[0045] After determining the image shift parameters, the electronic device can perform spatial shift processing on the third image frame according to the image shift parameters to obtain the fourth image frame. Specifically, the electronic device can amplify the image shift parameters Vx and Vy to the same size as the second image frame. The same size, then, according to the image offset parameters of each pixel point, the third image frame Move to the second image frame The position of the same spatial perspective, thereby obtaining the fourth image frame

[0046] Optionally, the electronic device may amplify the image offset parameter by a bilinear interpolation method.

[0047] Optionally, the electronic device may use a bicubic interpolation algorithm to convert the third image frame Move to the second image frame Positions with the same airspace perspective.

[0048] Step 103: Determine a time domain weighting coefficient according to the second image frame and the fourth image frame, and perform weighted processing on the second image frame and the fourth image frame according to the time domain weighting coefficient to obtain a fifth image frame.

[0049] The fifth image frame is the image frame after noise reduction processing at the second moment.

[0050] Optionally, determining the fourth image frame Afterwards, the electronic device can and the fourth image frame Determine the time domain weighting coefficient, such as Figure 3 As shown, the process of the electronic device determining the time domain weighting coefficient may include S301-S304:

[0051] S301: Determine a scaling factor image according to the second image frame and the fourth image frame;

[0052] S302: Determine a residual image according to a difference between the second image frame and the fourth image frame;

[0053] S303: Performing point-to-point multiplication on the scaling coefficient image and the residual image to obtain a residual scaling image;

[0054] S304: Determine a time-domain weighting coefficient according to the residual scaling image.

[0055] Specifically, the electronic device can and the fourth image frame Determine the scaling factor image Im ratio , and according to the second image frame and the fourth image frame Determine the residual image I dif ,in, Afterwards, the electronic device can convert the scaling factor image Im into ratio and the residual image I dif Perform point-to-point multiplication to obtain the residual scaled image Iamp (x,y); Finally, the electronic device can scale the image I according to the residual amp (x,y) determines the time domain weighting coefficient;

[0056] I amp (x,y)=Im ratio (x,y)·I dif (x,y) (7)

[0057] Based on the above solution, since the time domain weighting coefficient can be determined according to the second image frame and the fourth image frame, it provides a basis for the electronic device to perform weighted processing on the second image frame and the fourth image frame according to the time domain weighting coefficient.

[0058] Optionally, the electronic device may and the fourth image frame Determine the scaling factor image Im ratio The process may specifically include: filtering the second image frame by the first filter function respectively and the fourth image frame Perform filtering processing to obtain a second filtered image and a fourth filtered image, wherein the second filtered image corresponds to the second image frame, and the fourth filtered image corresponds to the fourth image frame; then, perform weighted processing on the second filtered image and the fourth filtered image to obtain a weighted image; and determine a scaling factor image based on the weighted image.

[0059] Specifically, the electronic device can filter the second image frame by the first filter function. Perform filtering to obtain the second filtered image The fourth image frame is filtered by the first filter function Perform filtering to obtain the fourth filtered image in, f lowpass (x) is the low-pass filter function, and δ is the filter strength. Afterwards, the electronic device can filter the second filtered image according to formula (8) and the fourth filtered image Perform weighted processing to obtain weighted image I lum Finally, the electronic device can be used according to formula (9) and the weighted image I lum Determine the scaling factor image Im ratio .

[0060]

[0061]

[0062] Wherein, α is a weighting coefficient, and α∈[0,1], Q is the input data bit width, and γ is a nonlinear coefficient greater than 1.

[0063] Optionally, the embodiment of the present application adopts mean filtering, the above δ can be the filtering radius, the above α can be 0.5, and γ can be 2.

[0064] Based on the above solution, since the scaling coefficient image can be determined by filtering and weighting the second image frame and the fourth image frame, it can provide a basis for obtaining the residual scaling image.

[0065] Optionally, the process of the electronic device determining the time domain weighting coefficient according to the residual scaled image may specifically include: the electronic device may use a second filter function to scale the residual scaled image I amp (x,y) is filtered to obtain the residual scaled filtered image The electronic device can then scale the residual filtered image The pixel values ​​are normalized to obtain the time domain weighting coefficient A(x,y).

[0066] Optionally, the electronic device can scale the residual image I according to formula (10): amp (x,y) is filtered, δ amp is the filter strength.

[0067]

[0068] Optionally, the electronic device may be configured to Scale the filtered image to the residual The pixel values ​​of A(x,y) are normalized, where ω is the normalization coefficient. The closer the pixel value in A(x,y) is to 1, the higher the similarity between the two images at that position. Otherwise, the similarity is lower.

[0069] Based on the above solution, since the time domain weighting coefficient can be obtained by filtering and normalizing the residual scaled image, it can provide a basis for weighting the second image frame and the fourth image frame according to the time domain weighting coefficient.

[0070] Optionally, the electronic device may calculate the second image frame according to formula (11) and A(x, y). and the fourth image frame Perform weighted processing to obtain the fifth image frame I out (x,y).

[0071]

[0072] Optionally, the first filter function and the second filter function may be any spatial domain low-pass filter function such as a Gaussian filter function, a threshold filter function, etc. The first filter function and the second filter function may be the same filter function or different filter functions.

[0073] In the embodiment of the present application, on the one hand, since the image frames in the video to be processed can be directly subjected to noise reduction processing, not only can the ISP preprocessing steps be reduced and image deviation caused by ISP preprocessing be avoided, but the computing power overhead of video noise reduction can also be reduced; on the other hand, since Raw domain images that have not been ISP preprocessed are used, it has higher robustness to environmental changes; on the other hand, since the second image frame at the second moment can be subjected to noise reduction processing based on the third image frame, the image quality of the image frames in the video to be processed can be improved.

[0074] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the image processing device executing the image processing method as an example.

[0075] like Figure 4 As shown, an embodiment of the present application further provides an image processing device 400, comprising: an acquisition module 410, a spatial registration module 420, and a temporal noise reduction module 430. The acquisition module 410 is configured to acquire a first image frame at a first moment and a second image frame at a second moment from a video to be processed, wherein the first moment and the second moment are adjacent moments; the spatial registration module 420 is configured to perform spatial displacement processing on a third image frame according to image offset parameters of the first image frame and the second image frame to obtain a fourth image frame, wherein the third image frame is an image frame after noise reduction processing of the image frame at the first moment; the temporal noise reduction module 430 is configured to determine a temporal weighting coefficient based on the second image frame and the fourth image frame, and perform weighted processing on the second image frame and the fourth image frame according to the temporal weighting coefficient to obtain a fifth image frame, wherein the fifth image frame is an image frame after noise reduction processing of the image frame at the second moment.

[0076] Alternatively, as Figure 5 As shown, the temporal noise reduction module 430 may include a scaling coefficient estimation submodule 431, an inter-frame residual submodule 432, a residual scaling submodule 433, and a coefficient conversion submodule 434. The scaling coefficient estimation submodule 431 is configured to determine a scaling coefficient image based on the second image frame and the fourth image frame; the inter-frame residual submodule 432 is configured to determine a residual image based on the difference between the second image frame and the fourth image frame; the residual scaling submodule 433 is configured to perform a point-to-point multiplication on the scaling coefficient image and the residual image to obtain a residual scaled image; and the coefficient conversion submodule 434 is configured to determine a temporal weighting coefficient based on the residual scaled image.

[0077] Optionally, the scaling coefficient estimation submodule 431 is specifically used to filter the second image frame and the fourth image frame respectively through a first filtering function to obtain a second filtered image and a fourth filtered image, where the second filtered image corresponds to the second image frame and the fourth filtered image corresponds to the fourth image frame; weighting the second filtered image and the fourth filtered image to obtain a weighted image; and determining the scaling coefficient image based on the weighted image.

[0078] Optionally, the coefficient conversion submodule 434 is specifically configured to filter the residual scaled image using a second filter function to obtain a residual scaled filtered image; and normalize the pixel values ​​of the residual scaled filtered image to obtain the time domain weighting coefficient.

[0079] Optionally, the spatial registration module 420 is further used to determine the image offset parameters between the first image frame and the second image frame by a grayscale-based image matching algorithm before performing spatial displacement processing on the third image frame according to the image offset parameters of the first image frame and the second image frame to obtain the fourth image frame; wherein the image matching algorithm is one of a block matching algorithm, a mean absolute difference algorithm, an error sum of squares algorithm, an average error sum of squares algorithm, a normalized product correlation algorithm, and a sequential similarity algorithm.

[0080] In the embodiment of the present application, on the one hand, since the image frames in the video to be processed can be directly subjected to noise reduction processing, not only can the ISP preprocessing steps be reduced and image deviation caused by ISP preprocessing be avoided, but the computing power overhead of video noise reduction can also be reduced; on the other hand, since Raw domain images that have not been ISP preprocessed are used, it has higher robustness to environmental changes; on the other hand, since the second image frame at the second moment can be subjected to noise reduction processing based on the third image frame, the image quality of the image frames in the video to be processed can be improved.

[0081] The image processing device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0082] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0083] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0084] Alternatively, as Figure 6 As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores a program or instruction that can be run on the processor 601, and when the program or instruction is executed by the processor 601, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0085] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0086] Figure 7 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0087] The electronic device 1000 includes but is not limited to components such as a radio frequency unit 1001 , a network module 1002 , an audio output unit 1003 , an input unit 1004 , a sensor 1005 , a display unit 1006 , a user input unit 1007 , an interface unit 1008 , a memory 1009 , and a processor 1010 .

[0088] Those skilled in the art will understand that the electronic device 1000 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0089] Among them, the sensor 1005 is used to obtain a first image frame at a first moment and a second image frame at a second moment from the video to be processed, wherein the first moment and the second moment are adjacent moments; the processor 1010 is used to perform spatial displacement processing on the third image frame according to the image offset parameters of the first image frame and the second image frame to obtain a fourth image frame, wherein the third image frame is the image frame after the image frame at the first moment is denoised; the processor 1010 is also used to determine a time domain weighting coefficient based on the second image frame and the fourth image frame, and perform weighted processing on the second image frame and the fourth image frame according to the time domain weighting coefficient to obtain a fifth image frame, wherein the fifth image frame is the image frame after the image frame at the second moment is denoised.

[0090] In the embodiment of the present application, on the one hand, since the image frames in the video to be processed can be directly subjected to noise reduction processing, not only can the ISP preprocessing steps be reduced and image deviation caused by ISP preprocessing be avoided, but the computing power overhead of video noise reduction can also be reduced; on the other hand, since Raw domain images that have not been ISP preprocessed are used, it has higher robustness to environmental changes; on the other hand, since the second image frame at the second moment can be subjected to noise reduction processing based on the third image frame, the image quality of the image frames in the video to be processed can be improved.

[0091] Optionally, the processor 1010 is used to determine a scaling factor image based on the second image frame and the fourth image frame; determine a residual image based on the difference between the second image frame and the fourth image frame; perform point-to-point multiplication on the scaling factor image and the residual image to obtain a residual scaling image; and determine a time domain weighting coefficient based on the residual scaling image.

[0092] In an embodiment of the present application, since the time domain weighting coefficient can be determined based on the second image frame and the fourth image frame, it provides a basis for the electronic device to perform weighted processing on the second image frame and the fourth image frame according to the time domain weighting coefficient.

[0093] Optionally, the processor 1010 is specifically used to filter the second image frame and the fourth image frame respectively through a first filtering function to obtain a second filtered image and a fourth filtered image, where the second filtered image corresponds to the second image frame and the fourth filtered image corresponds to the fourth image frame; perform weighted processing on the second filtered image and the fourth filtered image to obtain a weighted image; and determine a scaling coefficient image based on the weighted image.

[0094] In the embodiment of the present application, since the scaling coefficient image can be determined by filtering and weighting the second image frame and the fourth image frame, it can provide a basis for obtaining the residual scaling image.

[0095] Optionally, the processor 1010 is specifically configured to filter the residual scaled image using a second filter function to obtain a residual scaled filtered image; and normalize pixel values ​​of the residual scaled filtered image to obtain the time domain weighting coefficient.

[0096] In the embodiment of the present application, since the time domain weighting coefficient can be obtained by filtering and normalizing the residual scaled image, it can provide a basis for weighting the second image frame and the fourth image frame according to the time domain weighting coefficient.

[0097] Optionally, the processor 1010 is further used to determine the image offset parameters between the first image frame and the second image frame by a grayscale-based image matching algorithm before performing spatial displacement processing on the third image frame according to the image offset parameters of the first image frame and the second image frame to obtain the fourth image frame; wherein the image matching algorithm is one of a block matching algorithm, a mean absolute difference algorithm, an error sum of squares algorithm, an average error sum of squares algorithm, a normalized product correlation algorithm, and a sequential similarity algorithm.

[0098] In an embodiment of the present application, since the image offset parameters between the first image frame and the second image frame can be determined by a grayscale-based image matching algorithm, the offset between the first image frame and the second image frame can be determined, thereby providing a basis for obtaining the fourth image frame based on the third image frame.

[0099] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0100] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0101] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0102] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0103] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0104] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0105] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0106] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0107] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0108] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0109] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, characterized in that: include: Acquire a first image frame at a first moment and a second image frame at a second moment from a video to be processed, where the first moment and the second moment are adjacent moments; performing spatial displacement processing on a third image frame according to image offset parameters of the first image frame and the second image frame to obtain a fourth image frame, wherein the third image frame is an image frame after noise reduction processing on the image frame at the first moment; determining a time-domain weighting coefficient based on the second image frame and the fourth image frame, and performing weighted processing on the second image frame and the fourth image frame according to the time-domain weighting coefficient to obtain a fifth image frame, wherein the fifth image frame is an image frame after the noise reduction processing of the image frame at the second moment; Determining the temporal weighting coefficient based on the second image frame and the fourth image frame includes: determining a scaling factor image based on the second image frame and the fourth image frame; determining a residual image based on a difference between the second image frame and the fourth image frame; performing a point-to-point multiplication calculation on the scaling factor image and the residual image to obtain a residual scaling image; and determining the temporal weighting coefficient based on the residual scaling image. Determining the scaling coefficient image based on the second image frame and the fourth image frame includes: filtering the second image frame and the fourth image frame using a first filtering function to obtain a second filtered image and a fourth filtered image, wherein the second filtered image corresponds to the second image frame and the fourth filtered image corresponds to the fourth image frame; performing weighted processing on the second filtered image and the fourth filtered image to obtain a weighted image; and determining the scaling coefficient image based on the weighted image; Determining the time domain weighting coefficient based on the residual scaled image includes: filtering the residual scaled image through a second filter function to obtain a residual scaled filtered image; and normalizing the pixel values ​​of the residual scaled filtered image to obtain the time domain weighting coefficient.

2. The image processing method according to claim 1, wherein: Before performing spatial displacement processing on the third image frame according to the image offset parameters of the first image frame and the second image frame to obtain the fourth image frame, the method further includes: determining an image offset parameter between the first image frame and the second image frame by using a grayscale-based image matching algorithm; The image matching algorithm is one of a block matching algorithm, a mean absolute difference algorithm, a sum of squared errors algorithm, a mean sum of squared errors algorithm, a normalized product correlation algorithm, and a sequential similarity algorithm.

3. An image processing device, characterized in that: include: Acquisition module, spatial registration module and temporal noise reduction module; The acquisition module is configured to acquire a first image frame at a first moment and a second image frame at a second moment from the video to be processed, wherein the first moment and the second moment are adjacent moments; The spatial registration module is configured to perform spatial displacement processing on a third image frame according to image offset parameters of the first image frame and the second image frame to obtain a fourth image frame, wherein the third image frame is an image frame after noise reduction processing of the image frame at the first moment; The temporal noise reduction module is configured to determine a temporal weighting coefficient based on the second image frame and the fourth image frame, and perform weighted processing on the second image frame and the fourth image frame according to the temporal weighting coefficient to obtain a fifth image frame, wherein the fifth image frame is the image frame after the noise reduction processing of the image frame at the second moment; The time domain noise reduction module includes a scaling coefficient estimation submodule, an inter-frame residual submodule, a residual scaling submodule and a coefficient conversion submodule; The scaling factor estimation submodule is configured to determine a scaling factor image based on the second image frame and the fourth image frame; The inter-frame residual submodule is configured to determine a residual image based on a difference between the second image frame and the fourth image frame; The residual scaling submodule is configured to perform point-to-point multiplication on the scaling coefficient image and the residual image to obtain a residual scaling image; The coefficient conversion submodule is used to determine the time domain weighting coefficient according to the residual zoom image; The scaling coefficient estimation submodule is specifically configured to filter the second image frame and the fourth image frame using a first filtering function to obtain a second filtered image and a fourth filtered image, wherein the second filtered image corresponds to the second image frame and the fourth filtered image corresponds to the fourth image frame; perform weighted processing on the second filtered image and the fourth filtered image to obtain a weighted image; and determine a scaling coefficient image based on the weighted image; The coefficient conversion submodule is specifically used to filter the residual scaling image through a second filtering function to obtain a residual scaling filtered image; and normalize the pixel values ​​of the residual scaling filtered image to obtain the time domain weighting coefficient.

4. The image processing device according to claim 3, wherein The spatial registration module is further used to determine the image offset parameters between the first image frame and the second image frame by a grayscale-based image matching algorithm before performing spatial displacement processing on the third image frame according to the image offset parameters of the first image frame and the second image frame to obtain the fourth image frame; wherein the image matching algorithm is one of a block matching algorithm, a mean absolute difference algorithm, an error sum of squares algorithm, a mean error sum of squares algorithm, a normalized product correlation algorithm, and a sequential similarity algorithm.

Images