Image noise reduction device and method, processing chip and electronic device

Abstract

An embodiment of the present application discloses an image noise reduction device and method, a processing chip, and an electronic device. The image noise reduction device includes: an image processing module, a filtering processing module, and a noise reduction processing module; wherein, the image processing module is configured to match a first feature point data window in an image block of a reference frame within an image block of the current frame to output image affine transformation information, and parallelly perform feature point detection on the image block of the current frame to obtain a second feature point data window, and the reference frame is temporally before the current frame; the filtering processing module is configured to perform spatio-temporal domain smoothing filtering processing on the image affine transformation information to output motion adjustment information; the noise reduction processing module is configured to perform rotation alignment processing on the reference frame by using the motion adjustment information to obtain a first image frame, and perform noise reduction filtering processing on the current frame and / or the first image frame to output a second image frame, thereby improving the stability and accuracy of image noise reduction processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to an image denoising device and method, a processing chip, and an electronic device. Background Art

[0002] Currently, in video denoising technology, it is usually necessary to perform image warping on image blocks of image frames in a video stream to obtain image warping information, so as to perform operations such as rotation alignment processing on the image frame by using the image warping information. However, in the actual video denoising process, directly using the image warping information to perform rotation alignment processing on the image frame results in a poor rotation alignment effect of the image frame, thereby affecting the stability and accuracy of the image denoising process. Summary of the Invention

[0003] Embodiments of this application provide an image denoising device and method, a processing chip, and an electronic device, which are expected to perform spatio-temporal domain smoothing filtering on the image warping information to output motion adjustment information, so as to obtain more accurate and reasonable motion adjustment information, thereby improving the accuracy of reference frame rotation alignment, as well as enhancing the stability and accuracy of the image denoising process.

[0004] In a first aspect, an embodiment of this application provides an image denoising device, including: an image processing module, a filtering processing module, and a denoising processing module; where

[0005] The image processing module is configured to match a first feature point data window in an image block of a reference frame within an image block of a current frame to output image warping information, and parallelly perform feature point detection on the image block of the current frame to obtain a second feature point data window, where the reference frame is temporally before the current frame;

[0006] The filtering processing module is configured to perform spatio-temporal domain smoothing filtering on the image warping information to output motion adjustment information;

[0007] The denoising processing module is configured to perform rotation alignment processing on the reference frame by using the motion adjustment information to obtain a first image frame, and perform denoising filtering processing on the current frame and / or the first image frame to output a second image frame.

[0008] It can be seen that, firstly, the image processing module concurrently executes the matching of the first feature point data window in the image block of the reference frame within the image block of the current frame and the feature point detection of the image block of the current frame; secondly, the filtering processing module performs spatio-temporal domain smoothing filtering on the image affine transformation information to output motion adjustment information; finally, the noise reduction processing model uses the motion adjustment information to perform rotation alignment processing on the reference frame to obtain the first image frame, and performs noise reduction filtering on the current frame and / or the first image frame to output the second image frame. Since the spatio-temporal domain smoothing filtering is performed on the image affine transformation information to obtain the motion adjustment information, more accurate and reasonable motion adjustment information can be obtained, thereby improving the accuracy of the rotation alignment of the reference frame, enhancing the stability and accuracy of the noise reduction filtering of the current frame and / or the reference frame after rotation alignment, making the time domain noise reduction effect presented by the second image frame better, and improving the stability and accuracy of the image noise reduction processing.

[0009] In a second aspect, an embodiment of the present application provides an image noise reduction method, including:

[0010] Obtain a first feature point data window in an image block of a reference frame and a current frame, where the reference frame is temporally before the current frame;

[0011] Match the first feature point data window within the image block of the current frame to obtain image affine transformation information, and concurrently perform feature point detection on the image block of the current frame to obtain a second feature point data window;

[0012] Perform spatio-temporal domain smoothing filtering on the image affine transformation information to obtain motion adjustment information;

[0013] Use the motion adjustment information to perform rotation alignment processing on the reference frame to obtain a first image frame;

[0014] Perform noise reduction filtering on the current frame and / or the first image frame to obtain a second image frame.

[0015] It can be seen that, firstly, directly obtain the first feature point data window in the image block of the reference frame (i.e., without performing feature point detection) and the current frame; secondly, perform the matching of the first feature point data window of the reference frame within the image block of the current frame and the feature point detection of the image block of the current frame in parallel; thirdly, perform spatio-temporal domain smoothing filtering on the image affine transformation information to obtain motion adjustment information; finally, use the motion adjustment information to perform rotational alignment on the reference frame to obtain the first image frame, and perform noise reduction filtering on the current frame and / or the first image frame to output the second image frame. Since the motion adjustment information is obtained by performing spatio-temporal domain smoothing filtering on the image affine transformation information, more accurate and reasonable motion adjustment information can be obtained, thereby improving the accuracy of the rotational alignment of the reference frame, enhancing the stability and accuracy of the noise reduction filtering performed on the current frame and / or the reference frame after rotational alignment, making the time-domain noise reduction effect presented by the second image frame better, and improving the stability and accuracy of the image noise reduction processing.

[0016] In a third aspect, an embodiment of the present application provides a processing chip, including the image noise reduction device in the first aspect of the embodiments of the present application.

[0017] It can be seen that the image noise reduction device in the processing chip outputs motion adjustment information by performing spatio-temporal domain smoothing filtering on the image affine transformation information, so as to obtain more accurate and reasonable motion adjustment information, thereby improving the accuracy of the rotational alignment of the reference frame, enhancing the stability and accuracy of the noise reduction filtering performed on the current frame and / or the reference frame after rotational alignment, making the time-domain noise reduction effect presented by the second image frame better, and improving the stability and accuracy of the image noise reduction processing.

[0018] In a fourth aspect, an embodiment of the present application provides an electronic device, including a memory, a communication interface, and the image noise reduction device in the first aspect of the embodiments of the present application.

[0019] It can be seen that the image noise reduction device in the electronic device outputs motion adjustment information by performing spatio-temporal domain smoothing filtering on the image affine transformation information, so as to obtain more accurate and reasonable motion adjustment information, thereby improving the accuracy of the rotational alignment of the reference frame, enhancing the stability and accuracy of the noise reduction filtering performed on the current frame and / or the reference frame after rotational alignment, making the time-domain noise reduction effect presented by the second image frame better, and improving the stability and accuracy of the image noise reduction processing. Description of the Drawings

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following described drawings are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a schematic structural diagram of an image denoising device provided by an embodiment of the present application;

[0022] Figure 2 It is a schematic flow diagram of an image denoising provided by an embodiment of the present application;

[0023] Figure 3 It is a schematic structural diagram of detecting feature points of an image block of a current frame provided by an embodiment of the present application;

[0024] Figure 4 It is a schematic structural diagram of searching and matching a current frame and a reference frame provided by an embodiment of the present application;

[0025] Figure 5 It is a schematic structural diagram of the relative displacement from a feature point data window of a current frame to a target feature point data window provided by an embodiment of the present application;

[0026] Figure 6 It is a schematic structural diagram of another image denoising device provided by an embodiment of the present application;

[0027] Figure 7 It is a schematic structural diagram of the exposure time of an image block in a row block of an image frame provided by an embodiment of the present application;

[0028] Figure 8 It is a schematic structural diagram of an image block in a row block of a current frame provided by an embodiment of the present application;

[0029] Figure 9 It is a schematic structural diagram of an image block in a row block of a current frame and an image block in a row block of a reference frame provided by an embodiment of the present application;

[0030] Figure 10 It is a schematic flow diagram of another image denoising provided by an embodiment of the present application;

[0031] Figure 11 It is a schematic flow diagram of an image denoising method provided by an embodiment of the present application;

[0032] Figure 12 It is a schematic structural diagram of a processing chip provided by an embodiment of the present application;

[0033] Figure 13 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0034] For those skilled in the art to better understand the technical solution of this application, the technical solution in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, rather than all embodiments. Based on the description of the embodiments of this application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope protected by this application.

[0035] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, software, product or device that includes a series of steps or units is not limited to the listed steps or units, but also includes unlisted steps or units, or other steps or units inherent to these processes, methods, products or devices.

[0036] In the embodiments of this application, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a coupled connection, a detachable connection; it can be directly connected, indirectly connected through an intermediate medium, or in spaced contact; it can also be a physical connection or an electrical connection, etc.

[0037] Referring to "embodiments" in the embodiments of this application means that the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of this application. The appearance of this phrase at various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0038] Before describing the technical solution of the embodiments of this application, the relevant concepts that this application may involve will be introduced below.

[0039] (1) Video noise reduction

[0040] Since imaging devices, such as complementary metal-oxide-semiconductor (CMOS) sensors, charge-coupled device (CCD) sensors, etc., will be affected by noise during the data acquisition process, resulting in random noise in the video stream, it is necessary to use image noise reduction technology to remove the noise.

[0041] Video noise reduction methods can be classified into spatial domain, frequency domain, wavelet domain, time domain, spatio-temporal domain, etc. according to different processing domains. Among them, there will be an overlapping phenomenon in the noise reduction methods between different processing domains, or a noise reduction method will involve multiple processing domains. For example, in the noise reduction methods in the time domain or spatio-temporal domain, the noise reduction method of frequency domain filtering can be used, that is, all or part of the image frames in the video stream are transformed to the frequency domain through Fourier transform, and then time domain filtering or spatio-temporal domain filtering is used for noise reduction processing.

[0042] Ways such as spatial domain filtering, frequency domain filtering, and wavelet domain filtering in video noise reduction are the same as those in image noise reduction, but they are extended to process multiple frames of images, and at the same time, more redundant information in the video signal is used for optimization, so as to achieve a better noise reduction effect. Among them, spatial domain filtering directly performs algebraic operations on the pixel values of each frame of image in the video stream, only considering the correlation of the image frames in the spatial domain. Frequency domain filtering transforms the image frames in the video stream to the frequency domain through Fourier transform, and then attenuates the frequencies representing noise to retain the original information in the video stream to the greatest extent. Wavelet domain filtering performs noise reduction processing after transforming the image frames in the video stream to the time-frequency domain. Time domain filtering performs noise reduction filtering by considering the correlation of the image frames in the video stream in the time dimension, with simple operations, high efficiency, and no phenomenon of spatial blurring introduced.

[0043] (2) 3-Dimension Digital Noise Reduction

[0044] 3DNR is a noise reduction method that combines spatial domain filtering and time domain filtering. Its general idea is to detect the motion vectors between multiple frames of images in the video stream through motion estimation, then perform alignment processing according to the motion vectors, and finally perform fusion noise reduction filtering on the aligned multiple frames of images to output the filtered image.

[0045] Currently, in video noise reduction technology, it is usually necessary to perform image warping on the image blocks of the image frames in the video stream to obtain image warping information, so as to use this image warping information to perform operations such as alignment processing on the image frame. However, in the actual process of video noise reduction, directly using the image warping information to perform alignment processing on the image frame will result in poor alignment effect of the image frame, thus affecting the stability and accuracy of the image noise reduction processing.

[0046] Combined with the above description, the image noise reduction device of the embodiments of the present application will be introduced below. Please refer to Figure 1 . Figure 1It is a schematic structural diagram of an image noise reduction device provided by an embodiment of the present application. The image noise reduction device 10 includes an image processing module 110, a filtering processing module 120, and a noise reduction processing module 130. The image processing module 110 is respectively connected to the filtering processing module 120 and the noise reduction processing module 130, and the filtering processing module 120 is connected to the noise reduction processing module 130. Among them,

[0047] The image processing module 110 is configured to match (also referred to as search and match) the first feature point data window in the image block of the reference frame within the image block of the current frame to output image affine transformation information, and perform feature point detection on the image block of the current frame in parallel to obtain a second feature point data window, where the reference frame is an image frame located before the current frame in time sequence.

[0048] The filtering processing module 120 is configured to perform spatio-temporal domain smoothing filtering on the image affine transformation information to output motion adjustment information.

[0049] The noise reduction processing module 130 is configured to perform rotation alignment processing on the reference frame using the motion adjustment information to obtain a first image frame, and perform noise reduction filtering on the current frame and / or the first image frame to output a second image frame.

[0050] It should be noted that, in order to improve the stability and accuracy of image noise reduction processing, the image noise reduction device 10 of the embodiment of the present application includes corresponding hardware and / or software modules for performing various functions. Those skilled in the art should easily realize that, combining the functions of the modules described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain module is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the functions of the various modules described in the embodiments of the present application by calling specific programs or algorithms through a processor (such as a CPU, etc.), but this implementation should not be considered to exceed the scope of the present application. In addition, the embodiment of the present application can divide the function modules of the image noise reduction device 10. For example, each function module can be divided according to functions, or two or more functions can be integrated into one function module. The above integrated function modules can be implemented in the form of hardware or software. At the same time, the division of function modules in the embodiment of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation, and no specific limitation is made thereto.

[0051] In Figure 2In the image denoising method shown, a reference frame and a current frame to be denoised are obtained from a video stream, and the current frame is uploaded to the bus for storage. The reference frame is the previous frame or the next frame of the current frame; the current frame is subjected to smoothing filtering, and feature point detection is performed on the smoothed current frame to obtain a feature point data window; the feature point data window of the current frame is searched and matched with the reference frame to obtain a motion vector, that is, a motion matching result, and motion adjustment information is calculated through the motion matching result, and the motion adjustment information is used to perform rotational alignment processing on the reference frame; the current frame and the reference frame are read from the bus, and the reference frame is rotationally aligned through the motion adjustment information; the current frame and the rotationally aligned reference frame are subjected to two-frame fusion denoising processing to obtain a second image frame, thereby achieving temporal domain denoising of the current frame.

[0052] For feature point detection of the current frame, the current frame can be first divided into image blocks to obtain N*N image blocks with equal size and non-overlapping, and the value of N can be 2, 4, 8 or 16, etc., and then feature point detection is performed on each image block to obtain a plurality of feature point data windows. For example, please refer to Figure 3 , the current frame is divided into 4*4 image blocks, and then feature point detection is performed on each image block to obtain a plurality of feature point data windows.

[0053] For searching and matching the feature point data window of the current frame with the reference frame to obtain the best motion vector, data windows similar to the feature point data window of the current frame can be first searched and matched within the reference frame to obtain the target feature point data window in the reference frame, as shown in Figure 4 . Figure 4 The area 410 in represents the range of search and matching, and then the relative displacement of the feature point data window of the current frame to the target feature point data window is calculated to obtain the motion vector, as shown in Figure 5 .

[0054] In Figure 3 the feature point detection process shown, usually feature point detection needs to be performed on 4 image blocks (i.e., ¼ frame of the current frame) in the first row block of the divided current frame, and then feature point detection is performed on 4 image blocks in the second row block of the current frame, and so on. Therefore, in the feature point detection process, a processing delay of 1 / N frame is usually introduced, that is, Figure 3 the processing delay of ¼ frame in. At the same time, in Figure 4 the search and matching process shown, usually the feature point data window of ¼ frame of the current frame (i.e., the first row block of the current frame) is searched and matched in ¼ frame of the reference frame (i.e., the first row block of the reference frame), and so on. Therefore, a processing delay of 1 / N frame is also introduced in the search and matching process.

[0055] Because the fusion noise reduction process between the current frame and the rotationally aligned reference frame still requires reading both the current frame and the reference frame from the bus, a processing delay of one frame is introduced during the fusion noise reduction process. This indicates that significant processing delay is often present in the actual video noise reduction process.

[0056] In addition, in the process of rotational alignment of the reference frame, since the image affine transformation information is usually directly used as motion adjustment information, that is, the reference frame is directly rotationally aligned using the image affine transformation information, the rotational alignment effect of the reference frame is poor, which in turn affects the stability and accuracy of the fusion noise reduction processing of the current frame and the reference frame after rotational alignment, resulting in poor time domain noise reduction effect presented by the second image frame.

[0057] In combination with the above description, in order to improve the accuracy of image frame rotation alignment and enhance the stability and accuracy of image noise reduction processing, the technical solution of the embodiment of the present application is described in detail below.

[0058] Specifically, the current frame can be used to represent an image frame to be denoised in the video stream.

[0059] Specifically, the reference frame may be located before the current frame in terms of time sequence. For example, the reference frame may be an image frame preceding the current frame.

[0060] Specifically, matching or search matching can be understood as determining a search range in the image block of the current frame using a search matching algorithm, then finding the most similar (or matching) data window within the search range using the first feature point data window, and finally obtaining motion adjustment information using the motion vector (MV) from the first feature point data window to the most similar data window. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous screening algorithm, a multi-layer screening algorithm, a partial distortion elimination algorithm, and the like.

[0061] Specifically, feature point detection can include Harris corner detection, where corners in an image frame can have the following characteristics: intersections between contours; stability for the same scene even when the viewing angle changes; and significant changes in the gradient direction and magnitude of pixels in the vicinity of the corner.

[0062] It should be noted that hash corner detection can be performed by moving a local window over the image patches of the current frame to determine whether there are significant changes in the pixel values within the local window. Among them, whether there are significant changes in the pixel values can be described by the image gradient. If the image gradient is larger, it indicates that the pixel values within the local window change more significantly (or the change rate of the grayscale values is larger). Therefore, if there are significant changes in the grayscale values (in terms of the image gradient) within the local window, then there are corners in the area where the local window is located.

[0063] It can be understood that if hash corner detection is used to detect feature points for the image patches of the current frame to obtain a feature point data window, then the feature point data window can be used to represent the corners in the image patch, or the feature point data window can be used to represent the image area composed of the corners in the image patch and the pixel points around the corners, etc.

[0064] Specifically, the image patches of the reference frame can be used to represent at least one image patch in the reference frame that has equal size and does not overlap with each other.

[0065] It can be understood that in the embodiments of the present application, the obtained reference frame can be first subjected to image block processing to obtain N*N image patches with equal size and non-overlapping. The value of N can be 2, 4, 8, or 16, etc. Then, feature point detection is performed on each image patch to obtain a plurality of feature point data windows, and finally the plurality of feature point data windows are stored, which is beneficial to directly obtaining the feature point data window of the reference frame in subsequent search and matching without having to perform feature point detection again, thereby reducing the processing delay in the video denoising process.

[0066] Specifically, the image patches of the current frame can be used to represent at least one image patch in the current frame that has equal size and does not overlap with each other.

[0067] It can be understood that in the embodiments of the present application, the obtained current frame can be subjected to image block processing to obtain N*N image patches with equal size and non-overlapping. The value of N can be 2, 4, 8, or 16, etc.

[0068] Combined with the above description, compared with Figure 2In the prior art, the current frame is first subjected to feature point detection, and then the feature point data window of the current frame is searched and matched with the reference frame for distributed processing. In the embodiments of the present application, the image processing module 110 directly obtains the first feature point data window of the reference frame (i.e., without performing feature point detection) and the current frame, and parallelly executes the search and matching of the first feature point data window within the image block of the current frame and the feature point detection of the image block of the current frame, so as to realize the parallel processing of search and matching and feature point detection by directly obtaining the first feature point data window and the current frame, and further reduce the processing delay in the video noise reduction process and improve the processing efficiency through parallel processing. At the same time, by directly uploading the second feature point data window of the current frame, the rest of the redundant data in the current frame is avoided from being uploaded and stored to reduce the data upload bandwidth.

[0069] In addition, compared with Figure 2 in which the image affine transformation information is directly used as the motion adjustment information, that is, the reference frame is directly rotated and aligned by using the image affine transformation information, in the embodiments of the present application, the filtering processing module 120 performs spatio-temporal domain smoothing filtering on the image affine transformation information to output the motion adjustment information, so as to obtain more accurate and reasonable motion adjustment information, and further improve the accuracy of the reference frame rotation alignment, enhance the stability and accuracy of the noise reduction filtering processing of the current frame and / or the rotated and aligned reference frame, make the time-domain noise reduction effect presented by the second image frame better, and improve the stability and accuracy of the image noise reduction processing.

[0070] Combined with the above description, the embodiments of the present application will specifically describe each module in the image noise reduction device 10 below Figure 6 with reference to

[0071] Specifically, the filtering processing module 120 may specifically include: a spatio-temporal domain smoothing filtering module 1201 and a motion adjustment module 1202; wherein,

[0072] The spatio-temporal domain smoothing filtering module 1201 can be used to perform spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristics of the rolling shutter to output the smoothed filtering information.

[0073] The motion adjustment module 1202 can be used to determine the motion adjustment information based on the smoothed filtering information.

[0074] Among them, determining the motion adjustment information based on the smoothed filtering information may include using the smoothed filtering information as the motion adjustment information, or performing linear weighting processing on the smoothed filtering information to obtain the motion adjustment information.

[0075] Among them, spatial domain smoothing filtering can be understood as performing smoothing filtering between the image affine transformation information corresponding to the previous and / or subsequent image blocks in the same image frame (current frame), or performing smoothing filtering between the image affine transformation information corresponding to the previous and / or subsequent image blocks in the same row block of the same image frame.

[0076] Among them, temporal domain smoothing filtering can be understood as performing smoothing filtering between the image affine transformation information corresponding to the image blocks in different image frames (between the current frame and the reference frame), or performing smoothing filtering between the image affine transformation information corresponding to the image blocks in different row blocks of the same image frame.

[0077] It should be noted that compared with the one-time exposure of the global shutter, the exposure characteristics of the rolling shutter satisfy one of the following: serial exposure, top-to-bottom exposure, and left-to-right exposure. It can be seen that the exposure characteristics of the rolling shutter can form image blocks at different exposure times. Based on this, in the embodiments of the present application, based on the exposure characteristics of the rolling shutter, each row block from top to bottom of the current frame (reference frame) (as Figure 3 shown, after the current frame (reference frame) is subjected to image block processing, the current frame includes 4 row blocks from top to bottom, and each row block includes 4 image blocks from left to right) is represented as row blocks at different exposure times, and each image block from left to right in the same row block is represented as image blocks at different exposure times.

[0078] For example, as Figure 7 shown, the image frame includes 4 row blocks from top to bottom, and each row block includes 4 image blocks from left to right. Based on the exposure characteristics of the rolling shutter, the 4 row blocks are exposed sequentially from top to bottom, and the 4 image blocks in each row block are exposed sequentially from left to right.

[0079] In addition, in the embodiments of the present application, the feature point data window of the directly obtained reference frame can be used to search and match within each image block (or each row block of the current frame) of the current frame, so as to obtain the image affine transformation information corresponding to each image block of the current frame. From the above exposure characteristics of the rolling shutter (rolling shutter), it can be seen that since each image block of the current frame has a different exposure time, the image affine transformation information corresponding to different image blocks is also obtained at different exposure times.

[0080] In summary, in the embodiments of the present application, based on the exposure characteristics of the rolling shutter, the image affine transformation information corresponding to the image blocks of the current frame is subjected to spatial domain smoothing filtering and / or temporal domain smoothing filtering, so as to obtain more accurate and reasonable motion adjustment information, which is beneficial to improving the accuracy of image frame rotation alignment.

[0081] Combined with the above description, the following describes three cases of performing spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image blocks of the current frame based on the exposure characteristics of the rolling shutter.

[0082] Case 1:

[0083] Specifically, the spatio-temporal domain smoothing filtering module 1201 can be used to: perform smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the second image affine transformation information corresponding to the image blocks located at the previous position and / or the next position of the first image block within the first row block to output the first smoothing filtering information in the smoothing filtering information.

[0084] Among them, the image blocks of the current frame are located in different row blocks of the current frame.

[0085] Among them, the second image affine transformation information may include at least one of the following: the image affine transformation information corresponding to the image block located at the previous position of the first image block within the first row block, the image affine transformation information corresponding to the image block located at the next position of the first image block within the first row block, and the image affine transformation information corresponding to the image blocks located at the previous position and the next position of the first image block within the first row block.

[0086] Among them, the first row block may be one row block among the row blocks of the current frame, the first image block may be at least one image block in the first row block, and the image affine transformation information may include the first image affine transformation information and the second image affine transformation information.

[0087] Among them, the motion adjustment module 1202 can be used to use the first smoothing filtering information in the smoothing filtering information as the motion adjustment information.

[0088] It should be noted that in "Case 1", the spatio-temporal domain smoothing filtering module 1201 can perform smoothing filtering on the image affine transformation information corresponding to at least one image block in a certain row block of the current frame and the image affine transformation information corresponding to the image blocks located at the previous position and / or the next position of the at least one image block within the row block, and so on, so as to realize performing spatial domain smoothing filtering on the image affine transformation information corresponding to the image blocks of the current frame to obtain the first smoothing filtering information, and then using the first smoothing filtering information as the motion adjustment information through the motion adjustment module 1202, which is beneficial to ensuring that the obtained motion adjustment information is more accurate and reasonable.

[0089] For example, as Figure 8As described above, if the first image block is the second image block in the first row block of the current frame (i.e., image block 801), image block 802 is the image block at the previous position of image block 801, and image block 803 (or image block 804) is the image block at the next position of image block 801, then in the manner of performing spatial domain smoothing filtering on the image affine transformation information, the spatio-temporal domain smoothing filtering module 1201 performs smoothing filtering on the image affine transformation information corresponding to image block 801, the image affine transformation information corresponding to image block 802, and the image affine transformation information corresponding to image block 803 (or image block 804).

[0090] For another example, as Figure 8 described above, if the first image block is the first image block in the first row block of the current frame (i.e., image block 802), and image block 801 (or image block 803, or image block 804) is the image block at the next position of image block 801, then in the manner of performing spatial domain smoothing filtering on the image affine transformation information, the spatio-temporal domain smoothing filtering module 1201 performs smoothing filtering on the image affine transformation information corresponding to image block 802 and the image affine transformation information corresponding to image block 801 (or image block 803, or image block 804).

[0091] For another example, as Figure 8 described above, if the first image block is the last image block in the first row block of the current frame (i.e., image block 804), and image block 803 (or image block 802, or image block 801) is the image block at the previous position of image block 804, then in the manner of performing spatial domain smoothing filtering on the image affine transformation information, the spatio-temporal domain smoothing filtering module 1201 performs smoothing filtering on the image affine transformation information corresponding to image block 804 and the image affine transformation information corresponding to image block 803 (or image block 802, or image block 801).

[0092] Case 2:

[0093] Specifically, the spatio-temporal domain smoothing filtering module 1201 can be used to perform smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the third image affine transformation information corresponding to the second image block in the row block before the first row block of the current frame to output the second smoothing filtering information in the smoothing filtering information.

[0094] Among them, the image blocks of the current frame are located in different row blocks of the current frame.

[0095] Among them, the first row block can be one of the row blocks in the row blocks of the current frame, the first image block can be at least one image block in the first row block, the image affine transformation information can include first image affine transformation information and second image affine transformation information, the second image block can be at least one image block in the image blocks of the current frame or at least one image block in the image blocks of the reference frame, and the image affine transformation information can further include third image affine transformation information.

[0096] Among them, the motion adjustment module 1202 is configured to use the second smoothing filter information in the smoothing filter information as the motion adjustment information.

[0097] It should be noted that in "Case 2", the spatio-temporal domain smoothing filter module 1201 can perform smoothing filtering on the image affine transformation information corresponding to at least one image block in a certain row block of the current frame and the image affine transformation information corresponding to at least one image block in the previous row block of this row block, and so on, so as to realize performing temporal domain smoothing filtering on the image affine transformation information corresponding to the image blocks of the current frame to obtain the second smoothing filter information, and then using the second smoothing filter information as the motion adjustment information through the motion adjustment module 1202, which is further beneficial to ensuring that the obtained motion adjustment information is more accurate and reasonable.

[0098] Furthermore, if the first row block is the first row block in the row blocks of the current frame, then the previous row block of the first row block is the last row block in the row blocks of the reference frame.

[0099] Among them, the image blocks of the reference frame are located in the row blocks of different reference frames.

[0100] It should be noted that the previous row block of the first row block can be a row block of the current frame or a row block of the reference frame. When the first row block is the first row block in the row blocks of the current frame, since the reference frame is the previous image frame of the current frame, and the last row block in the row blocks of the reference frame is closest to the first row block in the row blocks of the current frame in terms of the exposure time, the smoothing filter information output by performing smoothing filtering on the image affine transformation information corresponding to the image blocks in the first row block of the current frame and the image affine transformation information corresponding to the image blocks in the last row block of the reference frame will be more accurate, thereby further ensuring that the obtained motion adjustment information is more accurate and reasonable.

[0101] For example, as Figure 9As shown, if the first image block is the first image block in the first row block of the current frame (such as image block 901), and image block 902 is the first image block in the last row block of the reference frame, then in the manner of performing temporal smoothing filtering on the image affine transformation information, the spatio-temporal smoothing filtering module 1201 performs smoothing filtering on the image affine transformation information corresponding to image block 901 and the image affine transformation information corresponding to image block 902 (or another image block in the last row block of the reference frame).

[0102] Case 3:

[0103] Specifically, the spatio-temporal smoothing filtering module 1201 can be used to: perform smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the second image affine transformation information corresponding to the image block at the previous position and / or the next position of the first image block within the first row block to output the first smoothing filtering information in the smoothing filtering information; perform smoothing filtering on the first image affine transformation information corresponding to the first image block and the third image affine transformation information corresponding to the second image block in the previous row block of the first row block to output the second smoothing filtering information in the smoothing filtering information.

[0104] Among them, the image blocks of the current frame are located in different row blocks of the current frame.

[0105] Among them, the motion adjustment module 1202 can be used to perform linear weighted processing on the first smoothing filtering information and the second smoothing filtering information to obtain motion adjustment information.

[0106] It should be noted that in "Case 3", the spatio-temporal smoothing filtering module 1201 realizes performing spatial smoothing filtering and temporal smoothing filtering on the image affine transformation information corresponding to the image block of the current frame to obtain the first smoothing filtering information and the second smoothing filtering information, and then the motion adjustment module 1202 performs linear weighted processing on the first smoothing filtering information and the second smoothing filtering information to obtain motion adjustment information, so as to further ensure that the obtained motion adjustment information is more accurate and reasonable.

[0107] Combined with the above description, the storage module further included in the image noise reduction device 10 will be specifically described below.

[0108] In a possible example, the image noise reduction device 10 may further include a storage module. Among them, the storage module can be used to store at least one of the current frame, the reference frame, the first feature point data window, and the second feature point data window.

[0109] Specifically, the storage module may include a double data rate DDR memory.

[0110] Among them, the DDR memory may include a double data rate synchronous dynamic random access memory (DDR SDRAM).

[0111] It should be noted that the image processing module 110 in the embodiments of the present application can directly obtain the first feature point data window and the current frame in the image block of the reference frame from the storage module, and then upload the current second feature point data window to the storage module for storage, so that the second feature point data window can be directly read from the storage module later without performing feature point detection again for search and matching in parallel processing, thereby reducing the processing delay in the video noise reduction process. At the same time, by directly obtaining the first feature point data window and the current frame to implement parallel processing of search and matching and feature point detection, the processing delay in the video noise reduction process is reduced through parallel processing, and the processing efficiency is improved. In addition, by directly uploading the second feature point data window of the current frame, the remaining redundant data in the current frame is avoided from being uploaded and stored to reduce the data upload bandwidth.

[0112] Combined with the above description, the image processing module 110 will be specifically described below.

[0113] In a possible example, the image processing module 110 may include a search and matching module 1101, an affine transformation module 1102, and a feature point detection module 1103; among them,

[0114] The search and matching module 1101 can be used to match the first feature point data window within the image block of the current frame to output a motion vector.

[0115] The affine transformation module 1102 can be used to perform data fitting processing on the motion vector to output image affine transformation information.

[0116] The feature point detection module 1103 can be used to perform feature point detection on the image block of the current frame to obtain a second feature point data window.

[0117] It should be noted that first, the embodiments of the present application can perform parallel processing of the search and matching of the feature point data window of the reference frame within the image block of the current frame and the feature point detection of the current frame through the image processing module 110, so as to reduce the processing delay in the video noise reduction process through the parallel processing of search and matching and feature point detection.

[0118] Secondly, the feature point detection in the embodiments of the present application includes hash corner detection, so the corner points in the image frame of the current frame can be detected through the feature point detection module 1103.

[0119] Next, search matching is an important part of video denoising technology. By calculating the relative motion offset of a pixel point in two adjacent frames of images, its motion vector (MV) can be obtained. Among them, the search matching module 1101 can determine the search range in the image block of the current frame through a search matching algorithm, and then find the most similar (or matching) data window within this search range. The search matching algorithm can include full search matching algorithm, three-step search algorithm, diamond search algorithm, four-step search algorithm, successive elimination algorithm, multi-layer elimination algorithm, partial distortion elimination algorithm, etc. Therefore, through the search matching module 1101, the feature point data window of the reference frame can be searched and matched in the image block of the current frame to obtain the motion vector.

[0120] Finally, the data fitting process of the motion vector in the embodiment of the present application can include the least squares method. The motion vector is processed by the least squares method to obtain the image warping information. Among them, the image affine transformation can include a homography matrix. The homography transformation can be used to describe the position mapping relationship between an object in the world coordinate system and the pixel coordinate system, and the corresponding homography transformation is called a homography matrix. Therefore, through the affine transformation module 1102, the motion vector is processed by data fitting to obtain the motion adjustment information for rotating and aligning the reference frame.

[0121] Specifically, the search matching module 1101 can be specifically used to: calculate the sum of absolute errors between the gray value of each pixel point in the first feature point data window and the gray value of each pixel point in the data window within the search range of the image block of the current frame to obtain a parameter value; use the data window corresponding to the minimum value in the parameter values as the target data window; and calculate the relative displacement from the first feature point data window to the target data window to output the motion vector.

[0122] It should be noted that in the embodiment of the present application, it is considered to use the sum of absolute difference (SAD) criterion in the first feature point data window of the reference frame to perform search matching in the image block of the current frame, and find the data window corresponding to the minimum value of the SAD value (i.e., the parameter value) in the first feature point data window. At this time, this data window is the data window that best matches the first feature point data window, that is, the target data window. Therefore, the first feature point data window and the target data window form a relative position in the time domain to obtain the motion vector, and thus the motion adjustment information for rotating and aligning the reference frame is constructed through this motion vector.

[0123] Combined with the above description, the modules further included in the image processing module 110 will be specifically described below.

[0124] Specifically, the image processing module 110 may further include a smoothing filter module 1104. Among them,

[0125] The smoothing filter module 1104 can be used to perform smoothing filter processing on the current frame and output the current frame after the smoothing filter processing.

[0126] It should be noted that the smoothing filter module 1104 can be used to reduce noise and artifacts in the current frame. Among them, the smoothing filter module 1104 may include a mean filter, a box filter, a median filter, a Gaussian filter, a bilateral filter, etc.

[0127] Specifically, the image processing module 110 may further include an image block division module 1105. Among them,

[0128] The image block division module 1105 can be used to perform image block division processing on the current frame and output the current frame after the image block division processing.

[0129] It should be noted that in the embodiments of the present application, the image block division module 1105 can divide the image frames (including the current frame or the reference frame) of the acquired video stream into N*N image blocks with the same size and non-overlapping, so as to facilitate subsequent direct feature point detection on the image blocks and improve the feature point detection efficiency of the image frames.

[0130] Specifically, the image processing module 110 may further include a data window upload module 1106. Among them,

[0131] The data window upload module 1106 can be used to upload the second feature point data window for storage.

[0132] Specifically, the data window upload module 1106 can be used to upload the second feature point data window to the storage module.

[0133] It should be noted that in the embodiments of the present application, the data window upload module 1106 uploads the current second feature point data window to the storage module for storage, so that the second feature point data window can be directly read from the storage module later without performing feature point detection again for search and matching in parallel processing, thereby reducing the processing delay in the video noise reduction process. In addition, by directly uploading the second feature point data window of the current frame, uploading and storing the remaining redundant data in the current frame is avoided to reduce the data upload bandwidth.

[0134] Combined with the above description, the noise reduction processing module 130 will be specifically described below.

[0135] In a possible example, the noise reduction processing module 130 may specifically include a rotation alignment module 1301, a filtering decision module 1302, and a fusion filtering module 1303. Among them,

[0136] The rotation alignment module 1301 can be used to perform rotation alignment processing on a reference frame based on motion adjustment information to output a first image frame.

[0137] The filtering decision module 1302 can be used to determine noise reduction filtering strategy information for the current frame and the first image frame based on the gray values of each pixel point in the image block of the current frame, the gray values of each pixel point in the image block of the first image frame, and a preset threshold. The noise reduction filtering strategy information includes time-domain filtering or spatial-domain filtering.

[0138] The fusion filtering module 1303 can be used to perform noise reduction filtering processing on the current frame and / or the first image frame based on the noise reduction filtering strategy information to output a second image frame.

[0139] Among them, determining the noise reduction filtering strategy information for the current frame and the first image frame based on the gray values of each pixel point in the image block of the current frame, the gray values of each pixel point in the image block of the first image frame, and a preset threshold may include: calculating the mean absolute error based on the gray values of each pixel point in the image block of the current frame and the gray values of each pixel point in the image block of the first image frame to obtain a parameter value; determining the noise reduction filtering strategy information for the current frame and the first image frame according to the comparison result between the parameter value and the preset threshold.

[0140] It should be noted that, first, in order to reduce the residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers performing rotation alignment processing on the reference frame through the rotation alignment module 1301.

[0141] Second, in order to overcome the "trailing" phenomenon of fast-moving objects easily caused by time-domain filtering, the embodiment of the present application considers performing motion intensity detection through the mean absolute differences (MAD) algorithm, so that for objects with different motion intensities, the filter adopts different filtering intensities, which is beneficial to avoiding the "trailing" phenomenon caused by fast-moving objects and improving the image filtering effect. Among them, the filtering decision module 1302 calculates the gray values of each pixel point in the third image block of the current frame and the gray values of each pixel point in the fourth image block of the first image frame through the MAD algorithm to obtain a MAD value (i.e., the parameter value).

[0142] Third, the embodiment of the present application tests a large number of image sequences to determine an empirical value for measuring motion intensity, that is, a preset threshold, and then determines the motion intensity of the image block of the current frame on the motion trajectory represented by the motion vector according to the comparison result between the parameter value and the preset threshold.

[0143] If the comparison result between the parameter value and the preset threshold is that the parameter value is greater than the preset threshold, it indicates that the image block has a strong motion intensity on the motion trajectory. At this time, the filtering intensity of the filter can be lowered (or it is determined that the noise reduction filtering strategy information is spatial domain filtering), so that the original information of the image block after filtering is retained as much as possible to avoid the generation of the "trailing" phenomenon.

[0144] If the comparison result between the parameter value and the preset threshold is that the parameter value is less than the preset threshold, it indicates that the image block has a weak motion intensity on the motion trajectory, that is, a stable and slow motion. For example, the background picture in the video stream generally does not move. At this time, the filtering intensity of the filter can be increased (or it is determined that the noise reduction filtering strategy information is time domain filtering), so as to effectively remove noise. At the same time, the low motion intensity of the image block will not cause the "trailing" phenomenon, which is beneficial to ensuring a better filtering effect.

[0145] Finally, in order to perform noise reduction filtering on the current frame, the fusion filtering module 1303 of the embodiment of the present application performs fusion noise reduction filtering on the current frame and / or the first image frame according to time domain filtering or spatial domain filtering.

[0146] Specifically, spatial domain filtering may include mean filtering, median filtering, Gaussian filtering, or bilateral filtering, etc. Therefore, noise reduction filtering is performed on the current frame according to spatial domain filtering to output a second image frame, so as to filter the image block with a strong motion intensity on the motion trajectory through spatial domain filtering, and retain the original information in the image block as much as possible to avoid the generation of the "trailing" phenomenon.

[0147] Specifically, time domain filtering may include linear weighted filtering. Therefore, the second image frame output by performing fusion noise reduction filtering on the current frame and the first image frame according to linear weighted filtering satisfies the following formula:

[0148] F out =ω*F t +(1 - ω)*F t-1 ;

[0149] Wherein, F out represents the pixel values of each pixel in the image block of the second image frame, F t represents the pixel values of each pixel in the image block of the current frame, F t-1 represents the pixel values of each pixel in the image block of the first image frame, and ω represents a preset weight. In addition, the filtering intensity of the filter can be dynamically adjusted through the preset weight ω.

[0150] In short, filtering is performed on the image block with a weak motion intensity on the motion trajectory through linear weighted filtering, so as to effectively remove the noise in the image block.

[0151] Specifically, the image blocks of the first image frame can be used to represent at least one image block in the first image frame that are equal in size and non-overlapping. It can be understood that the filtering decision module 1302 in the embodiments of the present application can perform image block processing on the input first image frame to obtain N*N image blocks that are equal in size and non-overlapping. The value of N can be 2, 4, 8, 16, etc.

[0152] In summary, the image noise reduction process of the embodiments of the present application will be exemplarily described below, as Figure 10 shown. In Figure 10 , the feature point data window of the reference frame and the current frame to be denoised are directly obtained from the DDR. The reference frame is the previous frame of the current frame, and the current frame is subjected to smoothing filtering; the feature point detection of the current frame after smoothing filtering and the search and matching of the feature point data window of the reference frame within the image blocks of the current frame after smoothing filtering are executed in parallel; the best motion vector, that is, the motion matching result, is obtained through the search and matching from the feature point data window of the reference frame to the current frame, and the image affine transformation information is obtained by performing data fitting processing on the motion matching result; the image affine transformation information is subjected to spatio-temporal domain smoothing filtering to obtain motion adjustment information, and the motion adjustment information is used to perform rotation alignment processing on the reference frame; the feature point data window of the current frame obtained by the feature point detection of the current frame is uploaded to the DDR for storage; the current frame and the reference frame are read from the DDR, and the reference frame is subjected to rotation alignment processing through the motion adjustment information; the current frame and the reference frame after rotation alignment are fused and denoised filtered to obtain the second image frame.

[0153] It can be seen that by directly obtaining the feature point data window of the reference frame and the current frame to achieve parallel processing of search matching and feature point detection, and then reducing the processing delay in the video noise reduction process and improving the processing efficiency through parallel processing. At the same time, by only directly uploading the feature point data window of the current frame and avoiding uploading and storing the remaining redundant data in the current frame to reduce the data upload bandwidth. In addition, by performing spatio-temporal domain smoothing filtering on the image affine transformation information to obtain motion adjustment information, more accurate and reasonable motion adjustment information can be obtained, and the stability and accuracy of the denoising filtering process of the current frame and / or the reference frame after rotation alignment can be improved, so that the time-domain denoising effect presented by the second image frame is better.

[0154] Combined with the above description, the image noise reduction method of the embodiments of the present application will be introduced below. Please refer to Figure 11 .

[0155] It should be noted that in Figure 2In the image denoising method shown, a reference frame and a current frame to be denoised are obtained from a video stream, and the current frame is uploaded to a bus for storage. The reference frame is the previous frame or the next frame of the current frame; the current frame is subjected to smoothing filtering, and feature point detection is performed on the smoothed current frame to obtain a feature point data window; the feature point data window of the current frame is searched and matched with the reference frame to obtain a motion vector, that is, a motion matching result, and motion adjustment information is calculated through the motion matching result, and the motion adjustment information is used to perform rotational alignment processing on the reference frame; the current frame and the reference frame are read from the bus, and the reference frame is rotationally aligned through the motion adjustment information; the current frame and the rotationally aligned reference frame are subjected to two-frame fusion denoising processing to obtain a second image frame, thereby realizing time-domain denoising of the current frame.

[0156] For feature point detection of the current frame, the current frame can be first subjected to image block processing to obtain N*N image blocks with equal size and non-overlapping, and the value of N can be 2, 4, 8, or 16, etc., and then feature point detection is performed on each image block to obtain a plurality of feature point data windows. For example, please refer to Figure 3 , the current frame is divided into 4*4 image blocks, and then feature point detection is performed on each image block to obtain a plurality of feature point data windows.

[0157] For searching and matching the feature point data window of the current frame with the reference frame to obtain the best motion vector, a data window similar to the feature point data window of the current frame can be first searched and matched within the reference frame to obtain a target feature point data window in the reference frame, as Figure 4 shown, Figure 4 the area 410 in represents the search and matching range, and then the relative displacement of the feature point data window of the current frame to the target feature point data window is calculated to obtain a motion vector, as Figure 5 shown.

[0158] In Figure 3 the feature point detection process shown, usually feature point detection needs to be performed on 4 image blocks (i.e., 1 / 4 frame of the current frame) in the first row block of the current frame after block division, and then feature point detection is performed on 4 image blocks in the second row block of the current frame, and so on. Therefore, in the feature point detection process, usually a processing delay of 1 / N frame will be introduced, that is, Figure 3 the processing delay of 1 / 4 frame in. At the same time, in Figure 4 the search and matching process shown, usually the feature point data window of 1 / 4 frame of the current frame (i.e., the first row block of the current frame) is searched and matched in 1 / 4 frame of the reference frame (i.e., the first row block of the reference frame), and so on. Therefore, a processing delay of 1 / N frame will also be introduced in the search and matching process.

[0159] Because the fusion noise reduction process between the current frame and the rotationally aligned reference frame still requires reading both the current frame and the reference frame from the bus, a processing delay of one frame is introduced during the fusion noise reduction process. This indicates that significant processing delay is often present in the actual video noise reduction process.

[0160] In addition, in the process of rotational alignment of the reference frame, since the image affine transformation information is usually directly used as motion adjustment information, that is, the reference frame is directly rotationally aligned using the image affine transformation information, the rotational alignment effect of the reference frame is poor, which in turn affects the stability and accuracy of the fusion noise reduction processing of the current frame and the reference frame after rotational alignment, resulting in poor time domain noise reduction effect presented by the second image frame.

[0161] In summary, in order to improve the accuracy of image frame rotation alignment and enhance the stability and accuracy of image noise reduction processing, the following is a detailed description.

[0162] S1110 , obtaining a first feature point data window in an image block of a reference frame and a current frame.

[0163] The reference frame is located before the current frame in terms of time sequence.

[0164] Specifically, the current frame can be used to represent an image frame to be denoised in the video stream.

[0165] Specifically, the reference frame may be an image frame before the current frame.

[0166] S1120 , matching the first feature point data window within the image block of the current frame to obtain image affine transformation information, and concurrently performing feature point detection on the image block of the current frame to obtain a second feature point data window.

[0167] Specifically, matching or search matching can be understood as determining a search range in the image block of the current frame using a search matching algorithm, then finding the most similar (or matching) data window within the search range using the first feature point data window, and finally obtaining motion adjustment information using the motion vector (MV) from the first feature point data window to the most similar data window. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous screening algorithm, a multi-layer screening algorithm, a partial distortion elimination algorithm, and the like.

[0168] Specifically, feature point detection may include Harris corners detection. Among them, the corners in the image frame may have the following characteristics: the intersection points between contours; for the same scene, even if the viewing angle changes, they still have stable properties; the pixel points in the vicinity of the corner have large changes both in the gradient direction and its gradient magnitude.

[0169] It should be noted that if Harris corners detection is used to detect feature points in the image block of the current frame to obtain a feature point data window, the feature point data window can be used to represent the corners in the image block, or the feature point data window can be used to represent the image area composed of the corners in the image block and the pixel points around the corners, etc.

[0170] Specifically, the image blocks of the reference frame can be used to represent at least one image block with equal size and non-overlapping in the reference frame.

[0171] It can be understood that in the embodiments of the present application, the obtained reference frame can be first subjected to image block processing to obtain N*N image blocks with equal size and non-overlapping, where the value of N can be 2, 4, 8, or 16, etc. Then, feature point detection is performed on each image block to obtain a plurality of feature point data windows, and finally the plurality of feature point data windows are stored, which is beneficial to directly obtaining the feature point data window of the reference frame in subsequent search and matching without performing feature point detection again, thereby reducing the processing delay in the video noise reduction process.

[0172] Specifically, the image blocks of the current frame can be used to represent at least one image block with equal size and non-overlapping in the current frame.

[0173] It can be understood that in the embodiments of the present application, the obtained current frame can be subjected to image block processing to obtain N*N image blocks with equal size and non-overlapping, where the value of N can be 2, 4, 8, or 16, etc.

[0174] S1130. Perform spatio-temporal domain smoothing filtering on the image affine transformation information to obtain motion adjustment information.

[0175] Specifically, the motion adjustment information can be used to perform rotation alignment processing on the reference frame.

[0176] S1140. Perform rotation alignment processing on the reference frame based on the motion adjustment information to obtain the first image frame.

[0177] It should be noted that in order to reduce the residual difference between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiments of the present application consider performing rotation alignment processing on the reference frame through the motion adjustment information.

[0178] S1150. Perform noise reduction filtering on the current frame and / or the first image frame to obtain a second image frame.

[0179] It can be seen that in the embodiments of the present application, first, directly obtain the first feature point data window in the image block of the reference frame (i.e., without performing feature point detection) and the current frame; second, perform the matching of the first feature point data window of the reference frame within the image block of the current frame and the feature point detection of the image block of the current frame in parallel; third, perform spatio-temporal domain smoothing filtering on the image affine transformation information to obtain motion adjustment information; finally, use the motion adjustment information to perform rotational alignment on the reference frame to obtain the first image frame, and perform noise reduction filtering on the current frame and / or the first image frame to output the second image frame. Since the motion adjustment information is obtained by performing spatio-temporal domain smoothing filtering on the image affine transformation information, more accurate and reasonable motion adjustment information can be obtained, thereby improving the accuracy of the rotational alignment of the reference frame, and enhancing the stability and accuracy of the noise reduction filtering of the current frame and / or the reference frame after rotational alignment, making the time-domain noise reduction effect presented by the second image frame better.

[0180] Combined with the above description, the technical solutions of the embodiments of the present application will be specifically described below.

[0181] Specifically, performing spatio-temporal domain smoothing filtering on the image affine transformation information in S1130 to obtain motion adjustment information may include: performing spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristics of the rolling shutter to obtain smoothing filtering information; determining the motion adjustment information based on the smoothing filtering information, or performing linear weighting processing on the smoothing filtering information to obtain the motion adjustment information.

[0182] Among them, determining the motion adjustment information based on the smoothing filtering information may include: using the smoothing filtering information as the motion adjustment information, or performing linear weighting processing on the smoothing filtering information to obtain the motion adjustment information.

[0183] Further, spatial domain smoothing filtering can be understood as performing smoothing filtering between the image affine transformation information corresponding to the previous and / or subsequent image blocks in the same image frame (current frame), or performing smoothing filtering between the image affine transformation information corresponding to the previous and / or subsequent image blocks in the same row block of the same image frame.

[0184] Further, temporal domain smoothing filtering can be understood as performing smoothing filtering between the image affine transformation information corresponding to the image blocks in different image frames (between the current frame and the reference frame), or performing smoothing filtering between the image affine transformation information corresponding to the image blocks in different row blocks of the same image frame.

[0185] It should be noted that, compared with the one-time exposure of the global shutter, the exposure characteristics of the rolling shutter satisfy one of the following: serial exposure, top-to-bottom exposure, and left-to-right exposure. It can be seen that the exposure characteristics of the rolling shutter can form image blocks at different exposure times. Therefore, in the embodiments of the present application, considering the exposure characteristics of the rolling shutter, each row block from top to bottom of the current frame (reference frame) (such as Figure 3 As shown, after the current frame (reference frame) is subjected to image block processing, the current frame includes 4 row blocks from top to bottom, and each row block includes 4 image blocks from left to right) is represented as row blocks at different exposure times, and the image blocks from left to right in the same row block are represented as image blocks at different exposure times.

[0186] In summary, in the embodiments of the present application, based on the exposure characteristics of the rolling shutter, spatial domain smoothing filtering and / or temporal domain smoothing filtering are performed on the image affine transformation information corresponding to the image blocks of the current frame, so as to obtain more accurate and reasonable motion adjustment information, which is beneficial to improving the accuracy of image frame rotation alignment.

[0187] Combined with the above description, the following describes three cases of performing spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image blocks of the current frame based on the exposure characteristics of the rolling shutter.

[0188] Case 1:

[0189] Specifically, performing spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image blocks of the current frame based on the exposure characteristics of the rolling shutter to obtain smoothed filtering information may include: performing smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the second image affine transformation information corresponding to the image blocks located before and after the first image block in the first row block to obtain the first smoothed filtering information in the smoothed filtering information.

[0190] Among them, the image blocks of the current frame are located in different row blocks of the current frame.

[0191] Among them, the second image affine transformation information may include at least one of the following: the image affine transformation information corresponding to the image block located at the previous position of the first image block in the first row block, the image affine transformation information corresponding to the image block located at the next position of the first image block in the first row block, and the image affine transformation information corresponding to the image blocks located at the previous and next positions of the first image block in the first row block.

[0192] Among them, the first row block can be one of the row blocks in the row blocks of the current frame, the first image block can be at least one image block in the first row block, and the image affine transformation information includes the first image affine transformation information and the second image affine transformation information.

[0193] It should be noted that in "Case 1", by performing smoothing filtering on the image affine transformation information corresponding to at least one image block in a certain row block of the current frame and the image affine transformation information corresponding to the image block at the previous position and / or the next position of the at least one image block in the same row block, and so on, the image affine transformation information corresponding to the image blocks of the current frame is smoothed in the spatial domain to obtain the first smoothed filtering information, and then the first smoothed filtering information is used as the motion adjustment information, which is beneficial to ensuring that the obtained motion adjustment information is more accurate and reasonable.

[0194] Case 2:

[0195] Specifically, performing spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image blocks of the current frame based on the exposure characteristics of the rolling shutter to obtain the smoothed filtering information may include: performing smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the previous frame and the third image affine transformation information corresponding to the second image block in the row block before the first row block to obtain the second smoothed filtering information in the smoothed filtering information.

[0196] Among them, the image blocks of the current frame are located in different row blocks of the current frame.

[0197] Among them, the first row block can be one of the row blocks in the row blocks of the current frame, the first image block can be at least one image block in the first row block, the image affine transformation information includes the first image affine transformation information and the second image affine transformation information, the second image block can be at least one image block in the image blocks of the current frame or at least one image block in the image blocks of the reference frame, and the image affine transformation information may further include the third image affine transformation information.

[0198] It should be noted that in "Case 2", by performing smoothing filtering on the image affine transformation information corresponding to at least one image block in a certain row block of the current frame and the image affine transformation information corresponding to at least one image block in the row block before the same row block, and so on, the image affine transformation information corresponding to the image blocks of the current frame is smoothed in the temporal domain to obtain the second smoothed filtering information, and then the second smoothed filtering information is used as the motion adjustment information, which is beneficial to ensuring that the obtained motion adjustment information is more accurate and reasonable.

[0199] Furthermore, if the first row block is the first row block in the row blocks of the current frame, then the row block before the first row block is the last row block in the row blocks of the reference frame.

[0200] Among them, the image blocks of the reference frame are located in the row blocks of different reference frames.

[0201] It should be noted that the previous row block of the first row block can be the row block of the current frame or the row block of the reference frame. When the first row block is the first row block in the row blocks of the current frame, since the reference frame is the previous image frame of the current frame, and the last row block in the row blocks of the reference frame is closest to the first row block in the row blocks of the current frame in terms of the exposure time, the image affine transformation information corresponding to the image block in the first row block of the current frame and the image affine transformation information corresponding to the image block in the last row block of the reference frame are smoothed to output smoother filtering information, which will be more accurate, thus further ensuring that more accurate and reasonable motion adjustment information can be obtained.

[0202] Case 3:

[0203] Specifically, performing spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image blocks of the current frame based on the exposure characteristics of the rolling shutter to obtain smoothing filtering information may include: performing smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the second image affine transformation information corresponding to the image blocks at the previous position and / or the next position of the first image block within the first row block to obtain the first smoothing filtering information in the smoothing filtering information; performing smoothing filtering on the first image affine transformation information corresponding to the first image block and the third image affine transformation information corresponding to the second image block in the previous row block of the first row block to obtain the second smoothing filtering information in the smoothing filtering information.

[0204] Among them, the image blocks of the current frame are located in the row blocks of different current frames.

[0205] It should be noted that in "Case 3", the embodiments of the present application implement performing spatial domain smoothing filtering and temporal domain smoothing filtering on the image affine transformation information corresponding to the image blocks of the current frame to obtain the first smoothing filtering information and the second smoothing filtering information, and then performing linear weighted processing on the first smoothing filtering information and the second smoothing filtering information to obtain motion adjustment information, thereby further ensuring that the obtained motion adjustment information is more accurate and reasonable.

[0206] Combined with the above description, the following describes how to match the first feature point data window within the image blocks of the current frame to obtain image affine transformation information.

[0207] Specifically, the matching of the first feature point data window within the image block of the current frame to obtain the image affine transformation information in S1120 may include: matching the first feature point data window within the image block of the current frame to obtain a motion vector; performing data fitting processing on the motion vector to obtain the image affine transformation information.

[0208] It should be noted that the data fitting processing of the motion vector in the embodiment of the present application may include the least squares method. The least squares method is used to perform data fitting processing on the motion vector to obtain the image warping information. Among them, the image affine transformation may include a homography matrix. The homography transformation can be used to describe the position mapping relationship between an object in the world coordinate system and the pixel coordinate system, and the corresponding homography transformation is called a homography matrix. Therefore, the motion adjustment information for performing rotation alignment processing on the reference frame is obtained by performing data fitting processing on the motion vector.

[0209] Further, the matching of the first feature point data window within the image block of the current frame to obtain a motion vector may include: calculating the sum of absolute errors between the gray value of each pixel point in the first feature point data window and the gray value of each pixel point in the data window within the search range of the image block of the current frame to obtain a parameter value; using the data window corresponding to the minimum value in the parameter values as the target data window; calculating the relative displacement from the first feature point data window to the target data window to obtain a motion vector.

[0210] It should be noted that the embodiment of the present application considers using the sum of absolute difference (SAD) criterion in the first feature point data window of the reference frame to perform search matching in the image block of the current frame, and finding the data window corresponding to the minimum value in the SAD values (i.e., parameter values) of the first feature point data window. At this time, this data window is the data window that best matches the first feature point data window, that is, the target data window. Therefore, the first feature point data window and the target data window form a relative position in the time domain to obtain a motion vector, so as to construct the motion adjustment information for performing rotation alignment processing on the reference frame through this motion vector.

[0211] Combined with the above description, a specific description of how to perform noise reduction filtering on the current frame and / or the first image frame to obtain the second image frame will be given below.

[0212] Specifically, the noise reduction filtering process of the current frame and / or the first image frame in S1150 to obtain the second image frame may include: determining noise reduction filtering strategy information for the current frame and the first image frame based on the gray value of each pixel in the image block of the current frame, the gray value of each pixel in the image block of the first image frame, and a preset threshold, where the noise reduction filtering strategy information includes temporal domain noise reduction filtering or spatial domain noise reduction filtering; and performing noise reduction filtering on the current frame and / or the first image frame based on the noise reduction filtering strategy information to obtain the second image frame.

[0213] Among them, determining the noise reduction filtering strategy information for the current frame and the first image frame based on the gray value of each pixel in the image block of the current frame, the gray value of each pixel in the image block of the first image frame, and a preset threshold may include: calculating the mean absolute error to obtain a parameter value based on the gray value of each pixel in the image block of the current frame and the gray value of each pixel in the image block of the first image frame; and determining the noise reduction filtering strategy information for the current frame and the first image frame according to the comparison result between the parameter value and the preset threshold.

[0214] Among them, the image block of the first image frame may be used to represent at least one non-overlapping image block with equal size in the first image frame. It can be understood that the filtering decision module 1302 in the embodiment of the present application may perform image block processing on the input first image frame to obtain N*N non-overlapping image blocks with equal size, and the value of N may be 2, 4, 8, 16, etc.

[0215] It should be noted that, first, in order to overcome the "trailing" phenomenon of fast-moving objects easily caused by temporal filtering, the embodiment of the present application considers detecting the motion intensity through the mean absolute differences (MAD) algorithm, so that for objects with different motion intensities, the filter adopts different filtering intensities, which is beneficial to avoiding the "trailing" phenomenon caused by fast-moving objects and improving the image filtering effect. Among them, the filtering decision module 1302 calculates the gray value of each pixel in the third image block of the current frame and the gray value of each pixel in the fourth image block of the first image frame through the MAD algorithm to obtain the MAD value (i.e., the parameter value).

[0216] Second, the embodiment of the present application tests a large number of image sequences to determine the empirical value for measuring the motion intensity, that is, the preset threshold, and then determines the motion intensity of the image block of the current frame on the motion trajectory represented by the motion vector according to the comparison result between the parameter value and the preset threshold.

[0217] If the comparison result between the parameter value and the preset threshold is that the parameter value is greater than the preset threshold, it indicates that the image block has a strong motion intensity on the motion trajectory. At this time, the filtering intensity of the filter can be adjusted lower (or it is determined that the noise reduction filtering strategy information is spatial domain filtering), so that the filtered image block retains as much original information as possible to avoid the generation of the "trailing" phenomenon.

[0218] If the comparison result between the parameter value and the preset threshold is that the parameter value is less than the preset threshold, it indicates that the image block has a weak motion intensity on the motion trajectory, that is, a stable and slow motion. For example, the background picture in the video stream generally does not move. At this time, the filtering intensity of the filter can be adjusted higher (or it is determined that the noise reduction filtering strategy information is time domain filtering), so as to effectively remove noise. At the same time, the low motion intensity of the image block will not cause the "trailing" phenomenon, which is beneficial to ensuring a better filtering effect.

[0219] Finally, in order to perform noise reduction filtering on the current frame, the embodiment of the present application performs noise reduction filtering on the current frame and / or the first image frame according to time domain filtering or spatial domain filtering.

[0220] Further, the spatial domain filtering may include mean filtering, median filtering, Gaussian filtering, or bilateral filtering, etc. Therefore, noise reduction filtering is performed on the current frame according to spatial domain filtering to output a second image frame, so as to filter the image block with a strong motion intensity on the motion trajectory through spatial domain filtering, and retain as much original information in the image block as possible to avoid the generation of the "trailing" phenomenon.

[0221] Further, the time domain filtering may include linear weighted filtering. Therefore, the second image frame output by performing fusion noise reduction filtering on the current frame and the first image frame according to linear weighted filtering satisfies the following formula:

[0222] F out =ω*F t +(1 - ω)*F t-1 ;

[0223] Wherein, F out represents the pixel values of each pixel in the image block of the second image frame, F t represents the pixel values of each pixel in the image block of the current frame, F t-1 represents the pixel values of each pixel in the image block of the first image frame, and ω represents a preset weight. In addition, the filtering intensity of the filter can be dynamically adjusted through the preset weight ω.

[0224] In short, filtering is performed on the image block with a weak motion intensity on the motion trajectory through linear weighted filtering, so as to effectively remove the noise in the image block.

[0225] Based on the above description, the technical solutions of the embodiments of the present application will be further described below.

[0226] Specifically, after obtaining the first feature point data window in the image block of the reference frame and the current frame in S1110, the method further includes: uploading the second feature point data window to the bus for storage.

[0227] Among them, the bus may include a DDR memory.

[0228] It should be noted that in the embodiments of the present application, only the feature point data window of the current frame is considered to be uploaded to the bus for storage, and the remaining redundant data features of the current frame are not uploaded and stored. This is not only beneficial to avoiding uploading and storing the redundant data in the current frame to reduce the data upload bandwidth, but also beneficial to directly reading the feature point data window of the current frame from the storage module when searching and matching the next frame image of the current frame in the subsequent process, without having to perform feature point detection again, thereby achieving a reduction in the processing delay during the video noise reduction process and an improvement in the processing efficiency.

[0229] Specifically, after obtaining the first feature point data window in the image block of the reference frame and the current frame in S1110, the method further includes: performing a smoothing filtering process on the current frame and outputting the current frame after the smoothing filtering process.

[0230] Specifically, after obtaining the first feature point data window in the image block of the reference frame and the current frame in S1110, the method further includes: performing a block processing on the current frame to obtain the image blocks of the current frame.

[0231] It should be noted that the descriptions of the embodiments of the present application each have their own focuses. Therefore, Figure 11 For the technical solutions not described in detail in the above embodiments, reference can be made to Figure 1 and Figure 6 the relevant descriptions in the above embodiments, and details will not be repeated here.

[0232] Based on the above description, the embodiments of the present application further provide a processing chip. Among them, the processing chip includes the above-mentioned Figure 1 or Figure 6 image noise reduction device. The image noise reduction device includes: an image processing module, a filtering processing module, and a noise reduction processing module; among them,

[0233] The image processing module is configured to match the first feature point data window in the image block of the reference frame within the image block of the current frame to output image affine transformation information, and parallelly perform feature point detection on the image block of the current frame to obtain a second feature point data window. The reference frame is temporally before the current frame;

[0234] A filtering processing module, configured to perform spatio-temporal domain smoothing filtering on the image affine transformation information to output motion adjustment information;

[0235] A noise reduction processing module, configured to perform rotation alignment processing on a reference frame by using the motion adjustment information to obtain a first image frame, and perform noise reduction filtering on the current frame and / or the first image frame to output a second image frame.

[0236] It should be noted that the descriptions of the embodiments in this application have their own emphases. Therefore, for the specific implementation functions of the various modules of the image noise reduction device in the processing chip, reference can be made to the relevant descriptions in the above embodiments, which will not be elaborated here.

[0237] It can be seen that the image noise reduction device in the processing chip directly obtains the first feature point data window and the current frame to implement parallel processing of search matching and feature point detection, and further reduces the processing delay in the video noise reduction process and improves the processing efficiency through parallel processing. At the same time, by directly uploading the second feature point data window of the current frame, the upload and storage of the remaining redundant data in the current frame are avoided to reduce the data upload bandwidth.

[0238] In addition, the image noise reduction device in the processing chip performs spatio-temporal domain smoothing filtering on the image affine transformation information to output motion adjustment information, so as to obtain more accurate and reasonable motion adjustment information, and further improve the accuracy of reference frame rotation alignment, and enhance the stability and accuracy of noise reduction filtering on the current frame and / or the rotation-aligned reference frame, so that the time-domain noise reduction effect presented by the second image frame is better, and the stability and accuracy of image noise reduction processing are improved.

[0239] Exemplarily, please refer to Figure 12 , Figure 12It is a schematic structural diagram of a processing chip provided by an embodiment of the present application. Among them, the processing chip 1200 includes an image noise reduction device 1210 and a communication bus for connecting the image noise reduction device 1210. Specifically, the processing chip 1200 may be a processor. Among them, the processor may include a central processing unit (CPU), an application processor (AP), a modulation and demodulation processor, a graphics processing unit (GPU), an image signal processor (ISP), a video codec, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processors may be independent devices or integrated in the same processor.

[0240] Furthermore, the processing chip 1200 may further include a memory for storing instructions and data. In some embodiments, the memory in the processing chip 1200 may be a cache memory. This memory can save the instructions or data that the processor has just used or recycled. If the processing chip 1200 needs to use the instruction or data again, it can be directly called from this memory, thus avoiding repeated accesses, reducing the waiting time of the processing chip 1200, and improving the system efficiency.

[0241] Further, the processing chip 1200 may further include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0242] Specifically, the filtering processing module specifically includes: a spatio-temporal domain smoothing filtering module and a motion adjustment module; where

[0243] The spatio-temporal domain smoothing filtering module is configured to perform spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristics of the rolling shutter to output smoothed filtering information;

[0244] The motion adjustment module is configured to determine motion adjustment information based on the smoothed filtering information.

[0245] Specifically, the spatio-temporal domain smoothing filtering module is configured to: perform smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the second image affine transformation information corresponding to the image block at the previous position and / or the next position of the first image block within the first row block to output the first smoothed filtering information in the smoothed filtering information;

[0246] Wherein, the image affine transformation information includes first image affine transformation information and second image affine transformation information.

[0247] Specifically, the spatio-temporal domain smoothing filtering module is configured to: perform smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the third image affine transformation information corresponding to the second image block in the previous row block of the first row block to output the second smoothed filtering information in the smoothed filtering information;

[0248] Wherein, the image affine transformation information further includes third image affine transformation information.

[0249] Specifically, if the first line block is the first line block in the line blocks of the current frame, the previous line block of the first line block is the last line block in the line blocks of the reference frame.

[0250] Specifically, the image processing module specifically includes a search and matching module, an affine transformation module, and a feature point detection module; among them,

[0251] The search and matching module is used to match the first feature point data window within the image block of the current frame to output a motion vector;

[0252] The affine transformation module is used to perform data fitting processing on the motion vector to output image affine transformation information;

[0253] The feature point detection module is used to detect feature points in the image block of the current frame to obtain a second feature point data window.

[0254] Specifically, the noise reduction processing module specifically includes a rotation alignment module, a filtering decision module, and a fusion filtering module; among them,

[0255] The rotation alignment module is used to perform rotation alignment processing on the reference frame based on the motion adjustment information to output a first image frame;

[0256] The filtering decision module is used to determine noise reduction filtering strategy information for the current frame and the first image frame based on the gray value of each pixel point in the image block of the current frame, the gray value of each pixel point in the image block of the first image frame, and a preset threshold, and the noise reduction filtering strategy information includes time-domain noise reduction filtering or spatial-domain noise reduction filtering;

[0257] The fusion filtering module is used to perform noise reduction filtering processing on the current frame and / or the first image frame based on the noise reduction filtering strategy information to output a second image frame.

[0258] Combined with the above description, an embodiment of the present application further provides an electronic device, where the electronic device includes a memory, a communication interface, and the above-mentioned image noise reduction device. The image noise reduction device includes: an image processing module, a filtering processing module, and a noise reduction processing module; among them,

[0259] The image processing module is used to match the first feature point data window in the image block of the reference frame within the image block of the current frame to output image affine transformation information, and parallelly detect feature points in the image block of the current frame to obtain a second feature point data window, and the reference frame is temporally before the current frame;

[0260] The filtering processing module is used to perform spatio-temporal domain smoothing filtering processing on the image affine transformation information to output motion adjustment information;

[0261] A noise reduction processing module is configured to perform rotation alignment processing on a reference frame by using motion adjustment information to obtain a first image frame, and perform noise reduction filtering processing on the current frame and / or the first image frame to output a second image frame.

[0262] It should be noted that the descriptions of the embodiments in this application have their own emphases. Therefore, for the specific implementation functions of each module in the image noise reduction device of the electronic device, reference can be made to the relevant descriptions in the above embodiments, which will not be elaborated here.

[0263] It can be seen that the image noise reduction device in the electronic device directly obtains the first feature point data window and the current frame to implement parallel processing of search matching and feature point detection, and then reduces the processing delay in the video noise reduction process through parallel processing to improve the processing efficiency. At the same time, by directly uploading the second feature point data window of the current frame, uploading and storing the remaining redundant data in the current frame is avoided to reduce the data upload bandwidth.

[0264] In addition, the image noise reduction device in the electronic device performs spatio-temporal domain smoothing filtering processing on the image affine transformation information to output motion adjustment information, so as to obtain more accurate and reasonable motion adjustment information, and then improve the accuracy of reference frame rotation alignment, and enhance the stability and accuracy of noise reduction filtering processing on the current frame and / or the reference frame after rotation alignment, so that the time-domain noise reduction effect presented by the second image frame is better, and the stability and accuracy of image noise reduction processing are improved.

[0265] Exemplarily, please refer to Figure 13 , Figure 13 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Among them, the electronic device 1300 includes a memory 1401, a communication interface 1402, an image noise reduction device 1403, and a communication bus for connecting the memory 1401, the communication interface 1402, and the image noise reduction device 1403.

[0266] Specifically, the electronic device 1300 according to the embodiments of the present application may be a handheld device, a vehicle-mounted device, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, or other devices connected to a wireless modem, and may also be various specific forms of user equipment (UE), terminal device, smart phone, smart screen, smart TV, smart watch, laptop computer, station (STA), access point (AP), mobile station (MS), personal digital assistant (PDA), personal computer (PC), or relay device, etc.

[0267] Specifically, the image noise reduction device 1403 may be a processor. The processor may include a CPU, a GPU, an ISP, a DSP, an FPGA, an ASIC, a baseband processor, and / or an NPU, etc. Wherein, when the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0268] Specifically, the memory 1401 may include a double data rate synchronous dynamic random access memory (DDR SDRAM), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), and / or a compact disc read-only memory (CD-ROM), and the memory 1401 is used to store relevant instructions and data.

[0269] Specifically, the communication interface 1402 is used to receive and send data, signals, instructions, etc.

[0270] Specifically, the filtering processing module specifically includes: a spatio-temporal domain smoothing filtering module and a motion adjustment module; wherein,

[0271] A spatio-temporal domain smoothing filtering module, which is used to perform spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristics of the rolling shutter to output smoothing filtering information;

[0272] A motion adjustment module, which is used to determine motion adjustment information based on the smoothing filtering information.

[0273] Specifically, the spatio-temporal domain smoothing filtering module is used to: perform smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the second image affine transformation information corresponding to the image block at the previous position and / or the next position of the first image block within the first row block to output the first smoothing filtering information in the smoothing filtering information;

[0274] Among them, the image affine transformation information includes the first image affine transformation information and the second image affine transformation information.

[0275] Specifically, the spatio-temporal domain smoothing filtering module is used to: perform smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the third image affine transformation information corresponding to the second image block in the previous row block of the first row block to output the second smoothing filtering information in the smoothing filtering information;

[0276] Among them, the image affine transformation information further includes the third image affine transformation information.

[0277] Specifically, if the first row block is the first row block among the row blocks of the current frame, then the previous row block of the first row block is the last row block among the row blocks of the reference frame.

[0278] Specifically, the image processing module specifically includes a search and matching module, an affine transformation module, and a feature point detection module; among them,

[0279] The search and matching module is used to match the first feature point data window within the image block of the current frame to output a motion vector;

[0280] The affine transformation module is used to perform data fitting processing on the motion vector to output image affine transformation information;

[0281] The feature point detection module is used to perform feature point detection on the image block of the current frame to obtain a second feature point data window.

[0282] Specifically, the noise reduction processing module specifically includes a rotation alignment module, a filtering decision module, and a fusion filtering module; among them,

[0283] The rotation alignment module is used to perform rotation alignment processing on the reference frame based on the motion adjustment information to output a first image frame;

[0284] A filtering decision module, configured to determine noise reduction filtering strategy information for the current frame and the first image frame based on the gray values of each pixel point in the image block of the current frame, the gray values of each pixel point in the image block of the first image frame, and a preset threshold, where the noise reduction filtering strategy information includes time-domain noise reduction filtering or spatial-domain noise reduction filtering;

[0285] A fusion filtering module, configured to perform noise reduction filtering processing on the current frame and / or the first image frame based on the noise reduction filtering strategy information to output a second image frame.

[0286] It should be noted that for the above embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should know that the present application is not limited by the described action sequence, because some steps in the embodiments of the present application can be performed in other sequences or simultaneously. In addition, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of the present application.

[0287] In the above embodiments, the present application focuses on the descriptions of the respective embodiments. For parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0288] In several embodiments provided by the present application, those skilled in the art should know that the described device can be implemented in other ways. It can be understood that the described device embodiments are only illustrative. For example, the above module division is only a logical function division, and there can be other division methods in practice. That is to say, multiple modules can be combined or integrated into the same module, and some features can be ignored or not executed. In addition, the connections between the modules shown or discussed can be indirect coupling connections, direct coupling connections, or communication connection methods, etc., can be through some interfaces, or in electrical or other forms.

[0289] If the above modules are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. It can be understood that the technical solution of the present application (the part that contributes to the prior art or all or part of the technical solution) can be embodied in the form of a computer-readable storage medium. The computer-readable storage medium can be stored in a memory, including several instructions for causing a computer device (such as a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. In addition, the above computer-readable storage medium can be stored in various memories such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disc.

[0290] The above has made a specific introduction to the embodiments of the present application. Those skilled in the art should be aware that the embodiments of the present application are only used to help understand the core idea of the technical solution of the present application. Therefore, there will be changes in the specific implementation manner and application scope of the embodiments of the present application. So far, the content recorded in this specification should not be construed as a limitation on the protection scope of the present application. In addition, any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. An image noise reduction device, characterized in that, Including: An image processing module, a filtering processing module, and a noise reduction processing module; wherein, The image processing module is configured to match a first feature point data window in an image block of a reference frame within an image block of the current frame to output image affine transformation information, and concurrently perform feature point detection on the image block of the current frame to obtain a second feature point data window, and the reference frame is temporally before the current frame; The filtering processing module is configured to perform spatio-temporal domain smoothing filtering on the image affine transformation information to output motion adjustment information; The noise reduction processing module is configured to perform rotational alignment processing on the reference frame using the motion adjustment information to obtain a first image frame, and perform noise reduction filtering on the current frame and / or the first image frame to output a second image frame.

2. The device according to claim 1, characterized in that, The filtering processing module includes: a spatio-temporal domain smoothing filtering module and a motion adjustment module; wherein, The spatio-temporal domain smoothing filtering module is configured to perform spatial domain smoothing filtering and / or temporal domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristics of a rolling shutter to output smoothing filtering information; The motion adjustment module is configured to determine the motion adjustment information based on the smoothing filtering information.

3. The device according to claim 2, characterized in that, The spatio-temporal domain smoothing filtering module is configured to: Perform smoothing filtering on the first image affine transformation information corresponding to a first image block in the first row block of the current frame and the second image affine transformation information corresponding to an image block at a previous position and / or a subsequent position of the first image block within the first row block to output first smoothing filtering information in the smoothing filtering information; the first row block is one of the row blocks of the current frame, and the first image block is at least one image block in the first row block; Wherein, the image affine transformation information includes the first image affine transformation information and the second image affine transformation information.

4. The device according to claim 2, characterized in that The spatio-temporal domain smoothing filtering module is configured to: Perform smoothing filtering on the first image affine transformation information corresponding to a first image block in the first row block of the current frame and the third image affine transformation information corresponding to a second image block in the row block before the first row block of the current frame to output second smoothing filtering information in the smoothing filtering information; the first row block is one of the row blocks of the current frame, and the first image block is at least one image block in the first row block; Wherein, the image affine transformation information further includes the third image affine transformation information.

5. The device according to claim 4, wherein If the first row block is the first row block among the row blocks of the current frame, then the row block before the first row block is the last row block among the row blocks of the reference frame.

6. The device according to any one of claims 1-5, characterized in that The image processing module includes a search and matching module, an affine transformation module, and a feature point detection module; wherein, The search and matching module is configured to match the first feature point data window within the image block of the current frame to output a motion vector; The affine transformation module is configured to perform data fitting processing on the motion vector to output the image affine transformation information; The feature point detection module is configured to perform feature point detection on the image block of the current frame to obtain the second feature point data window.

7. The device according to any one of claims 1-5, characterized in that The noise reduction processing module includes a rotation alignment module, a filtering decision module, and a fusion filtering module; wherein, the rotation alignment module is configured to perform rotation alignment processing on the reference frame based on the motion adjustment information to output the first image frame; the filtering decision module is configured to determine noise reduction filtering strategy information for the current frame and the first image frame based on the gray value of each pixel point in the image block of the current frame, the gray value of each pixel point in the image block of the first image frame, and a preset threshold, and the noise reduction filtering strategy information includes time-domain noise reduction filtering or spatial-domain noise reduction filtering; the fusion filtering module is configured to perform noise reduction filtering processing on the current frame and / or the first image frame based on the noise reduction filtering strategy information to output the second image frame.

8. An image denoising method, characterized in that, including: acquiring a first feature point data window in an image block of a reference frame and a current frame, where the reference frame is temporally before the current frame; matching the first feature point data window in the image block of the current frame to obtain image affine transformation information, and simultaneously detecting feature points in the image block of the current frame to obtain a second feature point data window; performing spatio-temporal domain smoothing filtering on the image affine transformation information to obtain motion adjustment information; performing rotation alignment processing on the reference frame based on the motion adjustment information to obtain the first image frame; performing noise reduction filtering processing on the current frame and / or the first image frame to obtain the second image frame.

9. The method according to claim 8, wherein The performing spatio-temporal domain smoothing filtering on the image affine transformation information to obtain motion adjustment information includes: performing spatial-domain smoothing filtering and / or temporal-domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristics of a rolling shutter to obtain smoothing filtering information; determining the motion adjustment information based on the smoothing filtering information.

10. The method according to claim 9, wherein The performing spatial-domain smoothing filtering and / or temporal-domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristics of a rolling shutter to obtain smoothing filtering information includes: performing smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the second image affine transformation information corresponding to the image block at the previous position and / or the next position of the first image block in the first row block to obtain the first smoothing filtering information in the smoothing filtering information; the first row block is one of the row blocks of the current frame, and the first image block is at least one image block in the first row block; wherein, the image affine transformation information includes the first image affine transformation information and the second image affine transformation information.

11. The method according to claim 9, wherein The performing spatial-domain smoothing filtering and / or temporal-domain smoothing filtering on the image affine transformation information corresponding to the image block of the current frame based on the exposure characteristics of a rolling shutter to obtain smoothing filtering information includes: Performing smoothing filtering on the first image affine transformation information corresponding to the first image block in the first row block of the current frame and the third image affine transformation information corresponding to the second image block in the previous row block of the first row block to obtain the second smoothing filtering information in the smoothing filtering information; the first row block is one of the row blocks of the current frame, and the first image block is at least one image block in the first row block; Wherein, the image affine transformation information further includes the third image affine transformation information.

12. The method according to claim 11, wherein If the first row block is the first row block among the row blocks of the current frame, then the previous row block of the first row block is the last row block among the row blocks of the reference frame.

13. The method according to any one of claims 8-12, characterized in that, The matching of the first feature point data window within the image block of the current frame to obtain the image affine transformation information includes: Matching the first feature point data window within the image block of the current frame to obtain a motion vector; Performing data fitting processing on the motion vector to obtain the image affine transformation information.

14. The method according to any one of claims 8 - 12, characterized in that, The noise reduction filtering process of the current frame and / or the first image frame to obtain the second image frame includes: Determining noise reduction filtering strategy information for the current frame and the first image frame based on the gray value of each pixel point in the image block of the current frame, the gray value of each pixel point in the image block of the first image frame, and a preset threshold, where the noise reduction filtering strategy information includes time-domain noise reduction filtering or spatial-domain noise reduction filtering; Performing noise reduction filtering on the current frame and / or the first image frame based on the noise reduction filtering strategy information to obtain the second image frame.

15. A processing chip, characterized in that, Including the image noise reduction device according to any one of claims 1-7.

16. An electronic device, characterized in that, Including a memory, a communication interface, and the image noise reduction device according to any one of claims 1-7.

Images