Image processing device and method, processing chip and electronic device

Abstract

The embodiment of the present application discloses an image processing device and method, a processing chip and an electronic device. The image processing device includes: a delay processing module and a noise reduction processing module; wherein the delay processing module is used to search and match the first feature point data window in the first image block of the acquired reference frame in the second image block of the acquired current frame to output motion adjustment information to the noise reduction processing module, and to perform feature point detection on the second image block in parallel to obtain a second feature point data window, and the reference frame is used to represent an image frame that is located before the current frame in time sequence; the noise reduction processing module is used to perform block rotation and alignment processing on the reference frame using the motion adjustment information to obtain an aligned frame, and to perform noise reduction filtering processing on the current frame and / or the aligned frame to output a target frame, thereby reducing the processing delay in the video noise reduction process.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing device and method, a processing chip and an electronic device. Background Art

[0002] At present, in video noise reduction technology, fusion noise reduction filtering is performed through the correlation of image frames in the video stream in the time domain. At the same time, in order to overcome the non-stationary characteristics of the video stream in the time domain caused by object motion, motion estimation and motion compensation are used to find the corresponding pixels of the current pixel in the previous and next frames, so as to ensure the filtering efficiency by maintaining the stability in the time domain. However, in the actual video noise reduction process, there is usually a certain processing delay problem between the processing steps. Therefore, how to reduce the processing delay in the video noise reduction process and thus improve the processing efficiency requires further analysis. Summary of the invention

[0003] The embodiments of the present application provide an image processing device and method, a processing chip and an electronic device, in the hope of reducing the processing delay in the video noise reduction process and improving the processing efficiency through the parallel processing of search matching and feature point detection.

[0004] In a first aspect, an embodiment of the present application provides an image processing device, comprising: a delay processing module and a noise reduction processing module; wherein:

[0005] The delay processing module is used to search and match a first feature point data window in a first image block of a reference frame in a second image block of a current frame to output motion adjustment information to the noise reduction processing module, and to perform feature point detection on the second image block in parallel to obtain a second feature point data window, wherein the reference frame is used to represent an image frame that is located before the current frame in time sequence;

[0006] The noise reduction processing module is used to perform block rotation and alignment processing on the reference frame using the motion adjustment information to obtain an aligned frame, and perform noise reduction filtering processing on the current frame and / or the aligned frame to output a target frame.

[0007] It can be seen that, first, the delay processing module obtains the first feature point data window in the first image block of the reference frame and the current frame, that is, directly obtains the first feature point data window (without performing feature point detection) and the current frame; secondly, the delay processing module performs search and matching of the first feature point data window of the reference frame in the second image block of the current frame and feature point detection of the second image block of the current frame in parallel; thirdly, the delay processing module outputs motion adjustment information to the noise reduction processing module; finally, the noise reduction processing model uses the motion adjustment information to rotate and align the reference frame to obtain an aligned frame, and performs noise reduction filtering on the current frame and / or the aligned frame to output a target frame, thereby directly obtaining the first feature point data window and the current frame to achieve parallel processing of search and matching and feature point detection, and then reducing the processing delay in the video noise reduction process through parallel processing to improve processing efficiency.

[0008] In a second aspect, an embodiment of the present application provides an image processing method, including:

[0009] Acquire a first feature point data window in a first image block of a reference frame and a current frame, wherein the reference frame is used to represent an image frame that is located before the current frame in time sequence;

[0010] Searching and matching the first feature point data window in the second image block of the current frame to obtain motion adjustment information, and performing feature point detection on the second image block in parallel to obtain a second feature point data window;

[0011] Using the motion adjustment information, block rotation and alignment processing are performed on the reference frame to obtain an aligned frame;

[0012] The current frame and / or the aligned frame are subjected to noise reduction filtering to obtain a target frame.

[0013] It can be seen that, first, the first feature point data window in the first image block of the reference frame (i.e., no feature point detection is required) and the current frame are directly acquired; secondly, the search and matching of the first feature point data window of the reference frame in the second image block of the current frame and the feature point detection of the second image block of the current frame are performed in parallel; finally, the reference frame is rotationally aligned using the motion adjustment information to obtain an aligned frame, and the current frame and / or the aligned frame are subjected to denoising filtering to output a target frame, thereby directly acquiring the first feature point data window and the current frame to achieve parallel processing of search and matching and feature point detection, and then reducing the processing delay in the video denoising process through parallel processing to improve processing efficiency.

[0014] In a third aspect, an embodiment of the present application provides a processing chip, including the image processing device in the first aspect of the embodiment of the present application.

[0015] It can be seen that the image processing device in the processing chip directly obtains the first feature point data window and the current frame to realize parallel processing of search matching and feature point detection, thereby reducing the processing delay in the video denoising process and improving the processing efficiency through parallel processing.

[0016] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a communication interface, and the image processing device in the first aspect of the embodiment of the present application.

[0017] It can be seen that the image processing device in the electronic device directly obtains the first feature point data window and the current frame to realize parallel processing of search matching and feature point detection, and then reduces the processing delay in the video noise reduction process through parallel processing to improve the processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art description are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without paying creative work.

[0019] Figure 1 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of a process of video noise reduction provided by an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of performing feature point detection on an image block of a current frame provided by an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of searching and matching a current frame and a reference frame provided in an embodiment of the present application;

[0023] Figure 5 It is a structural schematic diagram of the relative displacement from a feature point data window of a current frame to a target feature point data window provided by an embodiment of the present application;

[0024] Figure 6 is a structural schematic diagram of another image processing device provided in an embodiment of the present application;

[0025] Figure 7 is a structural schematic diagram of another image processing device provided in an embodiment of the present application;

[0026] Figure 8 is a structural schematic diagram of another image processing device provided in an embodiment of the present application;

[0027] Fig. 9 This is another schematic diagram of a process of video noise reduction provided by an embodiment of the present application;

[0028] Fig.10 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0029] Fig.11 is a schematic diagram of the structure of a processing chip provided in an embodiment of the present application;

[0030] Fig.12 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to help those skilled in the art better understand the technical solutions of the present application, the following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the description of the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0032] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, software, product, or device that includes a series of steps or units is not limited to the listed steps or units, but also includes steps or units that are not listed, or also includes other steps or units inherent to these processes, methods, products, or devices.

[0033] In the embodiments of the present application, unless otherwise clearly specified and limited, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a coupled connection, a detachable connection; it can be a direct connection, an indirect connection through an intermediate medium, or an interval contact; it can also be a physical connection or an electrical connection, etc.

[0034] Mentioning "embodiment" in the embodiments of the present application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] Before describing the technical solutions of the embodiments of the present application, the following first introduces relevant concepts that may be involved in the present application.

[0036] (1) Video noise reduction

[0037] Since camera devices, such as complementary metal-oxide-semiconductor (CMOS) sensors and charge-coupled device (CCD) sensors, will be affected by noise during the data acquisition process, resulting in random noise in the video stream, it is necessary to use image noise reduction technology to remove the noise.

[0038] Video denoising methods can be divided into spatial domain, frequency domain, wavelet domain, time domain, spatiotemporal domain, etc. according to different processing domains. Among them, denoising methods between different processing domains may overlap, or a denoising method may involve multiple processing domains. For example, a frequency domain filtering denoising method can be used in a time domain or spatiotemporal domain denoising method, that is, all or part of the image frames in the video stream are converted to the frequency domain through Fourier transform, and then the time domain filtering or spatiotemporal domain filtering is used to perform denoising.

[0039] The spatial domain filtering, frequency domain filtering and wavelet domain filtering in video denoising are the same as those in image denoising, but they are extended to process multiple frames of images and optimize by using more redundant information in the video signal, so as to achieve better denoising effect. Among them, spatial domain filtering directly performs algebraic operations on pixel values ​​for each frame of the video stream, and only considers the correlation of the image frames in the spatial domain. Frequency domain filtering converts the image frames in the video stream into the frequency domain through Fourier transform, and then attenuates the frequencies representing the noise to retain the original information in the video stream to the greatest extent. Wavelet domain filtering converts the image frames in the video stream into the time-frequency domain before performing denoising. Time domain filtering considers the correlation of the image frames in the video stream in the time dimension for denoising filtering, which is simple in operation, highly efficient, and does not introduce spatial blur.

[0040] (2) 3-Dimension Digital Noise Reduction

[0041] 3DNR is a noise reduction method that combines spatial domain filtering and temporal domain filtering. The general idea is to detect the motion vectors between multiple frames in the video stream through motion estimation, then align them according to the motion vectors, and finally perform fusion noise reduction filtering on the aligned multiple frames to output the filtered image.

[0042] At present, in video noise reduction technology, fusion noise reduction filtering is performed through the correlation of image frames in the video stream in the time domain. At the same time, in order to overcome the non-stationary characteristics of the video stream in the time domain caused by object motion, motion estimation and motion compensation are used to find the corresponding pixels of the current pixel in the previous and next frames, so as to ensure the filtering efficiency by maintaining the stability in the time domain. However, in the actual video noise reduction process, there is usually a certain processing delay problem between the processing steps. Therefore, how to reduce the processing delay in the video noise reduction process and thus improve the processing efficiency requires further analysis.

[0043] In combination with the above description, the image processing device of the embodiment of the present application is introduced below. Figure 1 . Figure 1 1 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device 10 includes a delay processing module 100 and a noise reduction processing module 200. The delay processing module 100 is connected to the noise reduction processing module 200.

[0044] The delay processing module 100 is used to search and match the first feature point data window in the first image block of the acquired reference frame in the second image block of the acquired current frame to output motion adjustment information to the noise reduction processing module, and to perform feature point detection on the second image block in parallel to obtain a second feature point data window. The reference frame is used to represent an image frame that is located before the current frame in time sequence.

[0045] The noise reduction processing module 200 is used to perform block rotation and alignment processing on the reference frame using the motion adjustment information to obtain an aligned frame, and perform noise reduction filtering processing on the current frame and / or the aligned frame to output a target frame.

[0046] It should be noted that, first of all, in order to realize the functions of time domain noise reduction and low processing delay, the image processing device 10 of the embodiment of the present application includes hardware and / or software modules for executing the corresponding functions. It should be easy for those skilled in the art to realize that, in combination with the functions of the modules described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a module is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can call a specific program or algorithm through a processor (such as a CPU, etc.) to implement the functions of the modules described in the embodiment of the present application, but such implementation should not be considered to be beyond the scope of the present application. In addition, the embodiment of the present application can divide the image processing device 10 into functional modules, for example, each functional module can be divided according to function, or two or more functions can be integrated into one functional module. The above-mentioned integrated functional module can be implemented in the form of hardware or software. At the same time, the division of the functional modules in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation, and no specific restrictions are made on this.

[0047] Secondly, in Figure 2 In the video denoising method shown, a reference frame and a current frame to be denoised are obtained from a video stream, and the current frame is uploaded to a bus for storage, where the reference frame is a previous frame or a next frame of the current frame; the current frame is smoothed and filtered, and feature point detection is performed on the smoothed and filtered current frame to obtain a feature point data window; the feature point data window of the current frame is searched and matched to the reference frame to obtain a motion vector, i.e., a motion matching result, and motion adjustment information is calculated through the motion matching result, and the motion adjustment information is used to rotationally align the reference frame; the current frame and the reference frame are read from the bus, and the reference frame is rotationally aligned through the motion adjustment information; the current frame and the rotationally aligned reference frame are subjected to a two-frame fusion denoising process to obtain a target frame, thereby achieving time domain denoising of the current frame.

[0048] Next, for feature point detection of the current frame, the current frame can be first divided into blocks to obtain N*N image blocks of equal size and non-overlapping, where N can be 2, 4, 8 or 16, etc., and then feature point detection is performed on each image block to obtain multiple feature point data windows. For example, see Figure 3 , divide the current frame into 4*4 image blocks, and then perform feature point detection on each image block to obtain multiple feature point data windows.

[0049] For searching and matching the feature point data window of the current frame to the reference frame to obtain the best motion vector, a data window similar to the feature point data window of the current frame can be searched and matched in the reference frame to obtain the target feature point data window in the reference frame, such as Figure 4 As shown, Figure 4 The area 410 in the figure represents the search matching range, and then the relative displacement from the feature point data window of the current frame to the target feature point data window is calculated to obtain the motion vector, such as Figure 5 shown.

[0050] Finally, in Figure 3 In the feature point detection process shown in FIG. 1 , it is usually necessary to perform feature point detection on the four image blocks in the first row block of the current frame after segmentation (i.e., 1 / 4 frame of the current frame), and then perform feature point detection on the four image blocks in the second row block of the current frame, and so on. Therefore, a processing delay of 1 / N frame is usually introduced in the feature point detection process, i.e. Figure 3 The processing delay of 1 / 4 frame in Figure 4 In the search and matching process shown, the feature point data window of the 1 / 4 frame of the current frame (i.e., the first row block of the current frame) is usually searched and matched in the 1 / 4 frame of the reference frame (i.e., the first row block of the reference frame), and so on. Therefore, a processing delay of 1 / N frame will be introduced in the search and matching process.

[0051] In addition, when the current frame and the reference frame after rotation alignment are fused for noise reduction, it is still necessary to read one frame of the current frame and one frame of the reference frame from the bus completely. Therefore, a processing delay of one frame will be introduced in the fusion noise reduction process. It can be seen that there are usually many processing delays in the actual video noise reduction processing steps.

[0052] In summary, in order to reduce processing delay, the technical solution of the embodiment of the present application is specifically described below.

[0053] Specifically, the current frame can be used to represent a frame of an image to be denoised in the video stream.

[0054] Specifically, the reference frame can be used to represent an image frame that is located before the current frame in time sequence. For example, the reference frame can be an image frame before the current frame.

[0055] Specifically, search matching can be understood as determining a search range in the image block of the current frame through a search matching algorithm, then finding the most similar (or matching) data window within the search range through the first feature point data window, and finally obtaining motion adjustment information through a motion vector (motion vector, MV) from the first feature point data window to the most similar data window. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous screening algorithm, a multi-layer screening algorithm, a partial distortion elimination algorithm, and the like.

[0056] Specifically, the motion adjustment information may be used to perform block rotation and alignment processing on the reference frame.

[0057] Specifically, feature point detection may include Harris corners detection, where the corners in the image frame may have the following features: intersections between contours; for the same scene, they remain stable even when the viewing angle changes; and pixels in the vicinity of the corners have large changes in both the gradient direction and the gradient amplitude.

[0058] It should be noted that hash corner detection can be performed by moving a local window on the image block of the current frame to determine whether the pixel value in the local window has changed significantly. Among them, whether the pixel value has changed significantly can be described by the image gradient. The larger the image gradient, the greater the change in the pixel value in the local window (or the greater the rate of change of the grayscale value). Therefore, if the grayscale value in the local window (in the image gradient) has a large change, there is a corner point in the area where the local window is located.

[0059] It can be understood that if hash corner point detection is used to perform feature point detection on the image block of the current frame to obtain a feature point data window, then the feature point data window can be used to represent the corner point in the image block, or, the feature point data window can be used to represent the image area composed of the corner point in the image block and the pixel points around the corner point, etc.

[0060] Specifically, the first image block may be used to represent at least one image block among a plurality of image blocks of the reference frame that are equal in size and do not overlap with each other.

[0061] It can be understood that the embodiments of the present application can first perform image block processing on the acquired reference frame to obtain N*N image blocks of equal size and non-overlapping, where the value of N can be 2, 4, 8 or 16, etc., and then perform feature point detection on each image block to obtain multiple feature point data windows, and finally store the multiple feature point data windows, which is conducive to directly obtaining the feature point data window of the reference frame in subsequent search and matching without the need to perform feature point detection, thereby reducing the processing delay in the video denoising process.

[0062] Specifically, the second image block may be used to represent at least one image block among a plurality of image blocks of the current frame that are equal in size and do not overlap with each other.

[0063] It can be understood that the embodiment of the present application can perform image block processing on the acquired current frame to obtain N*N image blocks of equal size and non-overlapping, and the value of N can be 2, 4, 8 or 16, etc.

[0064] Specifically, the position of the first image block in the reference frame corresponds to the position of the second image block in the current frame.

[0065] It can be understood that if the first image block is an image block in the first row block of the reference frame, then the second image block is an image block in the first row block of the current frame, and the rest are similar.

[0066] It can be seen that compared with Figure 2 In the distributed processing of first performing feature point detection on the current frame and then transferring the feature point data window of the current frame to the reference frame for search and matching, the embodiment of the present application directly obtains the first feature point data window of the reference frame (i.e., no feature point detection is required) and the current frame through the delay processing module 100, performs search and matching of the first feature point data window in the second image block of the current frame and feature point detection of the second image block of the current frame in parallel, and outputs motion adjustment information to the noise reduction processing module 200 through the delay processing module 100, thereby directly obtaining the first feature point data window and the current frame to realize parallel processing of search and matching and feature point detection, and then reduces the processing delay in the video noise reduction process through parallel processing, thereby improving processing efficiency. At the same time, by directly uploading the second feature point data window of the current frame, it is avoided to upload and store the remaining redundant data in the current frame to reduce the bandwidth for data upload.

[0067] In combination with the above description, the following embodiments of the present application will be combined with Figure 6 Each module in the image processing device 10 will be described in detail.

[0068] In a possible example, the delay processing module 100 specifically includes a feature point detection module 130, a motion estimation module 110 and a motion adjustment module 120; wherein,

[0069] The feature point detection module 130 is used to perform feature point detection on the second image block to obtain a second feature point data window.

[0070] The motion estimation module 110 is used to search and match the first feature point data window in the second image block to obtain a motion vector, and output the motion vector to the motion adjustment module.

[0071] The motion adjustment module 120 is used to perform data fitting processing on the input motion vector to obtain a homography matrix, use the homography matrix as motion adjustment information, and output the motion adjustment information to the noise reduction processing module 200 .

[0072] It should be noted that, firstly, the embodiment of the present application can directly obtain the first feature point data window of the reference frame and the current frame through the delay processing module 100, and perform the search and matching of the first feature point data window in the current frame and the feature point detection of the current frame in parallel, thereby directly obtaining the first feature point data window and the current frame to realize parallel processing of search and matching and feature point detection, and then reduce the processing delay in the video noise reduction process through parallel processing.

[0073] Secondly, the feature point detection in the embodiment of the present application includes hash corner point detection, so the feature point detection module 130 can detect the corner points in the image frame of the current frame.

[0074] Again, motion estimation is one of the important components of video noise reduction technology. By calculating the relative motion offset of the pixel point in two adjacent frames of images, its motion vector (MV) is obtained. Among them, the motion estimation algorithm may include a block matching algorithm, a pixel method, a feature method, and a phase method. Therefore, the motion estimation module 110 can realize the search and matching of the reference frame and the current frame to obtain the motion vector. Among them, the motion estimation module 110 can determine the search range in the image block of the current frame through the search matching algorithm, and then find the most similar (or matching) data window within the search range. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous screening algorithm, a multi-layer screening algorithm, a partial distortion elimination algorithm, and the like.

[0075] Finally, the data fitting process of the motion vector of the embodiment of the present application may include the least square method, and the motion vector is subjected to data fitting process by the least square method to obtain a homography matrix. Among them, the homography transformation can be used to describe the position mapping relationship between the world coordinate system and the pixel coordinate system of the object, and the corresponding homography transformation is called a homography matrix. Therefore, the motion vector is calculated by the motion adjustment module 120 to obtain the motion adjustment information for performing block rotation and alignment processing on the reference frame.

[0076] Specifically, the motion estimation module 110 is specifically used to: calculate the absolute error sum between the grayscale value of each pixel in the first feature point data window and the grayscale value of each pixel in the data window within the search range of the second image block to obtain a first parameter value; use the data window corresponding to the minimum value in the first parameter value as the target data window; and calculate the relative displacement from the first feature point data window to the target data window to obtain a motion vector, and output the motion vector.

[0077] It should be noted that the embodiment of the present application considers using the minimum sum of absolute difference (SAD) criterion in the first feature point data window of the reference frame to search and match in the second image block of the current frame, and finds the data window corresponding to the minimum value of the SAD value (i.e., the first parameter value) of the first feature point data window. At this time, the data window is the data window that best matches the first feature point data window, i.e., the target data window. Therefore, the first feature point data window and the target data window form a relative position in the time domain to obtain a motion vector, thereby constructing motion adjustment information for block rotation and alignment processing of the reference frame through the motion vector.

[0078] In a possible example, the delay processing module 100 further includes a smoothing filter module 140.

[0079] The smoothing filter module 140 may be used to perform smoothing filter processing on the current frame and output the current frame after the smoothing filter processing.

[0080] It should be noted that the smoothing filter module 140 can be used to reduce noise and artifacts in the current frame. The smoothing filter module 140 can include a mean filter, a box filter, a median filter, a Gaussian filter, a bilateral filter, and the like.

[0081] In a possible example, the delay processing module 100 further includes an image segmentation module 150.

[0082] The image blocking module 150 may be used to perform image blocking processing on the current frame and output the current frame after the image blocking processing.

[0083] It should be noted that the embodiment of the present application can divide the image frames (including the current frame or the reference frame) of the acquired video stream into N*N image blocks of the same size and non-overlapping through the image segmentation module 150, thereby facilitating the subsequent direct feature point detection of the image blocks and improving the efficiency of feature point detection of the image frames.

[0084] In a possible example, the noise reduction processing module 200 specifically includes a rotation alignment module 210, a filtering decision module 220 and a fusion filtering module 230.

[0085] The rotation alignment module 210 may be used to call the motion adjustment information to perform block rotation and alignment processing on the read reference frame to obtain an aligned frame, and output the aligned frame.

[0086] The filtering decision module 220 is used to calculate the mean absolute error of the grayscale value of each pixel in the third image block of the read current frame and the grayscale value of each pixel in the fourth image block of the input aligned frame to obtain a second parameter value; and determine the noise reduction filtering strategy information for the current frame and the aligned frame according to the comparison result between the second parameter value and a preset threshold value, and output the noise reduction filtering strategy information to the fusion filtering module 230, wherein the noise reduction filtering strategy information includes time domain filtering or spatial domain filtering.

[0087] The fusion filtering module 230 may be configured to perform fusion denoising filtering processing on the read current frame and / or the input aligned frame according to the input denoising filtering strategy information to output a target frame.

[0088] It should be noted that, firstly, in order to reduce the residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers performing block rotation and alignment processing on the reference frame through the rotation alignment module 210.

[0089] Secondly, in order to overcome the "tailing" phenomenon of fast-moving objects that is easily caused by time domain filtering, the embodiment of the present application considers using the mean absolute differences (MAD) algorithm to detect motion intensity, so as to ensure that the filter uses different filtering intensities for objects with different motion intensities, thereby helping to avoid the "tailing" phenomenon caused by fast-moving objects and improve the image filtering effect. Among them, the filtering decision module 220 calculates the grayscale value of each pixel in the third image block of the current frame and the grayscale value of each pixel in the fourth image block of the aligned frame through the MAD algorithm to obtain the MAD value (i.e., the second parameter value).

[0090] Again, the embodiment of the present application tests a large number of image sequences to determine an empirical value for measuring motion intensity, i.e., a preset threshold, and then determines the motion intensity of the third image block of the current frame on the motion trajectory represented by the running vector through a comparison result between the second parameter value and the preset threshold.

[0091] If the comparison result between the second parameter value and the preset threshold value is that the second parameter value is greater than the preset threshold value, it means that the third image block has a strong motion intensity on the motion trajectory. At this time, the filtering intensity of the filter can be lowered (or the noise reduction filtering strategy information is determined to be spatial domain filtering), so that the filtered third image block retains the original information as much as possible to avoid the generation of the "tailing" phenomenon.

[0092] If the comparison result between the second parameter value and the preset threshold value is that the second parameter value is less than the preset threshold value, it means that the third image block has a weaker motion intensity on the motion trajectory, that is, a steady and slow motion. For example, the background picture in the video stream generally does not move. At this time, the filtering intensity of the filter can be increased (or the noise reduction filtering strategy information is determined to be time domain filtering) to effectively remove noise. At the same time, the low motion intensity of the third image block will not cause the "tailing" phenomenon, which is conducive to ensuring a better filtering effect.

[0093] Finally, in order to implement noise reduction filtering processing on the current frame, the fusion filtering module 230 of the embodiment of the present application performs fusion noise reduction filtering processing on the current frame and / or the aligned frame according to time domain filtering or spatial domain filtering.

[0094] Specifically, spatial filtering may include mean filtering, median filtering, Gaussian filtering or bilateral filtering, etc. Therefore, the current frame is subjected to noise reduction filtering according to spatial filtering to output the target frame, thereby filtering the image blocks with strong motion intensity on the motion trajectory through spatial filtering, and retaining the original information in the image blocks as much as possible to avoid the generation of the "tailing" phenomenon.

[0095] Specifically, the time domain filtering may include linear weighted filtering. Therefore, the current frame and the aligned frame are subjected to fusion noise reduction filtering according to the linear weighted filtering to output a target frame that satisfies the following formula:

[0096] F out =ω*F t +(1-ω)*F t-1 ;

[0097] Among them, F out Represents the pixel values ​​in the image block of the target frame, F t Represents the pixel values ​​in the image block of the current frame, F t-1represents the pixel values ​​in the image block of the aligned frame, and ω represents the preset weight. In addition, the filtering strength of the filter can be dynamically adjusted by the preset weight ω.

[0098] In summary, the image block with weak motion intensity on the motion trajectory is filtered by linear weighted filtering, thereby effectively removing the noise in the image block.

[0099] Specifically, the third image block can be used to represent at least one image block of a plurality of image blocks of equal size and non-overlapping in the current frame. It is understandable that the filtering decision module 220 of the embodiment of the present application can perform image block processing on the read current frame to obtain N*N image blocks of equal size and non-overlapping.

[0100] Specifically, the fourth image block may be used to represent at least one image block of a plurality of image blocks of equal size and non-overlapping in the aligned frame. It is understandable that the filtering decision module 220 of the embodiment of the present application may perform image block processing on the input aligned frame to obtain N*N image blocks of equal size and non-overlapping, where the value of N may be 2, 4, 8 or 16, etc.

[0101] Specifically, the position of the third image block in the current frame corresponds to the position of the fourth image block in the aligned frame. It can be understood that if the third image block is an image block in the first row block of the current frame, then the second image block is an image block in the first row block of the aligned frame, and the rest are similar.

[0102] In a possible example, the image processing device 10 further includes a storage module, wherein the storage module can be used to store at least one of the current frame, the reference frame, the first feature point data window, and the second feature point data window.

[0103] Specifically, the storage module may include a double data rate DDR memory.

[0104] The DDR memory may include a double data rate synchronous dynamic random access memory (DDR SDRAM).

[0105] It should be noted that the delay processing module 100 of the embodiment of the present application can directly obtain the first feature point data window and the current frame in the image block of the reference frame from the storage module, and then upload the current second feature point data window to the storage module for storage, so that the second feature point data window can be directly read from the storage module later without having to perform feature point detection for search and matching in parallel processing, thereby reducing the processing delay in the video noise reduction process. At the same time, by directly obtaining the first feature point data window and the current frame to achieve parallel processing of search matching and feature point detection, the processing delay in the video noise reduction process is reduced through parallel processing, thereby improving processing efficiency. In addition, by uploading the second feature point data window of the current frame, the remaining redundant data in the current frame is avoided from being uploaded and stored to reduce the bandwidth for data upload.

[0106] In summary, the image processing process of the embodiment of the present application is further exemplified below. Among them, the feature point data window of the reference frame and the current frame to be denoised are obtained, the reference frame is the previous frame of the current frame, and the current frame is smoothed and filtered; the feature point detection of the current frame after smooth filtering and the search and matching of the feature point data window of the reference frame in the image block of the current frame after smooth filtering are performed in parallel; the best motion vector, that is, the motion matching result, is obtained by searching and matching the feature point data window of the reference frame to the current frame, and the motion adjustment information is calculated by the motion matching result, and the motion adjustment information is used to rotationally align the reference frame; the current frame and the reference frame are obtained, and the reference frame is rotationally aligned by the motion adjustment information; the current frame and the reference frame after rotation alignment are subjected to noise reduction filtering to obtain the target frame, so as to directly obtain the feature point data window of the reference frame and the current frame to realize the parallel processing of search matching and feature point detection, and then reduce the processing delay in the video noise reduction process through parallel processing to improve the processing efficiency.

[0107] Consistent with the above embodiment, the image processing device of the embodiment of the present application is introduced below. Figure 7 . Figure 7 7 is a structural diagram of another image processing device provided in an embodiment of the present application. The image processing device 70 includes a delay processing module 710, a storage module 720 and a noise reduction processing module 730. The delay processing module 710 is connected to the storage module 720, the noise reduction processing module 730 is connected to the storage module 720, and the delay processing module 710 is connected to the noise reduction processing module 730.

[0108] The delay processing module 710 is used to read the first feature point data window in the first image block of the current frame and the reference frame from the storage module 720, where the reference frame is used to represent the image frame that is located before the current frame in time sequence; search and match the first feature point data window in the second image block of the current frame to output motion adjustment information, and perform feature point detection on the second image block in parallel to obtain a second feature point data window; and upload the second feature point data window to the storage module 720.

[0109] The noise reduction processing module 730 is used to read the current frame and the reference frame from the storage module 720; call the motion adjustment information output by the delay processing module 710 to perform block rotation and alignment processing on the reference frame to obtain an aligned frame; and perform noise reduction filtering processing on the current frame and / or the aligned frame to output a target frame.

[0110] It should be noted that, in order to realize the functions of time domain noise reduction and low processing delay, the image processing device 70 of the embodiment of the present application includes hardware and / or software modules for executing the corresponding functions. It should be easy for those skilled in the art to realize that, in combination with the functions of the modules described in the embodiments provided herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a module is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can call a specific program or algorithm through a processor (such as a CPU, etc.) to implement the functions of the modules described in the embodiment of the present application, but such implementation should not be considered to be beyond the scope of the present application. In addition, the embodiment of the present application can divide the image processing device 70 into functional modules. For example, each functional module can be divided according to function, or two or more functions can be integrated into one functional module. The above-mentioned integrated functional module can be implemented in the form of hardware or software. At the same time, the division of the functional modules in the embodiment of the present application is schematic, which is only a logical function division, and there may be other division methods in actual implementation, and no specific restrictions are made on this.

[0111] It can be seen that compared to the distributed processing of first performing feature point detection on the current frame and then transferring the feature point data window of the current frame to the reference frame for search and matching, in the embodiment of the present application, first, the delay processing module 710 directly reads the first feature point data window in the first image block of the reference frame and the current frame from the storage module 720, that is, directly obtains the first feature point data window (without performing feature point detection) and the current frame; secondly, the delay processing module performs the search and matching of the first feature point data window of the reference frame in the second image block of the current frame and the feature point detection of the second image block of the current frame in parallel; thirdly, the delay processing module 710 outputs the motion adjustment information matching result to the noise reduction processing module 730 and uploads the second feature point data window to the storage module 720 for storage; finally, the noise reduction processing model uses the motion adjustment information to rotate and align the reference frame to obtain an aligned frame, and performs noise reduction filtering on the current frame and / or the aligned frame to output a target frame, thereby directly obtaining the first feature point data window and the current frame to realize parallel processing of search and matching and feature point detection, and then reduces the processing delay in the video noise reduction process through parallel processing, thereby improving processing efficiency. In addition, by uploading the second feature point data window of the current frame to the storage module 720, it is avoided to upload and store the remaining redundant data in the current frame to reduce the bandwidth of data upload.

[0112] The following embodiments of the present application will be combined with Figure 8 Each module in the image processing device 70 will be described in detail.

[0113] In a possible example, the delay processing module 710 may specifically include a parallel processing module 711 and a data window uploading module 712.

[0114] The parallel processing module 711 can be used to search and match the first feature point data window of the parameter frame read from the storage module 720 within the second image block to output motion adjustment information, and to perform feature point detection on the second image block of the current frame read from the storage module 720 in parallel to obtain a second feature point data window.

[0115] The data window uploading module 712 may be configured to upload the second feature point data window to the storage module 720 .

[0116] It should be noted that the embodiment of the present application can use the parallel processing module 711 to parallelly execute the search and matching of the feature point data window of the reference frame in the current frame and the feature point detection of the current frame, thereby reducing the processing delay in the video denoising process through the parallel processing of the search and matching and the feature point detection.

[0117] In addition, the feature point data window of the current frame is uploaded to the storage module for storage through the data window upload module 712, while the remaining redundant data features of the current frame are not uploaded and stored. This is not only beneficial for avoiding uploading and storing redundant data in the current frame to reduce the bandwidth for data upload, but also beneficial for directly reading the feature point data window of the current frame from the storage module when searching and matching the next frame image of the current frame, without having to perform feature point detection, thereby reducing the processing delay in the video denoising process and improving processing efficiency.

[0118] In a possible example, the parallel processing module 711 specifically includes a feature point detection module 7111, a motion estimation module 7113 and a motion adjustment module 7115.

[0119] The feature point detection module 7111 may be used to perform feature point detection on the read second image block to obtain a second feature point data window.

[0120] The motion estimation module 7113 may be used to search and match the read first feature point data window in the second image block to obtain a motion vector, and output the motion vector to the motion adjustment module 7115 .

[0121] The motion adjustment module 7115 may be used to perform data fitting processing on the input motion vector to obtain a homography matrix, use the homography matrix as motion adjustment information, and output the motion adjustment information to the noise reduction processing module 730 .

[0122] It should be noted that, firstly, the feature point detection in the embodiment of the present application includes hash corner point detection, so the corner point detection in the image frame of the current frame can be achieved through the feature point detection module 7111.

[0123] Secondly, motion estimation is one of the important components of video noise reduction technology. By calculating the relative motion offset of the pixel point in two adjacent frames of images, the motion vector (MV) thereof is obtained. Among them, the motion estimation algorithm may include a block matching algorithm, a pixel method, a feature method, and a phase method. Therefore, the motion estimation module 7113 can realize the search and matching of the reference frame and the current frame to obtain the motion vector. Among them, the motion estimation module 7113 can determine the search range in the image block of the current frame through the search matching algorithm, and then find the most similar (or matching) data window within the search range. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous screening algorithm, a multi-layer screening algorithm, a partial distortion elimination algorithm, and the like.

[0124] Finally, the data fitting process of the motion vector of the embodiment of the present application may include the least square method, and the motion vector is subjected to data fitting process by the least square method to obtain a homography matrix. Among them, the homography transformation can be used to describe the position mapping relationship between the world coordinate system and the pixel coordinate system of the object, and the corresponding homography transformation is called a homography matrix. Therefore, the motion vector is calculated by the motion adjustment module 7115 to obtain the motion adjustment information for block rotation and alignment of the reference frame.

[0125] Specifically, the motion estimation module 7113 is specifically used to: calculate the absolute error sum between the grayscale value of each pixel in the read first feature point data window and the grayscale value of each pixel in the data window within the search range of the second image block to obtain a first parameter value; use the data window corresponding to the minimum value in the first parameter value as the target data window; and calculate the relative displacement from the first feature point data window to the target data window to obtain a motion vector, and output the motion vector.

[0126] It should be noted that the embodiment of the present application considers using the minimum sum of absolute difference (SAD) criterion in the first feature point data window of the reference frame to search and match in the second image block of the current frame, and finds the data window corresponding to the minimum value of the SAD value (i.e., the first parameter value) of the first feature point data window. At this time, the data window is the data window that best matches the first feature point data window, i.e., the target data window. Therefore, the first feature point data window and the target data window form a relative position in the time domain to obtain a motion vector, thereby constructing motion adjustment information for block rotation and alignment processing of the reference frame through the motion vector.

[0127] In a possible example, the delay processing module 710 further includes a smoothing filter module 713.

[0128] The smoothing filter module 713 may be used to perform smoothing filter processing on the read current frame and output the current frame after the smoothing filter processing.

[0129] It should be noted that the smoothing filter module 713 can be used to reduce noise and artifacts in the current frame. The smoothing filter module 713 can include a mean filter, a box filter, a median filter, a Gaussian filter, a bilateral filter, and the like.

[0130] In a possible example, the delay processing module 710 further includes an image segmentation module 714.

[0131] The image block module 714 may be used to perform image block processing on the current frame and output the current frame after the image block processing.

[0132] It should be noted that the embodiment of the present application can divide the image frames (including the current frame or the reference frame) of the video stream read from the storage module 720 into N*N image blocks of the same size and non-overlapping through the image segmentation module 714, thereby facilitating the subsequent direct feature point detection of the image blocks and improving the efficiency of feature point detection of the image frames.

[0133] In a possible example, the noise reduction processing module 730 specifically includes a rotation alignment module 731, a filtering decision module 732 and a fusion filtering module 733.

[0134] The rotation alignment module 731 may be used to call the motion adjustment information to perform block rotation and alignment processing on the read reference frame to obtain an aligned frame, and output the aligned frame.

[0135] The filtering decision module 732 is used to calculate the mean absolute error of the grayscale value of each pixel in the third image block of the read current frame and the grayscale value of each pixel in the fourth image block of the input aligned frame to obtain a second parameter value; and determine the noise reduction filtering strategy information for the current frame and the aligned frame according to the comparison result between the second parameter value and a preset threshold, and output the noise reduction filtering strategy information to the fusion filtering module 733, and the noise reduction filtering strategy information includes time domain filtering or spatial domain filtering.

[0136] The fusion filtering module 733 may be configured to perform fusion denoising filtering processing on the read current frame and / or the input aligned frame according to the input denoising filtering strategy information to output a target frame.

[0137] It should be noted that, firstly, in order to reduce the residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers performing block rotation and alignment processing on the reference frame through the rotation alignment module 731.

[0138] Secondly, in order to overcome the "tailing" phenomenon of fast-moving objects that is easily caused by time domain filtering, the embodiment of the present application considers using the mean absolute differences (MAD) algorithm to detect motion intensity, so as to ensure that the filter uses different filtering intensities for objects with different motion intensities, thereby helping to avoid the "tailing" phenomenon caused by fast-moving objects and improve the image filtering effect. Among them, the filtering decision module 732 calculates the grayscale value of each pixel in the third image block of the current frame and the grayscale value of each pixel in the fourth image block of the aligned frame through the MAD algorithm to obtain the MAD value (i.e., the second parameter value).

[0139] Again, the embodiment of the present application tests a large number of image sequences to determine an empirical value for measuring motion intensity, that is, a preset threshold, and then determines the motion intensity of the third image block of the current frame on the motion trajectory represented by the running vector through the comparison result between the second parameter value and the preset threshold. Wherein, if the comparison result between the second parameter value and the preset threshold is that the second parameter value is greater than the preset threshold, it means that the third image block has a strong motion intensity on the motion trajectory. At this time, the filtering intensity of the filter can be lowered (or the noise reduction filtering strategy information is determined to be spatial domain filtering), so that the filtered third image block retains the original information as much as possible to avoid the generation of the "tailing" phenomenon. If the comparison result between the second parameter value and the preset threshold is that the second parameter value is less than the preset threshold, it means that the third image block has a weaker motion intensity on the motion trajectory, that is, a steady and slow motion. For example, the background picture in the video stream generally does not move. At this time, the filtering intensity of the filter can be increased (or the noise reduction filtering strategy information is determined to be time domain filtering), so as to effectively remove noise. At the same time, the low motion intensity of the third image block will not cause the "tailing" phenomenon, which is conducive to ensuring a better filtering effect.

[0140] Finally, in order to implement noise reduction filtering processing on the current frame, the fusion filtering module 733 of the embodiment of the present application performs fusion noise reduction filtering processing on the current frame and / or the aligned frame according to time domain filtering or spatial domain filtering. Among them, spatial domain filtering can include mean filtering, median filtering, Gaussian filtering or bilateral filtering. Therefore, according to spatial domain filtering, the current frame is subjected to noise reduction filtering to output the target frame, so that the image block with strong motion intensity on the motion trajectory is filtered by spatial domain filtering, and the original information in the image block is retained as much as possible to avoid the generation of "tailing" phenomenon.

[0141] In addition, the time domain filtering may include linear weighted filtering. Therefore, the current frame and the aligned frame are subjected to fusion noise reduction filtering according to the linear weighted filtering to output the target frame to satisfy the following formula:

[0142] F out =ω*F t +(1-ω)*F t-1 ;

[0143] Among them, F out Represents the pixel values ​​in the image block of the target frame, F t Represents the pixel values ​​in the image block of the current frame, F t-1 represents the pixel values ​​in the image block of the aligned frame, and ω represents the preset weight. In addition, the filtering strength of the filter can be dynamically adjusted by the preset weight ω.

[0144] In summary, the image block with weak motion intensity on the motion trajectory is filtered by linear weighted filtering, thereby effectively removing the noise in the image block.

[0145] Specifically, the third image block can be used to represent at least one image block of a plurality of image blocks of equal size and non-overlapping in the current frame. It is understandable that the filtering decision module 732 of the embodiment of the present application can perform image block processing on the read current frame to obtain N*N image blocks of equal size and non-overlapping.

[0146] Specifically, the fourth image block may be used to represent at least one image block of a plurality of image blocks of equal size and non-overlapping in the aligned frame. It is understandable that the filtering decision module 732 of the embodiment of the present application may perform image block processing on the input aligned frame to obtain N*N image blocks of equal size and non-overlapping, where the value of N may be 2, 4, 8 or 16, etc.

[0147] Specifically, the position of the third image block in the current frame corresponds to the position of the fourth image block in the aligned frame. It can be understood that if the third image block is an image block in the first row block of the current frame, then the second image block is an image block in the first row block of the aligned frame, and the rest are similar.

[0148] In one possible example, the memory module 720 may include a double data rate DDR memory.

[0149] Specifically, the DDR memory may include a double data rate synchronous dynamic random access memory (DDR SDRAM).

[0150] It should be noted that the description of each embodiment in this application has different focuses. Therefore, the specific implementation functions of each module in the image processing device 70 can be found in Figure 1 The relevant description in the above-mentioned embodiment will not be repeated here.

[0151] In summary, the image processing process of the embodiment of the present application is exemplarily described below. Fig. 9 As shown. Fig. 9In the method, the feature point data window of the reference frame and the current frame to be denoised are directly read from the DDR, the reference frame is the previous frame of the current frame, and the current frame is smoothed and filtered; the feature point detection of the current frame after smoothing filtering and the search and matching of the feature point data window of the reference frame in the image block of the current frame after smoothing filtering are performed in parallel; the best motion vector, that is, the motion matching result, is obtained by searching and matching the feature point data window of the reference frame to the current frame, and the motion adjustment information is calculated by the motion matching result, and the motion adjustment information is used to rotationally align the reference frame; the feature point data window of the current frame obtained by the feature point detection of the current frame is uploaded to the DDR for storage; the current frame and the reference frame are read from the DDR, and the reference frame is rotationally aligned by the motion adjustment information; the current frame and the reference frame after rotation alignment are fused and denoised to obtain the target frame, so that the parallel processing of search matching and feature point detection is realized by directly obtaining the feature point data window of the reference frame and the current frame, and then the processing delay in the video denoising process is reduced through parallel processing, and the processing efficiency is improved. In addition, by directly uploading only the feature point data window of the current frame, it is avoided to upload and store the remaining redundant data in the current frame, so as to reduce the bandwidth of data upload.

[0152] It can be seen that in the embodiment of the present application, first, the first feature point data window in the first image block of the current frame and the reference frame is directly read from the storage module through the delay processing module; secondly, the feature point detection of the second image block of the current frame and the search and matching of the first feature point data window of the reference frame in the second image block of the current frame are performed in parallel; thirdly, the motion adjustment information matching result is output to the noise reduction processing module through the delay processing module, and the second feature point data window is uploaded to the storage module for storage; finally, the reference frame is rotationally aligned to obtain an aligned frame through the noise reduction processing model, and the current frame and / or the aligned frame is subjected to noise reduction filtering to output a target frame, thereby reducing the processing delay in the video noise reduction process and improving the processing efficiency through the parallel processing of search matching and feature point detection, and reducing the bandwidth for data upload by only uploading the feature point data window of the current frame and avoiding uploading and storing the remaining redundant data in the current frame.

[0153] Consistent with the above embodiment, the image processing method of the embodiment of the present application is introduced below. Fig.10 .

[0154] S1002: Acquire a first feature point data window in a first image block of a reference frame and a current frame.

[0155] The reference frame is used to represent an image frame that is located before the current frame in time sequence.

[0156] Specifically, the current frame can be used to represent a frame of an image to be denoised in the video stream.

[0157] Specifically, the reference frame may be an image frame before the current frame.

[0158] S1004: Search and match the first feature point data window in the second image block of the current frame to obtain motion adjustment information, and perform feature point detection on the second image block in parallel to obtain a second feature point data window.

[0159] Specifically, search matching can be understood as determining a search range in the image block of the current frame through a search matching algorithm, then finding the most similar (or matching) data window within the search range through the first feature point data window, and finally obtaining motion adjustment information through a motion vector (motion vector, MV) from the first feature point data window to the most similar data window. The search matching algorithm may include a full search matching algorithm, a three-step search algorithm, a diamond search algorithm, a four-step search algorithm, a continuous screening algorithm, a multi-layer screening algorithm, a partial distortion elimination algorithm, and the like.

[0160] Specifically, the motion adjustment information may be used to perform block rotation and alignment processing on the reference frame.

[0161] Specifically, feature point detection may include Harris corners detection, where the corners in the image frame may have the following features: intersections between contours; for the same scene, they remain stable even when the viewing angle changes; and pixels in the vicinity of the corners have large changes in both the gradient direction and the gradient amplitude.

[0162] It should be noted that if hash corner point detection is used to perform feature point detection on the image block of the current frame to obtain a feature point data window, then the feature point data window can be used to represent the corner point in the image block, or, the feature point data window can be used to represent the image area composed of the corner point in the image block and the pixel points around the corner point, etc.

[0163] Specifically, the first image block may be used to represent at least one image block among a plurality of image blocks of the reference frame that are equal in size and do not overlap with each other.

[0164] Specifically, the second image block may be used to represent at least one image block among a plurality of image blocks of the current frame that are equal in size and do not overlap with each other.

[0165] Specifically, the position of the first image block in the reference frame corresponds to the position of the second image block in the current frame.

[0166] S1006: Perform block rotation and alignment processing on the reference frame using the motion adjustment information to obtain an aligned frame.

[0167] It should be noted that, in order to reduce the residual value between each pixel in the image block of the current frame and each pixel in the image block of the reference frame, the embodiment of the present application considers performing block rotation and alignment processing on the reference frame through motion adjustment information.

[0168] S1008: Perform noise reduction filtering on the current frame and / or the aligned frame to obtain a target frame.

[0169] It can be seen that in the embodiment of the present application, first, the first feature point data window in the first image block of the reference frame and the current frame are obtained; secondly, the search and matching of the first feature point data window of the reference frame in the second image block of the current frame and the feature point detection of the second image block of the current frame are performed in parallel; finally, the reference frame is block rotated and aligned using the motion adjustment information to obtain an aligned frame, and the current frame and / or the aligned frame are subjected to noise reduction filtering to output a target frame, thereby directly obtaining the first feature point data window of the reference frame and the current frame to realize parallel processing of search and matching and feature point detection, thereby reducing the processing delay in the video noise reduction process and improving the processing efficiency through parallel processing.

[0170] In combination with the above description, the technical solution of the embodiment of the present application is specifically explained below.

[0171] Specifically, searching and matching the first feature point data window in the second image block to obtain motion adjustment information in S1004 may include: searching and matching the first feature point data window in the second image block to obtain a motion vector; performing data fitting processing on the motion vector to obtain a homography matrix, and obtaining motion adjustment information of the homography matrix.

[0172] Furthermore, searching and matching the first feature point data window within the second image block to obtain a motion vector may include: calculating the sum of absolute errors between the grayscale value of each pixel in the first feature point data window and the grayscale value of each pixel in the data window within the search range of the second image block to obtain a first parameter value; taking the data window corresponding to the minimum value in the first parameter value as the target data window; and calculating the relative displacement from the first feature point data window to the target data window to obtain a motion vector.

[0173] Specifically, performing noise reduction filtering on the current frame and / or aligned frame to obtain a target frame in S1008 may include: acquiring a third image block of the current frame and a fourth image block of the aligned frame; performing mean absolute error calculation on the grayscale value of each pixel in the third image block and the grayscale value of each pixel in the fourth image block to obtain a second parameter value; determining noise reduction filtering strategy information for the current frame and the aligned frame based on a comparison result between the second parameter value and a preset threshold, the noise reduction filtering strategy information including time domain filtering or spatial domain filtering; performing noise reduction filtering on the current frame and / or aligned frame according to the noise reduction filtering strategy information to obtain a target frame.

[0174] Specifically, after acquiring the first feature point data window in the first image block of the reference frame and the current frame in S1002, the method further includes: uploading the second feature point data window to the bus for storage.

[0175] Among them, the bus may include DDR memory.

[0176] It should be noted that the embodiment of the present application only considers uploading the feature point data window of the current frame to the bus for storage, while the remaining redundant data features of the current frame are not uploaded and stored. This is not only beneficial to avoid uploading and storing redundant data in the current frame to reduce the bandwidth for data upload, but also beneficial to directly read the feature point data window of the current frame from the storage module when searching and matching the next frame image of the current frame, without having to perform feature point detection, thereby reducing the processing delay in the video denoising process and improving processing efficiency.

[0177] Specifically, after acquiring the first feature point data window in the first image block of the reference frame and the current frame in S1002, the method further includes: performing smoothing filtering on the current frame, and outputting the current frame after the smoothing filtering.

[0178] Specifically, after obtaining the first feature point data window in the first image block of the reference frame and the current frame in S1002, the method further includes: performing block processing on the current frame to obtain at least one image block, wherein the at least one image block includes the second image block.

[0179] It should be noted that the description of each embodiment in this application has its own emphasis. Fig.10 The technical solutions not described in detail in the embodiments can be found in Figure 1 Or the relevant description in the embodiment described in 7 is not repeated here.

[0180] In combination with the above description, an embodiment of the present application further provides a processing chip, wherein the processing chip includes the above-mentioned image processing device.

[0181] It should be noted that the description of each embodiment in this application has different focuses. Therefore, the specific implementation functions of each module in the image processing device of the processing chip can refer to the relevant description in the above embodiments, which will not be repeated here.

[0182] In addition, the image processing device in the processing chip performs search matching and feature point detection in parallel and only uploads the feature point data window of the current frame, thereby reducing the processing delay in the video noise reduction process, improving processing efficiency, and avoiding uploading and storing the remaining redundant data in the current frame to reduce the bandwidth for data upload.

[0183] For example, see Fig.11 , Fig.11 1 is a schematic diagram of the structure of a processing chip provided in an embodiment of the present application. The processing chip 1100 includes an image processing device 1110 and a communication bus for connecting the image processing device 1110. Specifically, the processing chip 1100 may be a processor. The processor may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video codec, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a baseband processor and / or a neural network processing unit (NPU), etc. Different processors may be independent devices or may be integrated in the same processor.

[0184] Furthermore, the processing chip 1100 may also include a memory for storing instructions and data. In some embodiments, the memory in the processing chip 1100 may be a cache memory. The memory may store instructions or data that the processor has just used or circulated. If the processing chip 1100 needs to use the instruction or data again, it may be directly called from the memory, thereby avoiding repeated access, reducing the waiting time of the processing chip 1100, and improving system efficiency.

[0185] Furthermore, the processing chip 1100 may further include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0186] In combination with the above description, an embodiment of the present application further provides an electronic device, wherein the electronic device includes a memory, a communication interface and the above-mentioned image processing device.

[0187] It should be noted that the description of each embodiment in this application has different focuses. Therefore, the specific implementation functions of each module in the image processing device of the electronic device can refer to the relevant description in the above embodiments, which will not be repeated here.

[0188] In addition, the image processing device in the electronic device performs search matching and feature point detection in parallel and only uploads the feature point data window of the current frame, thereby reducing the processing delay in the video noise reduction process, improving processing efficiency, and avoiding uploading and storing the remaining redundant data in the current frame to reduce the bandwidth for data upload.

[0189] For example, see Fig.12 , Fig.12 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. The electronic device 1200 includes a memory 1201, a communication interface 1202, an image processing device 1203, and a communication bus for connecting the memory 1201, the communication interface 1202, and the image processing device 1203.

[0190] Specifically, the electronic device 1200 of the embodiment of the present application can be a handheld device, a vehicle-mounted device, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, or other devices connected to a wireless modem, or it can be various specific forms of user equipment (UE), terminal device, smart phone, smart screen, smart TV, smart watch, laptop computer, station (STA), access point (AP), mobile station (MS), personal digital assistant (PDA), personal computer (PC) or relay device, etc.

[0191] Specifically, the image processing device 1203 may be a processor. The processor may include a CPU, a GPU, an ISP, a DSP, an FPGA, an ASIC, a baseband processor and / or an NPU, etc. When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU.

[0192] Specifically, the memory 1201 may include a double data rate synchronous dynamic random access memory (DDR SDRAM), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read only memory (EPROM) and / or a portable read-only memory (CD-ROM), and the memory 1201 is used to store relevant instructions and data.

[0193] Specifically, the communication interface 1202 is used to receive and send data, signals or instructions, etc.

[0194] It should be noted that, for the above-mentioned embodiments, for the sake of simple description, they are all expressed as a series of action combinations. Those skilled in the art should be aware that the present application is not limited by the described order of actions, because some steps in the embodiments of the present application can be performed in other orders or simultaneously. In addition, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present application.

[0195] In the above embodiments, the present application has different focuses on the description of each embodiment. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0196] In several embodiments provided in the present application, those skilled in the art should be aware that the described device can be implemented in other ways. It is understandable that the device embodiments described above are merely schematic. For example, the division of the above modules is only a logical function division, and there may be other division methods in practice. In other words, multiple modules can be combined or integrated into the same module, and some features can be ignored or not executed. In addition, the connection between the modules shown or discussed can be indirect coupling connection, direct coupling connection or communication connection, and can be connected through some interfaces, or can be electrical or other forms.

[0197] If the above modules are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. It is understandable that the technical solution of the present application (the part of the technical solution that contributes to the prior art or all or part of the technical solution) can be embodied in the form of a computer-readable storage medium. The computer-readable storage medium can be stored in a memory, including several instructions to enable a computer device (a personal computer, an electronic device or a network device, etc.) to perform all or part of the steps of the embodiment of the present application. In addition, the above-mentioned computer-readable storage medium can be stored in various memories such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk or an optical disk.

[0198] The embodiments of the present application are specifically introduced above. Those skilled in the art should be aware that the embodiments of the present application are only used to help understand the core idea of ​​the technical solution of the present application, so the embodiments of the present application may be changed in the specific implementation and scope of application. So far, the contents recorded in this specification should not be understood as limiting the scope of protection of the present application. In addition, on the basis of the technical solution of the embodiments of the present application, any modification, equivalent replacement, improvement, etc. made shall be included in the scope of protection of the embodiments of the present application.

Claims

1. An image processing device, characterized in that: include: Delay processing module and noise reduction processing module; wherein, The delay processing module is used to search and match the first feature point data window in the first image block of the acquired reference frame in the second image block of the acquired current frame to output motion adjustment information to the noise reduction processing module, and perform feature point detection on the second image block in parallel to obtain the second feature point data window, so as to realize parallel processing of search and matching and feature point detection; the reference frame is used to represent an image frame that is located before the current frame in time sequence; the first feature point data window is directly acquired without performing feature point detection; The noise reduction processing module is used to perform block rotation and alignment processing on the reference frame using the motion adjustment information to obtain an aligned frame, and perform noise reduction filtering processing on the current frame and / or the aligned frame to output a target frame.

2. The device according to claim 1, characterized in that The delay processing module includes: a feature point detection module, a motion estimation module and a motion adjustment module; wherein, The feature point detection module is used to perform feature point detection on the second image block to obtain the second feature point data window; The motion estimation module is used to search and match the first feature point data window in the second image block to obtain a motion vector, and output the motion vector to the motion adjustment module; The motion adjustment module is used to perform data fitting processing on the input motion vector to obtain a homography matrix, use the homography matrix as the motion adjustment information, and output the motion adjustment information to the noise reduction processing module.

3. The device according to claim 2, characterized in that The motion estimation module is used for: Calculating the sum of absolute errors between the grayscale value of each pixel in the first feature point data window and the grayscale value of each pixel in the data window within the search range of the second image block to obtain a first parameter value; Taking the data window corresponding to the minimum value among the first parameter values ​​as the target data window; The relative displacement from the first feature point data window to the target data window is calculated to obtain the motion vector, and the motion vector is output.

4. The device according to claim 2, characterized in that The delay processing module also includes an image segmentation module; wherein, The image blocking module is used to perform image blocking processing on the current frame and output the current frame after the image blocking processing.

5. The device according to claim 1, characterized in that The noise reduction processing module includes: a rotation alignment module, a filtering decision module and a fusion filtering module; wherein, The rotation alignment module is used to call the motion adjustment information to perform block rotation and alignment processing on the read reference frame to obtain the aligned frame, and output the aligned frame; The filtering decision module is used to calculate the mean absolute error of the grayscale value of each pixel in the third image block of the current frame and the grayscale value of each pixel in the fourth image block of the aligned frame to obtain a second parameter value; and determine the noise reduction filtering strategy information for the current frame and the aligned frame according to the comparison result between the second parameter value and a preset threshold, and output the noise reduction filtering strategy information to the fusion filtering module, wherein the noise reduction filtering strategy information includes time domain filtering or spatial domain filtering; The fusion filtering module is used to perform noise reduction filtering on the read current frame and / or the input alignment frame according to the input noise reduction filtering strategy information to output the target frame.

6. The device according to any one of claims 1 to 5, characterized in that: Also includes: Storage module; The storage module is used to store at least one of the current frame, the reference frame, the first feature point data window, and the second feature point data window.

7. The device according to claim 6, characterized in that The memory module includes a double data rate DDR memory.

8. An image processing method, characterized in that: include: Acquire a first feature point data window in a first image block of a reference frame and a current frame, wherein the reference frame is used to represent an image frame that is located before the current frame in time sequence; the first feature point data window is directly acquired without performing feature point detection; Searching and matching the first feature point data window in the second image block of the current frame to obtain motion adjustment information, and performing feature point detection on the second image block in parallel to obtain a second feature point data window; so as to realize parallel processing of search and matching and feature point detection; Using the motion adjustment information, block rotation and alignment processing are performed on the reference frame to obtain an aligned frame; The current frame and / or the aligned frame are subjected to noise reduction filtering to obtain a target frame.

9. The method according to claim 8, characterized in that The step of searching and matching the first feature point data window in the second image block to obtain motion adjustment information includes: Searching and matching the first feature point data window in the second image block to obtain a motion vector; The motion vector is subjected to data fitting processing to obtain a homography matrix, and the homography matrix is ​​used as the motion adjustment information.

10. The method according to claim 9, characterized in that The step of searching and matching the first feature point data window in the second image block to obtain a motion vector comprises: Calculating the sum of absolute errors between the grayscale value of each pixel in the first feature point data window and the grayscale value of each pixel in the data window within the search range of the second image block to obtain a first parameter value; Taking the data window corresponding to the minimum value among the first parameter values ​​as the target data window; The relative displacement from the first feature point data window to the target data window is calculated to obtain the motion vector.

11. The method according to claim 8, characterized in that The performing noise reduction filtering on the current frame and / or the aligned frame to obtain a target frame includes: Acquire a third image block of the current frame and a fourth image block of the aligned frame; Performing mean absolute error calculation on the grayscale value of each pixel in the third image block and the grayscale value of each pixel in the fourth image block to obtain a second parameter value; Determine, according to a comparison result between the second parameter value and a preset threshold, noise reduction filtering strategy information for the current frame and the aligned frame, the noise reduction filtering strategy information including time domain filtering or space domain filtering; The current frame and / or the aligned frame are subjected to noise reduction filtering processing according to the noise reduction filtering strategy information to obtain the target frame.

12. The method according to any one of claims 8 to 11, characterized in that: After performing feature point detection on the second image block in parallel to obtain a second feature point data window, the method further includes: The second feature point data window is uploaded to the bus for storage.

13. The method according to any one of claims 8 to 12, characterized in that: After acquiring the first feature point data window in the first image block of the reference frame and the current frame, the method further includes: The current frame is subjected to block processing to obtain at least one image block, wherein the at least one image block includes the second image block.

14. A processing chip, characterized in that: The invention comprises an image processing device as described in any one of claims 1 to 7.

15. An electronic device, characterized in that: The invention comprises a memory, a communication interface and an image processing device as claimed in any one of claims 1 to 7.

Images