Multi-resolution multi-frame image denoising method, electronic device, storage medium

Abstract

The present application relates to a kind of multi-resolution multi-frame image noise reduction method, electronic equipment, storage medium, sequentially include the following steps: step 1, multi-resolution decomposition: to the input dynamic image sequence in each frame image is carried out Laplacian pyramid multi-resolution decomposition, obtain at least including low-resolution layer and high-resolution detail layer multi-layer resolution data;Step 2, multi-resolution motion estimation;Step 3, matching determination: for each macroblock of each layer, according to the minimum residual error of motion estimation, average weighting coefficient is calculated, form weighting coefficient table, and the greater the residual error is, the smaller the weighting coefficient is;Step 4, motion compensation and image reconstruction.The present application multi-resolution layered motion estimation: separate overall and local motion, still can work stably in 30nGy low dose X-ray image, and motion vector search range is expanded to 25x25 or more, realizes large range (25x25 or more) motion vector search under the premise of controllable amount of calculation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, in particular to a multi-resolution multi-frame image denoising method, an electronic device and a storage medium, and specifically relates to a multi-frame image time domain denoising method based on multi-resolution motion detection and compensation, an electronic device and a storage medium. BACKGROUND

[0002] In digital image processing, noise suppression is an important technology, especially in video monitoring, medical imaging, low-light photography and other scenarios, high-quality images are crucial for subsequent analysis and judgment.

[0003] The motion vector calculation used in image denoising processing can meet the real-time requirements of practical applications, and the search range of the macroblock is 7x7 in actual verification.

[0004] ‌Considering the superposition of target object motion and viewing angle motion in dynamic X-ray image sequences, and in order to reduce the radiation dose of patients under pulse fluoroscopy, the operator usually uses a lower acquisition frame rate (for example, 7.5 fps or lower) and as low a dose as possible per frame, resulting in a macroblock motion range > 7x7 between two frames, and the noise of each frame image is large, and the increase in noise will affect the accuracy of motion vector calculation.

[0005] In dynamic X-ray image sequences, the automatic dose control system will control the subsequent dose level according to the dose feedback of each frame to ensure that the incident dose of the detector is stable around the set target dose for different target objects, but this will result in different doses for each frame. In the "multi-frame image time domain denoising method based on motion detection and compensation", when fusing the current frame and the reference frame, if the X-ray doses of the current frame and the reference frame are different (i.e., the output image gray values are different), a mosaic effect will occur during fusion.

[0006] In summary, the defects of the prior art are as follows: Limited motion vector search range: To meet real-time requirements, the macroblock search range is usually 7x7, which cannot cover large-scale motion (motion range > 7x7) between adjacent frames in dynamic X-ray images due to low frame rate (e.g., 7.5 fps); Noise affects motion vector accuracy: Dynamic X-ray images have high noise due to low-dose acquisition, which interferes with motion vector calculation; Mosaic problem in fusion: Automatic dose control system causes different doses (gray value difference) between adjacent frames, and direct fusion will produce a mosaic effect.

[0007] In view of the above defects, the designer actively studies and innovates to create a multi-resolution multi-frame image denoising method, electronic equipment and storage medium, so that it has more industrial use value. SUMMARY

[0008] To solve the above technical problems, the purpose of the present application is to provide a multi-resolution multi-frame image denoising method, electronic equipment and storage medium.

[0009] To achieve the above purpose, the present application adopts the following technical solutions: One of the purposes of the present application is: A multi-resolution multi-frame image denoising method, comprising the following steps in sequence: Step 1, multi-resolution decomposition: performing Laplacian pyramid multi-resolution decomposition on each frame of image in the input dynamic image sequence to obtain multi-layer resolution data including at least a low-resolution layer and each resolution detail layer; Step 2, multi-resolution motion estimation: Step 21, overall motion estimation: in the low-resolution layer, the overall relative displacement of the adjacent two frames is calculated by searching for the minimum residual or phase matching method; Step 22, layer-by-layer motion estimation: from the low-resolution layer to the high-resolution detail layer, based on the motion estimation result of the low layer, the motion vector of each layer macroblock is calculated by using block matching or simplified optical flow method, forming the motion vector table of the high layer; Step 3, matching determination: for each macroblock of each layer, the average weighting coefficient is calculated according to the minimum residual of motion estimation, forming a weighting coefficient table, and the larger the residual is, the smaller the weighting coefficient is; Step 4, motion compensation and image reconstruction: Step 41, motion compensation: based on the motion vector table of each layer, the motion compensation is performed on the reference frame corresponding layer data, so that the motion area of adjacent frames is aligned; Step 42, weighted fusion: the compensated data of each layer is adaptively weighted and fused to obtain the denoising data of each layer; Step 43, reconstruction: from low to high, the low-resolution layer denoising data is up-sampled and superimposed with the high-resolution detail layer denoising data, and finally the denoising image of the original size is obtained.

[0010] As a further improvement of the present application, the data of each resolution detail layer in step 1 is the residual of the up-sampling of the current resolution image and the lower resolution image.

[0011] As a further improvement of the present application, in step 1, the sampling multiple of the Laplacian pyramid multi-resolution decomposition is 2 or 4 (or higher); the number of layer divisions is determined according to the image size and search range: 1Kx1K to 2Kx2K images use double resolution, and 3Kx3K or higher images use at least 3 layers of resolution.

[0012] As a further improvement of the present application, in step 22, the macroblock size is 8x8 pixels or 16x16 pixels; the low-resolution layer macroblock search range is mapped by the sampling multiple and superimposed with the upper layer resolution detail layer search range, so that the overall motion vector coverage reaches 25x25 pixels or higher.

[0013] As a further improvement of the present application, in step 4, the low-resolution layer can selectively skip the motion compensation and weighted fusion step and directly use the current frame data; if a certain layer uses the motion compensation and weighted fusion step, all higher resolution layers must perform the operation.

[0014] As a further improvement of the present application, in step 42, the calculation formula of the weighted fusion is: , wherein is the noise reduction data of the i-th layer, is the current frame i-th layer data, is the compensated reference frame i-th layer data, α is the weighted coefficient, i.e. , x , y is the pixel coordinate.

[0015] As a further improvement of the present application, in step 1, the dynamic image sequence is a dynamic X-ray image sequence, and the multi-resolution decomposition and motion estimation in steps 1 and 2 are performed on each resolution detail layer to reduce the interference of image noise and X-ray dose difference on motion vector calculation and weighted fusion.

[0016] The second object of the present application is: An electronic device comprising a memory and a processor, the memory storing a computer program, and the computer program being executed by the processor to make the processor perform the steps of the multi-resolution multi-frame image noise reduction method as described above.

[0017] The third object of the present application is: A computer readable storage medium storing a computer program, and the computer program being executed by the processor to implement the steps of the multi-resolution multi-frame image noise reduction method as described above.

[0018] By the above-mentioned scheme, the present application has at least the following advantages: Multi-resolution hierarchical motion estimation: separate global and local motion, still stable in low-dose X-ray images of 30nGy, motion vector search range expanded to 25x25 or more, large range (25x25 or more) motion vector search under the premise of controllable calculation amount.

[0019] High-resolution detail layer motion calculation: reduce the interference of noise on the accuracy of motion vector.

[0020] Hierarchical fusion mechanism: avoid global gray value changes caused by dose differences, solve the mosaic problem caused by low-dose noise interference and dose differences.

[0021] Real-time: algorithm complexity is reduced, and 3-frame temporal noise reduction real-time processing of 1K@60fps and 2K@30fps can be met.

[0022] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, and can be implemented according to the content of the specification, the following is the preferred embodiment of the present application and the detailed description of the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, it should be understood that the following drawings only show some embodiments of the present application, therefore should not be regarded as a limitation on the scope, for those skilled in the art, without creative labor, other related drawings can also be obtained according to these drawings.

[0024] Figure 1 is a flowchart of a multi-resolution multi-frame image noise reduction method of the present application; Figure 2 is a schematic diagram of Laplace pyramid decomposition of the present application; Figure 3 is a schematic diagram of Laplace pyramid reconstruction of the present application; Figure 4 is a flowchart of the second embodiment of the present application; Figure 5 is a flowchart of the third embodiment of the present application; Figure 6 is a flowchart of the fourth embodiment of the present application; Figure 7 is a flowchart of the seventh embodiment of the present application; Figure 8 is an effect diagram without applying the present application; Figure 9 is an effect diagram after applying the present application. DETAILED DESCRIPTION

[0025] The specific embodiments of the present application will be further described in details below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present application, but not to limit the scope of the present application.

[0026] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of the present application.

[0027] As Figures 1-6 shown: The first embodiment of the present application: The core scheme of the present embodiment is to separate the global and local motion by multi-resolution decomposition, and to perform motion estimation and compensation in layers. The specific steps are as follows: Multi-resolution decomposition: each frame of the input image is subjected to Laplacian pyramid multi-resolution decomposition. A simplified practical approach is to divide it into two resolutions, a high-resolution detail layer and a low-resolution; the following steps are described according to the case of double resolution (see Figure 5 ‌: a currently used double-resolution motion detection and compensation temporal noise reduction process): Multi-resolution motion estimation: Low-resolution global motion estimation: global matching between adjacent two frames in low resolution, based on search for minimum residual or phase matching method to obtain the relative displacement between two frames .

[0028] Low-resolution motion estimation: based on block matching (Block Matching) or simplified optical flow method, the motion vector between adjacent frames is calculated on the basis of the result of the resolution of the global motion estimation ( has been included ).

[0029] High-resolution motion estimation: based on block matching (Block Matching) or simplified optical flow method, the motion vector of the high-resolution detail layer between adjacent frames is calculated (refined) on the basis of the result of the low-resolution motion estimation .

[0030] Multi-resolution motion compensation and weighted fusion: according to the motion vector of different resolution layers, the reference frame is compensated, and the motion area in the multi-frame image is aligned. Then weighted fusion is carried out to achieve the purpose of noise reduction.

[0031] The high-resolution motion estimation can be directly based on Motion compensation is performed on the input original size image (reference frame), and weighted fusion is performed on the data of the current frame; The multi-resolution idea can also be used, and motion compensation and weighted fusion can be used for each layer, which can reduce the influence of image noise on motion estimation and reduce the influence of high-resolution detail layer motion estimation error on the result. Since the image noise of the dynamic X-ray image sequence is large, motion compensation is adopted. The specific steps are as follows: Low-resolution motion compensation and weighted fusion (optional): based on the motion vector between low-resolution adjacent frames Low-resolution motion compensation is performed on the reference frame, and the low-resolution data of the adjacent frames after compensation is adaptively weighted and fused. Since the purpose of motion compensation is to reduce noise, the low-resolution image is down-sampled, and the purpose of noise elimination has been achieved through sampling average, so this step can be skipped.

[0032] High-resolution detail layer motion compensation and weighted fusion: based on the motion vector between high-resolution adjacent frames High-resolution detail layer motion compensation is performed; the high-resolution data of the adjacent frames after compensation is adaptively weighted and fused.

[0033] Reconstruction of motion noise reduction data: the low-resolution data after up-sampling and the high-resolution detail layer data after compensation and weighted fusion are superimposed to obtain the reconstructed original size image noise reduction result.

[0034] Multi-resolution motion estimation: the motion is decomposed into global motion and local motion, and the global motion estimation and local motion preliminary estimation are performed on the basis of low resolution. The high-resolution motion estimation is refined based on the results of low-resolution motion estimation, and the motion vector results between adjacent frames in a larger range are obtained on the basis of controllable calculation amount. At the same time, the motion vector calculation is carried out on the high-resolution detail layer data, which can effectively reduce the influence of noise on the accuracy of motion vector calculation.

[0035] The second embodiment of the application: Motion compensation and time domain average noise reduction between two frames are calculated: The two frames participating in the calculation are defined as the current frame and the reference frame.

[0036] Current frame: image to be denoised; Reference frame: the frame which is adjacent to the current frame in time, and the information in this frame is used to participate in the noise reduction processing of the current frame.

[0037] ‌As Figure 1 And Figure 4 shown: Step 1: Multiresolution decomposition‌ ‌Step 1.1: Layer division: define the calculation to use several different resolution layers. How many resolution layers are used, and the setting of the amount of calculation and the trade-off of the complexity of the program, the search range and the size of the image, etc. determine various aspects. The larger the image, the more resolution can be used; the larger the search range, the more resolution can be used; the more resolution levels used, the smaller the amount of calculation under the same search range, but the higher the program complexity. But if the division level is too much, the detailed information of each layer is limited, which will affect the motion estimation (calculation of motion vector) later.

[0038] The current experience is that for 1Kx1K to 2Kx2K resolution images, double resolution can well balance between speed and effect, and for resolutions above 3Kx3K, as the search range of the macroblock needs to be further expanded, it can be considered to perform motion detection and compensation when the temporal noise reduction is performed at 3 resolution.

[0039] ‌Step 1.2: Image decomposition: use the Laplacian pyramid algorithm to decompose the current frame and the reference frame. The commonly used Laplacian pyramid algorithm performs image sampling with a scale of 2, that is, up-sampling is to double the image size, and down-sampling is to halve the image size. When the Laplacian pyramid algorithm is used here, the multiple of image sampling is set according to the results of step 1.1 layer division, and the commonly used is to perform image sampling with a scale of 4 (or more multiples). , That is , image size alignment is needed; ， That is , image size alignment is needed.

[0040] Wherein: n is the layer division defined in step 1.1; , ; and are symmetric sampling functions; see Figures 2-3 ‌Laplacian pyramid decomposition and reconstruction in

[0041] Step 2: Multi-resolution motion estimation The overall matching between two adjacent frames is performed at low resolution (e.g. 1 / 4 of the original resolution) The relative overall relative displacement between the reference frame and the current frame is obtained based on searching the minimum residual or phase matching method .

[0042] Step 3: Matching decision From the lowest resolution layer to the highest resolution layer, the following steps are repeated: Step 3.1: Motion estimation Block division: the current layer data is divided into multiple macroblocks (e.g. 8x8 pixels or 16x16 pixels).

[0043] Motion vector calculation: for each macroblock, search the matching block with minimum residual (SAD, Sum of Absolute Differences) in the vicinity of the macroblock basic displacement coordinates in the reference frame, forming a motion vector table .

[0044] Note: for the lowest resolution, the basic displacement is the overall relative displacement between the two frames (see Step 2), for other layers, the basic displacement is the motion estimation result of the adjacent low resolution layer (need to be scaled overall according to the sampling rate).

[0045] Step 3.2: Matching decision Matching degree decision: for each macroblock of the current layer, calculate the average weighting coefficient according to the value of the minimum residual, forming a weighting coefficient table . The larger the value of the residual (indicating the worse the matching degree), the smaller the weighting coefficient.

[0046] Step 4: Motion compensation and image reconstruction From the lowest resolution layer to the highest resolution layer, the following steps are repeated: Step 4.1: Motion compensation and temporal fusion Motion compensation and temporal fusion: the macroblocks of the reference frame current layer ( ) are aligned to the current frame current layer data according to the motion vector table, obtaining .

[0047] Another optional solution of Step 4.1: skip motion compensation and temporal fusion, directly use the data of the current frame current resolution layer.

[0048] ​​Note: This optional step is used when the data noise of the current resolution layer is low, and there is no need to reduce noise; in the best case, only the highest resolution layer needs to apply motion compensation and temporal fusion, and other low resolution layers do not need to apply motion compensation and temporal fusion. In addition, once a layer adopts motion compensation and temporal fusion, subsequent higher detail layers must all be motion compensated and temporally fused.

[0049] Step 4.2: Image reconstruction Fusion with low resolution data: According to the coefficients in the weighting coefficient table, calculate the image after fusion of the current frame and the reference frame.

[0050] , Where , is the image pixel coordinate.

[0051] The result of the highest resolution fusion is the final result of motion compensation and temporal average noise reduction between two frames.

[0052] , That is , image size alignment is needed.

[0053] Another optional solution for step 4: directly motion compensating and temporally fusing the original image The current resolution layer aligns the macroblock of the reference frame current layer ( ) to the current frame current layer data according to the motion vector table, obtaining ; ; Where , is the image pixel coordinate.

[0054] Calculate the motion compensation and temporal average noise reduction between multiple frames.

[0055] Explanation of the integrity of the steps of this embodiment: In the technical solution of this embodiment, each step is closely connected and interdependent, and together forms a complete and efficient multi-frame image temporal noise reduction system based on multi-resolution motion detection and compensation. The absence of any one step will result in the entire scheme failing to achieve the expected improvement effect.

[0056] 1、From the perspective of multi-resolution decomposition step, the Laplacian pyramid algorithm is used to decompose each frame of image, separating the image into low-resolution layer and high-resolution detail layer. The low-resolution layer can quickly capture the overall structure and general motion trend of the image, providing a basic and global reference framework for subsequent motion estimation. The high-resolution detail layer focuses on the fine texture and local change information of the image, which is crucial for accurately capturing small-scale motion and handling local motion in complex scenes. Without this step, the subsequent motion estimation cannot distinguish between global motion and local motion, making it difficult to expand the motion vector search range while controlling the computational load, and thus unable to accurately adapt to the complex motion situation caused by the superposition of target and viewing angle motion in dynamic X-ray images.

[0057] 2、The multi-resolution motion estimation step is based on multi-resolution decomposition. First, the overall relative displacement is calculated in the lowest resolution layer to obtain the overall motion trend of the image. This step provides important basic displacement information for subsequent accurate local motion estimation in the high-resolution layer. Then, in the low-resolution layer and the high-resolution detail layer, different basic displacements are used as the starting point, and specific search algorithms (such as block matching or simplified optical flow method) are used to calculate the motion vectors of macroblocks based on the characteristics of each layer. This hierarchical and step-by-step motion estimation method fully utilizes the advantages of different resolution layers, reducing the computational load and improving the accuracy of motion vector calculation. Without this step, it is impossible to accurately obtain the motion information of each part of the image, and the subsequent noise reduction operations based on motion compensation and fusion also lose a reliable basis, resulting in a significant reduction in noise reduction effect.

[0058] 3、The matching decision step is also indispensable. By calculating the minimum residual error of each macroblock, the average weighting coefficient is obtained, forming a weighting coefficient table. This weighting coefficient table plays a key guiding role in the subsequent motion compensation and weighted fusion in image reconstruction. It can adaptively adjust the weight of the current frame data and the compensated reference frame data in the fusion process according to the quality of macroblock motion matching. Without this step, it is impossible to accurately measure the quality of macroblock motion matching, and thus it is impossible to reasonably perform weighted fusion, which may lead to problems such as blurred, distorted or ineffective elimination of mosaic effect caused by dose difference in the fused image.

[0059] 4. The motion compensation and image reconstruction step is the final execution stage of the entire scheme. Building upon the preceding steps, it achieves image denoising. Based on the motion vector tables obtained from motion estimation, motion compensation is performed on the corresponding layer data of the reference frame to align the motion regions of adjacent frames. This step ensures that information on the same motion regions between different frames can accurately correspond during fusion, avoiding image ghosting or blurring caused by inconsistent motion. Then, using the weighted coefficient table obtained from matching determination, adaptive weighted fusion is performed on the compensated layer data to obtain the denoised data for each layer. Finally, the samples from the low-resolution layer denoised data are superimposed with the high-resolution detail layer denoised data to obtain the denoised image of the original size. This complex reconstruction process relies on the accurate information provided by the preceding multi-resolution decomposition, motion estimation, and matching determination steps. The absence of any preceding step will prevent the reconstruction process from executing correctly and will result in a low-quality denoised image.

[0060] In summary, all steps in this embodiment must be performed in a specific order to achieve the intended purpose. Multi-resolution decomposition is the first step; only by decomposing the image into different resolution layers can subsequent steps process the characteristics of each layer. If motion estimation is performed first, without multi-resolution decomposition, it is impossible to effectively distinguish between overall and local motion at different scales, severely impacting the accuracy and efficiency of motion estimation. Multi-resolution motion estimation must be performed after multi-resolution decomposition because it requires calculating motion vectors based on the data from the different resolution layers obtained from the decomposition. Matching determination relies on the results of motion estimation, determining weighting coefficients by calculating the residuals of motion matching. Finally, motion compensation and image reconstruction require the integration of motion vectors and weighting coefficients obtained from the preceding steps to complete image denoising. If this order is changed, for example, if motion compensation and image reconstruction are performed before motion estimation and matching determination are completed, the resulting image will inevitably fail to achieve effective denoising due to the lack of accurate motion information and weighting criteria, and may even exhibit serious errors and distortions.

[0061] Improvements in noise reduction performance: This embodiment can still ensure the accuracy of motion detection even under high noise conditions. In actual testing, it can still operate normally and stably with an incident dose of 30 nGy to the detector; the search and matching range of macroblocks in motion vector matrix calculation can be improved by more than 3 times.

[0062] Improvements in real-time performance: The algorithm complexity of this embodiment is significantly reduced, which can meet the real-time processing requirements of temporal denoising of 3 frames of DSA images at 1K@60fps and 2K@30fps.

[0063] The third embodiment of the present invention: like Figure 5Based on the first embodiment, the present embodiment is a dual-resolution motion detection and compensation temporal noise reduction process currently in use: The present embodiment searches the overall displacement of the reference frame relative to the current frame in the low-resolution 7x7 range, and then searches the 5x5 range of each macroblock of the current frame in the reference frame based on the overall displacement at low resolution, to obtain a macroblock motion vector matrix.

[0064] The macroblock motion vector matrix is mapped to high resolution, and based on the macroblock motion vector matrix, each macroblock of the current frame is searched in the 5x5 range of the reference frame at high resolution to obtain a macroblock motion vector matrix at high resolution. Considering that there is a 4-fold sampling difference between high-resolution data and low-resolution data, the 5x5 and 7x7 pixel ranges at low resolution are equivalent to cover the 20x20 and 28x28 pixel ranges at high resolution, and the refinement of the 5x5 pixel range at high resolution is superimposed, so that theoretically it can cover the overall translation of 28x28 pixels at high resolution superimposed with the local translation of 25x25 pixels.

[0065] The fourth embodiment of the present application: As Figure 6 , the present embodiment is a dual-resolution motion detection and compensation temporal noise reduction process currently in use: The process shown in the present embodiment cancels the overall translation search and local translation search at low resolution based on Figure 5 , so it degenerates to a single-resolution motion detection and compensation temporal noise reduction. Here, the search is performed at the detail layer of the current frame and the reference frame, which can reduce the interference of noise on one hand, and can avoid the interference of X-ray dose changes on calculation (the overall brightness of the image is different).

[0066] The fifth embodiment of the present application: Based on the fourth embodiment described above, the present embodiment is to cover the case that the overall translation between two frames is greater than 20 pixels, and at the same time improve the overall calculation speed.

[0067] Low-resolution only performs overall translation calculation (7x7 range), 1 / 8 sampling, which actually covers a search range of 56x56 at high resolution, supporting a translation of 28 pixels in each direction.

[0068] There is only one detail layer, and the motion vector of the detail layer is calculated as a search range of 9x9 for each block (macroblock). The calculation speed of the motion vector is improved by a residual registration algorithm of "dynamic resolution adjustment", which is less than that of a 5x5 full search. "Dynamic resolution adjustment": the resolution of the search area is adaptively adjusted according to the residual distribution, and the high residual area is switched to a line-by-line fine search, and the low residual area remains to be sampled by every other line. The specific method is to sample by every other line first, and then further search in a small range by line-by-line fine search after the best registration point is searched.

[0069] In fact, the above operation is optimized for 3-frame superposition, because the overall translation between N and N-1 frames is greater than 20 pixels, and the overall translation between N and N-2 frames is greater than 40 pixels. Therefore, an iterative scheme is adopted, the overall translation between N and N-1 frames is calculated first, and the overall translation between N and N-1 frames is input as a reference when calculating the overall translation between N and N-2 frames. Therefore, the algorithm only needs to consider the search range of the overall translation between N and N-1 frames, and when calculating the overall translation between N and N-2 frames, only the translation range between the adjacent two frames of N-1 and N-2 needs to be considered. Therefore, the overall translation search range of the algorithm is reduced.

[0070] The sixth embodiment of the present application: The method of the present application adopts a simplified method of dividing into two resolutions when performing multi-resolution decomposition. In fact, a multi-resolution scheme can be adopted to decompose into more resolutions, and the motion vector matrix is calculated gradually from low resolution to high resolution based on more resolutions. The motion vector matrix of a higher resolution layer is calculated based on the motion vector matrix of a lower layer, and the search range of the motion vector of each layer of resolution is greater than the sampling multiple between the current layer and the lower layer, so as to ensure the coverage of the search range of the multi-resolution. The sampling multiple between the two layers can be different in the X and Y axes of the image, for example, the sampling multiple in the X axis is 2, and the sampling multiple in the Y axis is 3 (the image matrix of the current layer is 200x300, and the image matrix of the next layer is 100x100). Therefore, the search range of the motion vector of the current layer is greater than (2, 3), and the search range of the motion vector of (3, 5) can be adopted.

[0071] The seventh embodiment of the present application: As Figure 7 , in order to improve the accuracy of motion detection and reduce the artifacts of motion compensation and adaptive fusion, the method of the present application adopts different detail layer data for motion detection and fusion when performing multi-resolution decomposition. That is, more detailed data (for example, containing information of two layers of details) is used for motion detection, and less detailed data (for example, containing information of only one layer of details) is used for fusion.

[0072] The embodiment calculates the motion vector of the current frame I0 with reference frame R 0 all are pyramidal decomposed, I 0 decomposed into DI 0 +DI 1 + I 2 , R 0 decomposed into DR 0 +DR 1 +R 2 ; meanwhile, by DI 0 and DI 1 is synthesized DI M 0 , i.e. DI M 0 =DI 0 +DI 1 (need up-sampling); DR M 0 by DR 0 and DR 1 is synthesized, i.e. DR M 0 =DR 0 +DR 1 (need up-sampling). Based on DI M 0 and DR M 0 the motion vector matrix is calculated (still adopting the multi-resolution idea, the overall translation is calculated based on the lower resolution to obtain V 1 , which is the calculation basis of the motion vector matrix of the lower resolution , and then is the calculation basis of ), and the calculation result is more accurate than that of the third embodiment . When adaptive image fusion is performed, still based on DI 0 and DR0 In this embodiment, the calculation of the global translation is also further improved in calculation performance by using the idea of multi-resolution, i.e. the preliminary calculation of the global translation (i.e. the image global registration) is continued on the basis of the lowest resolution to obtain V 2 and further calculation is performed on a higher resolution layer to obtain the result of the global translation at a higher resolution V 1 .

[0073] An eighth embodiment of the present application is as follows: An electronic device includes a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the multi-resolution multi-frame image denoising method as described above.

[0074] A ninth embodiment of the present application is as follows: A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the multi-resolution multi-frame image denoising method as described above.

[0075] As shown in Figure 8 , an effect diagram without applying the present application, in the diagram, the consecutive 3 frames of images are not denoised.

[0076] As shown in Figure 9 , an effect diagram after applying the present application, in the diagram, the consecutive 3 frames of images are denoised (the global translation is superimposed with the heartbeat and the local motion of the respiration).

[0077] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for the purpose of description and cannot be understood as indicating or implying relative importance or implying the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0078] In the description of the application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integrally connected, it can be mechanical connection, or electrical connection, it can be directly connected, or indirectly connected through intermediate medium, it can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the application can be understood according to the specific circumstances.

[0079] The above is only the preferred embodiment of the present application, and is not used to limit the present application. It should be noted that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should be considered as the protection scope of the present application.

Claims

1. A multi-resolution multi-frame image denoising method, characterized in that: Comprise the following steps in sequence: Step 1, multi-resolution decomposition: performing Laplacian pyramid multi-resolution decomposition on each frame of the input dynamic image sequence to obtain multi-layer resolution data including at least a low-resolution layer and each resolution detail layer; Step 2, multi-resolution motion estimation: Step 21, overall motion estimation: in the low-resolution layer, the overall relative displacement of the adjacent two frames is calculated by searching for the minimum residual or phase matching method; Step 22, layer-by-layer motion estimation: from the low-resolution layer to the high-resolution detail layer, based on the motion estimation result of the lower layer, the motion vector of each layer macroblock is calculated by using block matching or simplified optical flow method to form the motion vector table of the higher layer; Step 3, matching determination: for each macroblock of each layer, the average weighting coefficient is calculated according to the minimum residual of motion estimation to form a weighting coefficient table, and the larger the residual is, the smaller the weighting coefficient is; Step 4, motion compensation and image reconstruction: Step 41, motion compensation: based on the motion vector table of each layer, the motion compensation is performed on the corresponding layer data of the reference frame to align the motion area of the adjacent frames; Step 42, weighted fusion: the compensated data of each layer is adaptively weighted and fused to obtain the noise reduction data of each layer; Step 43, reconstruction: from low to high, the low-resolution layer noise reduction data is up-sampled and superimposed with the noise reduction data of the high-resolution detail layer, and finally the noise reduction image of the original size is obtained.

2. The method of claim 1, wherein, In the step 1, the resolution detail layer data is the residual of the current resolution image and the up-sampled lower resolution image.

3. The method of claim 1, wherein the method is a multi-resolution, multi-frame image denoising method, characterized in that, In the step 1, the sampling multiple of the Laplacian pyramid multi-resolution decomposition is 2 or 4; the number of layer divisions is determined according to the image size and the search range: 1Kx1K to 2Kx2K images use double resolution, and 3Kx3K or more images use at least 3 layers of resolution.

4. The method of claim 1, wherein the method is a multi-resolution, multi-frame image denoising method, characterized in that, In the step 22, the macroblock size is 8x8 pixels or 16x16 pixels; the search range of the low-resolution layer macroblock is mapped by the sampling multiple and superimposed with the search range of the upper resolution detail layer, so that the overall motion vector coverage reaches 25x25 pixels or more.

5. The method of claim 1, wherein the method is a multi-resolution, multi-frame image denoising method, characterized in that, In the step 4, the low-resolution layer can selectively skip the motion compensation and weighted fusion steps, and directly use the current frame data; if a certain layer uses the motion compensation and weighted fusion steps, all higher resolution layers must perform the operation.

6. The method of claim 1, wherein, The calculation formula of the weighted fusion in the step 42 is: wherein is the i-th layer noise reduction data, is the i-th layer data of the current frame, is the i-th layer data of the compensated reference frame, α is the weighting coefficient, that is, , x , y is the pixel coordinate.

7. The method of claim 1, wherein the method is a multi-resolution, multi-frame image denoising method, characterized in that, In the step 1, the dynamic image sequence is a dynamic X-ray image sequence, and the multi-resolution decomposition and motion estimation in the steps 1 and 2 are performed in each resolution detail layer to reduce the interference of image noise and X-ray dose difference on motion vector calculation and weighted fusion.

8. An electronic device comprising a memory and a processor, said memory having stored therein a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the multi-resolution multi-frame image noise reduction method according to any one of claims 1-7.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the multi-resolution multi-frame image noise reduction method according to any one of claims 1-7.

Images