Image processing method and device, computer device and storage medium

Abstract

The application relates to an image processing method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed image and a neighboring image corresponding to the to-be-processed image; constructing image pyramids of the to-be-processed image and the neighboring image respectively, the image pyramid corresponding to the to-be-processed image being a first image pyramid, and the image pyramid corresponding to the neighboring image being a second image pyramid; calculating image layer displacements between each layer of the first image pyramid and the second image pyramid, and a measurement index value corresponding to the image layer displacements; aligning the neighboring image and the to-be-processed image according to the image layer displacements and the measurement index value corresponding to the image layer displacements; and fusing the neighboring image and the to-be-processed image to acquire a processed image. The method can effectively reduce image noise and improve image quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the rise of short video software, people are increasingly fond of using video to capture and record life, and the quality requirements for video shooting are increasingly high. In particular, the visibility is relatively low at night, and the lighting conditions are complex, which can easily lead to low video image quality. Therefore, video shooting technology for night scenes has emerged to improve the quality of night video images.

[0003] Currently, in the video shooting technology for night scenes, image noise reduction is performed by using the video brightening and noise filtering method to improve image quality. However, using this method for noise reduction can cause jitter in the time domain, for example, in the case of vehicle light trailing and ground jitter, resulting in low image quality. SUMMARY

[0004] Therefore, it is necessary to provide an image processing method, device, computer equipment and storage medium capable of improving image quality to solve the above technical problems.

[0005] An image processing method, the method comprising:

[0006] obtaining a to-be-processed image and a neighboring image corresponding to the to-be-processed image;

[0007] respectively constructing image pyramids of the to-be-processed image and the neighboring image, the image pyramid corresponding to the to-be-processed image being a first image pyramid, and the image pyramid corresponding to the neighboring image being a second image pyramid;

[0008] calculating image layer displacements between each layer of the first image pyramid and the second image pyramid, and a measurement index value corresponding to the image layer displacements;

[0009] aligning the neighboring image and the to-be-processed image according to each of the image layer displacements and the measurement index value corresponding to the image layer displacements;

[0010] fusing the neighboring image and the to-be-processed image to obtain a processed image.

[0011] In one embodiment, after obtaining the to-be-processed image and the neighboring image corresponding to the to-be-processed image, before respectively constructing the image pyramids of the to-be-processed image and the neighboring image, the method further comprises:

[0012] performing quality optimization processing on the to-be-processed image and the adjacent image to obtain a to-be-processed image and an adjacent image after quality optimization processing.

[0013] In one of the embodiments, after the to-be-processed image and the adjacent image corresponding to the to-be-processed image are acquired, before the image pyramid of the to-be-processed image and the image pyramid of the adjacent image are respectively constructed, the method further includes:

[0014] performing primary filtering processing on the to-be-processed image and the adjacent image to obtain a to-be-processed image and an adjacent image after primary filtering processing;

[0015] determining a motion region in the to-be-processed image and the adjacent image after primary filtering processing according to pixel values of the to-be-processed image and the adjacent image after primary filtering processing;

[0016] calculating a boundary value of the to-be-processed image and the adjacent image after primary filtering processing;

[0017] determining a non-boundary motion region in the to-be-processed image and the adjacent image after primary filtering processing based on the boundary value;

[0018] performing secondary filtering processing on the non-boundary motion region to obtain a to-be-processed image and an adjacent image after noise processing.

[0019] In one of the embodiments, the calculating of the image layer displacement between each layer corresponding to the first image pyramid and the second image pyramid includes:

[0020] calculating an initial image block displacement of a first image layer, the initial image block displacement being a displacement between a first image block in the first image layer and a second image block in a corresponding second image layer, the first image layer being an image layer in the first image pyramid, the second image layer being an image layer in the second image pyramid, the first image block being an image block of a preset size in the first image layer, and the second image block being an image block of the preset size in the second image layer;

[0021] calculating an initial measurement index value corresponding to the initial image block displacement;

[0022] determining the image layer displacement between each layer corresponding to the first image pyramid and the second image pyramid according to the initial image block displacement and the initial measurement index value.

[0023] In one of the embodiments, the calculating of the initial image block displacement of the first image layer includes:

[0024] interpolating the image layer displacement of a previous layer image layer of the first image layer to obtain an interpolation result, the previous layer image layer being an image layer adjacent to the first image layer and having a lower resolution than the first image layer in the first image pyramid;

[0025] determining a first image block displacement of the first image layer according to the interpolation result, the initial image block displacement including the first image block displacement.

[0026] In one of the embodiments, the calculating the initial image block displacement of the first image layer includes:

[0027] statistically calculating the image layer displacement of a previous layer image layer of the first image layer to obtain a statistical calculation result, the previous layer image layer being an image layer adjacent to the first image layer and having a lower resolution than the first image layer in the first image pyramid;

[0028] determining a second image block displacement of the first image layer according to the statistical calculation result, the initial image block displacement including the second image block displacement.

[0029] In one of the embodiments, the calculating the initial image block displacement of the first image layer includes:

[0030] determining a determined image block displacement of the first image layer;

[0031] determining a third image block displacement of the first image layer according to the determined image block displacement, the initial image block displacement including the third image block displacement.

[0032] In one of the embodiments, the calculating the initial measurement index value corresponding to the initial image block displacement includes:

[0033] determining a second image block of the second image layer identical to the initial image block displacement of the first image layer;

[0034] calculating the initial measurement index value corresponding to the initial image block displacement according to the pixel value of the first image block in the first image layer and the pixel value of the second image block.

[0035] In one of the embodiments, the aligning the adjacent image with the image to be processed according to the image layer displacement and the measurement index value corresponding to the image layer displacement includes:

[0036] determining the image block displacement of the first image block in the first image layer from the initial image block displacement based on the initial measurement index value;

[0037] determine image layer displacements between each layer of the first image pyramid and the second image pyramid based on the image block displacement of the first image block in the first image layer, and a measurement index value corresponding to the image layer displacement;

[0038] align the neighboring image and the image to be processed according to each of the image layer displacements and the measurement index value corresponding to the image layer displacement.

[0039] In one of the embodiments, after the determination of the image layer displacements between each layer of the first image pyramid and the second image pyramid based on the image block displacement of the first image block in the first image layer, the method further comprises:

[0040] determine preset neighborhood image blocks adjacent to the first image block in the first image layer;

[0041] update the image block displacement of the first image block in the first image layer according to the image block displacement of each of the preset neighborhood image blocks and the measurement index value corresponding thereto, and determine an updated image layer displacement.

[0042] In one of the embodiments, the fusing of the neighboring image and the image to be processed to obtain a processed image comprises:

[0043] determine an initial weight of the neighboring image according to the measurement index value corresponding to the updated image layer displacement;

[0044] fuse a preset weight of the image to be processed and the initial weight of the neighboring image to obtain a fusion weight of the neighboring image and the image to be processed;

[0045] fuse the neighboring image and the image to be processed based on the fusion weight to obtain a processed image.

[0046] In one of the embodiments, after the obtaining of the processed image, the method further comprises:

[0047] correct the processed image according to a preset pixel value fluctuation range.

[0048] An image processing device, the device comprising:

[0049] an image acquisition module configured to acquire an image to be processed and a neighboring image corresponding to the image to be processed;

[0050] an image pyramid construction module configured to construct image pyramids of the image to be processed and the neighboring image respectively, the image pyramid corresponding to the image to be processed being a first image pyramid, and the image pyramid corresponding to the neighboring image being a second image pyramid;

[0051] The calculation module is used to calculate the image layer displacement between each corresponding layer of the first image pyramid and the second image pyramid, as well as the measurement index value corresponding to the image layer displacement;

[0052] An image alignment module is used to align the adjacent images with the image to be processed based on the displacement of each image layer and the measurement index value corresponding to the image layer displacement.

[0053] An image fusion module is used to fuse the adjacent images with the image to be processed to obtain the processed image.

[0054] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the image processing method described above.

[0055] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image processing method described above.

[0056] The aforementioned image processing method, apparatus, computer equipment, and storage medium acquire an image to be processed and its corresponding neighboring images; construct image pyramids for the image to be processed and its neighboring images, respectively, with the image pyramid for the image to be processed being the first image pyramid and the image pyramids for its neighboring images being the second image pyramid; calculate the image layer displacements between each corresponding layer of the first and second image pyramids, as well as the corresponding metric values ​​for these image layer displacements; align the neighboring images with the image to be processed based on each image layer displacement and its corresponding metric value; and fuse the neighboring images with the image to be processed to obtain the processed image. By constructing image pyramids for the image to be processed and its neighboring images, and then calculating the image layer displacements between each corresponding layer of the image pyramids, the accuracy of displacement calculation can be improved; aligning the neighboring images with the image to be processed based on each image layer displacement can improve the alignment degree between the neighboring images and the image to be processed; and fusing the aligned neighboring images with the image to be processed to obtain the processed image can effectively reduce image noise, improve image quality, and thus enhance overall image quality. Attached Figure Description

[0057] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0058] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0059] Figure 3 This is a schematic diagram of an image processing method in a specific embodiment;

[0060] Figure 4 Fig. 1 is a schematic diagram of an image to be processed in an embodiment;

[0061] Figure 5 Fig. 2 is a schematic diagram of a quality optimization process in an embodiment;

[0062] Figure 6 Fig. 3 is a schematic diagram of calculating image layer displacement in an embodiment;

[0063] Figure 7 Fig. 4 is a structural block diagram of an image processing device in an embodiment;

[0064] Figure 8 Fig. 5 is an internal structure diagram of a computer device in an embodiment;

[0065] Figure 9 Fig. 6 is an internal structure diagram of a computer device in another embodiment. DETAILED DESCRIPTION

[0066] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0067] In one embodiment, the image processing method provided by the present application can be applied in an application environment as shown in Fig. 1. The application environment involves a terminal 102 and a server 104. The terminal 102 can communicate with the server 104 through a network or other communication means such as a protocol. Specifically, the server 104 obtains a to-be-processed image and a neighboring image corresponding to the to-be-processed image through the terminal 102, constructs an image pyramid of the to-be-processed image and an image pyramid of the neighboring image, respectively, the image pyramid corresponding to the to-be-processed image is a first image pyramid, and the image pyramid corresponding to the neighboring image is a second image pyramid; calculates image layer displacement between each layer of the first image pyramid and the second image pyramid, and a measurement index value corresponding to the image layer displacement; aligns the neighboring image and the to-be-processed image according to the image layer displacement and the measurement index value corresponding to the image layer displacement; and fuses the neighboring image and the to-be-processed image to obtain a processed image. Figure 1 In one embodiment, the image processing method provided by the present application can be applied in an application environment as shown in Fig. 1. The application environment involves a terminal 102 and a server 104. The terminal 102 can communicate with the server 104 through a network or other communication means such as a protocol. Specifically, the server 104 obtains a to-be-processed image and a neighboring image corresponding to the to-be-processed image through the terminal 102, constructs an image pyramid of the to-be-processed image and an image pyramid of the neighboring image, respectively, the image pyramid corresponding to the to-be-processed image is a first image pyramid, and the image pyramid corresponding to the neighboring image is a second image pyramid; calculates image layer displacement between each layer of the first image pyramid and the second image pyramid, and a measurement index value corresponding to the image layer displacement; aligns the neighboring image and the to-be-processed image according to the image layer displacement and the measurement index value corresponding to the image layer displacement; and fuses the neighboring image and the to-be-processed image to obtain a processed image.

[0068]

[0069] ​In one of the embodiments, the image processing method provided by the present application can only involve the server 104. Specifically, the server 104 can directly acquire the image to be processed and the adjacent image corresponding to the image to be processed, perform image processing on the image to be processed and the adjacent image in the server 104, and finally acquire the processed image.

[0070] The terminal 102 can be, but is not limited to, various cameras, personal computers, notebook computers, smart phones, tablet computers, portable wearable devices, and the like. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0071] In one of the embodiments, as shown in Figure 2 , an image processing method is provided. The image processing method is applied to the terminal 102 and / or the server 104 in Figure 1 for example, and includes the following steps.

[0072] In step S202, the image to be processed and the adjacent image corresponding to the image to be processed are acquired.

[0073] In one of the embodiments, the image refers to an image obtained by a shooting device. The shooting device can be a camera or a camera head, etc. The image to be processed is referred to as the image to be processed, and the image adjacent to the image to be processed is referred to as the adjacent image. The adjacent image can be one image or multiple images.

[0074] In one of the embodiments, when the shooting device is a camera, the camera can directly obtain the image. The image to be processed obtained by shooting is referred to as the image to be processed. The previous one or multiple images of the image to be processed obtained by shooting are referred to as the adjacent image corresponding to the image to be processed, or the next one or multiple images of the image to be processed obtained by shooting are referred to as the adjacent image corresponding to the image to be processed.

[0075] In one of the embodiments, when the shooting device is a camera, the camera can obtain a video. The video frame image to be processed is obtained by extracting the video. The previous one or multiple frames of the image to be processed obtained by extraction are referred to as the adjacent image corresponding to the image to be processed, or the next one or multiple frames of the image to be processed obtained by extraction are referred to as the adjacent image corresponding to the image to be processed. The video can be a historical video or a real-time video, and the image can be a historical video frame image or a real-time video frame image.

[0076] Step S204: Construct image pyramids for the image to be processed and adjacent images respectively. The image pyramid corresponding to the image to be processed is the first image pyramid, and the image pyramid corresponding to the adjacent images is the second image pyramid.

[0077] In one embodiment, the image to be processed and its neighboring images can be downsampled or upsampled respectively to obtain image pyramids, such as Gaussian pyramids or Laplacian pyramids. The image pyramid corresponding to the image to be processed is the first image pyramid, and the image pyramids corresponding to the neighboring images are the second image pyramids. Downsampling refers to the process of downsampling the image. After downsampling or upsampling, the image pyramid corresponding to the image can be obtained. An image pyramid is a structure that describes an image at multiple resolutions; it is a collection of images arranged in a pyramid shape with progressively decreasing resolutions, all originating from the same image. The higher the level of the image pyramid, the smaller the image and the lower the resolution.

[0078] In one embodiment, there are multiple downsampling methods, and at least one can be selected arbitrarily. Specifically, downsampling can be performed using the OpenCV software library, or the image can be downsampled directly, or an improved weighted downsampling method can be used. The OpenCV software library is a computer vision and machine learning library that can run on operating systems such as Linux, Windows, Android, and Mac OS, and can implement general algorithms in computer vision and image processing.

[0079] In one embodiment, the downsampling method is the same for the image to be processed and the adjacent images. Specifically, the image to be processed is downsampled to obtain a first image pyramid, which includes first image layers. The adjacent images are downsampled to obtain a second image pyramid, which includes second image layers. Since the downsampling method is the same, and the image to be processed and the adjacent images are images obtained by the same imaging device, after downsampling the image to be processed and the adjacent images in the same way, each first image layer corresponds to each second image layer.

[0080] Step S206: Calculate the image layer displacement between each corresponding layer of the first image pyramid and the second image pyramid, and the corresponding measurement index value of the image layer displacement.

[0081] In one of the embodiments, since there is a shift between the to-be-processed image and the adjacent image, the displacement between the to-be-processed image and the adjacent image needs to be calculated, and the to-be-processed image and the adjacent image are aligned according to the displacement. Specifically, after down-sampling, the shift between the to-be-processed image and the adjacent image can be decomposed into image layer displacements between each layer of the first image pyramid and the second image pyramid, and the displacement between the to-be-processed image and the adjacent image is determined by the image layer displacements.

[0082] In one of the embodiments, when the adjacent image of the to-be-processed image is one image, the displacement between the adjacent image and the to-be-processed image only needs to be calculated once when aligning the adjacent image and the to-be-processed image. When the adjacent image of the to-be-processed image is multiple images, the displacement between the adjacent image and the to-be-processed image needs to be calculated multiple times when aligning the adjacent image and the to-be-processed image. For example, when the adjacent image of the to-be-processed image is two images, which are the previous image and the next image of the to-be-processed image, the displacement between the previous image and the to-be-processed image needs to be calculated to align the previous image and the to-be-processed image, and the displacement between the to-be-processed image and the next image needs to be calculated to align the next image and the to-be-processed image, that is, the displacement needs to be calculated twice.

[0083] In one of the embodiments, when calculating the image layer displacement between each layer of the first image pyramid and the second image pyramid, the measurement index value corresponding to each layer of the image layer displacement also needs to be calculated. The measurement index value can measure the good or bad degree of the image layer displacement, and the image layer displacement corresponds to the measurement index value. The smaller the value of the measurement index, the better the corresponding image layer displacement. When different calculation methods are used to obtain different image layer displacements corresponding to each layer, the optimal image layer displacement corresponding to each layer can be determined according to the measurement index value.

[0084] In step S208, the adjacent image and the to-be-processed image are aligned according to the image layer displacements and the measurement index values corresponding to the image layer displacements.

[0085] In one of the embodiments, after obtaining the image layer displacements and the measurement index values corresponding to the image layer displacements, the adjacent image and the to-be-processed image can be aligned according to the image layer displacements and the measurement index values corresponding to the image layer displacements. Specifically, the adjacent image and the to-be-processed image can be aligned layer by layer to improve the alignment degree of the images, so that the aligned adjacent image and the to-be-processed image are basically consistent.

[0086] In step S210, the adjacent image and the to-be-processed image are fused to obtain a processed image.

[0087] In one of the embodiments, the image fusion refers to fusing several processed images into one image, which can improve the accuracy of the image, and improve the resolution of the original image, etc. After aligning the adjacent image with the to-be-processed image, the adjacent image is fused with the to-be-processed image. Specifically, the fusion can be performed based on the weights of the to-be-processed image and the adjacent image. After the fusion, a fused image, referred to as a processed image, is obtained.

[0088] In the image processing method, the to-be-processed image and the adjacent image corresponding to the to-be-processed image are obtained; image pyramids of the to-be-processed image and the adjacent image are constructed respectively, the image pyramid corresponding to the to-be-processed image is a first image pyramid, and the image pyramid corresponding to the adjacent image is a second image pyramid; image layer displacements between each layer of the first image pyramid and the second image pyramid are calculated, and a measurement index value corresponding to the image layer displacement is calculated; the adjacent image is aligned with the to-be-processed image according to the image layer displacement and the measurement index value corresponding to the image layer displacement; and the adjacent image is fused with the to-be-processed image to obtain a processed image. By constructing the image pyramids of the to-be-processed image and the adjacent image, calculating the image layer displacements between each layer of the image pyramids, and aligning the adjacent image with the to-be-processed image according to the image layer displacements, the accuracy of displacement calculation can be improved, the alignment degree of the adjacent image and the to-be-processed image can be improved, the adjacent image and the to-be-processed image after alignment are fused to obtain a processed image, the image noise can be effectively reduced, the image quality can be improved, and thus the image quality can be improved.

[0089] In one of the embodiments, after obtaining the to-be-processed image and the adjacent image corresponding to the to-be-processed image in step S202, before constructing the image pyramids of the to-be-processed image and the adjacent image in step S204, the method further includes:

[0090] In step S302, the quality of the to-be-processed image and the adjacent image is optimized to obtain a to-be-processed image and an adjacent image after quality optimization processing.

[0091] In one of the embodiments, image contrast processing, image sharpening, etc. can be used to respectively perform quality optimization processing on the to-be-processed image and the adjacent image. Specifically, image contrast processing can be used. The image contrast refers to the size of the image gray scale contrast. When the image is an image captured at night, the overall image is dark due to the low visibility at night. Therefore, the image contrast can be reduced to improve the dark area details of the image. In addition, due to the complex lighting conditions at night, there are usually street lamps, neon lights, etc. Therefore, the image also inevitably has a region with a brightness higher than the surrounding brightness, which is referred to as a highlight region. After reducing the image contrast and improving the dark area details of the image, the highlight region is lost, and therefore the image contrast needs to be improved to restore the highlight region of the image.

[0092] In one of the embodiments, the to-be-processed image and the adjacent image are processed in the same way. The following embodiments take the processing process of the to-be-processed image as an example. The to-be-processed image can be processed by contrast processing through curve mapping. Specifically, a look-up table can be used to improve the image quality, improve the dark area details, reduce the image contrast, restore the highlight region of the image, and improve the image contrast. Specifically, the look-up table operation can be performed by using an image processing software, such as exposure adjustment, tone mapping, etc. in the image processing software PS. In addition, the highlight region of the image can be restored by using the dark channel defogging method to improve the image contrast.

[0093] In one of the embodiments, the to-be-processed image after contrast processing and the original to-be-processed image can be weighted to obtain a to-be-processed image with more highlight region details. Specifically, the to-be-processed image and the to-be-processed image after contrast processing can be weighted to obtain a weight map corresponding to the to-be-processed image. Since the brightness of the highlight region to the normal region of the image is gradually changed, the pixel value of the original to-be-processed image can be multiplied by a preset multiple and blurred to obtain the weight map corresponding to the to-be-processed image. The weight map can be represented as W, and the preset multiple can be 1.5 times, 2 times, or 3 times, etc. The blurring processing can be mean blurring processing, and the radius can be 21. The weight map corresponding to the to-be-processed image can also be obtained by using an image processing software. Specifically, the highlight extraction, etc. in the image processing software PS can be used.

[0094] In one of the embodiments, a preset formula can be used to restore the highlight region details of the weight map corresponding to the to-be-processed image to obtain the to-be-processed image after quality optimization processing. Specifically, the preset formula is as follows:

[0095]

[0096] In the formula, W(y, x) represents the value of the pixel point (y, x) in the weight map W corresponding to the image to be processed, r, g, and b represent the pixel values of the red, green, and blue channels of the original image to be processed, and r new , g new , and b new represent the pixel values of the red, green, and blue channels of the image to be processed after contrast processing.

[0097] In one embodiment, after the image to be processed and the adjacent image corresponding to the image to be processed are obtained in step S202, before the image pyramids of the image to be processed and the adjacent image are constructed in step S204, the method further includes:

[0098] In step S402, the image to be processed and the adjacent image are subjected to one-time filtering processing to obtain the image to be processed and the adjacent image after one-time filtering processing.

[0099] Pulse noise is a common noise in an image, which appears as a white point or a black point randomly appearing in the image. It can be a black pixel point existing in a highlight area, or a white pixel point existing in a dark area, or both. The cause of the pulse noise can be strong interference on the image signal, an analog-to-digital converter error, or a bit transmission error. Generally, filtering processing is used to remove the pulse noise, such as median filtering processing or other filtering processing for removing isolated points. When an original data is converted into an image by a shooting device, the pulse noise is expanded into several pixels, and multiple filtering processing is required. However, multiple filtering processing can blur the image, and thus targeted filtering processing is required, such as filtering processing for a flat area of the image or filtering processing for a motion area of the image, to effectively reduce the noise of the image.

[0100] In one embodiment, the same filtering processing is performed on the image to be processed and the adjacent image. Global filtering processing is performed on the entire image, and the global filtering processing is referred to as one-time filtering processing. Specifically, one-time filtering processing is performed on the image to be processed and the adjacent image to obtain the image to be processed and the adjacent image after one-time filtering processing. The median filtering processing or improved median filtering processing can be used.

[0101] In step S404, the motion area in the image to be processed and the adjacent image after one-time filtering processing is determined according to the pixel values of the image to be processed and the adjacent image after one-time filtering processing.

[0102] In one of the embodiments, since the noise of the non-motion region of the image can be removed through the alignment and fusion of the subsequent images, the motion region of the image needs to be determined for the filtering of the motion region of the image. Wherein, the motion region of the once-filtered image and the adjacent image is determined according to the pixel values of the once-filtered image and the adjacent image.

[0103] Specifically, when the difference between the pixel values of the current region of the image to be processed and the corresponding region of the adjacent image is greater than the preset motion threshold, the current region is determined as the motion region, and the motion region threshold is set as the first preset value. When the difference between the pixel values of the current region of the image to be processed and the corresponding region of the adjacent image is less than the preset motion threshold, the current region is determined as the non-motion region, and the non-motion region threshold is set as the second preset value. Wherein, the value range of the preset motion threshold is (0, 255), which can be specifically set as 35. The motion region threshold, i.e. the first preset value, can be set as 255, and the non-motion region threshold, i.e. the second preset value, can be set as 0.

[0104] Step S406, the boundary value of the once-filtered image to be processed and the adjacent image is calculated.

[0105] In one of the embodiments, since the filtering of the boundary of the image will exceed the image region, the filtering should not be at the boundary of the image, and the boundary value of the once-filtered image to be processed and the adjacent image needs to be calculated. Wherein, the boundary value of the once-filtered image to be processed and the adjacent image can be calculated by Sobel operator or edge detection algorithm.

[0106] Specifically, when the calculated boundary value of the current region of the image to be processed is greater than the preset boundary threshold, the current region is determined as the boundary region, and the boundary region threshold is set as the first preset value. When the calculated boundary value of the current region of the image to be processed is less than the preset boundary threshold, the current region is determined as the non-boundary region, and the non-boundary region threshold is set as the second preset value. Wherein, the value range of the preset boundary threshold is (0, 255), which can be specifically set as 35. The boundary region threshold, i.e. the first preset value, can be set as 255, and the non-boundary region threshold, i.e. the second preset value, can be set as 0.

[0107] Step S408, the non-boundary motion region of the once-filtered image to be processed and the adjacent image is determined based on the boundary value.

[0108] Specifically, the non-boundary motion region of the once-filtered image to be processed and the adjacent image is determined according to the calculated non-boundary region threshold and the motion region threshold.

[0109] In step S410, the non-boundary motion region is filtered again to obtain the noise-processed to-be-processed image and the adjacent image.

[0110] In step S410, the non-boundary motion region is filtered again to obtain the noise-processed to-be-processed image and the adjacent image.

[0111] In one embodiment, step S206 calculates the image layer displacement between each layer of the first image pyramid and the second image pyramid, including:

[0112] In step S502, the initial image block displacement of the first image layer is calculated, the initial image block displacement being the displacement between a first image block in the first image layer and a second image block in the corresponding second image layer, the first image layer being an image layer in the first image pyramid, the second image layer being an image layer in the second image pyramid, the first image block being a preset-size image block in the first image layer, and the second image block being a preset-size image block in the second image layer.

[0113] In one embodiment, the image layer displacement can be determined by calculating the image block displacement of each image block in each image layer of the image pyramid, so as to improve the accuracy of the calculation. After the to-be-processed image is down-sampled, each first image layer is obtained. Similarly, after the adjacent image is down-sampled in the same manner, each second image layer is obtained. A preset-size image block in the first image layer is referred to as a first image block, and each first image block of the preset size is obtained. Similarly, a preset-size image block in the second image layer is referred to as a second image block, and each second image block of the preset size is obtained. Since the first image layer corresponds to the second image layer, the obtained first image block of the preset size corresponds to the second image block of the preset size. The displacement between the calculated first image block and the corresponding second image block is referred to as the initial image block displacement. When calculating the alignment displacement, the calculation is performed according to each image block in each image layer. Before calculating the initial image block displacement of the first image layer, the displacement of the minimum-resolution image layer needs to be initialized to 0.

[0114] In step S504, the initial measurement index value corresponding to the initial image block displacement is calculated.

[0115] In one of the embodiments, a plurality of initial image block displacements of the first image layer can be calculated in a plurality of calculation manners, and then an optimal image block displacement can be determined according to the corresponding measurement index value. Specifically, an initial measurement index value corresponding to the initial image block displacement is calculated. The initial image block displacement corresponds to the initial measurement index value. The initial measurement index value can measure the goodness of the calculated initial image block displacement, and the optimal image block displacement can be determined from the plurality of initial image block displacements.

[0116] In step S506, the image layer displacement between each layer of the first image pyramid and the second image pyramid is determined according to the initial image block displacement and the initial measurement index value.

[0117] In one of the embodiments, the smaller the value of the initial measurement index is, the more optimal the corresponding image block displacement is. Specifically, the initial image block displacement corresponding to the initial measurement index value with the smallest value can be determined as the optimal image block displacement of the first image layer according to the initial image block displacement and the initial measurement index value. Then, the image layer displacement between each layer of the first image pyramid and the second image pyramid is determined through the image block displacement of the first image layer.

[0118] In one of the embodiments, step S502 of calculating the initial image block displacement of the first image layer comprises:

[0119] In step S602, the image layer displacement of the previous layer of the first image layer is calculated by interpolation to obtain an interpolation calculation result. The previous layer of the first image layer is an image layer adjacent to the first image layer and having a lower resolution than the first image layer in the first image pyramid.

[0120] In one of the embodiments, the first image pyramid is obtained after the down-sampling of the image to be processed. In the first image pyramid, the previous layer of the first image layer is an image layer adjacent to the first image layer and having a higher level and a lower resolution than the first image layer. The image layer displacement of the previous layer of the first image layer is essentially obtained based on the image layer displacement of the image layer with the minimum resolution.

[0121] In one of the embodiments, the image layer displacement of the previous layer of the first image layer is calculated by interpolation to obtain an interpolation calculation result. The interpolation calculation can be performed by Lagrange interpolation, Newton interpolation, piecewise interpolation, bilinear interpolation, etc.

[0122] In step S604, the first image block displacement of the first image layer is determined according to the interpolation calculation result, and the initial image block displacement includes the first image block displacement.

[0123] In one of the embodiments, since the resolutions of the first image layers in the first image pyramid are proportional, the interpolation calculation result of the image layer displacement of the previous image layer needs to be enlarged in resolution correspondingly before it can be used as the image block displacement of the first image block of the first image layer. The image block displacement determined by the steps of the embodiment is called the first image block displacement. In the embodiment, the resolution enlargement multiple is the resolution ratio of the first image layer to the previous image layer. For example, if the resolution of the first image layer is twice the resolution of the previous image layer, i.e. the corresponding resolution enlargement multiple is 2, then the interpolation calculation result needs to be multiplied by 2 before it can be used as the image block displacement of the first image block of the first image layer.

[0124] In one of the embodiments, the step S502 of calculating the initial image block displacement of the first image layer comprises:

[0125] The step S702 is to statistically calculate the image layer displacement of the previous image layer of the first image layer to obtain a statistical calculation result, the previous image layer being the first image layer in the first image pyramid that is adjacent to the first image layer and has a lower resolution than the first image layer.

[0126] In one of the embodiments, at least one of the mode, mean or median of the image block displacements in the image layer displacement of the previous image layer can be taken as the statistical calculation result. In the embodiment, the previous image layer is the image layer in the first image layers that is adjacent to the first image layer and has a lower resolution than the first image layer.

[0127] The step S704 is to determine the second image block displacement of the first image layer according to the statistical calculation result, the initial image block displacement comprising the second image block displacement.

[0128] In one of the embodiments, the statistical calculation result of the image layer displacement of the previous image layer needs to be enlarged in resolution correspondingly before it can be used as the image block displacement of the first image block of the first image layer. The image block displacement determined by the steps of the embodiment is called the second image block displacement.

[0129] In one of the embodiments, the step S502 of calculating the initial image block displacement of the first image layer comprises:

[0130] The step S802 is to determine the image block with the determined image block displacement in the first image layer.

[0131] In one of the embodiments, the initial image block displacement of the first image layer can also be determined based on the image block with the determined image block displacement in the first image layer. In the embodiment, the image block with the determined image block displacement also needs to be the image block adjacent to the first image block.

[0132] In one of the embodiments, when the image block displacement of each first image block of the first image layer is calculated, each first image block can be calculated in a preset manner. The preset manner can be calculated in a snake shape, specifically, the image blocks of the first row of the first image layer are calculated from left to right, the image blocks of the second row are calculated from right to left, and the calculation is repeated in turn. The image block displacement of the image block whose image block displacement has been determined can be determined according to steps S602-S604, steps S702-S704, or according to the steps of the present embodiment. Specifically, the image blocks adjacent to the first image block in the first image layer and whose image block displacement has been determined are a preset number of image blocks. The preset number can be multiple, for example, 2, 3, or 4, etc.

[0133] In step S804, the third image block displacement of the first image layer is determined according to the image block whose image block displacement has been determined, and the initial image block displacement includes the third image block displacement.

[0134] In one of the embodiments, the image block displacement of each first image block of the first image layer can be determined according to the image block whose image block displacement has been determined, and the image block displacement determined by the steps of the present embodiment is referred to as the third image block displacement.

[0135] In one of the embodiments, there is only one first image block in the first image layer, or in the first image layer, the image block whose image block displacement needs to be determined is the first calculated image block in the first image layer, that is, the first image block does not exist any image block adjacent to it and whose image block displacement has been determined. In this case, the image block displacement of the first image block can only be determined according to steps S602-S604 or steps S702-S704.

[0136] In one of the embodiments, step S504 calculates the initial measurement index value corresponding to the initial image block displacement, including:

[0137] In step S902, the second image block in the second image layer which is the same as the initial image block displacement of the first image layer is determined.

[0138] In one of the embodiments, the second image layer refers to the second image layer corresponding to the first image layer. Specifically, the second image layer corresponding to the first image layer can be determined, and in the second image layer, the second image block whose displacement is the same as the initial image block displacement of the first image block of the first image layer is determined.

[0139] In step S904, the initial measurement index value corresponding to the initial image block displacement is calculated according to the pixel value of the first image block and the pixel value of the second image block in the first image layer.

[0140] In one of the embodiments, the sum of absolute differences of pixel values between two image blocks can be used as the initial measurement index, or the correlation between two image blocks can be used as the initial measurement index. Specifically, when the sum of absolute differences of pixel values between two image blocks is used as the initial measurement index, the initial measurement index value corresponding to the initial image block displacement can be calculated according to the pixel values of the first image block in the first image layer and the pixel values of the second image block with the same initial image block displacement.

[0141] In one of the embodiments, the step S208 aligns the adjacent image and the image to be processed according to the image layer displacements and the measurement index values corresponding to the image layer displacements, including:

[0142] The step S1002 determines the image block displacement of the first image block in the first image layer from the initial image block displacements based on the initial measurement index values.

[0143] In one of the embodiments, the image block displacement of the first image block in the first image layer can be determined from the first image block displacement, the second image block displacement and the third image block displacement corresponding to the initial measurement index value with the minimum value.

[0144] The step S1004 determines the image layer displacements between each layer of the first image pyramid and the second image pyramid and the measurement index values corresponding to the image layer displacements based on the image block displacement of the first image block in the first image layer.

[0145] In one of the embodiments, the image layer displacements between each layer of the first image pyramid and the second image pyramid are determined through the image block displacement of the first image block in the first image layer. After the image layer displacements between each layer are determined, the measurement index values corresponding to the image layer displacements are determined.

[0146] The step S1006 aligns the adjacent image and the image to be processed according to the image layer displacements and the measurement index values corresponding to the image layer displacements.

[0147] In one of the embodiments, after the image layer displacements and the measurement index values corresponding to the image layer displacements are determined, the adjacent image and the image to be processed can be aligned according to the determined image layer displacements.

[0148] In one of the embodiments, after the step S1004 determines the image layer displacements between each layer of the first image pyramid and the second image pyramid based on the image block displacement of the first image block in the first image layer, the image layer displacements can be updated according to the image block displacements around the first image block, including:

[0149] Step S1102, determining a preset neighborhood image block adjacent to the first image block in the first image layer.

[0150] In one of the embodiments, the image block displacement of each first image block in the first image layer can be determined only according to one of the steps S602-S604, steps S702-S704 or steps S802-S804, or can be determined according to all the three ways. The preset neighborhood image block adjacent to the first image block in the first image layer is determined. Specifically, the preset neighborhood image block is the image block around the first image block and having a determined image block displacement. The number of the preset neighborhood image blocks can be multiple, for example, 4 around the first image block, 8 in a circle around the first image block or 24 in two circles outwardly from the first image block, and the determined image block displacement thereof can be determined according to one of the steps S602-S604, steps S702-S704 or steps S802-S804.

[0151] Step S1104, updating the image block displacement of the first image block in the first image layer according to the image block displacement of each preset neighborhood image block and the corresponding measurement index value, and determining the updated image layer displacement.

[0152] In one of the embodiments, the image block displacement of the first image block in the first image layer can be updated according to the determined image block displacement of each preset neighborhood image block and the corresponding measurement index value. Specifically, when there is a smaller and better measurement index value in the preset neighborhood image block of the first image block, the image block displacement of the first image block is updated to the image block displacement corresponding to the better measurement index value, otherwise, it is not updated.

[0153] In one of the embodiments, step S210 fuses the adjacent image and the image to be processed to obtain a processed image, including:

[0154] Step S1202, determining the initial weight of the adjacent image according to the measurement index value corresponding to the updated image layer displacement.

[0155] In one of the embodiments, after aligning the adjacent image and the image to be processed, the adjacent image and the image to be processed need to be fused into a denoising processed image. The image fusion can be performed based on the weight of the image. Since the initial image block displacement corresponds to the initial measurement index value, after determining the image block displacement of the first image block in the first image layer, there is only one corresponding initial measurement index value, which is the measurement index value.

[0156] In one of the embodiments, according to the relationship between the preset measurement index value and the threshold value, and the measurement index value of the first image block of the first image layer, the initial weight of the second image block of the adjacent image corresponding to the image to be processed is determined, that is, the initial weight of the adjacent image is determined. Specifically, when the threshold value of the second image block is greater than the measurement index value, the initial weight of the second image block is set to 0, at this time, the image to be processed is not well aligned with the adjacent image; when the threshold value of the second image block is less than the measurement index value, the initial weight of the second image block is set to 1-measurement index value / threshold value.

[0157] In one of the embodiments, in order to improve the quality of image fusion and obtain a higher initial weight, the determination of the initial weight can also be that when the threshold value of the second image block is greater than or equal to a preset larger threshold value, the initial weight of the second image block is set to 0, when the threshold value of the second image block is less than a preset smaller threshold value, the initial weight of the second image block is set to 1, and when the threshold value of the second image block is between the preset smaller threshold value and the preset larger threshold value, the initial weight of the second image block is linearly reduced in turn.

[0158] In step S1204, the preset weight of the image to be processed and the initial weight of the adjacent image are fused to obtain the fusion weight of the adjacent image and the image to be processed.

[0159] In one of the embodiments, the preset weight of the image to be processed and the initial weight of the corresponding adjacent image are weighted and summed to obtain the fusion weight of the adjacent image and the image to be processed. The preset weight of the image to be processed can be set to 1.

[0160] In step S1206, based on the fusion weight, the adjacent image and the image to be processed are fused to obtain a processed image.

[0161] In one of the embodiments, after the adjacent image and the image to be processed are fused based on the fusion weight to obtain a processed image, in order to improve the correlation between the processed image and the original image, the pixel value of the processed image should be within the value range of the pixel value of the original image.

[0162] In one of the embodiments, after the processed image is obtained, it further includes correcting the processed image according to a preset pixel value fluctuation range. The pixel value fluctuation range of the processed image is set in advance. Specifically, the pixel value fluctuation range can be set to ±10. For example, the pixel value in the original image is 50, the adjacent image and the image to be processed are fused according to the fusion weight to obtain a processed image, if the pixel value of the image is 62, it needs to be corrected to 60, if the pixel value of the image is 56, it does not need to be corrected.

[0163] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and one specific embodiment. It should be understood that the specific embodiment described herein is only used to explain the present application and does not limit the present application.

[0164] In one specific embodiment, a schematic diagram of the image processing method is shown in Figure 3 , wherein, after pre-processing the image, a pre-processed image is obtained, and then a frame of processed image is obtained by aligning and fusing multiple frames of images. A schematic diagram of the image to be processed is shown in Figure 4 , wherein, there are obvious impulse noises in the image. The specific steps of the image processing method are as follows:

[0165] I. Image pre-processing

[0166] Determine the image to be processed and the adjacent image corresponding to the image to be processed. Before performing multi-frame noise reduction processing on the image, image pre-processing can also be included, wherein the image pre-processing mainly includes at least one of quality optimization processing and noise processing. Taking the pre-processing process of the image to be processed as an example, the specific steps include:

[0167] Quality optimization pre-processing:

[0168] Use the curve mapping method shown in Figure 5 (a) to reduce the contrast of the image and improve the details of the dark area of the image, and use the curve mapping method shown in Figure 5 (b) to improve the contrast of the image to restore the highlight area of the image.

[0169] Multiply the pixel value of the original image to be processed by 2 times, and perform mean blur processing with a radius of 21 to obtain the weight map W corresponding to the image to be processed. Restore the highlight details of the weight map using the formula to obtain the image to be processed after quality optimization processing, and the formula is as follows:

[0170]

[0171] In the formula, W(y, x) represents the value of pixel point (y, x) in the weight map W corresponding to the image to be processed, r, g, b represent the pixel values of the red, green and blue channels of the original image to be processed, and r new , g new , b new represent the pixel values of the red, green and blue channels of the image to be processed after contrast processing.

[0172] Noise pre-processing:

[0173] Perform global filtering processing on the image to be processed to obtain the image to be processed after global filtering processing.

[0174] Determine the motion region in the image to be processed after global filtering processing, set the motion region threshold to 255, and set the non-motion region threshold to 0.

[0175] Calculate the boundary value of the image to be processed after global filtering processing by Sobel operator or edge detection algorithm, set the boundary region threshold to 255, and set the non-boundary region threshold to 0.

[0176] Determine the non-boundary motion region in the image to be processed after global filtering processing and perform filtering processing to obtain the image to be processed after noise processing.

[0177] II. Multi-frame image noise reduction processing

[0178] The multi-frame image noise reduction processing mainly includes: an alignment process of the image to be processed and the adjacent image, and a fusion process of the aligned images, and the specific steps include:

[0179] Image alignment process:

[0180] Downsample the image to be processed and the adjacent image respectively to obtain a first image pyramid corresponding to the image to be processed, which includes each first image layer; obtain a second image pyramid corresponding to the adjacent image, which includes each second image layer, and segment the first image layer and the second image layer according to a preset size to obtain each first image block and second image block respectively.

[0181] Determine the image layer displacement between each layer of the first image pyramid and the second image pyramid according to the initial image block displacement between the first image block and the corresponding second image block, and the initial measurement index value corresponding to the initial image block displacement.

[0182] The way of calculating the initial image block displacement includes:

[0183] Method ①: Interpolate the image layer displacement of the previous layer image layer of the first image layer to obtain an interpolation result, the previous layer image layer is an image layer in the first image pyramid adjacent to the first image layer and having a lower resolution than the first image layer, and the first image block displacement of the first image block of the first image layer is determined according to the interpolation result, and the initial image block displacement includes the first image block displacement.

[0184] Method ②: Statistically calculate the image layer displacement of the previous layer image layer of the first image layer to obtain a statistical calculation result, and determine the second image block displacement of the first image block of the first image layer according to the statistical calculation result, and the initial image block displacement includes the second image block displacement.

[0185] Method ③: Determine the image block adjacent to the first image block in the first image layer and having a determined image block displacement, specifically as Figure 6(a) shows, the arrow indicates the calculation order, ● represents the first image block of the first image layer, and × represents three image blocks adjacent to the first image block and for which the image block displacement has been determined, based on the image blocks for which the image block displacement has been determined, the third image block displacement of the first image block of the first image layer is determined, and the initial image block displacement includes the third image block displacement.

[0186] The second image block corresponding to the initial image block displacement of the first image layer in the second image layer corresponding to the first image layer is determined, and based on the pixel value of the first image block and the pixel value of the second image block, the initial measurement index value corresponding to the initial image block displacement is calculated, and the initial image block displacement corresponding to the minimum initial measurement index value is determined as the image block displacement of the first image block, based on which the image layer displacement between each layer corresponding to the first image pyramid and the second image pyramid is determined.

[0187] After determining the image block displacement of the first image block, the preset neighborhood image blocks adjacent to the first image block in the first image layer are determined, and the specific process is as follows Figure 6 (b) shows, ● represents the first image block of the first image layer, and × represents eight preset neighborhood image blocks adjacent to the first image block, based on the image block displacement and the corresponding measurement index value of each preset neighborhood image block, the image block displacement of the first image block is updated, and based on this, the updated image layer displacement is determined.

[0188] The fusion process of the image is as follows

[0189] Based on the measurement index value corresponding to the updated image layer displacement, the initial weight of the second image block of the adjacent image is determined, when the threshold value of the second image block is greater than the measurement index value, the initial weight of the second image block is 0, and when the threshold value of the second image block is less than the measurement index value, the initial weight of the second image block is equal to 1-measurement index value / threshold value.

[0190] The preset weight of the first image block of the image to be processed is 1, and it is fused with the initial weight of the second image block of the adjacent image to obtain the fusion weight of the adjacent image and the image to be processed.

[0191] Based on the fusion weight, the adjacent image and the image to be processed are fused to obtain the processed image, and the processed image is corrected according to the preset pixel value fluctuation range. When the pixel value fluctuation range is set to ±10, if the pixel value of the original image is 50, the pixel value of the processed image is 62, and the pixel value of the processed image needs to be corrected to 60, if the pixel value of the processed image is 56, no correction is needed.

[0192] Thus, a frame of denoising processed image is obtained.

[0193] It should be understood that, although Figure 2The steps in the flowchart are shown in sequence according to the arrows, but the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the steps are not strictly limited in sequence, and the steps can be executed in other sequences. Moreover, Figure 2 At least one of the steps in the flowchart can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with other steps or steps or stages in other steps.

[0194] In one embodiment, as shown in Figure 7 An image processing apparatus is provided, comprising: an image acquisition module 710, an image pyramid construction module 720, a calculation module 730, an image alignment module 740, and an image fusion module 750, wherein:

[0195] The image acquisition module 710 is configured to acquire a to-be-processed image and a corresponding adjacent image of the to-be-processed image.

[0196] The image pyramid construction module 720 is configured to construct image pyramids of the to-be-processed image and the adjacent image, respectively. The image pyramid corresponding to the to-be-processed image is a first image pyramid, and the image pyramid corresponding to the adjacent image is a second image pyramid.

[0197] The calculation module 730 is configured to calculate image layer displacements between each layer of the first image pyramid and the second image pyramid, and a measurement index value corresponding to the image layer displacements.

[0198] The image alignment module 740 is configured to align the adjacent image and the to-be-processed image according to the image layer displacements and the measurement index value corresponding to the image layer displacements.

[0199] The image fusion module 750 is configured to fuse the adjacent image and the to-be-processed image to obtain a processed image.

[0200] In one embodiment, the image processing apparatus further comprises:

[0201] A quality optimization processing module is configured to perform quality optimization processing on the to-be-processed image and the adjacent image after acquiring the to-be-processed image and the corresponding adjacent image of the to-be-processed image, and before constructing the image pyramids of the to-be-processed image and the adjacent image, respectively, to obtain a to-be-processed image and an adjacent image processed by quality optimization processing.

[0202] The noise reduction processing module is configured to perform noise reduction processing on the to-be-processed image and the adjacent image after the to-be-processed image and the adjacent image are acquired, to obtain a noise-processed to-be-processed image and a noise-processed adjacent image.

[0203] In one of the embodiments, the noise reduction processing module comprises the following units:

[0204] The first filtering processing unit is configured to perform first filtering processing on the to-be-processed image and the adjacent image, to obtain a first filtering-processed to-be-processed image and a first filtering-processed adjacent image.

[0205] The motion region determination unit is configured to determine a motion region in the first filtering-processed to-be-processed image and the first filtering-processed adjacent image according to pixel values of the first filtering-processed to-be-processed image and the first filtering-processed adjacent image.

[0206] The boundary value calculation unit is configured to calculate a boundary value of the first filtering-processed to-be-processed image and the first filtering-processed adjacent image.

[0207] The non-boundary motion region determination unit is configured to determine a non-boundary motion region in the first filtering-processed to-be-processed image and the first filtering-processed adjacent image based on the boundary value.

[0208] The second filtering processing unit is configured to perform second filtering processing on the non-boundary motion region, to obtain the noise-processed to-be-processed image and the noise-processed adjacent image.

[0209] In one of the embodiments, the calculation module 730 comprises the following units:

[0210] The initial image block displacement calculation unit is configured to calculate an initial image block displacement in a first image layer, the initial image block displacement being a displacement between a first image block in the first image layer and a second image block in a corresponding second image layer, the first image layer being an image layer in the first image pyramid, the second image layer being an image layer in the second image pyramid, the first image block being a preset-size image block in the first image layer, and the second image block being a preset-size image block in the second image layer.

[0211] The initial measurement index value calculation unit is configured to calculate an initial measurement index value corresponding to the initial image block displacement.

[0212] The image layer displacement determination unit is configured to determine an image layer displacement between each layer corresponding to the first image pyramid and the second image pyramid according to the initial image block displacement and the initial measurement index value.

[0213] In one of the embodiments, the initial image block displacement calculation unit comprises a first image block displacement calculation unit, which comprises the following units:

[0214] an interpolation calculation unit configured to perform interpolation calculation on the image layer displacement of a previous layer image layer of the first image layer to obtain an interpolation calculation result, the previous layer image layer being an image layer adjacent to the first image layer and having a lower resolution than the first image layer in the first image pyramid.

[0215] a first image block displacement determination unit configured to determine the first image block displacement of the first image layer according to the interpolation calculation result, the initial image block displacement comprising the first image block displacement.

[0216] In one of the embodiments, the initial image block displacement calculation unit comprises a second image block displacement calculation unit, which comprises the following units:

[0217] a statistical calculation unit configured to perform statistical calculation on the image layer displacement of a previous layer image layer of the first image layer to obtain a statistical calculation result, the previous layer image layer being an image layer adjacent to the first image layer and having a lower resolution than the first image layer in the first image pyramid.

[0218] a second image block displacement determination unit configured to determine the second image block displacement of the first image layer according to the statistical calculation result, the initial image block displacement comprising the second image block displacement.

[0219] In one of the embodiments, the initial image block displacement calculation unit comprises a third image block displacement calculation unit, which comprises the following units:

[0220] an image block determination unit configured to determine an image block in the first image layer having a determined image block displacement.

[0221] a third image block displacement determination unit configured to determine the third image block displacement of the first image layer according to the image block having the determined image block displacement, the initial image block displacement comprising the third image block displacement.

[0222] In one of the embodiments, the initial measurement index value calculation unit comprises the following units:

[0223] a second image block determination unit configured to determine a second image block in the second image layer having the same initial image block displacement as the first image layer.

[0224] An initial metric value determination unit is configured to calculate an initial metric value corresponding to the initial image block displacement based on pixel values of the first image block in the first image layer and pixel values of the second image block.

[0225] In one of the embodiments, the image alignment module 740 comprises the following units:

[0226] An image block displacement determination unit is configured to determine the image block displacement of the first image block in the first image layer from the initial image block displacement based on the initial metric value.

[0227] An image layer displacement determination unit is configured to determine image layer displacements between each layer of the first image pyramid and the second image pyramid corresponding to the image block displacement of the first image block in the first image layer and a metric value corresponding to the image layer displacement.

[0228] An image alignment unit is configured to align the adjacent image and the image to be processed based on the image layer displacements and the metric values corresponding to the image layer displacements.

[0229] In one of the embodiments, the calculation unit further comprises an image block displacement updating unit, which comprises the following units:

[0230] An image block second determination unit is configured to determine preset neighborhood image blocks adjacent to the first image block in the first image layer after determining the image layer displacements between each layer of the first image pyramid and the second image pyramid corresponding to the image block displacement of the first image block in the first image layer.

[0231] An image block displacement updating unit is configured to update the image block displacement of the first image block in the first image layer based on the image block displacements of the preset neighborhood image blocks and the corresponding metric values to determine updated image layer displacements.

[0232] In one of the embodiments, the image fusion module 760 comprises the following units:

[0233] An initial weight determination unit is configured to determine an initial weight of the adjacent image based on the metric value corresponding to the updated image layer displacement.

[0234] A fusion weight calculation unit is configured to fuse a preset weight of the image to be processed and the initial weight of the adjacent image to obtain a fusion weight of the adjacent image and the image to be processed.

[0235] An image fusion unit is configured to fuse the adjacent image and the image to be processed based on the fusion weight to obtain a processed image.

[0236] In one embodiment, the image processing apparatus further includes:

[0237] An image correction unit is used to correct the processed image according to a preset range of pixel value fluctuations after the processed image is acquired.

[0238] For specific limitations regarding the image processing apparatus, please refer to the limitations on the image processing method above, which will not be repeated here. Each module in the aforementioned image processing apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0239] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 8 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores image processing data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method.

[0240] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. Wireless mode can be achieved through WIFI, operator network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement an image processing method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0241] Those skilled in the art can understand that, Figures 8-9 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0242] In one of the embodiments, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the image processing method described above.

[0243] In one of the embodiments, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to implement the steps of the image processing method described above.

[0244] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0245] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0246] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An image processing method, the method comprising: obtaining a to-be-processed image and a neighboring image corresponding to the to-be-processed image; respectively constructing image pyramids of the to-be-processed image and the neighboring image, the image pyramid corresponding to the to-be-processed image being a first image pyramid, the image pyramid corresponding to the neighboring image being a second image pyramid, the first image pyramid being obtained by down-sampling the to-be-processed image, the second image pyramid being obtained by down-sampling the neighboring image, the to-be-processed image and the neighboring image being down-sampled in the same manner; calculating image layer displacements between each layer of the first image pyramid and the second image pyramid, and a measurement index value corresponding to the image layer displacements, the measurement index value being used to measure the degree of the image layer displacements, the image layer displacements corresponding to the measurement index value; aligning the neighboring image and the to-be-processed image according to each of the image layer displacements and the measurement index value corresponding to the image layer displacements; fusing the neighboring image and the to-be-processed image after the alignment to obtain a processed image according to an initial weight of the neighboring image and a preset weight of the to-be-processed image; the calculating of the image layer displacements between each layer of the first image pyramid and the second image pyramid comprises: after the down-sampling, decomposing an offset between the to-be-processed image and the neighboring image into the image layer displacements between each layer of the first image pyramid and the second image pyramid, so as to determine the displacement between the to-be-processed image and the neighboring image through the image layer displacements.

2. The image processing method of claim 1, wherein, after the obtaining of the to-be-processed image and the neighboring image corresponding to the to-be-processed image, and before the respectively constructing of the image pyramids of the to-be-processed image and the neighboring image, the method further comprises at least one of the following: a first item: performing quality optimization processing on the to-be-processed image and the neighboring image to obtain a to-be-processed image and a neighboring image after the quality optimization processing; a second item: performing primary filtering processing on the to-be-processed image and the neighboring image to obtain a to-be-processed image and a neighboring image after the primary filtering processing; determining a motion region in the to-be-processed image and the neighboring image after the primary filtering processing according to pixel values of the to-be-processed image and the neighboring image after the primary filtering processing; calculating a boundary value of the to-be-processed image and the neighboring image after the primary filtering processing; determining a non-boundary motion region in the to-be-processed image and the neighboring image after the primary filtering processing based on the boundary value; performing secondary filtering processing on the non-boundary motion region to obtain a to-be-processed image and a neighboring image after noise processing.

3. The image processing method of claim 1, wherein, the calculating of the image layer displacements between each layer of the first image pyramid and the second image pyramid comprises: calculating initial image block displacements of a first image layer, the initial image block displacements being displacements between first image blocks in the first image layer and second image blocks in a corresponding second image layer, the first image layer being an image layer in the first image pyramid, the second image layer being an image layer in the second image pyramid, the first image blocks being image blocks of a preset size in the first image layer, the second image blocks being image blocks of the preset size in the second image layer; calculating initial measurement index values corresponding to the initial image block displacements; determining image layer displacements between each layer of the first image pyramid and the second image pyramid according to the initial image block displacements and the initial measurement index values.

4. The image processing method of claim 3, wherein, The calculating of the initial image block displacements of the first image layer comprises at least one of the following: The first item is: performing interpolation calculation on image layer displacements of a previous layer image layer of the first image layer to obtain an interpolation calculation result, the previous layer image layer being an image layer adjacent to the first image layer and having a lower resolution than the first image layer in the first image pyramid; determining the first image block displacements of the first image layer according to the interpolation calculation result, the initial image block displacements comprising the first image block displacements; The second item is: performing statistical calculation on image layer displacements of a previous layer image layer of the first image layer to obtain a statistical calculation result, the previous layer image layer being an image layer adjacent to the first image layer and having a lower resolution than the first image layer in the first image pyramid; determining the second image block displacements of the first image layer according to the statistical calculation result, the initial image block displacements comprising the second image block displacements; The third item is: determining image blocks of which image block displacements have been determined in the first image layer; determining the third image block displacements of the first image layer according to the image blocks of which image block displacements have been determined, the initial image block displacements comprising the third image block displacements.

5. The image processing method of claim 4, wherein, The calculating of the initial measurement index values corresponding to the initial image block displacements comprises: determining second image blocks in the second image layer which have the same initial image block displacements as the first image layer; calculating the initial measurement index values corresponding to the initial image block displacements according to pixel values of the first image blocks in the first image layer and pixel values of the second image blocks.

6. The image processing method of claim 5, wherein, The aligning of the adjacent image and the to-be-processed image according to the image layer displacements and the measurement index values corresponding to the image layer displacements comprises: determining image block displacements of the first image blocks in the first image layer from the initial image block displacements based on the initial measurement index values; determining image layer displacements between each layer of the first image pyramid and the second image pyramid and measurement index values corresponding to the image layer displacements based on the image block displacements of the first image blocks in the first image layer; aligning the adjacent image and the to-be-processed image according to the image layer displacements and the measurement index values corresponding to the image layer displacements.

7. The image processing method of claim 6, wherein, After determining the image layer displacement between each layer of the first image pyramid and the second image pyramid based on the image block displacement of the first image block in the first image layer, the method further comprises: determining preset neighborhood image blocks adjacent to the first image block in the first image layer; updating the image block displacement of the first image block in the first image layer based on the image block displacement and the corresponding measurement index value of each preset neighborhood image block, and determining the updated image layer displacement.

8. The image processing method of claim 7, wherein, The method of fusing the adjacent image and the to-be-processed image based on the initial weight of the adjacent image and the preset weight of the to-be-processed image to obtain the processed image comprises: determining the initial weight of the adjacent image according to the measurement index value corresponding to the updated image layer displacement; fusing the preset weight of the to-be-processed image and the initial weight of the adjacent image to obtain the fusion weight of the adjacent image and the to-be-processed image; fusing the adjacent image and the to-be-processed image based on the fusion weight to obtain the processed image.

9. The image processing method of claim 8, wherein, After obtaining the processed image, the method further comprises: correcting the processed image according to a preset pixel value fluctuation range.

10. An image processing apparatus characterized by comprising: The device comprises: an image acquisition module configured to acquire a to-be-processed image and an adjacent image corresponding to the to-be-processed image; an image pyramid construction module configured to construct image pyramids of the to-be-processed image and the adjacent image respectively, wherein the image pyramid corresponding to the to-be-processed image is a first image pyramid, the image pyramid corresponding to the adjacent image is a second image pyramid, the first image pyramid is obtained by down-sampling the to-be-processed image, the second image pyramid is obtained by down-sampling the adjacent image, and the to-be-processed image and the adjacent image are down-sampled in the same manner; a calculation module configured to calculate image layer displacements between each layer of the first image pyramid and the second image pyramid, and measurement index values corresponding to the image layer displacements, wherein the measurement index values are used to measure the degree of goodness of the image layer displacements, and the image layer displacements correspond to the measurement index values; an image alignment module configured to align the adjacent image and the to-be-processed image according to each image layer displacement and the measurement index value corresponding to the image layer displacement; an image fusion module configured to fuse the adjacent image and the to-be-processed image after alignment according to an initial weight of the adjacent image and a preset weight of the to-be-processed image to obtain a processed image. The calculation module is specifically configured to: after down-sampling, decompose the offset between the to-be-processed image and the adjacent image into image layer displacements between each layer of the first image pyramid and the second image pyramid, so as to determine the displacement between the to-be-processed image and the adjacent image through the image layer displacements. 11.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-10. The processor implements the steps of the image processing method of any one of claims 1 to 9 when executing the computer program.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the image processing method of any one of claims 1 to 9.

Images