Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and computer readable storage medium, the method comprises: acquiring multiple different exposure images; calibrating the exposure images to obtain calibrated images; calculating the first weight corresponding to the calibrated images; determining the motion area of the calibrated images; calculating the second weight of the motion area of the calibrated images; according to the first weight and the second weight, the image synthesis is carried out on the multiple calibrated images to obtain a target synthesis image. The application matches the scene positions of the exposure images with different exposure times through image calibration, eliminates the differences caused by the movement of the image sensor during shooting, avoids the blur phenomenon caused by pixel-level fusion, thereby improving the image quality, and effectively eliminates the generation of ghosting in the synthesis image by recalculating the weights corresponding to the calibrated images and the motion area of the calibrated images, thereby further improving the image quality.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In related technologies, when the scene is a dynamic scene that is changing, the exposure time interval between different frames of exposed images will cause displacement deviation between different frames of exposed images due to moving objects, resulting in blurry ghosting phenomena, which will seriously affect the image quality. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0004] Therefore, one objective of this invention is to propose an image processing method that, through image calibration, matches the scene positions of exposed images with different exposure times, eliminates differences caused by image sensor movement during shooting, avoids blurring during pixel-level fusion, and thus improves image quality. By recalculating the weights corresponding to the calibration image and the motion region of the calibration image, the method performs image synthesis on the calibration image, effectively eliminating ghosting in the synthesized image, thereby further improving image quality.

[0005] Therefore, a second objective of the present invention is to provide an image processing apparatus.

[0006] Therefore, a third objective of the present invention is to provide an electronic device.

[0007] Therefore, a fourth object of the present invention is to provide a computer-readable storage medium.

[0008] To achieve the above objectives, a first aspect of the present invention provides an image processing method comprising the following steps: acquiring multiple frames of images with different exposures; calibrating the exposure images to obtain calibration images; calculating a first weight corresponding to the calibration images; determining the motion region of the calibration images; calculating a second weight of the motion region of the calibration images; and performing image synthesis on the multiple frames of the calibration images according to the first weight and the second weight to obtain a target synthesized image.

[0009] According to the image processing method of this invention, an exposed image is calibrated to obtain a calibration image. The motion region of the calibration image is determined, and a first weight corresponding to the calibration image and a second weight corresponding to the motion region of the calibration image are calculated. Then, multiple frames of calibration images are synthesized based on the first and second weights to obtain a target synthesized image. Thus, this invention takes into account the possibility of image sensor movement during shooting. Through image calibration, the scene positions of exposed images with different exposure times are matched, eliminating differences caused by image sensor movement during shooting and avoiding blurring during pixel-level fusion, thereby improving image quality. By recalculating the weights corresponding to the calibration image and the motion region of the calibration image, ghosting in the synthesized image is effectively eliminated, further improving image quality. Furthermore, the algorithm of this invention is simple, computationally inexpensive, easy to implement, and highly applicable.

[0010] In addition, the image processing method of the above embodiments of the present invention may also have the following additional technical features:

[0011] In one embodiment of the present invention, calibrating the exposed image includes: performing image registration on the exposed image to obtain a registered image; and performing image correction on the registered image to obtain the calibrated image.

[0012] In one embodiment of the present invention, image registration of the exposed image includes: setting the exposure of the exposed image to a preset exposure level according to a preset exposure curve to obtain an adjusted exposed image, wherein the adjusted exposed image includes a reference image and a pre-registered image; and obtaining feature point pairs between the pre-registered image and the reference image to obtain the registered image.

[0013] In one embodiment of the present invention, obtaining the feature point pair between the pre-registered image and the reference image includes: performing feature point detection on the pre-registered image and the reference image to obtain feature point information of the pre-registered image and the reference image; the reference image includes a preset origin, and starting from the preset origin, performing feature point matching on the pre-registered image and the reference image within a preset matching range to obtain the feature point pair.

[0014] In one embodiment of the present invention, correcting the registered image includes: calculating the distance between the feature point pairs; statistically calculating the distance between the feature point pairs, and using the distance between the feature point pairs that meet preset conditions as the image offset; calculating the offset direction required for the registered image; and performing an offset operation on the registered image according to the image offset and the offset direction to correct the registered image and obtain the calibration image.

[0015] In one embodiment of the present invention, the method further includes: if no feature point pair is matched within the preset matching range, the matching range is expanded according to a preset step size to re-match the feature points until the feature point pair is obtained.

[0016] In one embodiment of the present invention, calculating the distance between the feature point pairs includes: using the reference image as a reference, calculating the Euclidean distance between the matching feature points of the reference image and the registration image, and using the calculated Euclidean distance as the distance between the feature point pairs.

[0017] In one embodiment of the present invention, the distance between feature point pairs that satisfy the preset conditions includes the distance between the feature point pairs that appear most frequently.

[0018] In one embodiment of the present invention, calculating the offset direction required for the registration image includes: calculating the quadrant position of the corresponding pixel coordinates in the registration image with the pixel coordinates of the reference image as the origin; and determining the offset direction required for the registration image based on the quadrant position.

[0019] In one embodiment of the present invention, after offsetting the registered image according to the image offset and the offset direction, the method further includes: zero-padding the boundary of the offset registered image to make it the same size as the corresponding exposed image, so as to obtain the calibration image.

[0020] In one embodiment of the present invention, determining the motion region of the calibration image includes: acquiring binary images of the calibration image and the reference image; subtracting the binary images of the calibration image and the reference image and obtaining the absolute value to obtain the motion region of the calibration image.

[0021] In one embodiment of the present invention, before calculating the first weight corresponding to the calibration image, the method includes: determining the non-motion region of the calibration image.

[0022] In one embodiment of the present invention, determining the non-motion region of the calibration image includes: inverting the motion region of the calibration image to obtain the non-motion region.

[0023] In one embodiment of the present invention, the image of the non-motion region includes a long exposure image and a short exposure image. After determining the non-motion region of the calibration image, the method further includes: mapping the signal-to-noise ratio of the preset long exposure image onto the short exposure image of the non-motion region in the RGB color space to improve the signal-to-noise ratio of the short exposure image.

[0024] In one embodiment of the present invention, calculating the first weight corresponding to the calibration image includes: calculating the contrast, exposure, and image entropy of the calibration image; and calculating the weight corresponding to the calibration image using the contrast, exposure, and image entropy to obtain the first weight.

[0025] In one embodiment of the present invention, calculating the contrast of the calibration image includes: performing Laplacian filtering on the exposure image corresponding to the calibration image, and then taking the absolute value of the obtained filtering result to obtain the contrast of the calibration image.

[0026] In one embodiment of the present invention, calculating the exposure of the calibration image includes: calculating the exposure of the calibration image using a triangular weighting function.

[0027] In one embodiment of the present invention, calculating the image entropy of the calibration image includes: using a preset matrix as the range for entropy value calculation, calculating the entropy value of the intermediate point by calculating the probability of occurrence of all pixels within the preset matrix, and using the entropy value as the image entropy of the calibration image.

[0028] In one embodiment of the present invention, calculating the weight corresponding to the calibration image using the contrast, exposure, and image entropy includes: multiplying the contrast, exposure, and image entropy and normalizing them to obtain the weight of the calibration image.

[0029] In one embodiment of the present invention, calculating the second weight corresponding to the motion region of the calibration image includes: W′ x, =(W1*(I 1ex +I 2ex )+W2) / I 1ex , where W′ x, As the second weight, I 1ex I represents the exposure amount corresponding to the exposed image I1. 2ex Let I1 be the exposure amount corresponding to the exposure image I2, W1 be the first weight corresponding to the exposure image I1, and W2 be the first weight corresponding to the exposure image I2. Here, I1 and I2 are two adjacent exposure images, and (x,y) represents the coordinate point of the reference image.

[0030] In one embodiment of the present invention, image synthesis of multiple frames of the calibration images according to the first weight and the second weight includes: calculating a weighted average of the first weight and the second weight to obtain a weight map of the calibration images; obtaining the pixel value of each pixel of the image to be synthesized according to the weight map; and synthesizing the target synthesized image according to the pixel value of each pixel of the image to be synthesized.

[0031] In one embodiment of the present invention, a weighted average calculation of the first weight and the second weight includes: Among them, I (x,) Let i be the target image to be synthesized, n and m be the dimensions of the current image to be synthesized, and Value be the target image to be synthesized. i(,) B represents the pixel values ​​of the current image to be synthesized. i(,) For values ​​in the non-motion region, W i(x,y) To calibrate the first weights corresponding to the image, W′ i(,) This is the second weight corresponding to the motion region.

[0032] To achieve the above objectives, a second aspect of the present invention provides an image processing apparatus, comprising: an acquisition module for acquiring multiple frames of images with different exposures; a calibration module for calibrating the exposure images to obtain a calibration image; a first calculation module for calculating a first weight corresponding to the calibration image; a processing module for determining the motion region of the calibration image; a second calculation module for calculating a second weight of the motion region of the calibration image; and a synthesis module for synthesizing the multiple frames of the calibration images according to the first weight and the second weight to obtain a target synthesized image.

[0033] According to an embodiment of the image processing apparatus of the present invention, an exposure image is calibrated to obtain a calibration image, the motion region of the calibration image is determined, a first weight corresponding to the calibration image and a second weight corresponding to the motion region of the calibration image are calculated, and then multiple frames of calibration images are synthesized based on the first and second weights to obtain a target synthesized image. Thus, the present invention takes into account the possibility of image sensor movement during shooting. Through image calibration, the scene positions of exposure images with different exposure times are matched, eliminating differences caused by image sensor movement during shooting and avoiding blurring during pixel-level fusion, thereby improving image quality. By recalculating the weights corresponding to the calibration image and the motion region of the calibration image, ghosting in the synthesized image is effectively eliminated, further improving image quality. Furthermore, the algorithm of the present invention is simple, computationally inexpensive, easy to implement, and highly applicable.

[0034] To achieve the above objectives, a third aspect of the present invention provides an electronic device comprising an image processing apparatus as described in the second aspect of the present invention above; or, the electronic device comprising: a processor, a memory, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the image processing method as described in the first aspect of the present invention above.

[0035] According to an embodiment of the present invention, an electronic device obtains a calibration image by calibrating an exposed image, determines the motion region of the calibration image, calculates a first weight corresponding to the calibration image and a second weight corresponding to the motion region of the calibration image, and then synthesizes multiple frames of calibration images based on the first and second weights to obtain a target synthesized image. Thus, the present invention takes into account the possibility of image sensor movement during shooting. Through image calibration, the scene positions of exposed images with different exposure times are matched, eliminating differences caused by image sensor movement during shooting and avoiding blurring during pixel-level fusion, thereby improving image quality. By recalculating the weights corresponding to the calibration image and the motion region of the calibration image, the generation of ghosting in the synthesized image is effectively eliminated, further improving image quality. Furthermore, the algorithm of the present invention is simple, computationally inexpensive, easy to implement, and highly applicable.

[0036] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium storing an image processing program, which, when executed by a processor, implements the image processing method as described in the first aspect of the present invention.

[0037] According to an embodiment of the present invention, a computer-readable storage medium storing an image processing program thereon, when executed by a processor, calibrates an exposed image to obtain a calibration image, determines the motion region of the calibration image, calculates a first weight corresponding to the calibration image and a second weight corresponding to the motion region of the calibration image, and then performs image synthesis on multiple frames of calibration images based on the first and second weights to obtain a target synthesized image. Thus, the present invention takes into account the possibility of image sensor movement during shooting. Through image calibration, the scene positions of exposed images with different exposure times are matched, eliminating differences caused by image sensor movement during shooting and avoiding blurring during pixel-level fusion, thereby improving image quality. By recalculating the weights corresponding to the calibration image and the motion region of the calibration image, the generation of ghosting in the synthesized image is effectively eliminated, further improving image quality. Furthermore, the algorithm of the present invention is simple, computationally inexpensive, easy to implement, and highly applicable.

[0038] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0039] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0040] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of an image correction process according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram illustrating the process of determining the motion region of a calibration image according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram illustrating the process of calculating the first weight corresponding to the non-motion region in the calibration image according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0045] To provide a more detailed understanding of the features and technical content of the embodiments of the present invention, the implementation of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of the present invention. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be shown in a simplified manner to simplify the drawings.

[0046] Current image sensors cannot fully capture all the information in natural scenes and cannot effectively reproduce the true image information of natural scenes, resulting in low image quality. In order to improve image quality, such LDR (Low Dynamic Range) images need to be converted into HDR (High Dynamic Range) images through appropriate image processing techniques.

[0047] In related technologies, high dynamic range (HDR) image synthesis mainly involves calculating weights based on image information entropy, contrast, or descriptive operators such as Dense SIFT (Scale-invariant feature transform), and then fusing multi-exposure images to obtain HDR images. However, while this method can guarantee the quality of the synthesized image in static scenes, it cannot guarantee image quality in dynamic scenes.

[0048] Therefore, by image calibration, the scene positions of exposed images with different exposure times are matched, eliminating the differences caused by the movement of the image sensor during shooting and avoiding the blurring phenomenon caused by pixel-level fusion, thus improving image quality. By recalculating the weights corresponding to the moving and non-moving regions in the calibration image and performing image synthesis on the calibration image, the generation of ghosting in the synthesized image is effectively eliminated, thereby further improving image quality.

[0049] The following is for reference. Figures 1-5 Image processing methods, apparatus, electronic devices, and computer-readable storage media according to embodiments of the present invention are described.

[0050] Figure 1 This is a flowchart of an image processing method according to an embodiment of the present invention. Figure 1 As shown, the image processing method includes steps S1 to S6.

[0051] Step S1: Acquire multiple frames of images with different exposures.

[0052] In a specific embodiment, multiple frames of images with different exposures input from the image sensor can be acquired, for example, denoted as I. n Multiple frames with different exposures refer to multiple images with varying exposure levels. For example, during shooting, the image sensor may move or shake, resulting in exposure time intervals between different frames. This can cause mismatches in scene positions or differences in shooting angles, leading to displacement deviations between the different frames and thus blurring. The image sensor can be, for example, but not limited to, a CMOS image sensor.

[0053] Step S2: Calibrate the exposed image to obtain a calibrated image.

[0054] The calibration image is a registered and calibrated image. After acquiring multiple frames of images with different exposures, the images are calibrated to obtain the calibration image.

[0055] In this embodiment, as described above, acquiring multiple frames with different exposures results in exposure time intervals between them due to image sensor movement or jitter during shooting. This leads to varying exposure levels among the multiple frames, affecting image quality. Therefore, the exposure images are calibrated to bring them to the same shooting angle, thereby reducing the impact of differences in shooting angles on image quality and improving overall image quality. Thus, the calibrated images obtained after calibration have the same exposure level and shooting angle, which helps to eliminate blurring caused by pixel-level fusion during image synthesis, reducing ghosting and improving image quality.

[0056] Step S3: Calculate the first weight corresponding to the calibration image.

[0057] In this embodiment, after determining the calibration image, a first weight of the calibration image is calculated using contrast, exposure, and image entropy. It is understood that by calculating the first weight of the calibration image, it is easier to synthesize the calibration image based on the first weight.

[0058] Step S4: Determine the motion region of the calibration image.

[0059] Specifically, after obtaining the calibration image, the motion region of the calibration image is determined. When an image sensor captures a dynamic scene, there are moving objects in the dynamic scene. The motion region is the part of the image in the dynamic scene where the moving objects are located. Conversely, the part outside the motion region is the non-motion region, also known as the background region. Specifically, the aforementioned image calibration process solves the impact of image sensor movement on HDR images, but it still fails to eliminate the ghosting problem caused by moving objects in the image. Therefore, this embodiment of the invention obtains the motion map of the calibration image, determines its motion region, and after determining the motion region of the calibration image, inverts the motion region of the calibration image to obtain the non-motion region. After obtaining the non-motion region, it is easier to improve the signal-to-noise ratio of the short exposure image in the non-motion region. Then, the corresponding weights are reassigned to the motion region, and finally, a synthesized deghosting image is obtained, thereby improving image quality.

[0060] Step S5: Calculate the second weight of the motion region in the calibration image.

[0061] Specifically, the weight of the motion region is calculated, which is the second weight. Then, after calculating the first weight and the second weight, the multi-frame calibration image can be synthesized based on the first weight and the second weight.

[0062] Step S6: Perform image synthesis on multiple calibration images according to the first weight and the second weight to obtain the target synthesized image.

[0063] Specifically, during image synthesis, adjacent calibration images are synthesized based on the first and second weights, thereby achieving image synthesis of multiple calibration images corresponding to multiple different exposure images, resulting in a final target synthesized image with eliminated ghosting and high image quality.

[0064] Therefore, the image processing method described above, by calibrating the exposed image to obtain a calibration image, determining the motion region of the calibration image, calculating the first weight corresponding to the calibration image and the second weight corresponding to the motion region of the calibration image, and then synthesizing multiple frames of calibration images according to the first and second weights to obtain the target synthesized image. Thus, this invention takes into account the possibility of image sensor movement during shooting. Through image calibration, the scene positions of exposed images with different exposure times are matched, eliminating differences caused by image sensor movement during shooting and avoiding blurring during pixel-level fusion, thereby improving image quality. By recalculating the weights corresponding to the calibration image and the motion region of the calibration image, the generation of ghosting in the synthesized image is effectively eliminated, further improving image quality. Furthermore, the algorithm of this invention is simple, computationally inexpensive, easy to implement, and highly applicable.

[0065] In some embodiments, calibrating the exposed image includes: image registration of the exposed image to obtain a registered image; and image correction of the registered image to obtain a calibrated image.

[0066] Specifically, in this embodiment of the invention, image registration is performed on each exposure image during the synthesis of images with different exposure times, so that the exposure of each exposure image is at the same level, thereby matching the scene positions of each exposure image with different exposure times, thus avoiding the blurring phenomenon generated during pixel-level fusion, and thus helping to improve the image quality of the synthesized image.

[0067] As previously mentioned, image sensors may move or shake during shooting, resulting in exposure time intervals between exposed images. This causes the scene positions of images with different exposure times to be matched, and the shooting angles of multiple exposed images to differ, thus affecting image quality. Therefore, in embodiments of the present invention, after image registration, a registered image is obtained, and the registered image is further corrected to bring different exposed images to the same shooting angle. This reduces the impact of differences in shooting angles between multiple exposed images on image quality, thereby improving image quality. Thus, the multiple calibrated images obtained after registration and correction have the same exposure level and shooting angle, which helps to eliminate blurring caused by pixel-level fusion during image synthesis, eliminate ghosting, and improve image quality.

[0068] In one embodiment of the present invention, image registration of an exposed image includes: setting the exposure of the exposed image to a preset exposure level according to a preset exposure curve to obtain an adjusted exposed image, the adjusted exposed image including a reference image and a pre-registered image; and obtaining feature point pairs between the pre-registered image and the reference image to obtain a registered image.

[0069] Specifically, during image sensor capture, dynamic scenes not only involve moving objects but also the potential movement of the image sensor itself. Therefore, image registration is necessary when combining images with different exposure times to match the scene positions of the images, thus avoiding blurring during pixel-level fusion and improving image quality. The specific image registration process includes: placing the exposure of the images at a preset exposure level according to a preset exposure curve to obtain an adjusted exposure image. The adjusted exposure image includes a reference image and a pre-registered image. By acquiring feature point pairs between the pre-registered image and the reference image, a registered image is obtained based on the matching results of the feature point pairs. This allows multiple frames with different exposure times to be placed at the preset exposure level, ensuring they have the same exposure. In other words, for multiple input frames with different exposure times... n Multiple frames with different exposures are placed at the same exposure level according to a preset exposure curve to obtain an adjusted exposure pattern. This pattern is then used to obtain a registration image based on feature point pairs between the pre-registered image and the reference image. For example, different exposure images are placed at the same exposure level (e.g., I). ′ n = f(x,y), where f(x,y) is the preset exposure curve, I ′ n The image is a pre-registered image, and (x,y) are the pixel coordinates of the reference image.

[0070] In one embodiment of the present invention, obtaining a feature point pair between a pre-registered image and a reference image includes: performing feature point detection on the pre-registered image and the reference image to obtain feature point information of the pre-registered image and the reference image; the reference image may include a preset origin, and starting from the preset origin, performing feature point matching on the pre-registered image and the reference image within a preset matching range to obtain a feature point pair.

[0071] Specifically, that is, for the pre-registered image I ′ n Feature point detection is performed on both the pre-registered image and the reference image to obtain feature point information for the pre-registered image and the reference image, respectively. In an embodiment of the present invention, the SURF (Speeded Up Robust Features) algorithm can be used to perform feature point detection on the pre-registered image I. ′ n Feature point detection is performed on both the reference image and the image, thereby improving the efficiency and accuracy of feature point detection. Furthermore, the obtained feature point information includes, for example, the location of feature points in the image and the feature point information of the feature matrix.

[0072] After obtaining the feature point information, feature point matching is performed on the pre-registered image and the reference image starting from a preset origin in the reference image, within a preset matching range, to obtain feature point pairs. Specifically, the preset origin is a point pre-selected in the matching image. Feature point matching is performed using this point as the starting point or initial point, and the initial matching range is the preset matching range. When a feature point is matched, the corresponding feature point pair is recorded. In a specific embodiment, for example, I(0,0) on the reference image is used as the preset origin, and feature point matching starts from I(0,0), with an initial matching range R of 10. Compared to the method of selecting an initial point and then performing a global search and matching, this embodiment of the invention selects an initial point and sets a matching range, performing feature point matching within the matching range without performing global matching. This allows for faster feature point matching while saving resources, thus resulting in higher matching efficiency.

[0073] In one embodiment of the present invention, combined with Figure 2 As shown, the correction of the registered image includes: Step S32: Calculating the distance between feature point pairs; Step S33: Statistically calculating the distance between the calculated feature point pairs, and using the distance between feature point pairs that meet preset conditions as the image offset; Step S34: Calculating the offset direction required for the registered image; Step S35: Performing an offset operation on the registered image according to the image offset and offset direction to correct the registered image and obtain a calibrated image. Furthermore, through image correction, high dynamic range image synthesis by a moving image sensor has a good imaging effect, effectively avoiding differences in the synthesized image caused by the image sensor during movement, and thus improving the quality of the synthesized image.

[0074] Specifically, after image registration, the registered image undergoes further image correction to obtain a calibration image. This includes: calculating feature point matching between the reference image and the pre-registered image to obtain feature point pairs; calculating the distance between two matching feature points in each feature point pair. Since there are generally multiple feature point pairs, there are also multiple distances calculated for each pair. The distances of these multiple feature point pairs are statistically analyzed, and the distance of the feature point pair that meets the preset conditions is used as the image offset, which is the current required offset for the registered image. Furthermore, the required offset direction for the registered image is calculated. Then, after determining the image offset and offset direction, the registered image can be offset according to the image offset and offset direction to obtain the calibration image. It can be understood that the calibration image obtained after the offset operation eliminates image deviation, thus improving accuracy and enhancing the quality of the synthesized image. The reference image can be a standard image that meets the requirements from multiple exposed images.

[0075] In one embodiment of the present invention, the method further includes: if no feature point pair is matched within a preset matching range, the matching range is expanded according to a preset step size to re-match the feature points until a feature point pair is obtained.

[0076] Specifically, if no feature point is matched within the initial matching range, the matching range needs to be further expanded to ensure that feature point pairs are obtained, thereby improving the reliability of feature point matching. In this embodiment of the invention, the matching range is gradually expanded using a preset step size. In a specific embodiment, the initial matching range is, for example, 10, and the preset step size is, for example, 5. That is, when no feature point is matched, the matching range is expanded by a step size of 5 each time, thereby performing feature point matching as quickly as possible while saving resources and improving the reliability of feature point matching.

[0077] In one embodiment of the present invention, step S32 above, calculating the distance between feature point pairs, includes: using a reference image as a reference, calculating the Euclidean distance between matching feature points in the reference image and the registered image, and using the calculated Euclidean distance as the distance between feature point pairs.

[0078] Specifically, using a preset reference image as a benchmark, the Euclidean distance between matching feature points in the reference image and the registered image is calculated, which serves as the distance between feature point pairs between the reference image and the registered image. In a specific embodiment, the distance between feature point pairs can be calculated using the following formula: Dis(x,y)=|I(x1,y1)-I′(x2,y2)|, where Dis(x,y) is the calculated distance between feature point pairs, I(x1,y1) is the reference image, and I′(x2,y2) is the registered image.

[0079] In one embodiment of the present invention, in step S33, the distance between feature point pairs that satisfy the preset conditions includes the distance between the feature point pairs that appear most frequently.

[0080] Specifically, after calculating the distance between feature point pairs, the distances of the feature point pairs are statistically analyzed. The distances of the feature point pairs follow a normal distribution, and the distance value that appears most frequently is taken as the offset between the registered image and the reference image, i.e., the image offset.

[0081] In one embodiment of the present invention, step S34, calculating the offset direction required for the registration image, includes: calculating the quadrant position of the corresponding pixel coordinates in the registration image with the pixel coordinates of the reference image as the origin; and determining the offset direction required for the registration image based on the quadrant position.

[0082] Specifically, let I(x1,y1) be the reference image, with (x1,y1) being its pixel coordinates, and I′(x2,y2) be the registration image, with (x2,y2) being its pixel coordinates. Using (x1,y1) as the origin, calculate the quadrant position of (x2,y2) to determine the required offset direction for the registration image, and then perform the offset operation. In a specific embodiment, the required offset direction for the registration image can be calculated using the following formula: Where Dir(x,y) represents the offset direction.

[0083] In one embodiment of the present invention, after offsetting the registered image according to the image offset and offset direction, the method further includes: zero-padding the boundaries of the offset registered image to make it the same size as the corresponding exposed image, so as to obtain a calibration image. In other words, for the registered image after the offset operation, its boundaries are zero-padding to make it the same size as the initially acquired corresponding exposed image, so as to obtain a calibration image, thereby facilitating subsequent image synthesis.

[0084] In one embodiment of the present invention, such as Figure 3 As shown, determining the motion region of the calibration image includes: Step S41: acquiring binary images of the calibration image and the reference image; Step S42: subtracting the binary images of the calibration image and the reference image and obtaining the absolute value to get the motion region of the calibration image.

[0085] Specifically, the process involves first determining the corresponding binary images for the calibration image and the reference image, then subtracting the two binary images and taking the absolute value. The resulting difference map is the marker map of the relative motion between the calibration image and the reference image, which is also the motion region of the calibration image.

[0086] In one embodiment of the present invention, step S41, obtaining a binary image of the calibration image, includes: performing grayscale processing on the calibration image to obtain a first grayscale image; determining a first threshold for dividing the first grayscale image; and dividing the first grayscale image according to the first threshold to obtain a binary image of the calibration image.

[0087] Specifically, the calibration image is converted to grayscale using the formula: Gray1 = 0.3R + 0.59G + 0.11B, where Gray1 is the grayscale value of the first grayscale image, R is the red component, G is the green component, and B is the blue component. After converting the calibration image to grayscale, a first threshold, denoted as Threshold1, is determined. Threshold1 is used to segment the first grayscale image, and the binary image of the calibration image can be obtained based on this threshold.

[0088] In one embodiment of the present invention, determining a first threshold for dividing a first grayscale image includes: statistically analyzing the histogram of the first grayscale image, accumulating the number of occurrences of each pixel value to obtain a first total number of occurrences; and sequentially accumulating the number of occurrences of each pixel value on the statistical histogram from smallest to largest until the accumulated number of occurrences is greater than or equal to half of the first total number of occurrences, and then determining the current corresponding pixel value as the first threshold.

[0089] Specifically, the histogram of the first grayscale image is statistically analyzed, and the number of occurrences at each pixel level is accumulated to calculate the first total number of occurrences, denoted as SumAll1. SumHalf1 = SumAll1 / 2 is calculated, and the number of occurrences Value1 at each pixel level in the statistical histogram is accumulated sequentially from small to large until Value1 >= SumHalf1. The pixel level at this point is the first threshold Threshold1 for dividing the first grayscale image.

[0090] In one embodiment of the present invention, dividing the first grayscale image according to a first threshold includes: determining whether the pixel value of each pixel in the first grayscale image is greater than or equal to the first threshold; setting the pixel value of the pixel that is greater than or equal to the first threshold to 1, and setting the pixel value of the pixel that is less than the first threshold to 0, so as to obtain a binary image of the calibration image.

[0091] Specifically, it is determined whether the pixel value of each pixel in the first grayscale image is greater than or equal to the first threshold Threshold1. If the pixel value of a certain pixel is greater than or equal to Threshold1, then the pixel is represented by 1. If the pixel value of a certain pixel is less than Threshold1, then Threshold1 is represented by 0. Thus, the binary image divided by Threshold1 can be obtained, that is, the binary image corresponding to the calibration image can be obtained.

[0092] In one embodiment of the present invention, step S41, obtaining a binary image of a reference image, includes: performing grayscale processing on the reference image to obtain a second grayscale image; determining a second threshold for dividing the second grayscale image; and dividing the second grayscale image according to the second threshold to obtain a binary image of the reference image.

[0093] Specifically, the reference image is converted to grayscale using the formula: Gray2 = 0.3R + 0.59G + 0.11B, where Gray2 is the grayscale value of the second grayscale image, R is the red component, G is the green component, and B is the blue component. After converting the reference image to grayscale, a second threshold, denoted as Threshold2, is determined. Threshold2 is used to segment the second grayscale image, and the binary image of the reference image can be obtained by segmenting the second grayscale image accordingly.

[0094] In one embodiment of the present invention, determining a second threshold for dividing a second grayscale image includes: statistically analyzing the histogram of the second grayscale image, accumulating the number of occurrences of each pixel value to obtain a second total number of occurrences; and sequentially accumulating the number of occurrences of each pixel value on the statistical histogram from smallest to largest until the accumulated number of occurrences is greater than or equal to half of the second total number of occurrences, and then determining the current corresponding pixel value as the second threshold.

[0095] Specifically, the histogram of the second grayscale image is statistically analyzed, and the number of occurrences at each pixel level is accumulated to calculate the second total number of occurrences, denoted as SumAll2. SumHalf2 = SumAll2 / 2 is calculated, and the number of occurrences Value2 at each pixel level in the statistical histogram is accumulated from small to large until Value2 >= SumHalf2. The pixel level at this point is the second threshold Threshold2 for dividing the second grayscale image.

[0096] In one embodiment of the present invention, dividing the second grayscale image according to the second threshold includes: determining whether the pixel value of each pixel in the second grayscale image is greater than the second threshold; setting the pixel value of the pixel point that is greater than or equal to the second threshold to 1, and setting the pixel value of the pixel point that is less than the second threshold to 0, so as to obtain a binary image of the reference image.

[0097] Specifically, it involves determining whether the pixel value of each pixel in the second grayscale image is greater than or equal to the second threshold Threshold2. If the pixel value of a certain pixel is greater than or equal to Threshold2, then the pixel is represented by 1. If the pixel value of a certain pixel is less than Threshold2, then Threshold2 is represented by 0. In this way, the binary image divided by Threshold2 can be obtained, that is, the binary image corresponding to the reference image can be obtained.

[0098] Furthermore, after obtaining the corresponding images of the calibration image and the reference image, the absolute value of the difference between the binary images of the calibration image and the reference image is obtained. The resulting difference map is the marker map of the relative motion part between the calibration image and the reference image, that is, the motion region of the calibration image.

[0099] In one embodiment of the present invention, before calculating the first weight corresponding to the calibration image, the method includes: determining the non-motion region of the calibration image.

[0100] Specifically, before calculating the first weight of the calibration image, the non-motion region of the calibration image, i.e., the background region, is determined.

[0101] In one embodiment of the present invention, determining the non-motion region of a calibration image includes: inverting the motion region of the calibration image to obtain the non-motion region.

[0102] Specifically, the motion region of the previously obtained calibration image is a binary image. Inverting the motion region yields the non-motion region, i.e., the background region. It's understandable that since the background region is obtained by inverting the motion region, it is also a binary image, but opposite to the motion region's binary image. That is, a value of 1 in the motion region's binary image corresponds to a value of 0 in the background region, and vice versa.

[0103] In one embodiment of the present invention, the image of the non-motion region includes a long exposure image and a short exposure image. After determining the non-motion region of the calibration image, the method further includes: mapping the signal-to-noise ratio of the preset long exposure image onto the short exposure image in the RGB color space to improve the signal-to-noise ratio of the short exposure image.

[0104] Specifically, the process involves inverting the moving region to obtain the non-moving region, i.e., the background region. Then, the signal-to-noise ratio (SNR) of the short-exposure image is enhanced based on the background region's value of 1. This includes mapping the SNR of a preset long-exposure image onto the short-exposure image in the RGB color space to improve the SNR of the short-exposure image. Thus, by using long exposure to improve the SNR of short exposure, the noise level in the short-exposure image is reduced, which is beneficial for displaying detailed information in the final synthesized result and improving image quality.

[0105] The specific calculation and processing steps include: first calculating the ratio of the G component in long and short exposures. Among them, I1G (,) and I2G (,) These correspond to the magnitudes of the G components at a certain point in exposed images I1 and I2, respectively. I1 and I2 are two adjacent exposed images, and (x, y) represents the coordinates of the point. Furthermore, after obtaining RateG, the magnitudes of the R and B components of exposed images I1 and I2 can be calculated using RateG. Specifically, this can be achieved through… To calculate I1R (,) and I2R (,) It can be accessed through To calculate I1B (,) and I2B (,) Among them, I1R (,) and I2R (,) These correspond to the magnitudes of the R components of exposed images I1 and I2 at a certain point, respectively, I1B (,) and I2B (,) These correspond to the magnitudes of the B components at a certain point in the exposed images I1 and I2, respectively. Therefore, the components of the exposed images in each channel can be obtained, thereby enhancing the signal-to-noise ratio of short-exposure images where the background region has a value of 1.

[0106] In one embodiment of the present invention, such as Figure 4 As shown, calculating the first weight corresponding to the calibration image includes: calculating the contrast, exposure, and image entropy of the calibration image; and calculating the weight corresponding to the calibration image using the contrast, exposure, and image entropy to obtain the first weight.

[0107] In one embodiment of the present invention, such as Figure 4 As shown, the weights corresponding to the calibration image are calculated using contrast, exposure, and image entropy: the contrast, exposure, and image entropy are multiplied and normalized to obtain the weights corresponding to the calibration image.

[0108] Specifically, the weights calculated above, such as contrast, exposure, and image entropy, are multiplied and then normalized to obtain the weights corresponding to the calibrated image. A specific calculation formula is as follows:

[0109]

[0110] Among them, W x,y As the first weight, W CON For contrast, W EX For exposure, W EN Let W be the image entropy and MaxWeight be the maximum weight. CON maximum value, W EX The maximum value and W EN The product of the maximum values.

[0111] In one embodiment of the present invention, such as Figure 4 As shown, the weights corresponding to the calibration image are calculated, including: calculating the contrast, exposure, and image entropy of the calibration image; and calculating the weights corresponding to the calibration image using the contrast, exposure, and image entropy.

[0112] Specifically, after determining the calibration image, the first weight corresponding to the calibration image is calculated. After determining the motion region of the calibration image, the next compositing operation can be performed, which involves calculating the weight of the motion region. The two calculated weights are then combined and their weighted average is calculated to obtain the target composite image. In this embodiment of the invention, calculating the weight of the calibration image using contrast, exposure, and image entropy effectively improves the dynamic range of the composite image, thus enhancing the accuracy of image synthesis and ultimately improving image quality.

[0113] In one embodiment of the present invention, the process of calculating the contrast of the calibration image includes: performing Laplacian filtering on the exposure image corresponding to the calibration image, and then taking the absolute value of the obtained filtering result to obtain the contrast of the calibration image.

[0114] Specifically, that is, for the exposed image I nA Laplace filter is used for filtering, and the absolute value of the filtering result is taken to obtain the contrast, denoted as W. CON The Laplace kernel of the Laplace filter is represented by [0,1,0; 1,-4,1; 0,1,0].

[0115] In one embodiment of the present invention, calculating the exposure of the calibration image includes: using a triangular weighting function to calculate the exposure of the calibration image, which can effectively reduce the amount of calculation and improve the calculation efficiency.

[0116] Specifically, the principle behind using the triangular weighting function to calculate the exposure of a calibrated image is that the closer a pixel value is to half of its maximum value, the greater the weight. The specific calculation formula is as follows:

[0117]

[0118] Where is the calculated exposure value, Value is the pixel value of the pixel, and MaxValue is the maximum value among the pixel values.

[0119] In one embodiment of the present invention, the process of calculating the image entropy of the calibration image includes: using a preset matrix as the range for entropy value calculation, calculating the entropy value of the intermediate point by calculating the probability of occurrence of all pixels in the preset matrix, and using the entropy value as the image entropy of the calibration image.

[0120] In a specific embodiment, the preset matrix is, for example, a 3×3 matrix. That is, a 3×3 matrix is ​​used as the range for entropy calculation. The entropy value of the intermediate point is calculated by calculating the probability of the occurrence of nine pixels within the 3×3 matrix, denoted as W. EN The entropy value W EN Image entropy as a calibration image.

[0121] In one embodiment of the present invention, the second weight corresponding to the motion region can be calculated using the following formula:

[0122] W′ x, =(W1*(I 1ex +I 2ex )+W2) / I 1ex ,

[0123] Among them, W′ x, As the second weight, I 1ex I represents the exposure amount corresponding to the exposed image I1. 2ex Let I1 be the exposure amount corresponding to the exposure image I2, W1 be the first weight corresponding to the exposure image I1, which is also the first weight of the calibration image corresponding to the exposure image I1, and W2 be the first weight corresponding to the exposure image I2, which is also the first weight of the calibration image corresponding to the exposure image I2. Here, I1 and I2 are two adjacent exposure images, and (x,y) represents the coordinate point of the reference image.

[0124] In one embodiment of the present invention, image synthesis of multiple frames of calibration images is performed according to a first weight and a second weight, including: calculating a weighted average of the first weight and the second weight to obtain a weight map of the calibration image; obtaining the pixel value of each pixel of the image to be synthesized according to the weight map; and synthesizing a target synthesized image according to the pixel value of each pixel of the image to be synthesized.

[0125] Specifically, the first and second weights calculated are combined to form the final weight map. The pixel value of each pixel is calculated by weighted averaging to obtain the final ghost-free composite image, i.e., the target composite image.

[0126] In one embodiment of the present invention, a weighted average calculation of the first weight and the second weight includes:

[0127]

[0128] Among them, I (x,) Let i be the target image to be synthesized, n and m be the current size of the image to be synthesized, and Value be the target image to be synthesized. i(,) B represents the pixel values ​​of the current image to be synthesized. i(,) For non-motion values, W takes the value 0 or 1. i(x,y) To calibrate the first weights corresponding to the image, W′ i(,) The second weight corresponds to the motion region, and (x,y) represents the pixel coordinates of the reference image.

[0129] In summary, the image processing method of this invention addresses the root cause of ghosting in dynamic HDR images. It considers that dynamic scenes not only involve moving objects but also image sensor movement or jitter during shooting. First, images with different exposures are registered and corrected to the same exposure level and shooting angle. This image registration and correction eliminates differences caused by image sensor movement during shooting. Then, moving object detection is performed, the weights of moving objects are redefined, and image synthesis is performed accordingly to obtain a high-quality, ghost-free composite image. Specifically, the SURF algorithm is used for fast image registration. Euclidean distance is calculated by calculating the matching feature point pairs to obtain the image offset and direction. The registered image is then offset based on the offset and direction to achieve image correction. Further, the motion and background regions in the calibration image are identified. The weights of the background and motion regions are calculated using a redefined weighting method and fused into a weight map. Finally, ghost-free high dynamic range image synthesis is performed to improve image quality.

[0130] Compared to existing technologies, the weight calculation method in this invention is simple, consumes fewer resources, has high computational efficiency, and has good portability and applicability. Long exposure improves the signal-to-noise ratio of short exposures, reducing noise levels in short-exposure images and facilitating the display of detailed information in the final synthesized result. It provides excellent imaging effects for synthesizing high dynamic range images from moving image sensors, effectively avoiding differences in the synthesized image caused by sensor movement, thus improving image quality. Using contrast, exposure, and image entropy to determine the weights of the final fused image effectively improves the dynamic range of the synthesized image. By detecting moving objects, determining motion regions, recalculating the weight map, and using reference images, it effectively eliminates ghosting in the synthesized image, thereby improving the quality of the synthesized image.

[0131] In summary, the image processing method according to embodiments of the present invention involves acquiring an exposed image, performing image registration on the exposed image to obtain a registered image, performing image correction on the registered image to obtain a calibrated image, determining the motion region of the calibrated image, calculating a first weight corresponding to the calibrated image and a second weight corresponding to the motion region, and then synthesizing multiple frames of calibrated images based on the first and second weights to obtain a target synthesized image. Thus, the present invention considers the possibility of image sensor movement during shooting. Through image registration and correction, it matches the scene positions of exposed images with different exposure times, eliminating differences caused by image sensor movement during shooting and avoiding blurring during pixel-level fusion, thereby improving image quality. By detecting moving objects, determining motion regions, and recalculating the weight map for image synthesis, it effectively eliminates ghosting in the synthesized image, further improving image quality. Furthermore, the algorithm of the present invention is simple, computationally inexpensive, easy to implement, and highly applicable.

[0132] A further embodiment of the present invention also discloses an image processing apparatus.

[0133] Figure 5 This is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present invention, as shown below. Figure 5 As shown, the image processing device 100 includes: an acquisition module 110, a calibration module 120, a first calculation module 130, a processing module 140, a second calculation module 150, and a synthesis module 160.

[0134] Specifically, the acquisition module 110 is used to acquire multiple frames of images with different exposures.

[0135] The calibration module 120 is used to calibrate the exposed image to obtain a calibration image.

[0136] The first calculation module 130 is used to calculate the first weight corresponding to the calibration image;

[0137] The processing module 140 is used to determine the motion region of the calibration image.

[0138] The second calculation module 150 is used to calculate the second weight of the motion region of the calibration image.

[0139] The synthesis module 160 is used to synthesize multiple frames of calibration images according to the first weight and the second weight to obtain the target synthesized image.

[0140] In one embodiment of the present invention, the calibration module 120 is used to perform image registration on the exposed image to obtain a registered image; and to perform image correction on the registered image to obtain a calibration image.

[0141] In one embodiment of the present invention, the calibration module 120 is used to set the exposure of the exposure image to a preset exposure level according to a preset exposure curve to obtain an adjusted exposure image, the adjusted exposure image including a reference image and a pre-registered image; and to obtain feature point pairs between the pre-registered image and the reference image to obtain a registered image.

[0142] In one embodiment of the present invention, the calibration module 120 is used to perform feature point detection on the pre-registered image and the reference image to obtain feature point information of the pre-registered image and the reference image; the reference image includes a preset origin, and feature point matching is performed on the pre-registered image and the reference image from the preset origin with a preset matching range to obtain feature point pairs.

[0143] In one embodiment of the present invention, the calibration module 120 is used to calculate the distance between feature point pairs; statistically calculate the distance between the feature point pairs, and use the distance between feature point pairs that meet the preset conditions as the image offset; calculate the offset direction required for the registration image; and perform an offset operation on the registration image according to the image offset and the offset direction to correct the registration image and obtain a calibration image.

[0144] In one embodiment of the present invention, the calibration module 120 is further configured to expand the matching range by a preset step size if no feature point pair is matched within a preset matching range, so as to re-match the feature points until a feature point pair is obtained.

[0145] In one embodiment of the present invention, the calibration module 120 is used to calculate the Euclidean distance between matching feature points of the reference image and the registered image based on the reference image, and to use the calculated Euclidean distance as the distance between the feature point pairs.

[0146] In one embodiment of the invention, the calibration module 120 is used to determine the distance between the most frequently occurring feature point pairs.

[0147] In one embodiment of the present invention, the calibration module 120 is used to calculate the quadrant position of the corresponding pixel coordinates in the registration image with the pixel coordinates of the reference image as the origin; and to determine the offset direction required for the registration image based on the quadrant position.

[0148] In one embodiment of the present invention, the calibration module 120 is further configured to perform zero-padding on the boundaries of the offset registration image so that it is the same size as the corresponding exposure image, so as to obtain a calibration image.

[0149] In one embodiment of the present invention, the processing module 140 is used to acquire binary images of a calibration image and a reference image; and to obtain the motion region of the calibration image by subtracting the binary images of the calibration image and the reference image and taking the absolute value.

[0150] In one embodiment of the present invention, the first calculation module 130 is used to determine the non-motion region of the calibration image.

[0151] In one embodiment of the present invention, the processing module 140 is used to invert the motion region of the calibration image to obtain the non-motion region.

[0152] In one embodiment of the present invention, the processing module 140 is further configured to map the signal-to-noise ratio of a preset long-exposure image onto a short-exposure image in the RGB color space, so as to improve the signal-to-noise ratio of the short-exposure image.

[0153] In one embodiment of the present invention, the first calculation module 130 is used to calculate the contrast, exposure and image entropy of the calibration image; and calculate the weight corresponding to the calibration image through the contrast, exposure and image entropy to obtain the first weight.

[0154] In one embodiment of the present invention, the first calculation module 130 is used to perform Laplacian filtering on the exposure image corresponding to the calibration image, and then calculate the absolute value of the obtained filtering result to obtain the contrast of the calibration image.

[0155] In one embodiment of the present invention, the first calculation module 130 is used to calculate the exposure of the calibration image using a triangular weighting function.

[0156] In one embodiment of the present invention, the first calculation module 130 is used to calculate the entropy value using a preset matrix as the range of entropy value calculation, and calculates the entropy value of the intermediate point by calculating the probability of the occurrence of all pixels in the preset matrix, and uses the entropy value as the image entropy of the calibration image.

[0157] In one embodiment of the present invention, the first calculation module 130 is used to multiply the contrast, exposure, and image entropy and normalize them to obtain the weight of the calibration image.

[0158] In one embodiment of the present invention, the second calculation module 150 is used for W′x, =(W1*(I 1ex +I 2ex )+W2) / I 1ex , where W′ x, As the second weight, I 1ex I represents the exposure amount corresponding to the exposed image I1. 2ex Let I1 be the exposure amount corresponding to the exposure image I2, W1 be the first weight corresponding to the exposure image I1, and W2 be the first weight corresponding to the exposure image I2. Here, I1 and I2 are two adjacent exposure images, and (x,y) represents the coordinate point of the reference image.

[0159] In one embodiment of the present invention, the synthesis module 160 is used to perform a weighted average calculation on the first weight and the second weight to obtain a weight map of the calibration image; obtain the pixel value of each pixel of the image to be synthesized based on the weight map; and synthesize the target synthesized image based on the pixel value of each pixel of the image to be synthesized.

[0160] In one embodiment of the present invention, the synthesis module 160 is used for Among them, I (x,) Let i be the target image to be synthesized, n and m be the dimensions of the current image to be synthesized, and Value be the target image to be synthesized. i(,) B represents the pixel values ​​of the current image to be synthesized. i(,) For values ​​in the non-motion region, W i(x,y) To calibrate the first weights corresponding to the image, W′ i(,) The second weight corresponding to the motion region

[0161] It should be noted that the specific implementation of the image processing device 100 during image processing is similar to the specific implementation of the image processing method in any of the above embodiments of the present invention. Therefore, for a detailed exemplary description of the image processing device 100, please refer to the relevant description of the image processing method described above. To reduce redundancy, it will not be repeated here.

[0162] Therefore, according to the image processing apparatus 100 of this embodiment, an exposed image is acquired, an exposed image is registered to obtain a registered image, an exposed image is corrected to obtain a calibrated image, the motion region of the calibrated image is determined, a first weight corresponding to the calibrated image and a second weight corresponding to the motion region are calculated, and then multiple frames of calibrated images are synthesized according to the first weight and the second weight to obtain a target synthesized image. Thus, this invention takes into account the possibility of image sensor movement during shooting. Through image registration and correction, the scene positions of exposed images with different exposure times are matched, eliminating the differences caused by image sensor movement during shooting and avoiding blurring during pixel-level fusion, thereby improving image quality. By detecting moving objects, determining motion regions, and recalculating weight maps for image synthesis, the generation of ghosting in the synthesized image is effectively eliminated, further improving image quality. Furthermore, the algorithm of this invention is simple, computationally inexpensive, easy to implement, and highly applicable.

[0163] Further embodiments of the present invention also disclose an electronic device.

[0164] In one embodiment of the present invention, the electronic device includes the image processing apparatus 100 described in any of the above embodiments.

[0165] In another embodiment of the present invention, the electronic device includes: a processor, a memory, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the image processing method as described in any of the above embodiments of the present invention.

[0166] In a specific embodiment, the electronic device is, for example, an image sensing device, which includes, for example, a CMOS image sensor.

[0167] Therefore, when performing image processing, the specific implementation of the electronic device can be found in the aforementioned description of the image processing method or image processing device 100. To reduce redundancy, it will not be repeated here.

[0168] According to an embodiment of the present invention, an electronic device acquires an exposure image, performs image registration on the exposure image to obtain a registered image, performs image correction on the registered image to obtain a calibration image, determines the motion region of the calibration image, calculates a first weight corresponding to the calibration image and a second weight corresponding to the motion region, and then performs image synthesis on multiple frames of calibration images based on the first and second weights to obtain a target synthesized image. Thus, the present invention takes into account the possibility of image sensor movement during shooting. Through image registration and correction, the scene positions of exposure images with different exposure times are matched, eliminating differences caused by image sensor movement during shooting and avoiding blurring during pixel-level fusion, thereby improving image quality. By detecting moving objects, determining motion regions, and recalculating the weight map for image synthesis, the generation of ghosting in the synthesized image is effectively eliminated, further improving image quality. Furthermore, the algorithm of the present invention is simple, computationally inexpensive, easy to implement, and highly applicable.

[0169] Further embodiments of the present invention disclose a computer-readable storage medium storing an image processing program. When the image processing program is executed by a processor, it implements the image processing method as described in any of the above embodiments of the present invention. For a detailed description of the execution process of the image processing method, please refer to the relevant sections above, which will not be repeated here.

[0170] Therefore, according to the computer-readable storage medium of the present invention, when the image processing program stored thereon is executed by a processor, it can acquire an exposed image, perform image registration on the exposed image to obtain a registered image, perform image correction on the registered image to obtain a calibration image, determine the motion region of the calibration image, calculate the first weight corresponding to the calibration image and the second weight corresponding to the motion region, and then perform image synthesis on multiple frames of calibration images according to the first weight and the second weight to obtain a target synthesized image. Thus, the present invention takes into account the situation where the image sensor moves during shooting. Through image registration and correction, the scene positions of exposed images with different exposure times are matched, eliminating the differences caused by the movement of the image sensor during shooting, avoiding the blurring phenomenon caused by pixel-level fusion, thereby improving image quality. By detecting moving objects, determining the motion region, and recalculating the weight map for image synthesis, the generation of ghosting in the synthesized image is effectively eliminated, thereby further improving image quality. Furthermore, the algorithm of the present invention is simple, computationally inefficient, easy to implement, and highly applicable.

[0171] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example.

[0172] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An image processing method, characterized in that, Includes the following steps: Acquire multiple frames of images with different exposures; The calibration of the exposed image to obtain a calibration image includes: image registration of the exposed image to obtain a registered image, and image correction of the registered image to obtain the calibration image; wherein, image registration of the exposed image includes: placing the exposure of the exposed image at a preset exposure level according to a preset exposure curve to obtain an adjusted exposed image, the adjusted exposed image including a reference image and a pre-registered image; obtaining feature point pairs between the pre-registered image and the reference image to obtain the registered image; and calculating a first weight corresponding to the calibration image. Determine the motion region of the calibration image; Calculating the second weight of the motion region of the calibration image; wherein, calculating the second weight corresponding to the motion region of the calibration image includes: ,in, As the second weight, Exposure image The corresponding exposure, Exposure image The corresponding exposure, Exposure image The corresponding first weight, Exposure image The corresponding first weight, where, and For two adjacent exposure images, ( () represents the coordinates of the reference image; Based on the first weight and the second weight, the multiple frames of the calibration images are synthesized to obtain the target synthesized image.

2. The image processing method according to claim 1, characterized in that, Obtaining feature point pairs between the pre-registered image and the reference image includes: Feature point detection is performed on the pre-registered image and the reference image to obtain the feature point information of the pre-registered image and the reference image respectively; The reference image includes a preset origin. Starting from the preset origin, feature points are matched between the pre-registered image and the reference image within a preset matching range to obtain the feature point pair.

3. The image processing method according to claim 1, characterized in that, Correcting the registered image includes: Calculate the distance between the feature point pairs; The distance between feature point pairs obtained through statistical calculation is used as the image offset for feature point pairs that meet preset conditions. Calculate the offset direction required for the registered image; The registered image is offset according to the image offset and the offset direction to correct the registered image and obtain the calibration image.

4. The image processing method according to claim 2, characterized in that, Also includes: If no feature point pair is matched within the preset matching range, the matching range is expanded by a preset step size to re-match the feature points until the feature point pair is obtained.

5. The image processing method according to claim 3, characterized in that, The calculation of the distance between the feature point pairs includes: Using the reference image as a reference, the Euclidean distance between the matching feature points of the reference image and the registered image is calculated, and the calculated Euclidean distance is used as the distance between the feature point pairs.

6. The image processing method according to claim 3, characterized in that, The distance between feature point pairs that meet the preset conditions includes the distance between the feature point pairs that appear most frequently.

7. The image processing method according to claim 3, characterized in that, The calculation of the offset direction required for the registration image includes: The quadrant position of the corresponding pixel coordinate in the registered image is calculated using the pixel coordinate of the reference image as the origin; The required offset direction for the registration image is determined based on the quadrant position.

8. The image processing method according to claim 3, characterized in that, After performing an offset operation on the registered image based on the image offset and the offset direction, the method further includes: The boundaries of the offset registration image are zero-padded to make it the same size as the corresponding exposure image, thus obtaining the calibration image.

9. The image processing method according to claim 2, characterized in that, Determining the motion region of the calibration image includes: Obtain binary images of the calibration image and the reference image; The motion region of the calibration image is obtained by subtracting the binary images of the calibration image and the reference image and taking the absolute value.

10. The image processing method according to claim 1, characterized in that, Before calculating the first weight corresponding to the calibration image, the following steps are included: Identify the non-motion regions of the calibration image.

11. The image processing method according to claim 10, characterized in that, Determining the non-motion regions of the calibration image includes: The non-motion region is obtained by inverting the motion region of the calibration image.

12. The image processing method according to claim 10, characterized in that, The images of the non-motion regions include long-exposure images and short-exposure images. After determining the non-motion regions of the calibration image, the process further includes: In the RGB color space, the signal-to-noise ratio of a preset long-exposure image is mapped onto the short-exposure image to improve the signal-to-noise ratio of the short-exposure image.

13. The image processing method according to claim 1, characterized in that, Calculating the first weight corresponding to the calibration image includes: Calculate the contrast, exposure, and image entropy of the calibrated image; The weights corresponding to the calibration image are calculated using the contrast, exposure, and image entropy to obtain the first weight.

14. The image processing method according to claim 13, characterized in that, The calculation of the contrast of the calibrated image includes: After applying a Laplacian filter to the exposure image corresponding to the calibration image, the absolute value of the filtered result is calculated to obtain the contrast of the calibration image.

15. The image processing method according to claim 14, characterized in that, The calculation of the exposure of the calibration image includes: calculating the exposure of the calibration image using a triangular weighting function.

16. The image processing method according to claim 15, characterized in that, The calculation of the image entropy of the calibration image includes: A preset matrix is ​​used as the range for entropy calculation. The entropy value of the intermediate point is calculated by calculating the probability of all pixels appearing within the preset matrix. This entropy value is then used as the image entropy of the calibrated image.

17. The image processing method according to claim 16, characterized in that, The weights corresponding to the calibration image are calculated using the contrast, exposure, and image entropy, including: The weights of the calibrated image are obtained by multiplying the contrast, exposure, and image entropy and then normalizing the results.

18. The image processing method according to claim 1, characterized in that, Image synthesis is performed on multiple frames of the calibration images based on the first weight and the second weight, including: A weighted average is calculated on the first weight and the second weight to obtain the weight map of the calibration image; The pixel value of each pixel in the image to be synthesized is obtained based on the weight map. The target synthesized image is obtained by synthesizing the pixel value of each pixel in the image to be synthesized.

19. The image processing method according to claim 18, characterized in that, The weighted average of the first weight and the second weight is calculated, including: in, For the target synthesized image, Let n be the current image to be synthesized, and m be the size of the current image to be synthesized. The pixel values ​​of the current image to be synthesized. Values ​​for non-motion regions. To calibrate the first weight corresponding to the image, This is the second weight corresponding to the motion region.

20. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire multiple frames of images with different exposures; A calibration module is used to calibrate the exposed image to obtain a calibration image, including: image registration of the exposed image to obtain a registered image, and image correction of the registered image to obtain the calibration image; wherein, image registration of the exposed image includes: setting the exposure of the exposed image to a preset exposure level according to a preset exposure curve to obtain an adjusted exposed image, the adjusted exposed image including a reference image and a pre-registered image; and acquiring feature point pairs between the pre-registered image and the reference image to obtain the registered image; The first calculation module is used to calculate the first weight corresponding to the calibration image; A processing module is used to determine the motion region of the calibration image; The second calculation module is used to calculate the second weight of the motion region of the calibration image; wherein, calculating the second weight corresponding to the motion region of the calibration image includes: ,in, As the second weight, Exposure image The corresponding exposure, Exposure image The corresponding exposure, Exposure image The corresponding first weight, Exposure image The corresponding first weight, where, and For two adjacent exposure images, ( () represents the coordinates of the reference image; The synthesis module is used to synthesize multiple frames of the calibration images according to the first weight and the second weight to obtain the target synthesized image.

21. An electronic device, characterized in that, include: The image processing apparatus as described in claim 20, or, A processor, a memory, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the image processing method as described in any one of claims 1-19.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an image processing program, which, when executed by a processor, implements the image processing method as described in any one of claims 1-19.

Images