Image processing method and device, storage medium, and electronic device

Abstract

An image processing method and device, storage medium, and electronic device. The method includes: obtaining a current frame image and an image to be aligned; the current frame image and the image to be aligned are both images taken for the same scene containing a starry sky; performing starry sky registration on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned; obtaining a denoised frame image corresponding to the current frame image based on the current frame image and the aligned frame image, and outputting the denoised frame image. The above scheme can be used to improve the signal-to-noise ratio of the image containing the starry sky while solving the problem of image blur.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image processing method and device, a storage medium, and an electronic device. Background Art

[0002] When the camera is working at night, due to poor light, a longer exposure time is required. However, due to the shaking of the handheld camera and other reasons, the image containing the starry sky often has a low signal-to-noise ratio and a blurred picture. Summary of the invention

[0003] The problem to be solved by the present invention is to improve the signal-to-noise ratio of a starry sky image while solving the problem of image blur.

[0004] To solve the above problems, an embodiment of the present invention provides an image processing method, which includes: obtaining a current frame image and an image to be aligned; the current frame image and the image to be aligned are both images taken for the same scene containing a starry sky; performing starry sky alignment on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned; based on the current frame image and the aligned frame image, obtaining a denoised frame image corresponding to the current frame image, and outputting it.

[0005] An embodiment of the present invention also provides an image processing device, which includes: an acquisition unit, suitable for acquiring a current frame image and an image to be aligned; the current frame image and the image to be aligned are both images taken for the same scene containing a starry sky; an alignment unit, suitable for performing starry sky alignment on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned; a denoising unit, suitable for obtaining a denoised frame image corresponding to the current frame image based on the current frame image and the aligned frame image; and an output unit, suitable for outputting an image.

[0006] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the steps of any one of the methods described in the above embodiments.

[0007] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor runs the computer program, the steps of any one of the methods described in the above embodiments are executed.

[0008] Compared with the prior art, the technical solution of the embodiment of the present invention has the following advantages:

[0009] By applying the solution of the present invention, after obtaining the current frame image and the image to be aligned, the image to be aligned and the current frame image are firstly aligned with the starry sky to obtain the aligned frame image corresponding to the image to be aligned, and then the denoised frame image corresponding to the current frame image is obtained based on the current frame image and the aligned frame image. Since the denoised frame image corresponding to the current frame image is finally obtained, that is, the denoising operation is performed on the current frame image, the signal-to-noise ratio of the final output image can be improved. In addition, before the denoising operation is performed on the current frame image, the image to be aligned and the current frame image are firstly aligned with the starry sky to obtain the aligned frame image corresponding to the image to be aligned. At this time, since the aligned frame image is aligned with the current frame image, when the denoised frame image corresponding to the current frame image is obtained based on the current frame image and the aligned frame image, the image blurring problem of the denoised frame image can be solved. In addition, in the present invention, based on the starry sky registration method, the alignment frame image corresponding to the image to be aligned is obtained, that is, the alignment operation is performed. Compared with other image alignment methods, since the starry sky registration method is based on the star coordinates for registration, and the stars exist in the images taken at night, and their positions will not change in a short time, it is more suitable for aligning low-illuminance images, that is, it has a better alignment effect on low-illuminance images, thereby better solving the problem of blurred denoising frame images. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of an image processing method in an embodiment of the present invention;

[0011] Figure 2 is a flow chart of a method for performing starry sky registration of an image to be aligned and a current frame image in an embodiment of the present invention;

[0012] Figure 3 It is a schematic diagram of the process of star registration of the current frame image and the image to be aligned;

[0013] Figure 4 is a flow chart of a method for determining a translation vector in an embodiment of the present invention;

[0014] Figure 5 Schematic diagram of binary matrices corresponding to each matching window in an embodiment of the present invention;

[0015] Figure 6 It is a structural schematic diagram of an image processing device in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] Currently, when cameras work at night, due to poor light, a longer exposure time is required. However, due to factors such as shaking when shooting with a handheld camera, the images of the starry sky often have a low signal-to-noise ratio and are blurred.

[0017] In view of this problem, the present invention provides an image processing method. By applying the method, a noise reduction frame image corresponding to the current frame image is finally obtained, which can improve the signal-to-noise ratio of the final output image. Moreover, since an aligned frame image aligned with the current frame image is first obtained before the noise reduction operation is performed on the image, the problem of blurring of the noise reduction frame image can be solved. In addition, in the present invention, the aligned frame image is obtained based on the star registration method, which is more suitable for aligning low-illuminance images, that is, it has a better alignment effect on low-illuminance images, thereby better solving the problem of blurring of the noise reduction frame image.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0019] Reference Figure 1 , an embodiment of the present invention provides an image processing method, which may include the following steps:

[0020] Step 11, obtaining a current frame image and an image to be aligned; the current frame image and the image to be aligned are both images taken for the same scene containing a starry sky.

[0021] In a specific implementation, for an image captured with a starry sky scene, multiple frames of images may be captured continuously. The image to be aligned may be any frame of images except the current frame of images captured. The image to be aligned may be an adjacent frame of images to the current frame of images, or may be a non-adjacent frame of images to the current frame of images, which is not limited here. For example, the image to be aligned may be the previous frame of images of the current frame of images.

[0022] Since the current frame image and the image to be aligned are images taken for the same scene containing a starry sky, both images contain a starry sky region.

[0023] Step 12: perform starry sky registration on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned.

[0024] In the present invention, the image to be aligned is aligned with the current frame image by performing starry sky alignment to obtain an aligned frame image corresponding to the image to be aligned, wherein the image alignment is performed by using a starry sky alignment method, that is, the image alignment is performed based on the coordinates of the stars in the image.

[0025] The image to be aligned is subjected to starry sky registration with the current frame image to obtain an aligned frame image corresponding to the image to be aligned. The aligned frame image is aligned with the current frame image.

[0026] Step 13: Obtain a denoised frame image corresponding to the current frame image according to the current frame image and the aligned frame image, and output it.

[0027] In a specific implementation, a denoised frame image corresponding to the current frame image is obtained based on the aligned current frame image and the aligned frame image, that is, a denoising operation is performed on the current frame image, which can improve the signal-to-noise ratio of the current frame image without causing a blurred image.

[0028] In a specific implementation, according to the current frame image and the aligned frame image, a variety of methods can be used to obtain the noise reduction frame image corresponding to the current frame image, which is not limited here.

[0029] In one embodiment of the present invention, a weighted average operation may be performed on the current frame image and the aligned frame image to obtain a denoised frame image corresponding to the current frame image. Since the aligned frame image is aligned with the current frame image, the denoised frame image after weighted average eliminates the displacement between frames caused by camera shake, so that the denoised frame image does not produce a blurred image problem, and the weighted average operation improves the signal-to-noise ratio.

[0030] The image to be aligned is taken as the current frame image I n The previous frame image I n-1 For example, if the previous frame image I n-1 The corresponding aligned frame image is P n-1 , then for the current frame image I n And the aligned frame image is P n-1 Perform the weighted average operation, and the resulting denoised frame image A n = w·I n +(1-w)·P n-1 .

[0031] It should be noted that for the current frame image I n And the aligned frame image is P n-1 Perform weighted averaging operation, that is, the current frame image I n And the aligned frame image is P n-1 The weighted average operation is performed on the pixel values ​​at the same pixel position in the denoised frame image, and the weighted average operation result represents the pixel value of the pixel position in the denoised frame image.

[0032] Among them, w is the current frame image I n The assigned weight, 0<w<1, usually w=1 / 2.

[0033] The method for obtaining a noise-reduced frame image in the embodiment of the present invention can be easily extended to a scenario where multiple frame images are used to obtain an output image. For example, using n frames of continuously taken photos I1, I2, ..., I n To reduce noise, the images to be aligned I1, I2, ... n-1 Respectively with the current frame image In Alignment is performed to obtain n - 1 aligned images P1, P2......P n-1 , and then the n - 1 aligned images and the current frame image I n Perform a weighted average operation, and the resulting noise-reduced frame image is:

[0034] A n = w * I n +(1 - w) / (n - 1)*(P1 + P2+...+P n-1 )

[0035] where w is the weight assigned to the current frame image I n Allocated weight, 0 < w < 1, usually w = 1 / n at this time, and n is an integer greater than or equal to 2.

[0036] The larger w is, the greater the weight of the current frame image I n The greater the weight it occupies, when there are moving objects in the shooting scene, it is less likely to have motion blur. The smaller w is, the smaller the weight of the current frame image I n The smaller the weight it occupies, the higher the signal-to-noise ratio of the noise-reduced frame image A n . Therefore, when shooting moving objects, w should be set to a larger value, and when shooting still objects, w should be set to a smaller value, which can be specifically set by those skilled in the art according to actual needs.

[0037] In some embodiments, the corresponding relationship between the shooting scene and w can also be set, multiple shooting scenes and the w corresponding to each shooting scene. When running the image processing method, the w that is adapted can be automatically selected according to the shooting scene to improve the image processing efficiency.

[0038] In specific implementation, the jitter when holding a camera mainly causes the translation of the image frame, and the rotational component is very small. Therefore, referring to Figure 2 , the embodiment of the present invention also provides a method for performing starry sky registration on the to-be-aligned image and the current frame image. In the starry sky registration method of this embodiment, only the translation situation is considered, thereby reducing the calculation amount, and the image processing and output can be completed in real time on a handheld device, such as a mobile phone, smart glasses, smart helmets and other imaging devices that require real-time output of images.

[0039] Specifically, the method may include the following steps:

[0040] Step 21, segment the current frame image into a starry sky area and a non-starry sky area.

[0041] In specific implementation, the maximum inter-class variance (OTSU) algorithm can be used to segment the current frame image into a starry sky area and a non-starry sky area. The OTSU algorithm is an existing known algorithm, and the specific process of the OTSU algorithm will not be elaborated here.

[0042] Step 22, matching the starry sky region of the current frame image with the starry sky region of the image to be aligned in terms of star coordinates, to obtain a translation vector of the image to be aligned.

[0043] Figure 3 (a) is the current frame image, Figure 3 (b) is the image to be aligned. Figure 3 , the current frame image is translated compared to the field of view of the image to be aligned, so the translation vector of the image to be aligned needs to be calculated.

[0044] In a specific implementation, the coordinates of the stars in the starry sky area of ​​the current frame image and the image to be aligned are matched, and a translation vector is obtained based on the matching result. Since the positions of the stars in the starry sky area will not change in a short time, the translation vector obtained after the current frame image and the image to be aligned are matched based on the positions of the stars can be used as the image displacement deviation caused by camera shake between the two frames.

[0045] In a specific implementation, there may be multiple methods for matching the starry sky area of ​​the current frame image with the starry sky area of ​​the image to be aligned to obtain the translation vector of the image to be aligned, which are not specifically limited.

[0046] Step 23: translating the image to be aligned according to the translation vector to obtain an alignment frame image corresponding to the image to be aligned.

[0047] Assume that the image to be aligned is the current frame image I n The previous frame image I n-1 , the translation vector is obtained as T n-1 After that, the previous frame image I n-1 Each pixel coordinate plus the translation vector T n-1 , and the resulting image is the image that has been translated into the current frame image I n Aligned frame image P n-1 .

[0048] In practical applications, the image after translation may have some pixel coordinates that exceed the size range of the image before translation in the translation direction. For example, when translating to the upper right, some pixels on the right and upper sides of the image may exceed the size range of the image. These pixels that exceed the range will be cropped. The image after translation may have some pixels that are left blank in the opposite direction of translation. For example, when translating to the upper right, some pixels on the left and lower sides of the image may be left blank. These blank pixels are filled with pixels at the same coordinate position of the current frame image.

[0049] Reference Figure 4The embodiment of the present invention further provides a method for determining a translation vector, which may include the following steps:

[0050] Step 41: determine a first matching window in the current frame image.

[0051] In a specific implementation, the upper, lower, left and right edges of the first matching window are all spaced apart from the corresponding edges of the current frame image by a preset margin. The preset margins between the first matching window and each edge may be the same or different.

[0052] The size of the first matching window should be set so that when the camera is held normally, the translation between the two frames caused by camera shake does not exceed the margin between the first matching window and the edge of the current frame image (i.e., the preset margin). The jitter amplitude when holding the camera can be measured experimentally. For example, when measuring multiple people holding a certain camera, the average jitter amplitude is 3 pixels in the left and right directions and 2 pixels in the top and bottom directions. In this case, the first matching window can be set to leave 4 pixels of blank space on the left and right sides (i.e., the margin between the left and right edges of the first matching window and the current frame image is 4 pixels), and 3 pixels of blank space on the top and bottom (i.e., the margin between the top and bottom edges of the first matching window and the current frame image is 4 pixels). Figure 3 Shown as the dotted box pr1 in (a).

[0053] The above method is used to set the first matching window, which can help find several complete second matching windows in the image to be aligned to perform matching and summing operations with the first matching window when the user holds the camera and the camera shakes normally. It should be noted that the average jitter amplitude in this example is 3 pixels in the left and right directions, 2 pixels in the upper and lower directions, and the left and right blanks of the first matching window are 4 pixels, and the left and right blanks are 3 pixels. These are assumed parameters for the convenience of explanation through schematic diagrams. The actual jitter range when holding the camera is related to the focal length of the lens and the pixel size, and is usually dozens of pixels or even hundreds of pixels. It should be obtained through experiments based on the specific camera.

[0054] Step 42, determining reference stars based on the first matching window and the starry sky area of ​​the current frame image.

[0055] In a specific implementation, when determining the reference star, star detection can be first performed on the starry sky area of ​​the current frame image in the first matching window to obtain several detected stars of the current frame image, and then the star with the brightest brightness is selected from the detected stars of the several current frame images as the reference star.

[0056] In a specific implementation, the pixels belonging to the starry sky area in the first matching window can be traversed, and the mean value mean and the pixel standard deviation σ of the pixels belonging to the starry sky area in the first matching window can be counted. If the current pixel value is greater than (mean+c*σ), the current pixel is considered to be a pixel occupied by stars, not a sky background pixel. Among them, c is a parameter that controls the star detection threshold. The larger c is, the fewer the total number of stars detected, and the fewer pseudo stars detected due to noise; the smaller c is, the more the total number of stars detected, and the more pseudo stars detected due to noise. Usually c=3 can be taken.

[0057] After determining the pixels occupied by the stars, all the pixels occupied by the stars in the four directions of up, down, left and right constitute a detected star. For any detected star, the coordinates of all the pixels occupied by the detected star can be averaged as the coordinates of the detected star, and the brightness values ​​of the pixels of the detected star can be summed as the brightness of the detected star. In this way, the coordinates and brightness of all the detected stars in the first matching window can be obtained. The star with the largest brightness is selected as the reference star. If there is more than one brightest star with the same brightness in the first matching window, select any one of them as the reference star. Then the image in the first matching window can be binarized, so that the coordinate pixels where the star is located are equal to 1, and the other pixels are equal to 0.

[0058] For example, refer to Figure 3 (a), in the first matching window pr1, each grid represents a pixel, and the black solid pixel represents the position of the detected star. Star detection is performed on the starry sky area in the first matching window pr1, and a total of 7 detected stars can be obtained. If the detected star r1 has the highest brightness, the detected star r1 is used as the reference star.

[0059] Step 43, based on the first matching window, the reference star, the image to be aligned and the preset margin, determining a plurality of stars to be aligned in the image to be aligned and a second matching window corresponding to each of the stars to be aligned.

[0060] In a specific implementation, the size of the first matching window is the same as the size of the second matching window, and the first relative position relationship between each of the second matching windows and the corresponding star to be aligned is the same as the second relative position relationship between the first matching window and the reference star.

[0061] After obtaining the first matching window and the reference stars, the first translation window can be determined based on the first matching window, the reference stars and the preset margin, and then based on the image to be aligned, the starry sky area of ​​the image to be aligned is detected to obtain a number of detected stars of the image to be aligned, and then the area in the image to be aligned that is at the same position as the first translation window is used as the second translation window, and in the second translation window, the N detected stars of the image to be aligned with the highest brightness are selected as the stars to be aligned, and based on the stars to be aligned, the reference stars and the first matching window, the second matching window corresponding to each of the stars to be aligned is determined. Wherein, N is an integer greater than or equal to 1.

[0062] In a specific implementation, the position of the first translation window is related to the margin between the first matching window and the current frame image.

[0063] In one embodiment of the present invention, the distances from the upper, lower, left and right sides of the first translation window to the reference star are equal to the preset margin. Since the second translation window in the image to be aligned has the same position and size as the first translation window in the current frame image, and the star to be aligned is located in the second translation window, limiting the first translation window and thus limiting the size of the second translation window can reduce the number of stars to be aligned, thereby reducing the amount of calculation, which is conducive to the real-time processing and output of the image. In addition, since the distances from the upper, lower, left and right sides of the first translation window to the reference star are limited to be equal to the preset margin, and the distances from the upper, lower, left and right edges of the first matching window to the corresponding edges of the current frame image are also equal to the preset margin, and the preset margin is set to be greater than the translation distance between the two frames caused by camera shake when the camera is normally held, it is conducive to finding several complete second matching windows in the image to be aligned later to perform matching and summing operations with the first matching window.

[0064] Specifically, combined Figure 3 (a), after determining the reference star r1 and the first matching window pr1, since the margins between the first matching window pr1 and the left and right edges of the current frame image are 4 pixels each, and the margins between the first matching window pr1 and the upper and lower edges of the current frame image are 3 pixels each, the first translation window w1 can be: a rectangular window formed by taking the reference star r1 as the center, 4 pixels to the left and right of the reference star r1, and 3 pixels above and below the reference star r1.

[0065] When performing star detection on the starry sky area of ​​the image to be aligned, all pixels in the starry sky area of ​​the image to be aligned can be traversed according to the preset star detection window, and the mean value mean and standard deviation σ of each pixel in the star detection window can be counted, and the pixel value greater than (mean+c*σ) is used as the pixel occupied by the star, thereby obtaining the pixels occupied by all stars in the image to be aligned. All the pixels occupied by stars in the four directions of up, down, left and right constitute a detected star, and the coordinates of all the pixels occupied by the detected star can be averaged as the coordinates of the detected star, and the brightness values ​​of the pixels of the detected star can be summed as the brightness of the detected star, thereby obtaining the brightness and coordinates of each detected star in the image to be aligned.

[0066] For example, the image to be aligned (such as Figure 3 (b) Each small square represents a pixel, and each black solid pixel represents the coordinates of a detected star, and a total of 8 detected stars are obtained. Figure 3 (a)), the area in the image to be aligned with the same position as the first translation window w1 is used as the second translation window w2 (as shown in Figure 3 (b)). In the second translation window w2, the three detected stars s1, s2 and s3 with the highest brightness are selected as the stars to be registered.

[0067] Among them, it is assumed that the star to be aligned is s i , the corresponding second matching window is ps i , i = 1, 2..., k. The second matching window ps i and the stars to be aligned i The first relative position relationship between the first matching window pr1 and the reference star r1 is the same as the second relative position relationship between the first matching window pr1 and the reference star r1. Therefore, the second matching window can be obtained as ps i , i = 1, 2..., k. k is the number of stars to be registered, which is a preset parameter, k is a natural number, and k is less than or equal to the number of detected stars in the second matching window.

[0068] For example, suppose the pixel coordinates of the upper left corner of the first matching window pr1 are (pr1_x0, pr1_y0), the coordinates of the reference star r1 are (r1_x, r1_y), and the pixel coordinates of the second matching window ps are (r1_x0, r1_y0). i The pixel coordinates of the upper left corner are (psi_x0, psi_y0), and the star to be aligned is s i The coordinates are (si_x,si_y), then:

[0069] psi_x0-si_x=pr1_x0-r1_x; psi_y0-si_y=pr1_y0-r1_y.

[0070] Thus, the second matching window ps can be obtained i .

[0071] Specifically, the second matching window ps1 corresponding to the star to be aligned s1 should be a rectangle of the same size as the first matching window pr1, and the relative position relationship between the second matching window ps1 and the star to be aligned s1 is equal to the relative position relationship between the first matching window pr1 and the reference star r1. The second matching window ps2 corresponding to the star to be aligned s2 should be a rectangle of the same size as the first matching window pr1, and the relative position relationship between the second matching window ps2 and the star to be aligned s2 is equal to the relative position relationship between the first matching window pr1 and the reference star r1. The second matching window ps3 corresponding to the star to be aligned s3 should be a rectangle of the same size as the first matching window pr1, and the relative position relationship between the second matching window ps3 and the star to be aligned s3 is equal to the relative position relationship between the first matching window pr1 and the reference star r1.

[0072] Step 44: determining a star matching pair based on the reference star, the to-be-registered star, the first matching window, and the second matching window.

[0073] In a specific implementation, the first matching window may be matched and summed with each of the second matching windows, and then a star matching pair may be determined based on the matching and summing result, the reference star and the star to be aligned.

[0074] In a specific implementation, when matching and summing the first matching window with each of the second matching windows respectively, the binary matrix of the first matching window and the binary matrices corresponding to each of the second matching windows can be determined first, and then the binary matrix of the first matching window can be matched and summed with the binary matrices corresponding to each of the second matching windows respectively.

[0075] In a specific implementation, when determining a star matching pair based on the matching sum result, the reference star and the star to be aligned, the matching rate can be first calculated based on the matching sum result, and then the result with the highest matching rate is selected as the first matching rate, and finally the first matching rate is compared with a preset matching rate threshold. If the first matching rate is greater than or equal to the preset matching rate threshold, the star to be aligned corresponding to the first matching rate is used together with the reference star as the star matching pair.

[0076] In a specific implementation, when calculating the matching rate based on the matching sum result, the matching sum result may be divided by the total number of detected stars in the first matching window of the current frame image to obtain the matching rate.

[0077] In a specific implementation, each matching window can be converted into a binary matrix, in which the position where the star is detected is 1, and the other positions are 0. Figure 3 and Figure 5 , the binary matrix corresponding to the first matching window pr1 is as follows Figure 5 As shown in (a), the binary matrix corresponding to the second matching window ps1 is as follows Figure 5 As shown in (b), the binary matrix corresponding to the second matching window ps2 is as follows Figure 5 As shown in (c), the binary matrix corresponding to the second matching window ps3 is as follows Figure 5 (d) as shown.

[0078] All the second matching windows ps1, ps2, and ps3 in the image to be aligned are matched and summed with the first matching window pr1 in the current frame image. The matching and summing means that the corresponding pixels of the two windows are firstly subjected to a binary "AND" operation &, and the results of the AND operation are summed.

[0079] For example, the matching sum of the second matching window ps1 and the first matching window pr1 is: ∑ps1&pr1=7. The matching sum of the second matching window ps2 and the first matching window pr1 is: ∑ps2&pr1=1. The matching sum of the second matching window ps3 and the first matching window pr1 is: ∑ps3&pr1=1. Dividing the matching sum results by the total number of detected stars in the first matching window of the current frame image (i.e., "7"), the corresponding matching rate can be obtained, as shown in Table 1:

[0080] Table 1

[0081] Matching and summing objects Match Rate <![CDATA[ps1 and pr1]]> 7 / 7＝100％ <![CDATA[ps2 and pr1]]> 1 / 7＝14％ <![CDATA[ps3 and pr1]]> 1 / 7＝14％

[0082] In Table 1, the second matching window ps1 has the highest matching rate with the first matching window pr1, so this matching rate is used as the first matching rate. Assuming that the preset matching rate threshold thr=80%, since the first matching rate is greater than 80%, the to-be-aligned star s1 and the reference star r1 corresponding to the first matching rate are a star matching pair.

[0083] It should be noted that if all calculated matching rates are less than the preset matching rate threshold, the matching fails. The larger the preset matching rate threshold thr, the more accurate the matching result, but the more likely it is to cause matching failure. The smaller the preset matching rate threshold thr, the easier it is to successfully match, but the more likely it is to cause inaccurate matching results. The preset matching rate threshold is a manually set parameter that can be determined through experiments based on different cameras. If the matching fails, no noise reduction is performed and the current frame is directly output.

[0084] In a specific implementation, the number of detected stars that must be included in the second matching window is set as parameter m, where m is a natural number. The larger the value of m, the more accurate the matching result, but the greater the possibility of matching failure due to failure to meet the conditions; the smaller the value of m, the less likely the matching will fail, but the accuracy of the matching result will decrease. Usually, m can be set to an integer between 3 and 10.

[0085] Step 45: obtaining a translation vector of the image to be aligned based on the position coordinates of two stars in the star matching pair.

[0086] In a specific implementation, the difference between the coordinates of the reference star and the coordinates of the star to be aligned in the star matching pair can be used as the translation vector of the image to be aligned. For example, when the star to be aligned s1 and the reference star r1 are a star matching pair, the translation vector T = (r1_x-s1_x, r1_y-s1_y), wherein the coordinates of the reference star r1 are (r1_x, r1_y), and the coordinates of the star to be aligned s1 are (s1_x, s1_y).

[0087] It should be noted that the image size in this example is an assumed parameter for the convenience of explanation through the schematic diagram, and the size of an actual image is usually several thousand rows by several thousand columns, or several hundred rows by several hundred columns.

[0088] After obtaining the translation vector, add the translation vector to each pixel coordinate of the image to be aligned, and translate it. The resulting image is the aligned frame image aligned with the current frame image. In the translated image, some pixel coordinates in the translation direction will exceed the image size range before translation. For example, when translating to the upper right, some pixels on the right and top of the image will exceed the image size range. These pixels that exceed the range will be cropped. In the translated image, some pixels in the opposite direction of translation will be left blank. For example, when translating to the upper right, some pixels on the left and bottom of the image will be left blank. These blank pixels will be filled with pixels at the same coordinate position of the current frame image.

[0089] After weighted averaging the current frame image and the aligned frame image, a denoised frame image is obtained. Since the aligned frame image is aligned with the current frame image, the denoised frame image after weighted averaging eliminates the image displacement between frames caused by camera shake, thereby solving the problem of image blur after weighted averaging of the denoised frame image, and the weighted averaging operation improves the image signal-to-noise ratio.

[0090] From the above content, it can be seen that the image processing method in the embodiment of the present invention not only solves the problems of low signal-to-noise ratio and blurred image in the existing image, but also uses the star registration method to perform image alignment, and the alignment effect is better. In addition, when using the star registration method to perform image alignment, only the translation vector is calculated without considering the rotation, which is conducive to real-time image processing and output.

[0091] In order to enable those skilled in the art to better understand and implement the present invention, the user terminal and computer-readable storage medium corresponding to the above method are described in detail below.

[0092] Reference Figure 6 , an embodiment of the present invention provides an image processing device. The image processing device includes: an acquisition unit 61, an alignment unit 62, a noise reduction unit 63 and an output unit 64. Among them:

[0093] The acquisition unit 61 is adapted to acquire a current frame image and an image to be aligned; the current frame image and the image to be aligned are both images taken for the same scene containing a starry sky;

[0094] The alignment unit 62 is adapted to perform starry sky registration on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned;

[0095] The noise reduction unit 63 is adapted to obtain a noise reduction frame image corresponding to the current frame image according to the current frame image and the aligned frame image;

[0096] The output unit 64 is adapted to output images.

[0097] Regarding the acquisition unit 61, the alignment unit 62, the noise reduction unit 63 and the output unit 64, specific implementations may refer to the description of the corresponding steps in the above image processing method.

[0098] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of hardware such as circuits. The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0099] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the steps of the image processing method in the above embodiment, which will not be described in detail.

[0100] In a specific implementation, the computer-readable storage medium may include: ROM, RAM, magnetic disk or optical disk, etc.

[0101] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the image processing method in the above embodiment when running the computer program.

[0102] In a specific implementation, the electronic device includes but is not limited to a mobile phone, a camera, a camcorder, etc.

[0103] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. An image processing method, characterized in that: include: Get the current frame image and the image to be aligned; The current frame image and the image to be aligned are both images taken for the same scene containing stars; Performing starry sky registration on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned; According to the current frame image and the aligned frame image, a denoised frame image corresponding to the current frame image is obtained and outputted; The step of performing star registration on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned includes: dividing the current frame image into a starry sky area and a non-starry sky area; performing star coordinate matching on the starry sky area of ​​the current frame image and the starry sky area of ​​the image to be aligned to obtain a translation vector of the image to be aligned; and translating the image to be aligned according to the translation vector to obtain an aligned frame image corresponding to the image to be aligned; The step of matching the starry sky region of the current frame image with the starry sky region of the image to be aligned to obtain the translation vector of the image to be aligned comprises: determining a first matching window in the current frame image; four edges of the first matching window, namely, upper, lower, left and right, are all spaced apart from the corresponding edges of the current frame image by a preset margin; determining a reference star based on the first matching window and the starry sky region of the current frame image; determining a number of stars to be aligned in the image to be aligned and a second matching window corresponding to each star to be aligned based on the first matching window, the reference star, the image to be aligned and the preset margin; the size of the first matching window is the same as that of the second matching window; the first relative position relationship between each second matching window and the corresponding star to be aligned is the same as the second relative position relationship between the first matching window and the reference star; determining a star matching pair based on the reference star, the star to be aligned, the first matching window and the second matching window; and obtaining the translation vector of the image to be aligned based on the position coordinates of two stars in the star matching pair; The step of determining a star matching pair based on the reference star, the star to be aligned, the first matching window and the second matching window comprises: matching and summing the first matching window with each of the second matching windows respectively; calculating a matching rate based on the matching and summing results; selecting the result with the highest matching rate as the first matching rate; comparing the first matching rate with a preset matching rate threshold, and if the first matching rate is greater than or equal to the preset matching rate threshold, taking the star to be aligned corresponding to the first matching rate and the reference star as the star matching pair; The matching and summing the first matching window with each of the second matching windows respectively includes: determining a binary matrix of the first matching window and a binary matrix corresponding to each of the second matching windows; matching and summing the binary matrix of the first matching window with the binary matrix corresponding to each of the second matching windows respectively; The matching sum result is divided by the total number of detected stars in the first matching window of the current frame image to obtain the matching rate.

2. The image processing method according to claim 1, characterized in that: The step of determining a reference star based on the first matching window and the starry sky area of ​​the current frame image includes: Performing star detection on the starry sky area of ​​the current frame image within the first matching window to obtain a number of detected stars of the current frame image; From the detected stars in the current frame image, select the star with the greatest brightness as the reference star.

3. The image processing method according to claim 1, characterized in that: The step of determining a plurality of stars to be registered in the image to be registered and a second matching window corresponding to each of the stars to be registered based on the first matching window, the reference star, the image to be registered and the preset margin comprises: Determine a first translation window based on the first matching window, the reference star and the preset margin; Based on the image to be aligned, performing star detection on the starry sky area of ​​the image to be aligned to obtain a plurality of detected stars of the image to be aligned; Using the area in the image to be aligned, which is at the same position as the first translation window, as the second translation window; In the second translation window, select N detected stars of the to-be-aligned image with the highest brightness as the to-be-aligned stars, where N is a positive integer greater than or equal to 1; Based on the stars to be registered, the reference stars and the first matching window, a second matching window corresponding to each of the stars to be registered is determined.

4. The image processing method according to claim 3, characterized in that: The distances from the upper, lower, left and right sides of the first translation window to the reference star are equal to the preset margin.

5. The image processing method according to claim 1, wherein: The obtaining the translation vector of the image to be aligned based on the position coordinates of two stars in the star matching pair includes: The stars are matched and centered, and the difference between the coordinates of the reference star and the coordinates of the star to be aligned is used as the translation vector of the image to be aligned.

6. The image processing method according to claim 1, characterized in that: The step of obtaining the noise reduction frame image corresponding to the current frame image according to the current frame image and the aligned frame image includes: A weighted average operation is performed on the current frame image and the aligned frame image to obtain a denoised frame image corresponding to the current frame image.

7. An image processing device, characterized in that: include: An acquisition unit, adapted to acquire a current frame image and an image to be aligned; The current frame image and the image to be aligned are both images taken for the same scene containing stars; an alignment unit, adapted to perform starry sky registration on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned; A denoising unit, adapted to obtain a denoised frame image corresponding to the current frame image according to the current frame image and the aligned frame image; an output unit, adapted to output an image; The step of performing star registration on the image to be aligned and the current frame image to obtain an aligned frame image corresponding to the image to be aligned includes: dividing the current frame image into a starry sky area and a non-starry sky area; performing star coordinate matching on the starry sky area of ​​the current frame image and the starry sky area of ​​the image to be aligned to obtain a translation vector of the image to be aligned; and translating the image to be aligned according to the translation vector to obtain an aligned frame image corresponding to the image to be aligned; The step of matching the starry sky region of the current frame image with the starry sky region of the image to be aligned to obtain the translation vector of the image to be aligned comprises: determining a first matching window in the current frame image; four edges of the first matching window, namely, upper, lower, left and right, are all spaced apart from the corresponding edges of the current frame image by a preset margin; determining a reference star based on the first matching window and the starry sky region of the current frame image; determining a number of stars to be aligned in the image to be aligned and a second matching window corresponding to each star to be aligned based on the first matching window, the reference star, the image to be aligned and the preset margin; the size of the first matching window is the same as that of the second matching window; the first relative position relationship between each second matching window and the corresponding star to be aligned is the same as the second relative position relationship between the first matching window and the reference star; determining a star matching pair based on the reference star, the star to be aligned, the first matching window and the second matching window; and obtaining the translation vector of the image to be aligned based on the position coordinates of two stars in the star matching pair; The step of determining a star matching pair based on the reference star, the star to be aligned, the first matching window and the second matching window comprises: matching and summing the first matching window with each of the second matching windows respectively; calculating a matching rate based on the matching and summing results; selecting the result with the highest matching rate as the first matching rate; comparing the first matching rate with a preset matching rate threshold, and if the first matching rate is greater than or equal to the preset matching rate threshold, taking the star to be aligned corresponding to the first matching rate and the reference star as the star matching pair; The matching and summing the first matching window with each of the second matching windows respectively includes: determining a binary matrix of the first matching window and a binary matrix corresponding to each of the second matching windows; matching and summing the binary matrix of the first matching window with the binary matrix corresponding to each of the second matching windows respectively; The matching sum result is divided by the total number of detected stars in the first matching window of the current frame image to obtain the matching rate.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 6.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor runs the computer program, the steps of the method according to any one of claims 1 to 6 are performed.

Images