Image processing method, device, electronic device, storage medium and program product

Abstract

The present application provides an image processing method, apparatus, electronic device, storage medium, and program product. The method includes: acquiring at least two frames of initial images; determining a reference image and a target image from the at least two frames of initial images; acquiring an alignment detection result of the target image based on the target image and the reference image; the alignment detection result of the target image is used to indicate whether the target image and the reference image are aligned; when the alignment detection result indicates that the target image and the reference image are not aligned, acquiring a jitter pattern of the target image relative to the reference image; aligning the target image with the reference image according to an image alignment method corresponding to the jitter pattern to obtain a target image aligned with the reference image; performing image fusion on the reference image and the target image aligned with the reference image to obtain a target captured image; and outputting the target captured image. The present application improves image alignment efficiency and the clarity of the target captured image.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to an image processing method, device, electronic device, storage medium and program product. Background Art

[0002] When taking a photo, a camera can capture multiple frames continuously and then fuse them to create a single frame of the target image to improve its clarity. However, with handheld cameras like mobile phones, due to hand-held jitter, these multiple frames may often be misaligned due to issues such as translation and rotation.

[0003] However, if there is a misalignment problem between multiple frames of images, ghosting may occur in the target captured image obtained by fusion of the multiple frames of images. Summary of the Invention

[0004] The present application provides an image processing method, device, electronic device, storage medium and program product to solve the problem of misalignment of multiple frames of images.

[0005] In a first aspect, the present application provides an image processing method, the method comprising:

[0006] Acquire at least two frames of initial images;

[0007] Determining a reference image and a target image from the at least two frames of initial images;

[0008] Acquire an alignment detection result of the target image according to the target image and the reference image; the alignment detection result of the target image is used to indicate whether the target image is aligned with the reference image;

[0009] When the alignment detection result is used to indicate that the target image and the reference image are not aligned, obtaining a jitter pattern of the target image relative to the reference image;

[0010] aligning the target image with the reference image according to the image alignment method corresponding to the dithering pattern to obtain a target image aligned with the reference image;

[0011] Performing image fusion on the reference image and the target image aligned with the reference image to obtain a target captured image;

[0012] The target captured image is output.

[0013] Optionally, obtaining an alignment detection result of the target image according to the target image and the reference image includes:

[0014] Obtaining a first motion offset of the target image relative to the reference image in a first motion direction and a second motion offset in a second motion direction; the first motion direction and the second motion direction are not on the same straight line;

[0015] An alignment detection result of the target image is determined according to the first motion offset and the second motion offset.

[0016] Optionally, obtaining a first motion offset of the target image relative to the reference image in a first motion direction and a second motion offset in a second motion direction includes:

[0017] Performing image division on the target image and the reference image according to a preset image division method to obtain N first sub-images of the target image and N second sub-images of the reference image, where N is an integer greater than or equal to 2; any first sub-image corresponds to one second sub-image;

[0018] Obtaining, based on the M first sub-images and the M second sub-images corresponding to the M first sub-images, M first sub-motion offsets of the target image in a first motion direction and M second sub-motion offsets in a second motion direction relative to the reference image, where M is an integer greater than or equal to 2 and less than or equal to N;

[0019] Obtaining the first motion offset according to the M first sub-motion offsets;

[0020] The second motion offset is acquired according to the M second sub-motion offsets.

[0021] Optionally, the first motion direction is a positive direction indicated by an x-axis in an image coordinate system, and the second motion direction is a positive direction indicated by a y-axis in the image coordinate system. Obtaining, based on the M first sub-images and M second sub-images corresponding to the M first sub-images, M first sub-motion offsets of the target image relative to the reference image in the first motion direction includes:

[0022] For any first sub-image among the M first sub-images, obtain the sum of pixel values ​​in each row of the first sub-image to obtain a first projection vector of the first sub-image on the y-axis in the image coordinate system;

[0023] Performing an offset calculation on the first projection vector according to a preset offset step size and a preset maximum offset amount to obtain a plurality of first offset vectors corresponding to the first projection vector;

[0024] Obtaining a first absolute error and a SAD vector of the first sub-image relative to the second sub-image based on each of the first offset vectors and a second projection vector of the second sub-image corresponding to the first sub-image on the y-axis in the image coordinate system;

[0025] According to the first SAD vector, a first sub-motion offset of the target image relative to the reference image in a first motion direction is obtained.

[0026] Optionally, acquiring the first motion offset according to the M first sub-motion offsets includes:

[0027] taking an average of the absolute values ​​of the M first sub-motion offsets as the first motion offset;

[0028] or,

[0029] The acquiring the second motion offset according to the M second sub-motion offsets includes:

[0030] An average of the absolute values ​​of the M second sub-motion offsets is used as the second motion offset.

[0031] Optionally, acquiring a jitter pattern of the target image relative to the reference image includes:

[0032] Obtaining first variances corresponding to the M first sub-motion offsets and second variances corresponding to the M second sub-motion offsets;

[0033] If both the first variance and the second variance are smaller than a preset variance threshold, determining that the jitter pattern of the target image relative to the reference image is a first jitter pattern, where the first jitter pattern indicates that the target image is shifted relative to the reference image;

[0034] or,

[0035] If there is a variance between the first variance and the second variance that is greater than or equal to a preset variance threshold, it is determined that the jitter pattern of the target image relative to the reference image is a second jitter pattern, and the second jitter pattern indicates that the target image is rotated or distorted relative to the reference image.

[0036] Optionally, determining the alignment detection result of the target image according to the first motion offset and the second motion offset includes:

[0037] If both the first motion offset and the second motion offset are smaller than a preset motion offset threshold, it is determined that the alignment detection result indicates that the target image and the reference image are aligned.

[0038] Optionally, acquiring at least two frames of initial images includes:

[0039] In response to a user-triggered request to enter a photo-taking interface, a preview stream is captured; the preview stream includes at least one frame of a first initial image;

[0040] In response to a request to start taking a photo triggered by a user, performing image acquisition to obtain a second initial image;

[0041] The at least two frames of initial images are acquired based on the preview stream and the second initial image.

[0042] In a second aspect, the present application provides an image processing device, comprising:

[0043] An acquisition module, configured to acquire at least two frames of initial images;

[0044] a processing module configured to determine a reference image and a target image from the at least two frames of initial images; obtain an alignment detection result of the target image based on the target image and the reference image; obtain a jitter pattern of the target image relative to the reference image when the alignment detection result is used to indicate that the target image is not aligned with the reference image; align the target image with the reference image according to an image alignment method corresponding to the jitter pattern to obtain a target image aligned with the reference image; perform image fusion on the reference image and the target image aligned with the reference image to obtain a target captured image; wherein the alignment detection result of the target image is used to indicate whether the target image is aligned with the reference image;

[0045] An output module is used to output the target captured image.

[0046] In a third aspect, the present application provides an electronic device, comprising: at least one processor and a memory;

[0047] The memory stores computer-executable instructions;

[0048] The at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the method according to any one of the first aspects.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed by a processor, the method described in any one of the first aspects is implemented.

[0050] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the method described in any one of the first aspects when executed by a processor.

[0051] The image processing method, device, electronic device, storage medium and program product provided by the present application determine a reference image and a target image from at least two frames of initial images, so that the target image can be aligned with the reference image based on the reference image. The alignment detection result of the target image can be determined by the target image and the reference image, thereby determining whether image alignment processing is required based on the captured image. When the target image is not aligned with the reference image, by obtaining the jitter pattern of the target image relative to the reference image, the image alignment method for aligning the target image with the reference image can be determined based on the jitter pattern. Through the above method, different image alignment methods can be used for different jitter patterns, thereby improving the flexibility of the image alignment processing and thus improving the efficiency of the image alignment processing. Then, by fusing the aligned target image and the reference image, the target captured image can be obtained. Through the above method, image alignment can be achieved without the need for sensors such as gyroscopes, micro inertial sensors, or accelerometers, so that the method can be applied to camera equipment that is not equipped with the above sensors, and improves the clarity and efficiency of obtaining the target captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] Figure 1 This is a schematic diagram of a scene of taking photos with a mobile phone;

[0054] Figure 2 A flowchart of an image processing method provided in this application;

[0055] Figure 3 A schematic diagram of changes in the user interaction interface provided by this application;

[0056] Figure 4 A flowchart of a method for obtaining an alignment detection result of a target image based on a target image and a reference image provided in this application;

[0057] Figure 5 A schematic diagram of image division provided in this application;

[0058] Figure 6 A flowchart of another image processing method provided in this application;

[0059] Figure 7 A schematic structural diagram of an image processing device provided in this application;

[0060] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application.

[0061] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0062] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0063] In related technologies, in order to improve the clarity of the target image, the camera device can first collect multiple frames of images, and then perform image fusion on the multiple frames of images as the target image. In fact, taking the camera device as a mobile phone as an example, Figure 1 This is a schematic diagram of a scene where a mobile phone is used to take photos. Figure 1 As shown, when a user uses a mobile phone to take a photo, the user usually holds the mobile phone to take the photo.

[0064] However, when holding a camera, hand shaking can cause the camera to shake, leading to misalignment issues such as translation and rotation between the multiple frames captured by the camera. Misalignment between these multiple frames can lead to ghosting and other issues in the target image obtained by fusion of these multiple frames, resulting in poor clarity in the target image.

[0065] Some embodiments propose using data collected by sensors such as gyroscopes, micro-inertial sensors, or accelerometers installed in the camera device to determine the offset between the multiple frames, and then aligning the multiple frames based on this offset. However, the above methods are only applicable to camera devices equipped with such sensors. For camera devices without such sensors, existing technologies still cannot solve the problem of how to align multiple frames, and the target image captured based on the misaligned multiple frames still has poor clarity.

[0066] Taking into account the above-mentioned problems existing in the prior art, this application proposes a method for image alignment that is not based on sensors such as gyroscopes, micro inertial sensors, or accelerometers. For photographic devices that are not equipped with the above-mentioned sensors, image alignment processing can also be implemented, thereby improving the clarity of the target captured image.

[0067] It should be understood that Figure 1 This is merely an exemplary description of a camera device using a mobile phone as an example. This application does not limit the type of the above-mentioned camera device. For example, the camera device may also be a tablet computer, a camera, a video camera, or other camera device.

[0068] In addition, it should be understood that the present application does not limit the application scenarios of the above method. Optionally, the above method can be applied to any electronic device with processing functions. The electronic device can be any type of photographing device, or the electronic device can also be, for example, an electronic device such as a terminal or a server. Taking the electronic device as a photographing device as an example, the photographing device can perform alignment processing after taking multiple frames of images when the multiple frames of images are not aligned. Taking the electronic device as a terminal or a server as an example, the terminal or the server can, for example, receive multiple frames of images taken by the photographing device, and obtain a target captured image based on the aligned multiple frames of images and send it to the photographing device.

[0069] The following detailed description of the technical solution of the present application is provided in conjunction with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0070] Figure 2 This is a flow chart of an image processing method provided by this application. Figure 2 As shown, the method includes the following steps:

[0071] S101: Acquire at least two frames of initial images.

[0072] Optionally, the at least two frames of initial images may be continuous multiple frames of initial images taken by a photographing device, or may be discontinuous multiple frames of initial images, which is not limited in the present application.

[0073] Taking the electronic device as a camera as an example, as one possible implementation, the electronic device can respond to a user-triggered request to enter a camera interface by capturing a preview stream including at least one first initial image. Then, in response to a user-triggered request to start capturing, the electronic device can perform image capture to obtain a second initial image. Based on the preview stream and the second initial image, the electronic device can acquire at least two initial images.

[0074] For example, the electronic device is a mobile phone or a tablet computer. Figure 3 This is a schematic diagram of a user interaction interface change provided by this application. Figure 3 As shown, for example, the electronic device may have a pre-installed photo-taking application. The user may click an icon 1 corresponding to the photo-taking application to trigger a request to enter the photo-taking interface. In response to the user-triggered request to enter the photo-taking interface, the electronic device enters the photo-taking interface and begins capturing a preview stream.

[0075] Optional, such as Figure 3 As shown, the photo taking interface may include a photo taking control, and a user may trigger a request to start taking a photo by clicking the photo taking control. In response to the user-triggered request to start taking a photo, the electronic device may perform image capture to obtain a second initial image. The electronic device may then acquire at least two initial image frames based on the preview stream and the second initial image.

[0076] Alternatively, still taking the above-mentioned electronic device as a photographing device as an example, the electronic device may also capture at least two frames of initial images in response to a request to start photographing triggered by the user.

[0077] Taking the above-mentioned electronic device as a terminal or a server as an example, the electronic device may also receive at least two frames of initial images from a photographing device.

[0078] S102: Determine a reference image and a target image from at least two frames of initial images.

[0079] The reference image serves as a benchmark for aligning the target images. That is, to determine whether alignment is required for each target image is to determine whether each target image is aligned with the reference image.

[0080] Optionally, the electronic device may obtain the clarity of each initial image after obtaining the above-mentioned at least two frames of initial images. Then, the electronic device may, for example, use the initial image with the highest clarity as a reference image to improve the accuracy of determining whether the target image needs to be aligned based on the reference image, and to improve the accuracy of aligning the target image. Alternatively, the electronic device may, for example, use all initial images with a clarity greater than or equal to a preset clarity threshold as candidate reference images, and randomly determine a candidate reference image from multiple candidate reference images as a reference image. It should be understood that this application does not limit how the electronic device obtains the clarity of the initial image. Optionally, reference may be made to the existing implementation method of obtaining the clarity of the image, which will not be repeated here.

[0081] Alternatively, the electronic device may determine the reference image according to the arrangement order of the at least two frames of initial images. For example, the electronic device may use the first frame of the at least two frames of initial images, or the last frame of the at least two frames of initial images as the reference image.

[0082] Alternatively, the electronic device may also randomly determine an initial image frame from the at least two initial image frames as a reference image.

[0083] After the reference image is determined, optionally, each initial image in the at least two frames of initial images except the reference image may be used as a target image.

[0084] S103 : Acquire an alignment detection result of the target image according to the target image and the reference image.

[0085] The alignment detection result of the target image is used to indicate whether the target image is aligned with the reference image.

[0086] As a possible implementation method, the electronic device can, for example, obtain the horizontal motion offset of the target image relative to the reference image in the image coordinate system, or the vertical motion offset. Taking the example of the electronic device obtaining the horizontal motion offset of the target image relative to the reference image, if the horizontal motion offset is greater than or equal to a preset offset threshold, it means that the target image has a large offset relative to the reference image, and the electronic device can determine that the alignment detection result is used to characterize the misalignment between the target image and the reference image. If the horizontal motion offset is less than the preset offset threshold, it means that the target image has a small offset relative to the reference image, and the electronic device can determine that the alignment detection result is used to characterize the alignment between the target image and the reference image.

[0087] Alternatively, the electronic device can also obtain the horizontal motion offset of the target image relative to the reference image in the image coordinate system, as well as the vertical motion offset, and determine the alignment detection result of the target image based on the horizontal motion offset and the vertical motion offset to improve the accuracy of determining the alignment detection result of the target image.

[0088] If the alignment detection result of the target image is used to represent that the target image and the reference image are aligned, optionally, the electronic device can, for example, execute step S106 to obtain the target captured image based on the reference image and the target image, so as to improve the efficiency of the electronic device in obtaining the target captured image based on the fusion result of the target image and the reference image.

[0089] If the alignment detection result of the target image is used to indicate that the target image is not aligned with the reference image, the electronic device may execute step S104 to determine a jitter pattern of the target image relative to the reference image.

[0090] S104: Obtain a dithering pattern of the target image relative to the reference image.

[0091] Optionally, the aforementioned jitter pattern may be, for example, any one of the following: translation, rotation, and distortion. For example, if the target image is offset relative to the reference image in only one direction of motion, the electronic device may determine that the jitter pattern of the target image relative to the reference image is translation. Alternatively, if the target image is offset relative to the reference image in multiple directions of motion that are not on the same straight line, the electronic device may determine that the jitter pattern of the target image relative to the reference image is rotation or distortion.

[0092] S105 , aligning the target image with the reference image according to an image alignment method corresponding to the dithering pattern to obtain a target image aligned with the reference image.

[0093] It should be understood that the present application does not limit the image alignment method corresponding to each jitter mode. Optionally, for example, any existing image alignment method can be referenced. For example, the image alignment method corresponding to the translation jitter mode can be an image alignment method with higher image processing efficiency to improve the efficiency of image alignment. The image alignment method corresponding to the rotation and distortion jitter mode can be an image alignment method with higher image alignment accuracy to improve the accuracy of image alignment. The image alignment methods corresponding to the rotation and distortion jitter modes can be the same or different.

[0094] Optionally, the electronic device may determine the image alignment method corresponding to the dither pattern based on the dither pattern of the target image relative to the reference image, and the mapping relationship between the dither pattern and the image alignment method. The mapping relationship between the dither pattern and the image alignment method may be pre-stored in the electronic device by the user. For example, the mapping relationship between the dither pattern and the image alignment method may be as shown in Table 1 below:

[0095] Table 1

[0096] Dithering Mode Image alignment Dither Mode 1 Image alignment 1 Dither Mode 2 Image Alignment 2 Dither Mode 3 Image alignment 3

[0097] For example, if the electronic device determines that the jitter pattern of the target image relative to the reference image is jitter pattern 1, then according to the mapping relationship shown in Table 1, the electronic device can align the target image with the reference image through image alignment method 1 to obtain a target image aligned with the reference image.

[0098] S106 : Perform image fusion on the reference image and the target image aligned with the reference image to obtain a target captured image.

[0099] In some embodiments, the electronic device may obtain one or more frames of target captured images after performing image fusion on the reference image and the target image aligned with the reference image.

[0100] It should be understood that this application does not limit how to fuse a reference image and a target image aligned with the reference image. Alternatively, any existing image fusion method can be used to fuse the reference image and the target image aligned with the reference image to obtain the target captured image, which will not be described in detail here.

[0101] S107: Output the captured target image.

[0102] As a possible implementation method, the electronic device can output the target captured image through the display device installed in the electronic device. Figure 3 Taking the interface shown as an example, after the electronic device responds to the user's request to start taking a photo, it can execute steps S102-S106 and then display the target image on the photo interface. Furthermore, the electronic device can also store the target image.

[0103] Taking the electronic device as a terminal or a server as an example, after acquiring a target shot image, the electronic device may output the target shot image to a photographing device.

[0104] In this embodiment, by determining a reference image and a target image from at least two initial image frames, the target image can be aligned with the reference image based on the reference image. The target image alignment detection result can be determined using the target image and the reference image, thereby determining whether image alignment processing is required based on the captured image. When the target image is not aligned with the reference image, the jitter pattern of the target image relative to the reference image can be obtained, and an image alignment method for aligning the target image with the reference image can be determined based on the jitter pattern. This method allows different image alignment methods to be used for different jitter patterns, thereby increasing the flexibility and efficiency of the image alignment process. The aligned target image and reference image are then fused to obtain a captured target image. This method allows image alignment to be achieved without the need for sensors such as gyroscopes, micro inertial sensors, or accelerometers, making it applicable to camera devices that are not equipped with such sensors and improving the clarity and efficiency of capturing captured target images.

[0105] The following describes in detail how the electronic device obtains the alignment detection result of the target image based on the target image and the reference image:

[0106] Figure 4This is a flow chart of a method for obtaining alignment detection results of a target image based on a target image and a reference image provided by this application. Figure 4 As shown, as a possible implementation, the aforementioned step S103 may include the following steps:

[0107] S201 : Obtain a first motion offset of a target image relative to a reference image in a first motion direction and a second motion offset in a second motion direction.

[0108] The first and second motion directions are not co-linear. Optionally, the first and second motion directions may be, for example, two mutually perpendicular motion directions. As a possible implementation, the first motion direction may be the positive direction indicated by the x-axis in the image coordinate system, and the second motion direction may be the positive direction indicated by the y-axis in the image coordinate system. Alternatively, the angle between the first and second motion directions may be an acute angle or an obtuse angle, etc., which is not limited in this application.

[0109] As a possible implementation method, the electronic device can, for example, first divide the target image and the reference image into multiple image blocks, and then obtain a first motion offset of the target image relative to the reference image in a first motion direction and a second motion offset in a second motion direction based on the motion offset of each image block in the target image relative to the image block in the reference image.

[0110] Optionally, the electronic device may, for example, perform image segmentation on the target image and the reference image according to a preset image segmentation method to obtain N first sub-images of the target image and N second sub-images of the reference image. N is an integer greater than or equal to 2. Each of the first sub-images corresponds to one second sub-image.

[0111] For example, the above-mentioned preset image division method can be, for example, to evenly divide the target image and the reference image into N sub-images. In this application, the shapes of the first sub-image and the second sub-image are not limited. For example, taking the above-mentioned N equal to 9 as an example, Figure 5 This is a schematic diagram of image division provided by this application. Figure 5 As shown, the electronic device can evenly divide the target image and the reference image into 9 sub-images.

[0112] The electronic device can then determine M first sub-images from the N first sub-images and M second sub-images from the N second sub-images, so as to obtain, based on the M first sub-images and the M second sub-images corresponding to the M first sub-images, M first sub-motion offsets of the target image in the first motion direction and M second sub-motion offsets in the second motion direction relative to the reference image. M is an integer greater than or equal to 2 and less than or equal to N. Through the above method, the electronic device can obtain the first motion offset and the second motion offset based on a smaller number of sub-images, thereby reducing sub-image redundancy, improving the efficiency of the electronic device in obtaining the first motion offset and the second motion offset, and thereby improving the efficiency of the electronic device in performing image alignment processing.

[0113] For example, still Figure 5 For example, the electronic device may select five first sub-images, namely, first sub-image 1, first sub-image 3, first sub-image 5, first sub-image 7, and first sub-image 9, from the nine first sub-images of the target image, and obtain, based on the five first sub-images and the five second sub-images corresponding to the five first sub-images, M first sub-motion offsets of the target image in a first motion direction and M second sub-motion offsets in a second motion direction relative to the reference image. For example, the electronic device may obtain one first sub-motion offset and one second sub-motion offset based on first sub-image 1 and second sub-image 1.

[0114] The electronic device may obtain a first motion offset according to the M first sub-motion offsets, and obtain a second motion offset according to the M second sub-motion offsets.

[0115] Optionally, the electronic device may, for example, use the average of the absolute values ​​of the M first sub-motion offsets as the first motion offset. Calculating the first motion offset by the average value takes into account the impact of each of the M first sub-motion offsets on the first motion offset, thereby improving the accuracy of the electronic device in determining the first motion offset. Alternatively, the electronic device may, for example, use the mode or median of the absolute values ​​of the M first sub-motion offsets as the first motion offset.

[0116] For the second motion offset, the electronic device may also use the average of the absolute values ​​of the M second sub-motion offsets as the second motion offset. Alternatively, the electronic device may use the mode or median of the absolute values ​​of the M second sub-motion offsets as the second motion offset.

[0117] As another possible implementation, the electronic device may further, for example, obtain the first motion offset and the second motion offset of the target image relative to the reference image directly based on the N first sub-images and the N second sub-images corresponding to the N first sub-images after dividing both the target image and the reference image into N image blocks. Optionally, the specific implementation of the electronic device obtaining the first motion offset and the second motion offset of the target image relative to the reference image based on the N first sub-images and the N second sub-images corresponding to the N first sub-images can refer to the method described in the aforementioned embodiment and is not further described here.

[0118] S202: Determine an alignment detection result of the target image according to the first motion offset and the second motion offset.

[0119] Optionally, if both the first motion offset and the second motion offset are less than a preset motion offset threshold, indicating that the offset of the target image relative to the reference image in the first motion direction is small, and the offset of the target image relative to the reference image in the second motion direction is also small, the electronic device may determine that the alignment detection result indicates that the target image and the reference image are aligned.

[0120] The preset motion offset threshold may be pre-stored in the electronic device by a user. Optionally, the preset motion offset threshold corresponding to the first motion offset may be the same as or different from the preset motion offset threshold corresponding to the second motion offset.

[0121] If one of the first motion offset and the second motion offset is greater than or equal to a preset motion offset threshold, it indicates that the target image is significantly offset relative to the reference image in the first motion direction, and / or the target image is significantly offset relative to the reference image in the second motion direction. The electronic device may then determine that the alignment detection result indicates misalignment between the target image and the reference image.

[0122] In this embodiment, the alignment detection result of the target image is determined by the first motion offset of the target image relative to the reference image in the first motion direction and the second motion offset in the second motion direction, thereby improving the accuracy of whether image alignment is needed, and further improving the clarity of the target captured image obtained based on the aligned target image and the reference image.

[0123] When determining that the above-mentioned alignment detection result is used to characterize that the target image and the reference image are not aligned, the electronic device can obtain the jitter pattern of the target image relative to the reference image. As a possible implementation method, the electronic device can, for example, determine the jitter pattern of the target image relative to the reference image based on the first variance corresponding to the above-mentioned M first sub-motion offsets and the second variance corresponding to the M second sub-motion offsets. Among them, the first variance corresponding to the above-mentioned M first sub-motion offsets can be used to characterize the degree of deviation between the first sub-motion offsets. The larger the first variance, the greater the degree of deviation between the first sub-motion offsets, and therefore, the higher the possibility that the target image is rotated and distorted relative to the reference image in the first motion direction. The reason for obtaining the second variance is the same as that for the first variance and will not be repeated.

[0124] Optionally, after obtaining the M first sub-motion offsets and the M second sub-motion offsets, the electronic device may obtain a first variance corresponding to the M first sub-motion offsets and a second variance corresponding to the M second sub-motion offsets. Optionally, the electronic device may obtain the first variance and the second variance using a preset variance calculation formula.

[0125] If both the first variance and the second variance are less than a preset variance threshold, it indicates that the target image is less likely to be rotated or distorted relative to the reference image in the first and second motion directions. The electronic device can then determine that the jitter pattern of the target image relative to the reference image is a first jitter pattern. The first jitter pattern indicates that the target image is translated relative to the reference image.

[0126] If a variance between the first and second variances is greater than or equal to a preset variance threshold, it indicates that the target image is likely to be rotated or distorted relative to the reference image in the first and / or second motion directions. The electronic device may then determine that the jitter pattern of the target image relative to the reference image is a second jitter pattern. The second jitter pattern indicates that the target image is rotated or distorted relative to the reference image.

[0127] The preset variance threshold may be pre-stored in the electronic device by the user. Optionally, the preset variance threshold corresponding to the first variance and the preset variance threshold corresponding to the second variance may be the same or different.

[0128] The following describes in detail how an electronic device obtains M first sub-motion offsets of a target image relative to a reference image in the first motion direction and M second sub-motion offsets in the second motion direction based on M first sub-images and M second sub-images corresponding to the M first sub-images, taking the aforementioned first motion direction as the positive direction indicated by the x-axis in the image coordinate system and the second motion direction as the positive direction indicated by the y-axis in the image coordinate system as an example:

[0129] First, regarding how to obtain M first sub-motion offsets based on M first sub-images and M second sub-images corresponding to the M first sub-images, illustratively, the electronic device can obtain any first sub-motion offset through the following steps:

[0130] Step A: For any first sub-image among the M first sub-images, the electronic device may obtain the sum of pixel values ​​of each row of the first sub-image to obtain a first projection vector of the first sub-image on the y-axis in the above-mentioned image coordinate system.

[0131] In step B, the electronic device may perform an offset calculation on the first projection vector according to a preset offset step size and a preset maximum offset, to obtain a plurality of first offset vectors corresponding to the first projection vector.

[0132] The preset offset step size and the preset maximum offset may be pre-stored by the user in the electronic device. Optionally, the electronic device may offset the first projection vector in the first direction of motion and may also offset the first projection vector in a direction opposite to the first direction of motion to obtain multiple first offset vectors corresponding to the first projection vector.

[0133] Step C: Obtain a first Sum of Absolute Differences (SAD) vector of the first sub-image relative to the second sub-image based on the first offset vectors and the second projection vector of the second sub-image corresponding to the first sub-image on the y-axis in the image coordinate system.

[0134] Before step C, the electronic device may first obtain a second projection vector of each second sub-image on the y-axis in the image coordinate system. The specific implementation method for the electronic device to obtain the second projection vector of the second sub-image on the y-axis in the image coordinate system can refer to the method described in step A above and is not repeated here.

[0135] Optionally, the electronic device can, for example, input the above-mentioned first offset vectors and the second projection vector of the second sub-image corresponding to the first sub-image on the y-axis in the image coordinate system into a preset SAD algorithm to obtain the first SAD vector of the first sub-image relative to the second sub-image.

[0136] Step D: Obtain a first sub-motion offset of the target image relative to the reference image in a first motion direction according to the first SAD vector.

[0137] Optionally, the electronic device may, for example, use the minimum value in the first SAD vector as a first sub-motion offset of the target image relative to the reference image in the first motion direction. Following the above steps, the electronic device may obtain M first sub-motion offsets of the target image relative to the reference image in the first motion direction.

[0138] Regarding how to obtain M second sub-motion offsets based on the M first sub-images and the M second sub-images corresponding to the M first sub-images, reference may optionally be made to the method for obtaining M first sub-motion offsets described in the preceding embodiment. For example, for any first sub-image among the M first sub-images, the electronic device may obtain the sum of the pixel values ​​in each column of the first sub-image to obtain a third projection vector of the first sub-image on the x-axis in the aforementioned image coordinate system.

[0139] The electronic device may then perform an offset calculation on the third projection vector based on a preset offset step size and a preset maximum offset to obtain multiple second offset vectors corresponding to the third projection vector. A second SAD vector of the first sub-image relative to the second sub-image may be obtained based on each of the second offset vectors and a fourth projection vector of the second sub-image corresponding to the first sub-image on the x-axis in the image coordinate system.

[0140] Then, the electronic device may obtain a second sub-motion offset of the target image relative to the reference image in the second motion direction according to the second SAD vector.

[0141] Taking the electronic device as a photographing device as an example, Figure 6 This is a flow chart of another image processing method provided by this application. Figure 6 As shown, the camera device can first obtain at least two frames of initial images from the preview stream of the camera device. It should be understood that the initial images in the preview stream mentioned here may include the initial images captured by the camera device at the moment when the user triggers the request to start taking pictures.

[0142] Then, the camera device can determine the alignment detection result of the target image based on the at least two frames of initial image. The specific implementation method can refer to the method described in the above embodiment and will not be repeated here.

[0143] If the above alignment detection result is used to indicate that the target image and the reference image are aligned, the photographing device can determine that the target image and the reference image are aligned, and then perform image fusion based on the aligned target image and the reference image to obtain the target captured image.

[0144] If the alignment detection result indicates that the target image is misaligned with the reference image, the camera can determine whether the jitter pattern of the target image relative to the reference image is a translation jitter pattern, a rotation jitter pattern, or a twist jitter pattern. The specific determination method can refer to the method described in the previous embodiment and will not be repeated here.

[0145] If the target image's jitter pattern relative to the reference image is a translational jitter pattern, a translational alignment method is used to align the target image with the reference image. This translational alignment method may, for example, use a first motion offset of the target image relative to the reference image in a first motion direction and a second motion offset in a second motion direction as a global translational motion vector between the input image frames. Using this global translational motion vector, the target image is compensated for motion in the first and second motion directions to achieve rapid alignment of the target image with the reference image.

[0146] If the jitter pattern of the target image relative to the reference image is a rotation or distortion jitter pattern, the target image is aligned with the reference image using a rotation or distortion alignment method, wherein the rotation or distortion alignment method can be, for example:

[0147] Obtain a global homography matrix between the target image and the reference image. Using this global homography matrix, perform a homography transformation on the target image to obtain a target image aligned with the reference image. The method for obtaining the global homography matrix between the target image and the reference image by the camera device can refer to any method described in the prior art.

[0148] Exemplarily, the camera device may first obtain feature points in the target image and the reference image. Optionally, the feature points may be selected from, but not limited to, fast feature points and corresponding feature descriptor (Oriented FAST and Rotated BRIEF, ORB) feature points. Then, based on the feature points, the above-mentioned global homography matrix is ​​obtained. Exemplarily, the global homography matrix may be, for example, as shown in the following formula (1):

[0149]

[0150] Where H represents the global homography matrix, and H1-H9 are the global homography matrix elements obtained based on the above feature points.

[0151] Assume that the coordinates of each pixel point on the target image are (x, y), and the coordinates of each pixel point on the target image after alignment with the reference image are (x′, y′). Then, the relationship of the alignment transformation of the target image using the above global homography matrix can be expressed as follows:

[0152]

[0153] After obtaining the aligned target image and the reference image, the photographing device may perform image fusion on the aligned target image and the reference image to obtain the target captured image and display the output.

[0154] In this embodiment, at least two initial image frames are acquired based on the preview stream of the camera device, thereby improving the efficiency of the electronic device in acquiring initial images, thereby improving the efficiency of acquiring the target captured image and enhancing the user experience. The target image with translational jitter is aligned using the first motion offset and the second motion offset, without the need to acquire additional motion vectors. This improves the efficiency of image alignment and further improves the efficiency of acquiring the target captured image. The above method can achieve image alignment without the need for sensors such as gyroscopes, micro inertial sensors, or accelerometers, making the method applicable to camera devices that do not have such sensors installed.

[0155] Figure 7 This is a schematic diagram of the structure of an image processing device provided by this application. Figure 7 As shown, the device includes: an acquisition module 301, a processing module 302, and an output module 303.

[0156] The acquisition module 301 is configured to acquire at least two frames of initial images.

[0157] The processing module 302 is configured to determine a reference image and a target image from the at least two frames of initial images; obtain an alignment detection result of the target image based on the target image and the reference image; obtain a jitter pattern of the target image relative to the reference image when the alignment detection result indicates that the target image is not aligned with the reference image; align the target image with the reference image according to an image alignment method corresponding to the jitter pattern to obtain a target image aligned with the reference image; and perform image fusion on the reference image and the target image aligned with the reference image to obtain a target captured image. The alignment detection result of the target image indicates whether the target image is aligned with the reference image.

[0158] The output module 303 is configured to output the target captured image.

[0159] Optionally, the processing module 302 is specifically configured to obtain a first motion offset of the target image relative to the reference image in a first motion direction and a second motion offset in a second motion direction; and determine an alignment detection result of the target image based on the first motion offset and the second motion offset, wherein the first motion direction and the second motion direction are not collinear.

[0160] Optionally, the processing module 302 is specifically configured to perform image division on the target image and the reference image according to a preset image division method to obtain N first sub-images of the target image and N second sub-images of the reference image; obtain M first sub-motion offsets of the target image in the first motion direction and M second sub-motion offsets in the second motion direction relative to the reference image based on the M first sub-images and the M second sub-images corresponding to the M first sub-images; obtain the first motion offset based on the M first sub-motion offsets; and obtain the second motion offset based on the M second sub-motion offsets. N is an integer greater than or equal to 2; any one of the first sub-images corresponds to one second sub-image; and M is an integer greater than or equal to 2 and less than or equal to N.

[0161] Optionally, the processing module 302 is specifically used to obtain, for any first sub-image of the M first sub-images, the sum of the pixel values ​​of each row of the first sub-image, and obtain a first projection vector of the first sub-image in the first motion direction; perform offset calculation on the first projection vector according to a preset offset step size and a preset maximum offset, and obtain multiple first offset vectors corresponding to the first projection vector; obtain a first absolute error and SAD vector of the first sub-image relative to the second sub-image according to each of the first offset vectors and the second projection vector of the second sub-image corresponding to the first sub-image in the first motion direction; and obtain a first sub-motion offset of the target image relative to the reference image in the first motion direction according to the first SAD vector.

[0162] Optionally, the processing module 302 is specifically configured to use the average of the absolute values ​​of the M first sub-motion offsets as the first motion offset; or, the processing module 302 is specifically configured to use the average of the absolute values ​​of the M second sub-motion offsets as the second motion offset.

[0163] Optionally, the processing module 302 is specifically configured to obtain a first variance corresponding to the M first sub-motion offsets and a second variance corresponding to the M second sub-motion offsets. Alternatively, when both the first variance and the second variance are less than a preset variance threshold, the jitter pattern of the target image relative to the reference image is determined to be a first jitter pattern; and when a variance between the first variance and the second variance is greater than or equal to a preset variance threshold, the jitter pattern of the target image relative to the reference image is determined to be a second jitter pattern. The first jitter pattern indicates that the target image is translated relative to the reference image; and the second jitter pattern indicates that the target image is rotated or twisted relative to the reference image.

[0164] Optionally, the processing module 302 is specifically configured to, when both the first motion offset and the second motion offset are smaller than a preset motion offset threshold, determine that the alignment detection result indicates that the target image and the reference image are aligned.

[0165] Optionally, the acquisition module 301 is specifically configured to capture a preview stream in response to a user-triggered request to enter a photo-taking interface; perform image capture in response to a user-triggered request to start taking photos to obtain a second initial image; and acquire the at least two initial image frames based on the preview stream and the second initial image. The preview stream includes at least one first initial image frame.

[0166] The image processing device provided in this application is used to execute the aforementioned image processing method embodiment. Its implementation principle and technical effects are similar and will not be described in detail.

[0167] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application. The electronic device may be, for example, a camera device. Figure 8 As shown, the electronic device 400 may include: at least one processor 401 and a memory 402.

[0168] The memory 402 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer operation instructions.

[0169] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0170] Processor 401 is configured to execute computer-executable instructions stored in memory 402 to implement the image processing method described in the aforementioned method embodiment. Processor 401 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0171] Taking the electronic device as a camera device as an example, the electronic device may further include a camera sensor, a display device, etc. The camera sensor may be used to capture images, and the display device may be used to display images. For example, the electronic device may control the camera sensor to capture at least two frames of initial images, and after obtaining a target captured image using the aforementioned image processing method, the electronic device may control the display device to output the target captured image.

[0172] Optionally, the electronic device 400 may further include a communication interface 403. In a specific implementation, if the communication interface 403, the memory 402, and the processor 401 are implemented independently, the communication interface 403, the memory 402, and the processor 401 may be interconnected via a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, etc., but this does not mean that there is only one bus or only one type of bus.

[0173] Optionally, in a specific implementation, if the communication interface 403, the memory 402 and the processor 401 are integrated on a chip, the communication interface 403, the memory 402 and the processor 401 can complete communication through an internal interface.

[0174] The present application also provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above embodiments.

[0175] The present application also provides a program product, the program product including execution instructions stored in a readable storage medium. At least one processor of an electronic device can read the execution instructions from the readable storage medium, and the at least one processor executes the execution instructions so that the electronic device implements the image processing methods provided in the various embodiments described above.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire at least two frames of initial images; Determining a reference image and a target image from the at least two frames of initial images; Performing image division on the target image and the reference image according to a preset image division method to obtain N first sub-images of the target image and N second sub-images of the reference image, where N is an integer greater than or equal to 2; any first sub-image corresponds to one second sub-image; Obtaining, based on the M first sub-images and the M second sub-images corresponding to the M first sub-images, M first sub-motion offsets of the target image in a first motion direction and M second sub-motion offsets in a second motion direction relative to the reference image; M is an integer greater than or equal to 2 and less than or equal to N; the first motion direction and the second motion direction are not collinear; Obtaining the first motion offset according to the M first sub-motion offsets; obtaining the second motion offset according to the M second sub-motion offsets; determining an alignment detection result of the target image according to the first motion offset and the second motion offset; the alignment detection result of the target image is used to indicate whether the target image is aligned with the reference image; When the alignment detection result is used to indicate that the target image and the reference image are not aligned, obtaining a jitter pattern of the target image relative to the reference image; aligning the target image with the reference image according to the image alignment method corresponding to the dithering pattern to obtain a target image aligned with the reference image; Performing image fusion on the reference image and the target image aligned with the reference image to obtain a target captured image; Outputting the target captured image; The step of obtaining, based on the M first sub-images and the M second sub-images corresponding to the M first sub-images, M first sub-motion offsets of the target image relative to the reference image in the first motion direction includes: For any first sub-image among the M first sub-images, obtain the sum of pixel values ​​in each row of the first sub-image to obtain a first projection vector of the first sub-image on the y-axis in the image coordinate system; Performing an offset calculation on the first projection vector according to a preset offset step size and a preset maximum offset amount to obtain a plurality of first offset vectors corresponding to the first projection vector; Obtaining a first sum of absolute errors (SAD) of the first sub-image relative to the second sub-image, i.e., a first SAD vector, based on each of the first offset vectors and a second projection vector of the second sub-image corresponding to the first sub-image on the y-axis in the image coordinate system; According to the first SAD vector, a first sub-motion offset of the target image relative to the reference image in a first motion direction is obtained.

2. The method according to claim 1, characterized in that The first movement direction is the positive direction indicated by the x-axis in the image coordinate system, and the second movement direction is the positive direction indicated by the y-axis in the image coordinate system.

3. The method according to claim 1, characterized in that The acquiring the first motion offset according to the M first sub-motion offsets includes: taking an average of the absolute values ​​of the M first sub-motion offsets as the first motion offset; or, The acquiring the second motion offset according to the M second sub-motion offsets includes: An average of the absolute values ​​of the M second sub-motion offsets is used as the second motion offset.

4. The method according to claim 1, wherein The acquiring of the jitter pattern of the target image relative to the reference image includes: Obtaining first variances corresponding to the M first sub-motion offsets and second variances corresponding to the M second sub-motion offsets; If both the first variance and the second variance are smaller than a preset variance threshold, determining that the jitter pattern of the target image relative to the reference image is a first jitter pattern, where the first jitter pattern indicates that the target image is shifted relative to the reference image; or, If there is a variance between the first variance and the second variance that is greater than or equal to a preset variance threshold, it is determined that the jitter pattern of the target image relative to the reference image is a second jitter pattern, and the second jitter pattern indicates that the target image is rotated or distorted relative to the reference image.

5. The method according to any one of claims 1 to 4, characterized in that Determining an alignment detection result of the target image according to the first motion offset and the second motion offset includes: If both the first motion offset and the second motion offset are smaller than a preset motion offset threshold, it is determined that the alignment detection result indicates that the target image and the reference image are aligned.

6. The method according to any one of claims 1 to 4, characterized in that The acquiring of at least two frames of initial images comprises: In response to a user-triggered request to enter a photo-taking interface, a preview stream is captured; the preview stream includes at least one frame of a first initial image; In response to a request to start taking a photo triggered by a user, performing image acquisition to obtain a second initial image; The at least two frames of initial images are acquired based on the preview stream and the second initial image.

7. An image processing device, characterized in that: The device comprises: An acquisition module, configured to acquire at least two frames of initial images; A processing module is configured to determine a reference image and a target image from the at least two frames of initial image; perform image division on the target image and the reference image according to a preset image division method to obtain N first sub-images of the target image and N second sub-images of the reference image, where N is an integer greater than or equal to 2; any first sub-image corresponds to one second sub-image; obtain M first sub-motion offsets of the target image in a first motion direction and M second sub-motion offsets in a second motion direction relative to the reference image according to M first sub-images and M second sub-images corresponding to the M first sub-images; where M is an integer greater than or equal to 2 and less than or equal to N; the first motion direction and the second motion direction are not on the same straight line; and a first sub-motion offset, obtaining the first motion offset; obtaining the second motion offset based on the M second sub-motion offsets; determining an alignment detection result of the target image based on the first motion offset and the second motion offset; when the alignment detection result is used to indicate that the target image is not aligned with the reference image, obtaining a jitter pattern of the target image relative to the reference image; aligning the target image with the reference image according to an image alignment method corresponding to the jitter pattern to obtain a target image aligned with the reference image; performing image fusion on the reference image and the target image aligned with the reference image to obtain a target captured image; wherein the alignment detection result of the target image is used to indicate whether the target image is aligned with the reference image; An output module, configured to output the target captured image; When the processing module obtains M first sub-motion offsets of the target image relative to the reference image in the first motion direction based on the M first sub-images and the M second sub-images corresponding to the M first sub-images, the processing module is specifically used to obtain the sum of the pixel values ​​of each row of any first sub-image in the M first sub-images, and obtain the first projection vector of the first sub-image on the y-axis in the image coordinate system; perform offset calculation on the first projection vector according to a preset offset step size and a preset maximum offset, and obtain multiple first offset vectors corresponding to the first projection vector; obtain the first absolute error sum of the first sub-image relative to the second sub-image, that is, the first SAD vector, based on each of the first offset vectors and the second projection vector of the second sub-image corresponding to the first sub-image on the y-axis in the image coordinate system; and obtain one first sub-motion offset of the target image relative to the reference image in the first motion direction according to the first SAD vector.

8. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images