Image anti-shake method and device, equipment and medium

Abstract

The embodiment of the invention provides an image anti-shake method and device, equipment and a medium, and belongs to the technical field of image processing, and the method specifically comprises the steps: obtaining an original motion track of an image capturing device according to an original image; processing the original motion track according to noise to obtain a corrected image; and judging whether the corrected image meets an output requirement or not according to a preset rule, and adjusting the noise according to a judgment result until the corrected image meeting the output requirement is obtained. According to the scheme of the invention, the black edge detection is introduced to judge whether the corrected image has the black edge, the black edge constraint is triggered, the performance of the Kalman filter is effectively controlled and the black edge of the corrected image is removed only by adjusting the process noise or the measurement noise, and the black edge of the corrected image does not need to be removed by introducing additional post-processing. And meanwhile, the anti-shake effect is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image stabilization method, device, equipment and medium. Background Art

[0002] At present, digital image stabilization technology mainly includes motion estimation, motion smoothing, motion compensation and other modules. Motion estimation estimates the camera motion information based on the feature points of adjacent frame images; motion smoothing smoothes the estimated camera motion information to generate a new smooth camera motion trajectory; motion compensation uses the estimated camera motion trajectory and the smoothed motion trajectory to obtain compensation information and correct the input image.

[0003] Based on the real-time requirements of video stabilization, motion smoothing usually adopts a zero-delay Kalman filter method. Kalman filtering has two parameters that affect the filter strength, namely process noise Q and measurement noise R. One of the existing technologies is based on a fixed filter strength parameter, which results in the smoothing strength not being able to be adaptively adjusted according to the degree of jitter, limiting the stabilization effect, and the corrected image may have black edges, requiring additional overhead to be introduced for post-processing to remove the black edges. The other is to adjust the process noise Q and measurement noise R at the same time, but the adjustment of the two parameters makes it difficult to control the filter performance, affecting the stabilization effect.

[0004] It can be seen that there is an urgent need for a digital image stabilization method that can remove the black edges of the corrected image while effectively controlling the filter performance and improving the video stabilization effect. Summary of the invention

[0005] In view of this, the embodiments of the present disclosure provide an image stabilization method, device, equipment and medium, which can remove the black edges of the corrected image while effectively controlling the filter performance and improving the image stabilization effect.

[0006] In a first aspect, an embodiment of the present disclosure provides an image stabilization method, including:

[0007] Obtaining an original motion trajectory of the image capturing device according to the original image;

[0008] Processing the original motion trajectory according to the noise to obtain a corrected image; wherein the noise is either process noise or measurement noise;

[0009] Whether the corrected image meets the output requirement is determined according to a preset rule, and the noise is adjusted according to the determination result until a corrected image that meets the output requirement is obtained.

[0010] According to a specific implementation of the embodiment of the present disclosure, judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained includes:

[0011] According to the preset rule, it is determined whether there is a black edge between the corrected image and the preset output image, and the noise is adjusted according to the determination result until a corrected image that meets the output requirement is obtained.

[0012] According to a specific implementation of the embodiment of the present disclosure, judging whether there is a black edge between the corrected image and the preset output image according to the preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained, includes:

[0013] Determining whether there is a black edge between the corrected image and the preset output image according to the preset rule;

[0014] If it exists, the noise is adjusted to obtain the latest noise, the corrected image is obtained using the latest noise, and the process returns to the step of determining whether there is a black edge between the corrected image and the preset output image according to the preset rules; otherwise, the corrected image is determined to meet the output requirements and is output.

[0015] According to a specific implementation of the embodiment of the present disclosure, judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained includes:

[0016] Determine whether the noise reaches a first threshold, and if so, determine that the corrected image meets the output requirement and output it.

[0017] According to a specific implementation of the embodiment of the present disclosure, judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained includes:

[0018] It is determined whether the number of iterations reaches a second threshold value, and if so, it is determined that the corrected image meets the output requirement and is output.

[0019] According to a specific implementation manner of the embodiment of the present disclosure, the preset rule includes determining whether the coordinates of four image vertices of the corrected image exceed the boundaries of a preset output image.

[0020] According to a specific implementation method of an embodiment of the present disclosure, the original motion trajectory includes displacement data in the x direction, displacement data in the y direction, scaling data in the x direction, scaling data in the y direction, misalignment data in the x direction, misalignment data in the y direction and rotation angle data.

[0021] In a second aspect, an embodiment of the present disclosure provides an image stabilization device, including:

[0022] An estimation module, used for obtaining an original motion trajectory of the image capture device according to the original image;

[0023] A smoothing module processes the original motion trajectory according to noise to obtain a corrected image; wherein the noise is either process noise or measurement noise;

[0024] The correction module is used to determine whether the corrected image meets the output requirements according to a preset rule, and adjust the noise according to the judgment result until a corrected image that meets the output requirements is obtained.

[0025] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0026] at least one processor; and,

[0027] a memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image stabilization method in the aforementioned first aspect or any implementation manner of the first aspect.

[0029] In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the image stabilization method in the aforementioned first aspect or any implementation method of the first aspect.

[0030] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the image stabilization method in the aforementioned first aspect or any implementation of the first aspect.

[0031] The image stabilization scheme in the embodiment of the present disclosure includes: obtaining an original motion trajectory of an image capture device according to an original image; processing the original motion trajectory according to noise to obtain a corrected image; wherein the noise is either process noise or measurement noise; judging whether the corrected image meets output requirements according to preset rules, and adjusting the noise according to the judgment result until a corrected image that meets the output requirements is obtained.

[0032] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, black edge detection is introduced to determine whether black edges appear in the corrected image, black edge constraints are triggered, and the Kalman filter performance is effectively controlled only by adjusting process noise or measurement noise, and the black edges of the corrected image are removed at the same time. There is no need to introduce additional post-processing to remove the black edges of the corrected image, and the anti-shake effect is improved at the same time. In addition, considering the real-time requirements of video stabilization, the noise update will be stopped when the number of iterations exceeds a certain upper limit, which can remove the black edges of the corrected image while effectively controlling the filter performance and improving the video stabilization effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A schematic diagram of motion smoothing provided by an embodiment of the present disclosure;

[0035] Figure 2 A schematic diagram of an anti-shake image correction provided by an embodiment of the present disclosure;

[0036] Figure 3 A flowchart of a digital image anti-shake method provided by an embodiment of the present disclosure;

[0037] Figure 4 A schematic diagram of motion smoothing adaptive adjustment provided by an embodiment of the present disclosure;

[0038] Figure 5 A schematic diagram of a given output size provided by an embodiment of the present disclosure;

[0039] Figure 6 A schematic diagram of motion trajectory smoothing under different process noises provided by an embodiment of the present disclosure;

[0040] Figure 7 A schematic diagram of the structure of an image stabilization device provided by an embodiment of the present disclosure;

[0041] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0043] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0044] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0045] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0046] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0047] Motion estimation is to estimate the camera motion information based on the feature points of adjacent frame images; motion smoothing is to smooth the estimated camera motion information to generate a new smooth camera motion trajectory, such as Figure 1 As shown; motion compensation uses the estimated camera motion trajectory and the smoothed motion trajectory to obtain compensation information and correct the input image.

[0048] Based on the real-time requirements of video anti-shake, motion smoothing usually adopts a zero-delay Kalman filter method. Kalman filtering has two parameters that affect the filter strength, namely process noise Q and measurement noise R. One of the existing technologies is based on a fixed filter strength parameter, which results in the smoothing strength not being able to be adaptively adjusted according to the degree of jitter, limiting the anti-shake effect, and the corrected image may have black edges, such as Figure 2 As shown in the figure, additional overhead needs to be introduced for post-processing to remove the black edges. Another method is to adjust the process noise Q and the measurement noise R at the same time, but the adjustment of the two parameters is difficult to control the filter performance and affects the anti-shake effect.

[0049] The embodiment of the present disclosure provides a digital image stabilization method, which can be applied to an image stabilization process in an image processing scenario.

[0050] See also Figure 3 , which is a flow chart of a digital image stabilization method provided in an embodiment of the present disclosure.

[0051] like Figure 3 and Figure 4 As shown, the method mainly comprises the following steps:

[0052] S101, obtaining an original motion trajectory of an image capturing device according to an original image;

[0053] S102, processing the original motion trajectory according to the noise to obtain a corrected image; wherein the noise is either process noise or measurement noise;

[0054] S103, judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained.

[0055] Furthermore, the original motion trajectory includes displacement data in the x-direction, displacement data in the y-direction, scaling data in the x-direction, scaling data in the y-direction, shearing data in the x-direction, shearing data in the y-direction, and rotation angle data.

[0056] In a specific implementation, the image capture device may be a camera or an imaging device such as a video camera, and the original image is obtained by shooting the image capture device. Then, the motion information of the camera may be estimated according to the feature points of the adjacent frames of the original image to obtain the original motion trajectory of the image capture device, wherein the original motion trajectory includes displacement data in the x direction, displacement data in the y direction, scaling data in the x direction, scaling data in the y direction, shearing data in the x direction, shearing data in the y direction, and rotation angle data, wherein, in image processing, shearing data refers to the distortion or shearing of an image by shearing transformation, resulting in a change in the pixel position of the original image. In simple terms, shearing is to shift the pixels of an image in a certain direction (such as horizontally or vertically), which usually makes the image look stretched or tilted, such as a rectangle becoming a parallelogram. According to the initial process noise Q or measurement noise R, the original motion trajectory of the motion estimation is processed accordingly to obtain a corrected image. After obtaining the corrected image, it can be determined according to preset rules whether the corrected image meets the output requirements, and the noise can be adjusted according to the determination result until a corrected image that meets the output requirements is obtained.

[0057] In a specific embodiment, assume that we have a device for capturing images, such as a handheld camera or sensor. During operation, the device may produce certain motion trajectory errors due to various factors (such as device jitter, environmental interference, measurement errors, etc.), resulting in instability or distortion in the captured images.

[0058] Step 1: Get the original motion trajectory

[0059] First, the motion information of the image capture device can be estimated by using the feature points in the adjacent frame images captured by the image capture device to obtain the original motion trajectory of the image capture device, which includes displacement data in the x-direction, displacement data in the y-direction, scaling data in the x-direction, scaling data in the y-direction, shearing data in the x-direction, shearing data in the y-direction, and rotation angle data. These motion trajectory data usually contain errors or noise, which may come from mechanical inaccuracies of the device itself, external environmental factors (such as wind, vibration, etc.), or other measurement errors. Therefore, the original motion trajectory obtained may not be completely accurate.

[0060] For example, a preset feature point extraction algorithm is used to extract feature points of each frame of the image;

[0061] Calculate descriptors and match feature points of adjacent frames, and use a random sampling consistency algorithm to filter out the correct match;

[0062] Calculate the displacement and rotation of the camera based on the matched feature points as the camera’s position information;

[0063] The pose information of adjacent frames is accumulated from the camera motion estimation to obtain the original motion trajectory.

[0064] Step 2: Smoothing motion trajectory

[0065] Next, a noise processing algorithm is used to process the original motion trajectory. The source of noise can be process noise (process noise is a random interference that affects the system state itself. It usually originates from the uncertainty of the internal or external environment of the system, such as wear of mechanical parts, temperature changes, electrical noise, etc.) or measurement noise (measurement noise is a random interference introduced during the measurement process, which affects the accuracy and reliability of the measured value. This noise may come from the inaccuracy of the measuring equipment itself, the operator's measurement technology, environmental factors, etc.). At this stage, the system will filter, smooth or correct these noises to eliminate or reduce the impact of errors on the image. After this processing, a "corrected image" is obtained, that is, the corrected image obtained according to the motion trajectory after filtering and correction.

[0066] Step 3: Determine whether the output requirements are met

[0067] After the image is calibrated, the system will evaluate the calibrated image according to preset rules (e.g., image stability, clarity, degree of distortion, etc.). These rules can be defined by artificial intelligence algorithms or preset standards to ensure that the image quality meets certain requirements. If the calibrated image meets the output requirements, the image at this time can be used as the final output image.

[0068] Step 4: Adjust the noise until it meets the requirements

[0069] If the corrected image does not meet the output requirements (for example, there is still slight distortion, blur, black edges or instability), the system will adjust the noise processing algorithm based on the evaluation results. It may be necessary to further optimize the noise filtering or correction strategy and reprocess the original motion trajectory until the final image that meets the output quality requirements is obtained.

[0070] For example, if the stability of the corrected image is insufficient, the system may increase the strength of the filter to reduce the impact of noise; if the image does not meet the output requirements, the way the measurement noise is handled may be adjusted to reduce blur. After multiple iterations and adjustments, a high-quality image that meets the standards will eventually be obtained.

[0071] On the basis of the above embodiment, judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained, includes:

[0072] According to the preset rule, it is determined whether there is a black edge between the corrected image and the preset output image, and the noise is adjusted according to the determination result until a corrected image that meets the output requirement is obtained.

[0073] When implementing it, Figure 5 As shown, according to the preset rules, the system will determine whether there is a black border between the corrected image and the given output size of the preset output image. Specifically, the system will analyze whether the image edge meets the predetermined quality standards (for example, whether there is a black border or distortion). If the judgment result is that there is a black border, the system will adjust the noise processing algorithm, calculate the motion compensation information based on the latest motion trajectory obtained by motion smoothing to optimize the motion trajectory correction, and obtain the image correction matrix based on the motion compensation information and correct the image coordinates until a corrected image that meets the preset output requirements is obtained.

[0074] For example, compensation information is obtained according to the original motion trajectory and the smoothed motion trajectory, and each frame image of the original video sequence is corrected accordingly to obtain a stabilized image.

[0075] For example, the process of correcting the anti-shake is:

[0076] 1. Original motion trajectory

[0077] Suppose there is a video whose camera displacement and rotation data for each frame are as follows (unit: pixels and degrees):

[0078] Frame 1: Translation (0,0), Rotation (0°)

[0079] Frame 2: Translation (2,1), Rotation (1°)

[0080] Frame 3: Translation (3,2), Rotation (2°)

[0081] Frame 4: Translation (5,3), Rotation (4°)

[0082] Frame 5: Translation (6,6), Rotation (5°)

[0083] 2. Smoothed motion trajectory

[0084] After Kalman filtering, the smoothed motion trajectory is obtained:

[0085] Frame 1: Translation (0,0), Rotation (0°)

[0086] Frame 2: Translation (1.5, 0.5), Rotation (0.5°)

[0087] Frame 3: Translation (3,2), Rotation (1.5°)

[0088] Frame 4: Translation (4,3), Rotation (3°)

[0089] Frame 5: Translation (5, 5), Rotation (4°)

[0090] 3. Calculate compensation information

[0091] Next, calculate the compensation information, which is the difference between the original motion trajectory and the smoothed motion trajectory:

[0092] Frame 1: offset (0, 0), rotation (0°)

[0093] Frame 2: offset (0.5, 0.5), rotate (0.5°)

[0094] Frame 3: offset (0, 0), rotate (0.5°)

[0095] Frame 4: offset (1, 0), rotate (1°)

[0096] Frame 5: Compensation (1, 1), rotation (1°)

[0097] 4. Correct the original frame

[0098] According to the compensation information, an image correction matrix is ​​calculated, and the image is corrected according to the image correction matrix.

[0099] For example, the original motion trajectory and the smoothed motion trajectory are differentiated to obtain new motion information, and an image correction matrix is ​​generated accordingly, wherein the expression of the image correction matrix is:

[0100]

[0101] Where, M0 = sx*(cos(da)-py*sin(da))

[0102] M1=sy*(px*cos(da)-sin(da))

[0103] M2=dx

[0104] M3=sx*(sin(da)+py*cos(da))

[0105] M4=sy*(px*sin(da)+cos(da))

[0106] M5=dy

[0107] Wherein, sx represents the scaling in the x direction after differentiation, sy represents the scaling in the y direction after differentiation, px represents the shearing in the x direction after differentiation, py represents the shearing in the y direction after differentiation, dx represents the displacement in the x direction after differentiation, dy represents the displacement in the y direction after differentiation, and da represents the rotation angle after differentiation;

[0108] The coordinates corresponding to the four vertices of the corrected image are calculated according to the correction matrix and the preset correction formula, wherein the correction formula is:

[0109]

[0110] LT_x_corr=LT_x*M0+LT_y*M1+M2,

[0111] LT_y_corr=LT_x*M3+LT_y*M4+M5

[0112] RT_x_corr=RT_x*M0+RT_y*M1+M2,

[0113] RT_y_corr=RT_x*M3+RT_y*M4+M5

[0114] LB_x_corr=LB_x*M0+LB_y*M1+M2,

[0115] LB_y_corr=LB_x*M3+LB_y*M4+M5

[0116] RB_x_corr=RB_x*M0+RB_y*M1+M2,

[0117] RB_y_corr=RB_x*M3+RB_y*M4+M5.

[0118] 5. Output stable image

[0119] After the above correction steps, the images of all frames are adjusted to obtain a corrected image for each frame.

[0120] Furthermore, judging whether there is a black edge between the corrected image and the preset output image according to the preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained, includes:

[0121] Determining whether there is a black edge between the corrected image and the preset output image according to the preset rule;

[0122] If it exists, the noise is adjusted to obtain the latest noise, the corrected image is obtained using the latest noise, and the process returns to the step of determining whether there is a black edge between the corrected image and the preset output image according to the preset rules; otherwise, the corrected image is determined to meet the output requirements and is output.

[0123] For example, in the actual application of image processing, especially in the process of video stabilization, image stitching or motion compensation, black edges often appear after image correction. Black edges are usually caused by the fact that some pixel areas cannot completely cover the original image after the image is rotated, translated or deformed, causing these areas to appear as black or transparent areas. In order to remove these black edges, a series of judgment and adjustment steps are usually required. The following is a specific embodiment that illustrates how to judge whether there are black edges between the corrected image and the preset output image according to preset rules, and adjust the noise as needed to optimize the result.

[0124] Example: Black edge adjustment during image correction

[0125] Suppose we have an original image and a corrected image after motion compensation. After a certain rotation and translation, the corrected image may have black edges. Our goal is to determine whether there are black edges and get a new corrected image after adjusting the noise as needed.

[0126] 1. Definition of preset rules

[0127] Before we start adjusting the image, we first need to define some preset rules. Let's assume that these rules are as follows:

[0128] Rule 1: The calibrated image and the preset output image (ideal image) should have the same size and ratio.

[0129] Rule 2: If the edge area of ​​the corrected image displays black (or transparent) pixels exceeding a certain ratio, the image is considered to have black edges.

[0130] Rule 3: If the black border area of ​​the corrected image is large, you may need to adjust the noise to reduce anomalies caused by edge effects.

[0131] 2. Determine the black edge

[0132] First, we compare the rectified image with the preset output image to check whether the rectified image meets rule 1, that is, whether the size and proportion are consistent. If the size matches, then we check whether there is a black border.

[0133] We check the four edge areas of the corrected image and calculate the pixel ratio of the black area. If the area occupied by black pixels exceeds a certain threshold (such as 10%), the corrected image is considered to have black edges.

[0134] If black borders are detected, the system will record the size and position of the black borders to prepare for subsequent adjustments.

[0135] 3. Noise adjustment

[0136] Optionally, the step of adjusting the noise includes:

[0137] The latest process noise is obtained by increasing the first preset value to the process noise or the latest measurement noise is obtained by decreasing the second preset value to the measurement noise.

[0138] Kalman filtering has two parameters that affect the strength of the filter: process noise Q and measurement noise R. Q represents the degree of confidence in the uncertainty of the filtering process. The larger the Q, the closer the filtering result is to the measured value, and vice versa. R represents the degree of confidence in the measurement noise. The larger the R, the closer the filtering result is to the predicted value, and vice versa. In general, the noise can be adjusted by fixing one of the noises and adjusting the processing method of the other noise, or both noises can be adjusted at the same time. In order to effectively control the performance of the Kalman filter, only Q is adjusted to control the filtering result, such as Figure 6 The figures show the smooth motion trajectories of Q=0.4 and Q=0.004 respectively. Obviously, the smaller the Q, the smoother the motion trajectory and the better the anti-shake effect, but the probability of black edges appearing in the corrected image will also be greater.

[0139] For example, the initial process noise Q0 and measurement noise R0 are set to 0.004 and 0.5 respectively. When the image is corrected with black edges after the first motion smoothing using black edge detection, the process noise Q1 is adjusted to Q0+0.001. After the second motion smoothing, the corrected image still has black edges. The process noise Q2 is adjusted to Q1+0.001 until the Nth iteration process noise QN=QN-1+0.001. If no black edges appear in the corrected image, the updating of the process noise Q is stopped and the corrected image after anti-shake is output.

[0140] Alternatively, the initial process noise Q0 and R0 are set to 0.004 and 0.5 respectively. When the image is corrected with black edges after the first motion smoothing using black edge detection, the process noise R1 is adjusted to R0-0.001. After the second motion smoothing, the corrected image still has black edges. The process noise R2 is adjusted to R1-0.001 until the Nth iteration process noise RN is RN-1-0.001. If no black edges appear in the corrected image, the updating of the measurement noise R is stopped and the corrected image after anti-shake is output.

[0141] 4. Regenerate the corrected image

[0142] After the noise adjustment, the system will filter the motion trajectory according to the updated noise value, generate compensation information for the Nth smoothed motion trajectory and the original motion trajectory, obtain a new image correction matrix based on the compensation information, correct the image according to the latest image correction matrix, obtain the latest corrected image coordinate value, and then regenerate the latest corrected image. This process will include resampling and interpolation operations on the image to ensure that the image quality is as close as possible to the preset output image requirements.

[0143] The system will again determine whether the adjusted calibrated image still has black edges. If the black edges disappear after adjustment and the image quality meets the output requirements, it means that the current image has met the preset standards.

[0144] If black edges still exist, or the image quality does not meet expectations, the system may continue to perform further noise adjustments and image optimizations until the expected results are achieved.

[0145] 5. Output the final image

[0146] When the corrected image has been noise-adjusted and no black edges exist, the system determines that the image meets the preset output requirements. Finally, the processed and optimized image is returned as the output result.

[0147] result:

[0148] Through the above steps, the black edge problem in the image can be effectively determined and adjusted. If the corrected image has black edges, the system will make noise adjustments, optimize the image quality, and continuously check whether it meets the output requirements. In the end, we will get an image without black edges and with the expected quality.

[0149] This processing flow is ideal for video stabilization, image stitching, and other applications that require image correction.

[0150] On the basis of the above embodiment, judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained, includes:

[0151] Determine whether the noise reaches a first threshold, and if so, determine that the corrected image meets the output requirement and output it.

[0152] In a specific implementation, it can be determined whether there is a black edge between the corrected image and the preset output image according to the preset rule;

[0153] If it exists, the noise is adjusted to obtain the latest noise, the corrected image is obtained using the latest noise, and it is determined whether the latest noise is greater than or equal to the first threshold value. If so, it is determined that the corrected image meets the output requirement and is output, otherwise, returning to the step of determining whether there is a black edge between the corrected image and the preset output image according to the preset rule;

[0154] If not, it is determined that the corrected image meets the output requirement and is output.

[0155] Meanwhile, in specific implementation, it is also possible to first determine whether the latest noise is greater than or equal to the first threshold and then determine whether there is a black edge between the corrected image and the preset output image:

[0156] Determine whether the latest noise is greater than or equal to a first threshold; if less than the first threshold, determine whether there is a black edge between the corrected image and the preset output image according to the preset rule; if not, determine that the corrected image meets the output requirement and output it; otherwise, return to the step of adjusting the noise to obtain the latest noise, obtain the corrected image using the latest noise, and determine whether the latest noise is greater than or equal to the first threshold;

[0157] If it is greater than or equal to the first threshold, it is determined that the corrected image meets the output requirement and is output.

[0158] For example, in one embodiment, a Kalman filter algorithm may be used to smooth the motion trajectory, and the implementation of the Kalman filter is shown in Table 1 below. Figure 6 As shown, in order to effectively control the performance of the Kalman filter, the filtering result is controlled by adjusting Q only. Figure 6 The figure shows the smooth motion trajectories of Q=0.4 and Q=0.004. Obviously, the smaller the Q, the smoother the motion trajectory and the better the anti-shake effect, but the probability of black edges appearing in the corrected image will also be greater.

[0159] The initial process noise Q0 and R0 are set to 0.004 and 0.5 respectively. After the first motion smoothing, when the image is corrected by black edge detection, the process noise Q1 is adjusted to Q0+0.001. After the second motion smoothing, the corrected image still has black edges. The process noise Q2 is adjusted to Q1+0.001. The motion trajectory is smoothed according to the adjusted noise, and the smoothed motion trajectory and the original motion trajectory are used to generate compensation information. The image correction matrix is ​​generated according to the compensation information. The image is corrected according to the correction matrix to determine whether the corrected image has black edges. Until the Nth update of the process noise QN=QN-1+0.001, the corrected image does not have black edges, then the process noise Q is stopped from being updated. The Nth smoothed motion trajectory and the original motion trajectory are used to generate new motion information, and the corrected image after anti-shake is output. Considering the real-time nature of image anti-shake, when the process noise Q=4, the filtering result is basically the same as the original input, that is, there is almost no smoothing effect. Therefore, except for the correction image mentioned above, the Q is stopped from being updated when no black edges appear.

[0160] Table 1

[0161]

[0162] Furthermore, judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained, includes:

[0163] It is determined whether the number of iterations reaches a second threshold value, and if so, it is determined that the corrected image meets the output requirement and is output.

[0164] In specific implementation, the initial number of iterations may be set and the preset rule may be used to determine whether there is a black edge between the corrected image and the preset output image;

[0165] If it exists, the noise can be adjusted in the manner described in the above steps to obtain the latest noise (the process noise is increased by a first preset value to obtain the latest process noise or the measurement noise is reduced by a second preset value to obtain the latest measurement noise), the second preset value is added to the initial number of iterations to obtain the latest number of iterations, the corrected image is obtained using the latest noise, and it is determined whether the latest iteration value is greater than or equal to a second threshold value. If so, it is determined that the corrected image meets the output requirements and is output, otherwise, return to the step of determining whether there is a black edge between the corrected image and the preset output image according to the preset rule;

[0166] If not, the corrected image is determined to meet the output requirements and is output; it should be noted that the number of iterations is the number of times a black border is determined between the corrected image and the preset output image according to a preset rule.

[0167] Meanwhile, in specific implementation, it is also possible to first determine whether the latest iteration value is greater than or equal to the second threshold value and then determine whether there is a black edge between the corrected image and the preset output image:

[0168] Determine whether the latest iteration value is greater than or equal to a second threshold value; if less than the second threshold value, determine whether there is a black edge between the corrected image and the preset output image according to the preset rule; if not, determine that the corrected image meets the output requirement and output it; otherwise, return to adjust the noise to obtain the latest noise, add the second preset value to the initial iteration number to obtain the latest iteration number, use the latest noise to obtain the corrected image, and determine whether the latest iteration value is greater than or equal to the second threshold value;

[0169] If it is greater than or equal to the second threshold, it is determined that the corrected image meets the output requirement and is output.

[0170] For example, the number of iterations is the number of times of judging whether there is a black edge between the corrected image and the preset output image according to the preset rules. When the number of iterations exceeds N=4 / 0.001=4000, the noise is also stopped from being updated, and the correction matrix is ​​output for correction of the current frame image.

[0171] Furthermore, the preset rule includes determining whether the coordinates of four image vertices of the corrected image exceed a preset boundary of the output image.

[0172] When implementing it, Figure 5 As shown, W_ori and H_ori are the width and height of the original image, which are used as the set output image size. W_crop and H_crop are the width and height of the output image, respectively. The corresponding four vertex coordinates are:

[0173] LT_x=(W_ori-W_crop) / 2, LT_y=(H_ori-H_crop) / 2

[0174] RT_x=W_ori-(W_ori-W_crop) / 2, RT_y=(H_ori-H_crop) / 2

[0175] LB_x=(W_ori-W_crop) / 2, LB_y=H_ori-(H_ori-H_crop) / 2

[0176] RB_x=W_ori-(W_ori-W_crop) / 2, RB_y=H_ori-(H_ori-H_crop) / 2.

[0177] After obtaining the image correction matrix, the four corrected image vertices can be calculated using the correction formula The solution is as follows:

[0178] LT_x_corr=LT_x*M0+LT_y*M1+M2,

[0179] LT_y_corr=LT_x*M3+LT_y*M4+M5

[0180] RT_x_corr=RT_x*M0+RT_y*M1+M2,

[0181] RT_y_corr=RT_x*M3+RT_y*M4+M5

[0182] LB_x_corr=LB_x*M0+LB_y*M1+M2,

[0183] LB_y_corr=LB_x*M3+LB_y*M4+M5

[0184] RB_x_corr=RB_x*M0+RB_y*M1+M2,

[0185] RB_y_corr=RB_x*M3+RB_y*M4+M5

[0186] If LT_x_corr is less than 0, LT_y_corr is less than 0, RT_x_corr is greater than W_ori, RT_y_corr is less than 0, LB_x_corr is less than 0, LB_y_corr is greater than H_ori, RB_x_corr is greater than W_ori, and RB_y_corr is greater than H_ori. If any of the above conditions is met, the corrected image is judged to have black edges.

[0187] That is, if the upper left corner horizontal coordinate LT_x_corr of the corrected image is less than 0, or the upper left corner vertical coordinate LT_y_corr of the corrected image is less than 0, or the upper right corner horizontal coordinate RT_x_corr of the corrected image is greater than the set width W_ori of the output image size, or the upper right corner vertical coordinate RT_y_corr of the corrected image is less than 0, or the lower left corner horizontal coordinate LB_x_corr of the corrected image is less than 0, the lower left corner vertical coordinate LB_y_corr of the corrected image is greater than the set height H_ori of the output image size, or the lower right corner horizontal coordinate RB_x_corr of the corrected image is greater than the set width W_ori of the output image size, or the lower right corner vertical coordinate RB_y_corr of the corrected image is greater than the set height H_ori of the output image size, then it is judged that the corrected image has a black border.

[0188] The image stabilization method provided in this embodiment introduces black edge detection to determine whether black edges appear in the corrected image, triggers black edge constraints, and effectively controls the performance of the Kalman filter and removes the black edges of the corrected image only by adjusting process noise or measurement noise. There is no need to introduce additional post-processing to remove the black edges of the corrected image, while improving the stabilization effect. In addition, considering the real-time requirements of video stabilization, the noise will be stopped from being updated when the number of iterations exceeds a certain upper limit, which can remove the black edges of the corrected image while effectively controlling the filter performance and improving the video stabilization effect.

[0189] Corresponding to the above method embodiment, see Figure 7 The embodiment of the present disclosure further provides an image stabilization device 70, comprising:

[0190] An estimation module 701 is used to obtain an original motion trajectory of the image capture device according to the original image;

[0191] A smoothing module 702 processes the original motion trajectory according to noise to obtain a corrected image; wherein the noise is either process noise or measurement noise;

[0192] The correction module 703 is used to determine whether the corrected image meets the output requirement according to a preset rule, and adjust the noise according to the determination result until a corrected image that meets the output requirement is obtained.

[0193] Figure 7 The device shown can correspondingly execute the contents of the above method embodiment. For the parts not described in detail in this embodiment, refer to the contents recorded in the above method embodiment and will not be repeated here.

[0194] See also Figure 8 The embodiment of the present disclosure further provides an electronic device 80, which includes: at least one processor and a memory connected to the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image stabilization method in the aforementioned method embodiment.

[0195] The embodiment of the present disclosure further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the image stabilization method in the aforementioned method embodiment.

[0196] The present disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the image stabilization method in the aforementioned method embodiment.

[0197] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device 80 suitable for implementing the embodiment of the present disclosure. The electronic device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0198] like Figure 8 As shown, the electronic device 80 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 80 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0199] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 80 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 80 with various devices is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0200] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0201] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0202] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0203] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device can execute the relevant steps of the method embodiment.

[0204] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device can perform the relevant steps of the method embodiment.

[0205] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0206] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0207] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware.

[0208] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof.

[0209] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image stabilization method, characterized in that: include: Obtaining an original motion trajectory of the image capturing device according to the original image; Processing the original motion trajectory according to the noise to obtain a corrected image; wherein the noise is either process noise or measurement noise; It is determined whether the corrected image meets the output requirement according to a preset rule, and the noise is adjusted according to the determination result until a corrected image that meets the output requirement is obtained.

2. The image stabilization method according to claim 1, characterized in that: The step of judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained, comprises: According to the preset rule, it is determined whether there is a black edge between the corrected image and the preset output image, and the noise is adjusted according to the determination result until a corrected image that meets the output requirement is obtained.

3. The image stabilization method according to claim 2, characterized in that: The determining, according to the preset rule, whether there is a black edge between the corrected image and the preset output image, and adjusting the noise according to the determination result until a corrected image that meets the output requirement is obtained, includes: Determining whether there is a black edge between the corrected image and the preset output image according to the preset rule; If it exists, the noise is adjusted to obtain the latest noise, the corrected image is obtained using the latest noise, and the process returns to the step of determining whether there is a black edge between the corrected image and the preset output image according to the preset rules; otherwise, the corrected image is determined to meet the output requirements and is output.

4. The image stabilization method according to claim 1, characterized in that: The step of judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained, comprises: Determine whether the noise reaches a first threshold, and if so, determine that the corrected image meets the output requirement and output it.

5. The image stabilization method according to claim 1, characterized in that: The step of judging whether the corrected image meets the output requirement according to a preset rule, and adjusting the noise according to the judgment result until a corrected image that meets the output requirement is obtained, comprises: It is determined whether the number of iterations reaches a second threshold value, and if so, it is determined that the corrected image meets the output requirement and is output.

6. The image stabilization method according to claim 1, characterized in that: The preset rule includes determining whether the coordinates of four image vertices of the corrected image exceed the boundaries of a preset output image.

7. The image stabilization method according to any one of claims 1 to 6, characterized in that: The original motion trajectory includes displacement data in the x-direction, displacement data in the y-direction, scaling data in the x-direction, scaling data in the y-direction, shearing data in the x-direction, shearing data in the y-direction, and rotation angle data.

8. An image stabilization device, characterized in that: include: An estimation module, used for obtaining an original motion trajectory of the image capture device according to the original image; A smoothing module processes the original motion trajectory according to noise to obtain a corrected image; wherein the noise is either process noise or measurement noise; The correction module is used to determine whether the corrected image meets the output requirements according to a preset rule, and adjust the noise according to the judgment result until a corrected image that meets the output requirements is obtained.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image stabilization method according to any one of claims 1 to 7. 10 . A computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the image stabilization method according to claim 1 .

Images