Image alignment method and device, electronic equipment, chip and storage medium
By optimizing the estimation of the homography matrix in the video sequence and leveraging the correlation between video frames, the problem of unstable alignment accuracy in traditional image registration methods is solved, and better image alignment and video fusion effects are achieved.
Patent Information
- Application Number
- CN202510505630.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-21
- Publication Date
- 2025-08-08
AI Technical Summary
Traditional image registration methods have large differences in alignment accuracy in video sequences, resulting in jumping in the timing of the video, affecting the consistency and viewing of the video.
By determining the first matrix and updating it using the second matrix, the target matrix is generated, and the estimation of the homography matrix is optimized using the high correlation between video frames to reduce the difference in frame alignment, Kalman filtering and RANSAC algorithm are used to eliminate outliers of feature points and matrix optimization.
It achieves better image alignment effect, improves the visual stability after video fusion, reduces the difference in homography between images, and improves the consistency and visual quality of the video.
Smart Images
Figure CN120451230A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing, and in particular to an image alignment method, device, electronic device, chip, and storage medium. Background Art
[0002] In the field of image processing, image registration is a basic and critical technology. It involves aligning two or more images to achieve precise matching at the pixel level. Video registration is a subset of image registration, which requires the registration of consecutive frames in a video sequence to achieve temporal coherence and spatial consistency. Traditional registration methods have large differences in alignment accuracy between different images, which may cause jumps in the video timing. Summary of the Invention
[0003] The present disclosure provides an image alignment method, device, electronic device, chip, and storage medium to solve problems in related technologies.
[0004] A first aspect embodiment of the present disclosure proposes an image alignment method, which includes: determining a first matrix, where the first matrix is used to indicate the correspondence between at least one first feature point of a first image and at least one second feature point of a second image, where the first image is a subsequent frame image of the second image; updating the first matrix using a second matrix to obtain a target matrix, where the second matrix is used to indicate the correspondence between at least one second feature point and at least one third feature point of a third image, where the second image is a subsequent frame image of the third image; and aligning the first image and the second image using the target matrix.
[0005] In some embodiments of the present disclosure, determining the first matrix includes: generating a first set based on at least one first feature point and at least one second feature point; eliminating outliers in the first set to obtain a second set, the second set including at least one feature pair, each feature pair in the at least one feature pair including a first feature point in the at least one first feature point and a second feature point in the at least one second feature point; generating the first matrix based on the at least one feature pair included in the second set.
[0006] In some embodiments of the present disclosure, using the second matrix to update the first matrix to obtain the target matrix includes: determining the difference between the first matrix and the second matrix; determining the gain parameter corresponding to the first matrix; and updating the first matrix according to the difference between the first matrix and the second matrix, the gain parameter, and the second matrix to obtain the target matrix.
[0007] In some embodiments of the present disclosure, determining the gain parameters corresponding to the first matrix includes: determining a state covariance matrix corresponding to the first matrix; and determining the gain parameters corresponding to the first matrix based on the state covariance matrix corresponding to the first matrix and a preset noise parameter.
[0008] In some embodiments of the present disclosure, the method also includes: generating a state covariance matrix corresponding to a third matrix based on the gain parameters corresponding to the first matrix, the state covariance matrix corresponding to the first matrix, and the target matrix, the third matrix being used to indicate the correspondence between at least one feature point and at least one fourth feature point of a fourth image, where the fourth image is a subsequent frame image of the first image.
[0009] In some embodiments of the present disclosure, the method further includes: using the target matrix to verify at least one feature pair included in the second set; if there is a feature pair that fails verification in at least one feature pair, eliminating the feature pair that fails verification from the second set to obtain a third set; and using the third set to optimize the target matrix.
[0010] A second aspect embodiment of the present disclosure proposes an image alignment device, which includes: a first processing unit, used to determine a first matrix, the first matrix is used to indicate the correspondence between at least one first feature point of a first image and at least one second feature point of a second image, and the first image is a subsequent frame image of the second image; a second processing unit, used to update the first matrix using the second matrix to obtain a target matrix, the second matrix is used to indicate the correspondence between at least one second feature point and at least one third feature point of a third image, and the second image is a subsequent frame image of the third image; a third processing unit, used to align the first image and the second image using the target matrix.
[0011] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect embodiment of the present disclosure.
[0012] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.
[0013] The fifth aspect embodiment of the present disclosure proposes a chip, characterized in that it includes at least one processor and a communication interface; the communication interface is used to receive signals input into the chip or signals output from the chip, and the processor communicates with the communication interface and implements the method described in the first aspect embodiment of the present disclosure through logic circuits or executing code instructions.
[0014] In summary, the image alignment method proposed in the present disclosure can use the homography matrix of the previous frame to update the current homography matrix when determining the homography matrix corresponding to the current frame, thereby utilizing the high correlation between video frames to obtain better image alignment effects. When the aligned images are subsequently used for fusion, tracking or splicing, better video effects can be obtained.
[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.
[0017] Figure 1 A schematic diagram of the process of an image alignment method provided in an embodiment of the present disclosure Figure 1 ;
[0018] Figure 2 A schematic diagram of the process of an image alignment method provided in an embodiment of the present disclosure Figure 2 ;
[0019] Figure 3 A schematic diagram of the process of an image alignment method provided in an embodiment of the present disclosure Figure 3 ;
[0020] Figure 4 A flowchart of a video frame alignment method using previous frame information provided by an embodiment of the present disclosure;
[0021] Figure 5 A schematic diagram of a flow chart of a video frame image registration method provided in an embodiment of the present disclosure;
[0022] Figure 6 A schematic structural diagram of an image alignment device provided in an embodiment of the present disclosure;
[0023] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;
[0024] Figure 8 A schematic diagram of the chip structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.
[0026] Image registration is a fundamental and critical technology that involves aligning two or more images to achieve accurate matching at the pixel level. Video registration, as a subset of image registration, further extends this concept by requiring the registration of consecutive frames in a video sequence to achieve temporal coherence and spatial consistency.
[0027] Two-frame image alignment techniques typically involve modeling the homography matrix estimation problem. In one image alignment method, matching feature point pairs are obtained from the two images. The Random Sample Consensus (RANSAC) algorithm is then used to solve the optimal set of equations for these matching point pairs to calculate the homography matrix. This results in a projective transformation from one image to the other. Ultimately, the homography matrix implements the projective transformation from one image to the other, providing a precise geometric relationship for image alignment. For video image alignment, because it consists of multiple frames, it is generally separated into the previous and next frames, and then the two-frame image alignment technique is used for the previous and next frames.
[0028] The homography matrix is typically solved using the findHomography function in the OpenCV library, which implements a global alignment estimate between two image frames. The specific process involves first extracting feature point pairs from the two frames, then eliminating outliers using the RANSAC algorithm, and finally accurately estimating the homography matrix using the least squares method. In video image sequence alignment, this process is applied to each pair of consecutive frames to achieve independent alignment between them.
[0029] However, for video sequences, the calculated optimal alignment matrix can fluctuate in accuracy due to differences in feature point distribution between frames. When using these matrices for image fusion, this can cause visual flickering or localized lingering in the video image, affecting the video's coherence and visual quality. Therefore, to improve the quality of video alignment, further optimization of the homography matrix estimation method is needed to reduce inter-frame alignment variations and thus enhance the visual stability of the fused video.
[0030] Therefore, to address the above issues, this disclosure proposes an image alignment method. This method is not only applicable to the registration of static images, but is particularly well-suited for inter-frame registration of dynamic videos, providing an efficient and reliable solution for any visual task involving the registration of two or more multi-frame images. By accurately estimating the homography matrix, this technical solution can achieve geometric transformations between images, providing strong technical support in a wide range of application scenarios, such as feature matching, target tracking, and 3D reconstruction.
[0031] The specific contents of this method are as follows.
[0032] Figure 1 A schematic diagram of the process of an image alignment method provided in an embodiment of the present disclosure Figure 1 .like Figure 1 As shown, the method may include the following steps.
[0033] Step 101: Determine a first matrix.
[0034] In some embodiments, the first matrix is used to indicate the correspondence between at least one first feature point of the first image and at least one second feature point of the second image. The first image is the next frame image of the second image. For example, if the first image is image T, the second image is T-1. The first image and the second image may belong to the same video sequence. There is a time sequence change between the first image and the second image, that is, the first image is obtained after the time sequence change of the second image.
[0035] In some embodiments, any feature extraction algorithm can be used to extract at least one first feature point in the first image, and at least one second feature point in at least one second image. There is a corresponding relationship between the first feature point and the second feature point. For example, the vertex A of a triangle in the first image is extracted as the first feature point, and the triangle also exists in the second image. The vertex corresponding to the triangle can be extracted as the second feature point. Due to the time series changes between the first image and the second image, and the possible changes in the shooting angle between the first image and the second image, the positions of the first feature point and the second feature point will be different.
[0036] In some embodiments, the second feature point moves to the position of the first feature point on the image after timing changes and angle changes, that is, the first matrix can indicate how the second feature point changes to the position of the first feature point, that is, it can indicate how the second feature point is mapped to the first feature point.
[0037] In some embodiments, the first matrix and the second matrix of the present disclosure are both homography matrices, and determining the first matrix includes: generating a first set based on at least one first feature point and at least one second feature point; eliminating outliers in the first set to obtain a second set, the second set including at least one feature pair, each feature pair in the at least one feature pair including a first feature point in the at least one first feature point and a second feature point in the at least one second feature point; generating the first matrix based on the at least one feature pair included in the second set.
[0038] In other words, at least one feature pair can be generated based on the correspondence between at least one first feature point and at least one second feature point to form a first set. For example, vertex A of the triangle in the first image and vertex A of the triangle in the second image can form a feature pair.
[0039] In some embodiments, since the first feature point and the second feature point are both obtained using an extraction algorithm, there may be some errors. For example, the A vertex of the triangle in the first image is determined as the first feature point, and the B vertex of the quadrilateral in the second image is determined as the second feature point. At this time, the mapping relationship between the first feature point and the second feature point is wrong, and the deviation from the correct mapping relationship is large. Therefore, the outliers in the first set can be eliminated to obtain the second set S{curPt(t), prePt(t)}, where curPt(t) is the coordinate of the first feature point, and prePt(t) is the coordinate of the second feature point. Eliminating the outliers means eliminating the coordinate values of the feature points with larger deviations in the first set. Feature points with larger deviations can be called outliers. Optionally, the RANSAC algorithm can be used to eliminate the outliers, and then the least squares method can be used to use the data in the second set for estimation to obtain the first matrix.
[0040] Step 102: Use the second matrix to update the first matrix to obtain a target matrix.
[0041] In some embodiments, the second matrix is used to indicate a correspondence between at least one second feature point and at least one third feature point of a third image, where the second image is an image frame subsequent to the third image. Similarly, the third image and the first image belong to the same video sequence, where the first image is image T, the second image is image T-1, and the third image can be image T-2. Similarly, a feature extraction algorithm can be used to extract the at least one third feature point from the third image.
[0042] Among them, the second matrix is used to indicate the correspondence between at least one second feature point and at least one third feature point of the third image. The second matrix can be the target matrix corresponding to the second image. The first matrix is updated using the second matrix, and the target matrix obtained is the target matrix corresponding to the first image; in other words, if the second image is the second frame image, the method of obtaining the second matrix and the first matrix can be the same, that is, the second matrix can be determined by the method described in step 102. If the second image is neither the first frame nor the second frame image, the second matrix is the target matrix corresponding to the second image obtained by steps 101 and 102, that is, the second matrix is obtained after updating using the homography matrix corresponding to the third image.
[0043] In some embodiments, in the solution of the present disclosure, the sequence number of the first image in the video frame sequence is greater than or equal to 3, that is, the first image is neither the first frame nor the second frame image.
[0044] In some embodiments, for the same video sequence, the multiple images contained in the video sequence are interrelated, so the homography matrices between the multiple images are consistent, that is, the differences between the multiple homography matrices are not too large. If the differences between the multiple homography matrices are too large, it may cause the fused video to have jumps in accuracy and offset, and ultimately cause the output video screen to have a floating visual effect. Therefore, the solution of the present disclosure can use the homography matrix corresponding to the previous frame image to update the homography matrix corresponding to the current frame image, so that the error between the final target matrix and the second matrix is small, and when the images are subsequently fused, tracked, or spliced, better image or video effects can be obtained.
[0045] Step 103: align the first image and the second image using a target matrix.
[0046] In some embodiments, the first image and the second image are aligned using a target matrix to determine the correspondence between the second feature point and the first feature point, so that the position of the second feature point can be determined based on the position information of the first feature point and the target matrix, or the position of the first feature point can be determined based on the position information of the second feature point and the target matrix, so as to determine the correspondence between pixel points on different images, so that better image or video effects can be obtained when the images are subsequently fused, tracked or spliced.
[0047] Among them, fusion can be the fusion of two adjacent images to effectively reduce noise. For example, after determining the target matrix, the target matrix can be used to map the second image (i.e., warp), so that the second image can be transformed to the angle of the first image. At this time, the first image and the second image can be fused. For example, there is noise in the first image during acquisition, resulting in missing data in area A, but the second image has data in area A. Then, the data in area A in the second image can be moved to the position of area A in the first image. Optionally, the position of the pixel point in area A in the second image in the first image can be determined based on the moving distance of the pixel point between the first image and the second image, so as to achieve complementary noise reduction of the two frames of images and obtain better image quality.
[0048] Among them, tracking can be to determine the position of a point on the second image on the first image based on the target matrix to achieve tracking of the pixel point; stitching can mean that multiple image / video acquisition devices respectively collect image data of adjacent areas, and then determine the correspondence between the images of the two groups of video frames, and stitch the images collected by the two devices to obtain a larger image.
[0049] In summary, the above-mentioned embodiments of the present disclosure can use the second matrix to update the first matrix, thereby utilizing the high correlation between video frames and reducing the differences in homography matrices between different images, thereby obtaining a better image alignment effect. When the aligned images are subsequently used for fusion, tracking or splicing, a better video effect can be obtained.
[0050] Figure 2 A schematic diagram of the process of an image alignment method provided in an embodiment of the present disclosure Figure 2 .like Figure 2 As shown, based on Figure 1 In the illustrated embodiment, the method includes the following steps.
[0051] Step 201: Determine the difference between the first matrix and the second matrix.
[0052] In some embodiments, in the process of updating the first matrix to obtain the mother bird matrix, a Kalman filter method can be used for updating. For example, the first matrix can be expressed as H(t-1, t), and the second matrix can be expressed as H(t-2, t-1). Then, the difference between the first matrix and the second matrix can be expressed as:
[0053] y=H(t-1,t)-H(t-2,t-1)
[0054] Optionally, in the Kalman filtering process, the first matrix can be used as the observation quantity H_measure(t-1,t) of the Kalman filter, that is, H(t-1,t)=H_measure(t-1,t). At this time, the difference between the first matrix and the second matrix can also be expressed as:
[0055] y=H measure (t-1,t)-H(t-2,t-1)
[0056] Step 202: Determine the gain parameter corresponding to the first matrix.
[0057] In some embodiments, optionally, the gain parameter corresponding to the first matrix can be a Kalman gain, and determining the gain parameter corresponding to the first matrix includes: determining the state covariance matrix corresponding to the first matrix; and determining the gain parameter corresponding to the first matrix based on the state covariance matrix corresponding to the first matrix and a preset noise parameter.
[0058] Optionally, the gain parameter corresponding to the first matrix may refer to a gain parameter used when updating the first matrix; the state covariance matrix P is updated as the Kalman filter updates, that is, the state covariance matrix P used each time the target matrix is calculated may be different. For example, when calculating the state covariance matrix P(t) corresponding to the first matrix, it may be determined according to the following formula:
[0059] P(t)=P(t-1)-K(t-1)*H(t-2,t-1)*P(t-1)
[0060] Among them, P(t-1) is the state covariance matrix corresponding to the second matrix, K(t-1) is the gain parameter corresponding to the second matrix, and H(t-2, t-1) is the second matrix. That is, the gain parameter and state covariance matrix corresponding to the second matrix can be used to update the state covariance matrix P(t) corresponding to the first matrix.
[0061] In some embodiments, the preset noise parameter may be a preset observation noise R. In other words, the value of the observation noise may be preset. For example, the value of the observation noise R may be preset as:
[0062]
[0063] In some embodiments, the gain parameter corresponding to the first matrix is determined according to the state covariance matrix corresponding to the first matrix and the preset noise parameter. For example, the gain parameter corresponding to the first matrix can be determined according to the following formula:
[0064] K=P(t)*(P(t)+R) -1
[0065] Among them, K is the gain parameter corresponding to the first matrix, P(t) is the state covariance matrix corresponding to the first matrix, and R is the preset noise parameter.
[0066] Optionally, when the second matrix is a homography matrix between the second frame image and the first frame image, the state covariance matrix corresponding to the second matrix is P(t-1)=p+Q, where p is a unit matrix, for example, a third-order unit matrix, and Q is a preset process noise. For example, the value of the process noise Q can be preset as:
[0067]
[0068] Step 203: Update the first matrix according to the difference between the first matrix and the second matrix, the gain parameter, and the second matrix to obtain a target matrix.
[0069] In some embodiments, optionally, the first matrix may be updated according to the difference between the first matrix and the second matrix, the gain parameter, and the second matrix to obtain a target matrix according to the following formula:
[0070] H est (t-1,t)=H(t-2,t-1)+K*y
[0071] Among them, H est (t-1, t) is the target matrix, H(t-2, t-1) is the second matrix, K is the gain parameter corresponding to the first matrix, and y is the difference between the first matrix and the second matrix.
[0072] In some embodiments, the method also includes: generating a state covariance matrix corresponding to a third matrix based on the gain parameters corresponding to the first matrix, the state covariance matrix corresponding to the first matrix, and the target matrix, the third matrix being used to indicate the correspondence between at least one feature point and at least one fourth feature point of a fourth image, where the fourth image is a subsequent frame image of the first image.
[0073] Similarly, the fourth image and the first image belong to the same video sequence. For example, the first image may be image T, and the fourth image may be image T+1, that is, the third matrix is the homography matrix between the first image and the fourth image. In other words, after calculating the target matrix between the first image and the second image, the state covariance matrix may be updated to obtain the state covariance matrix corresponding to the third matrix. Optionally, the state covariance matrix corresponding to the third matrix may be determined according to the following formula:
[0074] P(t+1)=P(t)-K*H est (t-1,t)*P(t)
[0075] Among them, P(t+1) is the state covariance matrix corresponding to the third matrix, P(t) is the state covariance matrix corresponding to the first matrix, K is the gain parameter corresponding to the first matrix, H est (t-1,t) is the target matrix.
[0076] In summary, the above-mentioned embodiments of the present application can use the second matrix to update the first matrix, which can realize the consistency between video frames and effectively maintain the consistency of the estimated homography matrix, so that the aligned sequence has no sudden alignment differences at different positions, maintains relatively good consistency, and finally the fused output video will not have sudden image distortion.
[0077] Figure 3 A schematic diagram of the process of an image alignment method provided in an embodiment of the present disclosure Figure 3 .like Figure 3 As shown, based on Figure 1 In the illustrated embodiment, the method includes the following steps.
[0078] Step 301: Use a target matrix to verify at least one feature pair included in the second set.
[0079] In some embodiments, after determining the target matrix, the target matrix can be used to filter outliers from the multiple feature pairs contained in the second set, that is, a more accurate target matrix can be used to filter the second set. Optionally, the name of the second set can also be an outlier set, that is, the second set may still contain some abnormal feature points.
[0080] In some embodiments, a target matrix is used to verify at least one feature pair included in the second set. For example, the target matrix can be used to map the first feature point to the second image, determine the coordinates of the first feature point mapped on the second image, and compare them with the coordinates of the second feature point. When the difference between the mapped coordinates of the first feature point and the coordinates of the second feature point is large, it is determined that the feature pair verification has failed. When the difference between the mapped coordinates of the first feature point and the coordinates of the second feature point is small, it is determined that the feature pair verification has succeeded.
[0081] Alternatively, similarly, the target matrix is used to verify at least one feature pair included in the second set. For example, the target matrix can be used to map the second feature point to the first image, determine the coordinates of the second feature point mapped on the first image, and compare them with the coordinates of the first feature point. When the difference between the mapped coordinates of the second feature point and the coordinates of the first feature point is large, it is determined that the feature pair verification has failed. When the difference between the mapped coordinates of the second feature point and the coordinates of the first feature point is small, it is determined that the feature pair verification has succeeded.
[0082] Step 302: If there is a feature pair that fails verification in at least one feature pair, remove the feature pair that fails verification from the second set to obtain a third set.
[0083] In some embodiments, after determining the feature pairs that failed verification, the feature pairs that failed verification can be eliminated from the second set to obtain a third set to filter the second set. Optionally, the third set can be the inner point set S_inner{curPt(t),prePt(t)}.
[0084] Step 303: Use the third set to optimize the target matrix.
[0085] In some embodiments, the third set may be used to optimize the target matrix to obtain a homography matrix with higher accuracy. Optionally, the optimization method may use Gauss-Newton optimization or Levenberg-Marquardt optimization algorithm, which is not limited in this disclosure.
[0086] In summary, the above embodiments of the present disclosure can optimize the target matrix to obtain a higher-precision target matrix, achieve better image alignment effects, and obtain better image or video effects during subsequent fusion, tracking, and splicing operations.
[0087] The technical solution of the present disclosure is further described in detail below in conjunction with specific application examples.
[0088] The following is a video frame image registration method provided by an embodiment of the present disclosure. This method takes into account the correlation between frames in a video sequence and uses the results of the previous alignment to guide the current alignment, ensuring that the final alignment result has better consistency in the time domain and eliminating jitter or localized floating issues in the final fusion effect. The details of this method are as follows.
[0089] In one implementation, Figure 4 The figure shows a video frame alignment scheme that uses the information of the previous frame. However, due to time reasons, there is often a displacement between the previous frame and the current frame. Therefore, it is necessary to calculate the mapping relationship of the previous frame to the current frame, and then use this relationship to warp the previous frame to obtain an image aligned with the current frame. Then, use this image to compare with the current frame to obtain redundant information to complete operations such as fusion, tracking, or splicing to form a new video sequence.
[0090] The above solution has a problem: the previous and next frames in a video sequence are correlated. If the homography matrix is estimated independently for different groups of image pairs, the resulting H matrix will have different alignment accuracy or error offsets due to the different distribution and accuracy of feature point pairs at different times. When this aligned information is superimposed back onto the video sequence, it may cause jumps in the timing.
[0091] Taking time-domain fusion noise reduction as an example, we can estimate the alignment matrix of the previous and next frames, use this estimated H matrix to warp the previous frame, and then fuse it with the current frame to effectively reduce noise. However, if the H matrices of the previous and next frames of a video frame are estimated independently, although from an independent perspective, they are the optimal alignment of the two frames, the accuracy of the local alignment may be different from the previous one. This will cause the fused video to experience jumps in accuracy and offset, ultimately resulting in the output video having a floating visual effect.
[0092] The improvement of this solution is to consider the estimation of the previous homography matrix when estimating the homography matrix of the current frame. Because the homography matrices of two consecutive frames are consistent in time, that is, there will be no large jumps, the Kalman filter is introduced to make the homography matrix estimated before the final homography matrix maintain the consistency of the time domain. In this way, when the feature point pairs may change in different frames, the estimated homography matrix can maintain high accuracy and good time domain consistency. The specific flow chart is as follows: Figure 5 shown.
[0093] To estimate the homography matrix at time t-1 and time t, we need to know the set of feature point pairs S{curPt(t), prePt(t)} for the frames before and after time t-1 and t. We also know the result of the homography matrix estimation for the previous frame, as shown in the diagram H(t-2, t-1).
[0094] First, use S{curPt(t), prePt(t)} to estimate the homography matrix H(t-1,t). Because there may be outlier pairs in the matching point pairs, the RANSAC method is used to estimate and remove outlier pairs. The homography matrix can be used as the observation H_measure(t-1,t) of the Kalman filter. Then, the homography matrix before time t is used as the state matrix. Because the process noise Q and observation noise R of the Kalman filter are difficult to estimate, we can initially give an empirical value, such as
[0095] As the Kalman filter process is updated, the state covariance matrix P is also updated. The Kalman matrix update process is as follows:
[0096] First, let H(t-1,t)=H(t-2,t-1); the state covariance matrix P(t)=P(t-1)+Q, where when the state covariance matrix is first calculated, P(t-1) is the unit matrix.
[0097] Update process:
[0098] y=H measure(t-1,t)-H(t-2,t-1)
[0099] K=P(t)*(P(t)+R) -1
[0100] H est (t-1,t)=H(t-2,t-1)+K*y
[0101] P(t+1)=P(t)-K*H est (t-1,t)*P(t)
[0102] After the update, the Kalman estimation matrix state covariance matrix P is also updated and saved for use in the next estimation of the H(t, t+1) matrix. The obtained Kalman estimation matrix filters the set of input feature point pairs S{curPt(t), prePt(t)} to obtain the inner point set S_inner{curPt(t), prePt(t)}.
[0103] Finally, the Kalman estimation matrix is optimized using the interior point set S_inner{curPt(t), prePt(t)} to obtain the optimal solution for the interior points near the matrix. The optimization method can use Gauss-Newton optimization or Levenberg-Marquardt optimization algorithm.
[0104] In summary, the above examples of the present disclosure have an improvement effect on the floating stretching in the video sequence using time domain fusion noise reduction. Time domain fusion noise reduction of video is a technology that reduces noise by combining the information of adjacent frames in the time dimension. The basic idea of this method is to utilize the high correlation between consecutive frames in the video sequence. Due to the correlation between adjacent frames in space and time, the information they contain can be complementary, thereby improving the overall image quality. The basis of time domain fusion is to align and estimate adjacent frames, and then distort the previous frame according to the estimated homography matrix to align it with the current frame and then fuse it to achieve the purpose of reducing the signal-to-noise ratio. The use of this method can effectively maintain the consistency of the estimated homography matrix, so that the aligned sequence does not have sudden alignment differences at different positions, maintains relatively good consistency, and finally the fused output video will not have sudden image distortion.
[0105] Figure 6 FIG. 6 is a structural diagram of an image alignment device 600 provided in an embodiment of the present disclosure. Figure 6As shown, the device includes: a first processing unit 610, used to determine a first matrix, the first matrix is used to indicate the correspondence between at least one first feature point of the first image and at least one second feature point of the second image, and the first image is a subsequent frame image of the second image; a second processing unit 620, used to update the first matrix using the second matrix to obtain a target matrix, the second matrix is used to indicate the correspondence between at least one second feature point and at least one third feature point of the third image, and the second image is a subsequent frame image of the third image; a third processing unit 630, used to align the first image and the second image using the target matrix.
[0106] In some embodiments, the first processing unit is further used to generate a first set based on at least one first feature point and at least one second feature point; eliminate outliers in the first set to obtain a second set, the second set including at least one feature pair, each feature pair in at least one feature pair including a first feature point in at least one first feature point and a second feature point in at least one second feature point; generate a first matrix based on at least one feature pair included in the second set.
[0107] In some embodiments, the second processing unit is further used to determine the difference between the first matrix and the second matrix; determine the gain parameter corresponding to the first matrix; and update the first matrix according to the difference between the first matrix and the second matrix, the gain parameter, and the second matrix to obtain a target matrix.
[0108] In some embodiments, the second processing unit is further configured to determine a state covariance matrix corresponding to the first matrix; and determine a gain parameter corresponding to the first matrix according to the state covariance matrix corresponding to the first matrix and a preset noise parameter.
[0109] In some embodiments, the image alignment device also includes a fourth processing unit, which is used to generate a state covariance matrix corresponding to a third matrix based on the gain parameters corresponding to the first matrix, the state covariance matrix corresponding to the first matrix, and the target matrix. The third matrix is used to indicate the correspondence between at least one feature point and at least one fourth feature point of a fourth image, and the fourth image is the next frame image of the first image.
[0110] In some embodiments, the fourth processing unit is further used to verify at least one feature pair included in the second set using the target matrix; if there is a feature pair that fails verification in at least one feature pair, the feature pair that fails verification is eliminated from the second set to obtain a third set; and the target matrix is optimized using the third set.
[0111] In summary, the image alignment device 600 can use the homography matrix of the previous frame to update the current homography matrix when determining the homography matrix corresponding to the current frame, thereby utilizing the high correlation between video frames to obtain a better image alignment effect. When the aligned images are subsequently used for fusion, tracking or splicing, a better video effect can be obtained.
[0112] In the embodiments provided above, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the various functions of the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the aforementioned functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. One of the aforementioned functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.
[0113] Figure 7 FIG2 is a block diagram of an electronic device 700 for implementing the above method according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0114] Reference Figure 7 , electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .
[0115] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 702 may include one or more modules to facilitate interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate interaction between the multimedia component 708 and the processing component 702.
[0116] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.
[0117] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.
[0118] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.
[0119] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.
[0120] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.
[0121] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of components, such as the display and keypad of the electronic device 700. The sensor assembly 714 can also detect changes in the position of the electronic device 700 or a component of the electronic device 700, the presence or absence of user contact with the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and temperature changes of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0122] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0123] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0124] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by the processor 720 of the electronic device 700 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0125] The embodiments of the present disclosure further provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the above embodiments of the present disclosure.
[0126] Figure 8 FIG. 8 is a schematic diagram showing a structure of a chip 800 for implementing the above method according to an exemplary embodiment. Figure 8 The chip 800 includes a communication interface 801 and at least one processor 802. The communication interface 801 is used to receive signals input into the chip 800 or signals output from the above chip 800. The processor 802 communicates with the communication interface 801 and implements the method described in the above embodiments of the present disclosure through logic circuits or executing code instructions.
[0127] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.
[0128] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "example," "specific example," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in at least one embodiment or example.
[0129] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.
[0130] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection having at least one wire (control method), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.
[0131] It should be understood that various parts of the embodiments of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0132] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.
[0133] Furthermore, the functional units in the various embodiments of the present invention may be integrated into a single processing module, each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in either hardware or software functional modules. If the integrated modules are implemented as software functional modules and sold or used as standalone products, they may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.
[0134] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.
Claims
1. An image alignment method, characterized in that: The method comprises: Determine a first matrix, where the first matrix is used to indicate a correspondence between at least one first feature point of a first image and at least one second feature point of a second image, where the first image is a subsequent frame of the second image; updating the first matrix using a second matrix to obtain a target matrix, wherein the second matrix is used to indicate a correspondence between the at least one second feature point and at least one third feature point of a third image, where the second image is a subsequent frame of the third image; The first image and the second image are aligned using the target matrix.
2. The method according to claim 1, characterized in that Determining the first matrix includes: generating a first set according to the at least one first feature point and the at least one second feature point; Eliminating outliers in the first set to obtain a second set, where the second set includes at least one feature pair, each feature pair in the at least one feature pair includes a first feature point in the at least one first feature point and a second feature point in the at least one second feature point; The first matrix is generated according to the at least one eigenpair included in the second set.
3. The method according to claim 1, characterized in that The updating of the first matrix by using the second matrix to obtain a target matrix includes: determining a difference between the first matrix and the second matrix; Determining a gain parameter corresponding to the first matrix; The first matrix is updated according to the difference between the first matrix and the second matrix, the gain parameter, and the second matrix to obtain the target matrix.
4. The method according to claim 3, characterized in that Determining the gain parameter corresponding to the first matrix includes: Determining a state covariance matrix corresponding to the first matrix; A gain parameter corresponding to the first matrix is determined according to a state covariance matrix corresponding to the first matrix and a preset noise parameter.
5. The method according to claim 4, characterized in that The method further comprises: A state covariance matrix corresponding to a third matrix is generated according to the gain parameters corresponding to the first matrix, the state covariance matrix corresponding to the first matrix, and the target matrix. The third matrix is used to indicate the correspondence between the at least one feature point and at least one fourth feature point of a fourth image, where the fourth image is a subsequent frame image of the first image.
6. The method according to claim 2, characterized in that The method further comprises: verifying the at least one feature pair included in the second set using the target matrix; If there is a feature pair that fails verification in the at least one feature pair, remove the feature pair that fails verification from the second set to obtain a third set; The target matrix is optimized using the third set.
7. An image alignment device, comprising: a first processing unit, configured to determine a first matrix, where the first matrix is used to indicate a correspondence between at least one first feature point of a first image and at least one second feature point of a second image, where the first image is a frame subsequent to the second image; a second processing unit, configured to update the first matrix using a second matrix to obtain a target matrix, wherein the second matrix is used to indicate a correspondence between the at least one second feature point and at least one third feature point of a third image, where the second image is an image frame subsequent to the third image; A third processing unit is configured to align the first image and the second image using the target matrix.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.
10. A chip, characterized in that: The method comprises at least one processor and a communication interface; the communication interface is used to receive a signal input to the chip or a signal output from the chip, and the processor communicates with the communication interface and implements the method as described in any one of claims 1 to 6 through a logic circuit or executing code instructions.