Image alignment method, device and equipment

Abstract

The invention relates to the technical field of image processing, in particular to an image alignment method, device and equipment. The method comprises the following steps: acquiring an image frame sequence and an equipment pose sequence, wherein image frames in the image frame sequence and equipment pose data in the equipment pose sequence have a corresponding relationship; determining a reference frame and a frame to be aligned in the image frame sequence; determining an alignment matrix of the to-be-aligned frame relative to the reference frame according to the equipment pose data; performing coarse alignment between the to-be-aligned frame and the reference frame according to the alignment matrix; performing the same block processing on the reference frame and the to-be-aligned frame after the coarse alignment; determining in-block matching points from the block images of the reference frame and the frame to be aligned through image feature matching; determining a transformation matrix of the to-be-aligned frame relative to the reference frame according to the in-block matching points; and according to the transformation matrix, performing fine alignment between the to-be-aligned frame and the reference frame. According to the image alignment scheme provided by the embodiment of the invention, the alignment effect and the alignment efficiency of the image frames can be considered.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and more particularly to an image alignment method, apparatus, and device. Background Art

[0002] A single-frame image captured by a single camera usually only reflects a limited scene under the same scene, the same exposure level, and the same viewing angle. To make the image content richer, the image color range wider, the image noise smoother, and the image details clearer, image fusion technologies such as image stitching, image enhancement, image filtering, and image super-resolution based on multiple frames of images have emerged in the related art. However, since a single camera cannot capture multiple frames of images at the same time, in order to avoid problems such as ghosting in the fusion result that affect the fusion quality, it is necessary to perform inter-frame alignment on the image frames before fusing the image frames. The alignment result between the image frames directly affects the fusion effect of the image frames. Therefore, how to perform inter-frame alignment on the image frames has become a technical problem to be solved. Summary of the Invention

[0003] In view of this, this application provides an image alignment method, apparatus, and device that perform rough alignment on the image frames and then perform block-based fine alignment to balance the alignment effect and alignment efficiency of the image frames.

[0004] In a first aspect, an embodiment of the present invention provides an image alignment method, including:

[0005] Obtain an image frame sequence and a device pose sequence, where the image frames in the image frame sequence have a corresponding relationship with the device pose data in the device pose sequence;

[0006] Determine a reference frame and a frame to be aligned in the image frame sequence;

[0007] Determine an alignment matrix of the frame to be aligned relative to the reference frame according to the device pose data corresponding to the reference frame and the frame to be aligned respectively;

[0008] Perform rough alignment between the frame to be aligned and the reference frame according to the alignment matrix;

[0009] Perform the same block processing on the reference frame and the frame to be aligned after rough alignment;

[0010] Determine in-block matching points from the block images of the reference frame and the frame to be aligned through image feature matching;

[0011] Determine a transformation matrix of the frame to be aligned relative to the reference frame according to the in-block matching points;

[0012] Perform fine alignment between the frame to be aligned and the reference frame according to the transformation matrix.

[0013] In some embodiments, there is a correspondence between the image frames in the image frame sequence and the device pose data in the device pose sequence, including:

[0014] Predetermine the time difference value between the image frame timestamp and the device pose data timestamp;

[0015] Determine the correspondence between the image frames in the image frame sequence and the device pose data in the device pose sequence according to the time difference value.

[0016] In some embodiments, the predetermining the time difference value between the image frame timestamp and the device pose data timestamp includes:

[0017] Obtain continuous multiple-frame image frame samples and device pose data samples of a specified scene, where the specified scene includes a rich texture scene without local motion;

[0018] Calculate the inter-frame transformation matrix between the image frame samples through image feature matching;

[0019] Label different device pose data samples onto the image frame samples respectively, and calculate the inter-frame alignment matrix between the image frame samples by using the labeled device pose data samples respectively;

[0020] Determine the difference between the inter-frame transformation matrix and the inter-frame alignment matrix;

[0021] When the difference takes the minimum value, determine the time difference value according to the timestamp of the image frame sample and the timestamp of the device pose data sample labeled on the image frame sample.

[0022] In some embodiments, the determining the alignment matrix of the frame to be aligned relative to the reference frame according to the device pose data respectively corresponding to the reference frame and the frame to be aligned includes:

[0023] Determine the angular difference information between the frame to be aligned and the reference frame according to the time difference and the device pose data between the reference frame and the frame to be aligned;

[0024] Determine the external camera rotation matrix between the frame to be aligned and the reference frame according to the angular difference information;

[0025] Determine the alignment matrix of the frame to be aligned relative to the reference frame according to the external camera rotation matrix and the internal camera matrix.

[0026] In some embodiments, the determining the in-block matching points from the block images of the reference frame and the frame to be aligned through image feature matching includes:

[0027] Determine the texture complexity of each image block of the reference frame and the frame to be aligned;

[0028] According to the texture complexity, filter target image blocks from each image block of the reference frame and the frame to be aligned, wherein the target image blocks at the same position of the reference frame and the frame to be aligned form a target image block pair;

[0029] Determine the image feature matching point pairs in the target image block pair through image feature matching;

[0030] Filter the in-block matching points from the image feature matching point pairs.

[0031] In some embodiments, the filtering the in-block matching points from the image feature matching point pairs includes:

[0032] Determine the matching degree of the image feature matching point pairs in each target image block pair;

[0033] Determine the image feature matching point pairs with a matching degree greater than a set value in each target image block pair as the in-block matching points.

[0034] In some embodiments, the determining the transformation matrix of the frame to be aligned relative to the reference frame according to the in-block matching points includes:

[0035] Convert the coordinates of the in-block matching points into full-image coordinates;

[0036] Determine the transformation matrix of the frame to be aligned relative to the reference frame according to the full-image coordinates of the in-block matching points.

[0037] In some embodiments, before performing the same block processing on the reference frame and the frame to be aligned after rough alignment, the method further includes:

[0038] If there is a brightness difference between the reference frame and the frame to be aligned after rough alignment, perform brightness matching between the reference frame and the frame to be aligned; and / or,

[0039] If there is a sharpness difference between the reference frame and the frame to be aligned after rough alignment, improve the sharpness of the frame to be aligned with lower sharpness.

[0040] In a second aspect, an embodiment of the present invention provides an image alignment device, including:

[0041] An input module, configured to obtain an image frame sequence and a device pose sequence, and there is a corresponding relationship between the image frames in the image frame sequence and the device pose data in the device pose sequence;

[0042] An operation module, configured to determine a reference frame and a frame to be aligned in the image frame sequence; determine an alignment matrix of the frame to be aligned relative to the reference frame according to the device pose data corresponding to the reference frame and the frame to be aligned respectively; perform rough alignment between the frame to be aligned and the reference frame according to the alignment matrix; perform the same block processing on the reference frame and the frame to be aligned after rough alignment; determine in-block matching points from the block images of the reference frame and the frame to be aligned through image feature matching; determine a transformation matrix of the frame to be aligned relative to the reference frame according to the in-block matching points; and perform fine alignment between the frame to be aligned and the reference frame according to the transformation matrix.

[0043] An output module, configured to output the aligned image frame sequence.

[0044] In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is caused to execute the method described in the first aspect or any item of the first aspect above.

[0045] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the method described in the first aspect or any item of the first aspect above.

[0046] The solution of the embodiment of the present invention is an image alignment solution that integrates image features and device pose data. In the embodiment of the present invention, applying the device pose data corresponding to the image frame to the rough alignment process of the image frame can suppress the interference of large-scale motion on the rough alignment result. Moreover, based on the rough alignment, the embodiment of the present invention performs block feature extraction and matching on the image frame, which can avoid the perturbation of local motion on the alignment result and improve the fine degree of image frame alignment. Block alignment of the image frame based on rough alignment in the embodiment of the present invention is also beneficial to realizing parallel computing in the alignment process, can effectively reduce the computational amount of image feature extraction, and improve the alignment efficiency while taking into account the alignment effect. Description of the Drawings

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 It is a flowchart of an image alignment method provided by an embodiment of the present invention.

[0049] Figure 2 A flowchart of a method for determining a time difference value between an image frame and device pose data provided by an embodiment of the present invention;

[0050] Figure 3 A flowchart of another image alignment method provided by an embodiment of the present invention;

[0051] Figure 4 A schematic structural diagram of an image alignment device provided by an embodiment of the present invention;

[0052] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0053] For a better understanding of the technical solution of the present application, the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0054] It should be clear that the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.

[0055] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "the" and "said" used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0056] It should be understood that the term " / and / " used herein is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0057] Image alignment is a solution for aligning image data from different perspectives, different times, or different imaging conditions by warping and rotating (which can also be called affine transformation) to the same coordinate system to complete the alignment of image data.

[0058] Image alignment solutions may include image alignment solutions based on image content and image alignment solutions based on hardware output.

[0059] The image alignment scheme based on image content mainly calculates the transformation matrix between the frame to be aligned and the reference frame by using the image statistical values or image features of the frame to be aligned and the reference frame, and aligns the frame to be aligned by using the transformation matrix. This scheme has a strong correlation with the image content. Between image frames with local motion or large parallax, usually due to large deviations in image statistics and poor image feature matching, the problem of poor final correction and alignment effect is caused, making its application scenarios have certain limitations.

[0060] The image alignment scheme based on hardware output calculates the alignment matrix between the frame to be aligned and the reference frame by using the device pose data output by the hardware, and aligns the frame to be aligned by using the alignment matrix. The alignment accuracy of this scheme is restricted by the hardware output accuracy.

[0061] Considering the advantages and disadvantages of the above image alignment schemes and the fact that the hardware for outputting device pose data has been relatively popular in electronic devices, the embodiments of the present invention propose an image alignment scheme that fuses image content and hardware output. This scheme does not have very high requirements for the accuracy of hardware devices, and there will not be a large increase in device costs. Moreover, this scheme can improve the alignment effect and alignment efficiency, and the investment in technical costs is also relatively small.

[0062] See Figure 1 It is a flowchart of an image alignment method provided by an embodiment of the present invention. Figure 1 The method shown is an image alignment scheme that fuses image features and device pose data. As Figure 1 shown, the processing steps of this method include:

[0063] 101. Obtain an image frame sequence and a device pose sequence. The image frame sequence includes several image frames, and there is a corresponding relationship between each image frame in the image frame sequence and the device pose data in the device pose sequence. Among them, the difference between the timestamps of the corresponding image frame and the device pose data is less than a set value, that is, it can be considered that the timestamps of the corresponding image frame and the device pose data are relatively synchronized.

[0064] 102. Determine the reference frame and the frame to be aligned in the image frame sequence. Among them, the reference frame and the frame to be aligned can be determined according to the clarity and exposure of each frame image in the image frame sequence. Optionally, the image frame with the highest clarity and normal exposure in the image frame sequence can be determined as the reference frame, and the other frames in the image frame sequence can be determined as the frames to be aligned.

[0065] 103. Determine the alignment matrix of the frame to be aligned relative to the reference frame according to the device pose data corresponding to the reference frame and the frame to be aligned respectively.

[0066] 104. Coarsely align the frame to be aligned and the reference frame according to the alignment matrix. Optionally, apply the alignment matrix to the frame to be aligned to correct the frame to be aligned to the device pose where the reference frame is located, thereby reducing the alignment error caused by device movement or the movement of the photographed object.

[0067] 105. Perform the same block processing on the reference frame and the frame to be aligned after coarse alignment. An optional block method, for example, can be: perform the same number of block operations on the horizontal and vertical directions of the image frame. See Figure 2 , which is a schematic diagram of a reference frame and a frame to be aligned provided by an embodiment of the present invention. As Figure 2 shown, when performing block processing on the reference frame and the frame to be aligned, 4x4 block operations can be performed on the horizontal and vertical directions of the reference frame and the frame to be aligned.

[0068] 106. Determine the in-block matching points from the block images of the reference frame and the frame to be aligned through image feature matching. As Figure 2 shown, after performing the same block processing on the reference frame and the frame to be aligned, the image blocks at the same positions on the reference frame and the frame to be aligned form an image block pair. As Figure 2 shown, block a1 in the reference frame and block a2 in the frame to be aligned form an image block pair, block b1 in the reference frame and block b2 in the frame to be aligned form an image block pair, block c1 in the reference frame and block c2 in the frame to be aligned form an image block pair, and block d1 in the reference frame and block d2 in the frame to be aligned form an image block pair. Other image block pairs in the reference frame and the frame to be aligned are not listed one by one here. Through image feature matching, the in-block matching points in each image block pair of the reference frame and the frame to be aligned can be calculated.

[0069] 107. Determine the transformation matrix of the frame to be aligned relative to the reference frame according to the in-block matching points. Optionally, the in-block coordinates of the in-block matching points of the reference frame and the frame to be aligned can be converted into full-image coordinates to calculate the transformation matrix of the frame to be aligned relative to the reference frame.

[0070] 108. Perform fine alignment between the frame to be aligned and the reference frame according to the transformation matrix.

[0071] It can be seen from the above embodiments that the method of the embodiment of the present invention adds a step of coarse alignment according to device pose data before performing fine alignment according to image features. The method of the embodiment of the present invention is an image alignment scheme that integrates multiple information such as image features and device pose data.

[0072] In the method of the embodiment of the present invention, device pose data is introduced during the process of image alignment for rough alignment of image frames. Since the acquisition of device pose data is real-time, introducing less time cost for rough alignment of image frames, and the acquisition of device pose data is not affected by the image content, which can reduce the alignment error caused by the movement of the electronic device or the movement of the shooting object to a certain extent.

[0073] Based on the rough alignment, the method of the embodiment of the present invention introduces an accurate alignment scheme for image feature extraction and matching after dividing the image frames into blocks. Through the operation of image block alignment, this scheme avoids the possibility of local motion feature points crossing blocks for matching to a certain extent, reduces the influence of local motion on global alignment, and the block operation is also conducive to parallel computing to improve the computing speed.

[0074] Furthermore, the method of the embodiment of the present invention can suppress large-scale motion in the image frame based on device pose data for rough alignment of the image frame. After rough alignment of the image frame, fine alignment based on the image features of the divided images can reduce the number of pyramid layers and the number of feature points, reduce the amount of fine alignment computation, shorten the time required for fine alignment, and improve the overall alignment efficiency and alignment accuracy.

[0075] In the embodiment of the present invention, device pose data can be collected by a device pose sensor in the electronic device. The device pose sensor can be, for example, a gyroscope (Gyro) sensor and / or an accelerometer (ACC) sensor, etc. The collected device pose data can include, for example, measurement data such as angular velocity data and acceleration.

[0076] In the embodiment of the present invention, the electronic device captures an image frame through a camera module, and the device pose sensor synchronously collects device pose data during the process of capturing the image frame. Since the camera module and the device pose sensor belong to different hardware structures, the output image frame and device pose data have their own timestamps respectively. To ensure the time synchronization of the image frame and the device pose data, the time difference value between the image frame and the device pose data can be determined in advance. According to the timestamp of the image frame and the time difference value, the time range of the device pose data relatively synchronized with the image frame time can be determined, and then the corresponding device pose data can be selected to label the image frame.

[0077] In the embodiment of the present invention, the method of determining the time difference value between the image frame and the device pose data in advance can include:

[0078] 201. Obtain continuous multiple-frame image frame samples of a specified scenario and device pose data samples collected synchronously. The specified scenario can be a simple scenario, and a simple scenario can be a rich texture scenario without local motion to avoid the influence of local motion on the image frames. During data collection, the electronic device moves uniformly around the axis of the device pose sensor to ensure that the data obtained by the device pose sensor is uniformly and effectively acquired.

[0079] 202. Use an image feature matching algorithm to calculate the inter-frame transformation matrix between each image frame sample.

[0080] 203. Label different device pose data samples onto the image sample frames respectively, and use the labeled device pose data samples to calculate the inter-frame alignment matrix between the image frame samples respectively.

[0081] 204. Determine the difference between the inter-frame transformation matrix corresponding to the image frame sample and each inter-frame alignment matrix. In a specific example, the sampling frequency of the image frame sample is 30 frames per second, and the sampling frequency of the device pose data sample is 300 frames per second. Then, within the sampling time of one image frame sample, there are 10 device pose data corresponding to it. Then, the 10 device pose data samples can be respectively labeled for the image frame sample. Based on the device pose data samples respectively labeled for the image frame sample, the inter-frame alignment matrix corresponding to the image frame sample can be calculated respectively. Calculate the difference between the inter-frame alignment matrix corresponding to the image frame sample and the inter-frame transformation matrix calculated by the image feature matching algorithm respectively, and determine the minimum difference.

[0082] 205. When the above difference takes the minimum value, determine the above time difference value according to the time stamp of the image frame sample and the time stamp of the device pose data sample labeled for the image frame sample.

[0083] Using the above time difference value, the corresponding relationship between the data in the image frame sequence and the device pose sequence can be determined, and using this time difference value, pre-calibration of the image frames and device pose data can be achieved.

[0084] See Figure 3 , which is a flowchart of another image alignment method provided by an embodiment of the present invention. As Figure 3 shown, the processing steps of this method include:

[0085] 301. Input an image frame sequence and a device pose sequence collected by a single camera. There is a corresponding relationship between the image frames in the image frame sequence and the device pose data in the device pose sequence. In some embodiments, the corresponding relationship between the image frames and the device pose data can be determined according to the time difference value between the image frame time stamp and the device pose data time stamp.

[0086] 302. Determine the reference frame and the frame to be aligned in the image frame sequence. Optionally, the image frame with the highest clarity and a normal exposure in the image frame sequence can be determined as the reference frame, and the other frames can be determined as the frames to be aligned.

[0087] 303. Utilize the device pose data (ω x , ω y , ω z ) corresponding to the reference frame and the frame to be aligned, and the time difference Δt between the reference frame and the frame to be aligned, and calculate the angular difference information θ between the frame to be aligned and the reference frame. The angular difference information θ represents the rotation angle generated by the electronic device during the time of capturing the frame to be aligned and the reference frame, and this rotation angle brings about changes in the image features of the frame to be aligned relative to the reference frame.

[0088] In some embodiments, the angular difference information θ between the frame to be aligned and the reference frame can be calculated according to the formula θ = ωΔt.

[0089] 304. Determine the external camera rotation matrix (R x , R y , R z ) between the frame to be aligned and the reference frame according to the angular difference information θ between the frame to be aligned and the reference frame. Among them, the calculation formula of the external camera rotation matrix (R x , R y , R z ) is as shown below.

[0090]

[0091] 305. Determine the alignment matrix W of the frame to be aligned relative to the reference frame according to the external camera rotation matrix and the internal camera matrix.

[0092] In some examples, according to the formula W = K * R z * R y * R y * K -1 , calculate the alignment matrix W of the frame to be aligned relative to the reference frame, where K is the internal camera matrix.

[0093] 306. Apply the alignment matrix W to the matrix to be aligned to correct the frame to be aligned to the device pose where the reference frame is located, thereby achieving rough alignment of the matrix to be aligned.

[0094] 307. Perform the same block processing on the roughly aligned reference frame and the frame to be aligned. For example Figure 2As shown, when partitioning the reference frame and the frame to be aligned, 4x4 partitioning operations can be performed separately on the horizontal and vertical directions of the reference frame and the frame to be aligned. The reference frame and the frame to be aligned are each divided into 16 image blocks. After performing the same partitioning process on the reference frame and the frame to be aligned, the image blocks at the same positions on the reference frame and the frame to be aligned form image block pairs.

[0095] 308, determine the texture complexity of each image block of the reference frame or the frame to be aligned. In Figure 2 the given example, the texture complexity of 16 image blocks of the reference frame and the frame to be aligned can be calculated separately. Or, the texture complexity of only 16 image blocks of the reference frame can be calculated. Or, the texture complexity of only 16 image blocks of the frame to be aligned can be calculated. In some examples, the variance statistical information of the pixel values within the block can be used to determine the texture complexity of each image block.

[0096] 309, according to the texture complexity of each image block of the reference frame and the frame to be aligned, screen out target image blocks from each image block of the reference frame and the frame to be aligned. In Figure 2 the given example, the image blocks with texture complexity greater than a certain value can be determined as target image blocks. The target image blocks at the same positions on the reference frame and the frame to be aligned form target image block pairs. In some examples, if block a1 and c1 of the reference frame are determined as target image blocks, and block b2 and d2 of the frame to be aligned are determined as target image blocks, then a1 - a2 is determined as a target image block pair, b1 - b2 is determined as a target image block pair, c1 - c2 is determined as a target image block pair, and d1 - d2 is determined as a target image block pair. In some examples, only the texture complexity of each image block of the reference frame is calculated, and a1, b1, c1, and d1 are determined as target image blocks according to the texture complexity. Then, the image blocks a1 - a2 at the same positions on the reference frame and the frame to be aligned are determined as target image block pairs, b1 - b2 is determined as a target image block pair, c1 - c2 is determined as a target image block pair, and d1 - d2 is determined as a target image block pair. In some examples, only the texture complexity of each image block of the frame to be aligned is calculated, and a2, b2, c2, and d2 are determined as target image blocks according to the texture complexity. Then, the image blocks a1 - a2 at the same positions on the reference frame and the frame to be aligned are determined as target image block pairs, b1 - b2 is determined as a target image block pair, c1 - c2 is determined as a target image block pair, and d1 - d2 is determined as a target image block pair.

[0097] 310, determine the image feature matching point pairs in the target image block pairs through image feature matching. Optionally, the ORB features of each image block can be collected, and the image feature matching point pairs in the target image block pairs can be determined through the brute - force matching method.

[0098] 311. Determine the matching degree of the image feature matching point pairs in each target image block pair.

[0099] 312. Determine the image feature matching point pairs with a matching degree greater than a set value in each target image block pair as in-block matching points. Optionally, the top 10% of the image feature matching point pairs with the highest matching degrees in each target image block pair can be determined as in-block matching points.

[0100] 313. Convert the coordinates of the in-block matching points to full-image coordinates, and determine the transformation matrix of the frame to be aligned relative to the reference frame according to the full-image coordinates of the in-block matching points. In some examples, the RANSAC method can be used to calculate the transformation matrix using the full-image coordinates of the in-block matching points.

[0101] 314. Perform fine alignment between the frame to be aligned and the reference frame according to the transformation matrix, and output the image frame sequence that has been multi-information fused and aligned.

[0102] In the method of the embodiment of the present invention, if there is a brightness difference between the reference frame and the frame to be aligned after rough alignment, a brightness matching operation is performed between the reference frame and the frame to be aligned. For example, a histogram matching scheme can be used to perform brightness matching on the reference frame and the frame to be aligned.

[0103] In the method of the embodiment of the present invention, if there is a sharpness difference between the reference frame and the frame to be aligned after rough alignment, the sharpness of the image frame with low sharpness, usually the frame to be aligned, is improved. For example, the image frame with low sharpness can be sharpened to improve the sharpness of the corresponding image frame.

[0104] See Figure 4 , which is a schematic structural diagram of an image alignment device provided by an embodiment of the present invention. As Figure 4 shown, the device includes:

[0105] An input module 401, configured to obtain an image frame sequence and a device pose sequence, and there is a corresponding relationship between the image frames in the image frame sequence and the device pose data in the device pose sequence;

[0106] An operation module 402, configured to determine a reference frame and a frame to be aligned in the image frame sequence; determine an alignment matrix of the frame to be aligned relative to the reference frame according to the device pose data corresponding to the reference frame and the frame to be aligned; perform rough alignment between the frame to be aligned and the reference frame according to the alignment matrix; perform the same block processing on the reference frame and the frame to be aligned after rough alignment; determine in-block matching points from the block images of the reference frame and the frame to be aligned through image feature matching; determine the transformation matrix of the frame to be aligned relative to the reference frame according to the in-block matching points; perform fine alignment between the frame to be aligned and the reference frame according to the transformation matrix;

[0107] The output module 403 is configured to output the aligned image frame sequence.

[0108] The image alignment device according to the embodiment of the present invention can execute the image alignment method of the above - shown embodiment. For the parts not described in detail in the embodiment of the present invention, reference can be made to the relevant descriptions in the method embodiment. The execution process and technical effects of this technical solution can be seen in the description of the method embodiment, which will not be elaborated here.

[0109] See Figure 5 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. As Figure 5 shown, the electronic device 500 may include: a processor 501, a memory 502, and a communication unit 503. These components communicate through one or more buses. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiments of the present application. It can be a bus - shaped structure, a star - shaped structure, and may also include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0110] Among them, the communication unit 503 is configured to establish a communication channel, so that the electronic device can communicate with other devices. Receive user data sent by other devices or send user data to other devices.

[0111] The processor 501 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines. By running or executing software programs, instructions, and / or modules stored in the memory 502, and by calling data stored in the memory, it executes various functions of the electronic device and / or processes data. The processor can be composed of an integrated circuit (IC). For example, it can be composed of a single - packaged IC, or can be composed of multiple packaged ICs with the same or different functions connected together. For example, the processor 501 may include a central processing unit (CPU), a microcontroller unit (MCU), etc.

[0112] The memory 502 is used to store the execution instructions of the processor 501. The memory 502 can be implemented by any type of volatile or non - volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read - only memory (EEPROM), erasable programmable read - only memory (EPROM), programmable read - only memory (PROM), read - only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0113] When the execution instructions in the memory 502 are executed by the processor 501, the electronic device 500 is enabled to execute the image alignment method in the embodiments of the present invention.

[0114] In a specific implementation, the present application further provides a computer storage medium. The computer storage medium may store a program, and when the program is executed, it may include some or all of the steps in the embodiments of the image alignment method provided by the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), or the like.

[0115] In a specific implementation, the present application further provides a computer program product. The computer program product includes executable instructions, and when the executable instructions are executed on a computer, the computer is enabled to execute some or all of the steps in the embodiments of the image alignment method provided by the present application.

[0116] The embodiments of the present application further provide a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to execute the image alignment method provided by the embodiments of the present application.

[0117] The above non-transitory computer-readable storage medium may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0118] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0119] The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including—but not limited to—wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0120] Those skilled in the art can clearly understand that the technologies in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solutions in the embodiments of the present application, in essence, or the parts that contribute to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present application.

[0121] For the same or similar parts among the various embodiments in this specification, reference may be made to each other. In particular, for the apparatus embodiments and the terminal embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts may refer to the descriptions in the method embodiments.

Claims

1. An image alignment method, characterized in that: include: Acquire an image frame sequence and a device posture sequence, wherein the image frames in the image frame sequence and the device posture data in the device posture sequence have a corresponding relationship; Determining a reference frame and a frame to be aligned in the image frame sequence; Determine an alignment matrix of the frame to be aligned relative to the reference frame according to device pose data corresponding to the reference frame and the frame to be aligned respectively; Performing coarse alignment between the frame to be aligned and the reference frame according to the alignment matrix; Perform the same block processing on the reference frame and the frame to be aligned after rough alignment; Determine intra-block matching points from the block images of the reference frame and the frame to be aligned by image feature matching; Determining a transformation matrix of the frame to be aligned relative to the reference frame according to the intra-block matching points; According to the transformation matrix, the frame to be aligned is precisely aligned with the reference frame.

2. The method according to claim 1, characterized in that The image frames in the image frame sequence and the device posture data in the device posture sequence have a corresponding relationship, including: Predetermining a time difference value between an image frame timestamp and a device pose data timestamp; The corresponding relationship between the image frames in the image frame sequence and the device posture data in the device posture sequence is determined according to the time difference value.

3. The method according to claim 2, characterized in that The predetermining of the time difference value between the image frame timestamp and the device pose data timestamp includes: Acquire a plurality of continuous image frame samples and device posture data samples of a specified scene, wherein the specified scene includes a rich texture scene without local motion; Calculate the inter-frame transformation matrix between image frame samples by image feature matching; Different device pose data samples are respectively labeled on the image frame samples, and the inter-frame alignment matrices between the image frame samples are respectively calculated using the labeled device pose data samples; Determine a difference between the inter-frame transformation matrix and the inter-frame alignment matrix; When the difference value takes a minimum value, the time difference value is determined according to the timestamp of the image frame sample and the timestamp of the device posture data sample marked by the image frame sample.

4. The method according to claim 1, characterized in that The determining, according to the device pose data respectively corresponding to the reference frame and the frame to be aligned, an alignment matrix of the frame to be aligned relative to the reference frame, comprises: Determine angle difference information between the frame to be aligned and the reference frame according to the time difference between the reference frame and the frame to be aligned and the device posture data; Determine a camera extrinsic rotation matrix between the frame to be aligned and the reference frame according to the angle difference information; An alignment matrix of the matrix to be aligned relative to the reference frame is determined according to the camera extrinsic rotation matrix and the camera intrinsic parameter matrix.

5. The method according to claim 1, characterized in that The determining of intra-block matching points from the block images of the reference frame and the frame to be aligned by image feature matching includes: Determining the texture complexity of each image block of the reference frame and the frame to be aligned; According to the texture complexity, a target image block is selected from each image block of the reference frame and the frame to be aligned, wherein the target image blocks at the same position of the reference frame and the frame to be aligned constitute a target image block pair; Determining image feature matching point pairs in the target image block pair by image feature matching; The intra-block matching points are selected from the image feature matching point pairs.

6. The method according to claim 5, characterized in that The step of selecting the intra-block matching points from the image feature matching point pairs comprises: Determining the matching degree of the image feature matching point pairs in each of the target image block pairs; The image feature matching point pairs with a matching degree greater than a set value in each target image block pair are determined as the intra-block matching points.

7. The method according to claim 1, characterized in that The step of determining the transformation matrix of the frame to be aligned relative to the reference frame according to the intra-block matching points includes: Convert the coordinates of the matching points in the block to full-image coordinates; A transformation matrix of the frame to be aligned relative to the reference frame is determined according to the full-image coordinates of the matching points in the block.

8. The method according to claim 1, characterized in that Before performing the same block processing on the reference frame after the rough alignment and the frame to be aligned, the method further includes: If there is a brightness difference between the reference frame and the frame to be aligned after the rough alignment, performing brightness matching between the reference frame and the frame to be aligned; and / or, If there is a difference in clarity between the reference frame after the rough alignment and the frame to be aligned, the clarity of the frame to be aligned with lower clarity is improved.

9. An image alignment device, characterized in that: include: An input module, used for acquiring an image frame sequence and a device posture sequence, wherein the image frames in the image frame sequence and the device posture data in the device posture sequence have a corresponding relationship; A calculation module, used for determining a reference frame and a frame to be aligned in the image frame sequence; Determine an alignment matrix of the frame to be aligned relative to the reference frame according to device pose data corresponding to the reference frame and the frame to be aligned respectively; Performing coarse alignment between the frame to be aligned and the reference frame according to the alignment matrix; Performing the same block processing on the reference frame and the frame to be aligned after rough alignment; determining intra-block matching points from the block images of the reference frame and the frame to be aligned by image feature matching; determining the transformation matrix of the frame to be aligned relative to the reference frame according to the intra-block matching points; According to the transformation matrix, finely aligning the frame to be aligned with the reference frame; The output module is used to output the aligned image frame sequence.

10. An electronic device, characterized in that: include: A memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images