Method for fast stitching of aerial images using image transmission video coding information
Through image-transmitting video encoding information, the overall transformation matrix is constructed, and the matching point pair is directly constructed, which solves the problem of large amount of feature extraction calculation in traditional aerial image stitching, realizes rapid registration and efficient stitching of aerial images, and improves the stitching speed and image Fusion effect.
Patent Information
- Application Number
- CN202411148392.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-21
AI Technical Summary
In the panoramic stitching of traditional aerial images, the amount of feature extraction and matching calculations is large, which affects the stitching speed.
Use image-passing video encoding information to build an overall transformation matrix, project the image sequence to the same plane, and realize image stitching through the best suture search and fusion between adjacent frames, directly construct matching point pairs to avoid complex feature extraction and matching process.
It realizes rapid image registration, greatly improves the panoramic stitching speed, and quickly adjusts brightness and chromaticity through the weighted prediction weight of P frames to improve image fusion effect.
Smart Images

Figure CN118674618B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing and relates to a method for realizing rapid splicing of aerial images by using image transmission video coding information. Background Art
[0002] Drone aerial photography has the advantages of high safety, good stability, easy operation, and high image clarity, making it more and more widely used, especially for obtaining images of strip areas. At the same time, the captured images are small in width, large in number, and have a high repetition rate. Image stitching is required to obtain panoramic images. At the same time, during drone aerial photography, real-time image transmission has basically become a necessary function, allowing ground personnel to view drone images in real time.
[0003] The traditional panoramic stitching of aerial images is widely based on feature registration. However, the image feature extraction and matching calculation is very large, which greatly affects the stitching speed. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method for quickly stitching aerial images using image transmission video coding information, which can achieve rapid image registration and improve the speed of panoramic stitching.
[0005] In order to solve the above technical problems, the method of the present invention uses image transmission video coding information to realize rapid stitching of aerial images, extracts the coding information of real-time image transmission video and uses the coding information to construct an overall transformation matrix; uses the overall transformation matrix to project the image sequence onto the same plane, and then searches and fuses the best seam line between adjacent frames to realize image stitching.
[0006] The image transmission video adopts AVC, HEVC or VPx series encoding methods; the extracted real-time image transmission video encoding information includes sub-block division information of all P frames, prediction mode of sub-blocks, reference frame ID and motion vector; the overall transformation matrix construction method is as follows: for any sub-block in the current frame image, its center point is used as the current point, and the center point of the corresponding sub-block in the previous frame image calculated according to its motion vector is used as the reference point, and the current point and the reference point are matched to form a matching point pair; the matching point set is used to solve the perspective transformation matrix between adjacent frames and use it as the overall transformation matrix.
[0007] Furthermore, the perspective transformation matrix between adjacent frames is solved by singular value decomposition using the least squares method; or the perspective transformation matrix between adjacent frames is solved using a random consistency algorithm.
[0008] Furthermore, the current point and the reference point are matched to form matching point pairs, and then the mismatched point pairs are filtered out, and the remaining matching point pairs are used to calculate the perspective transformation matrix; the matching point filtering method is as follows:
[0009] 1) Convert the motion vectors of each sub-block in the current frame with reference to the previous frame into plane point coordinates; let the coordinates of the center point of the i-th sub-block in the current frame be (x di ,y di ), the corresponding reference point coordinates are (x ci , y ci ), i=1,2,…I; I is the number of sub-blocks in the current frame image that refer to the previous frame, then the plane point coordinates converted from the motion vector of the i-th sub-block in the current frame are (x ci -x di , y ci -y di ); add each plane point to point set A;
[0010] 2) Sort the x-coordinates and y-coordinates of all plane points in point set A, and take the x-coordinate x in the middle. pm and the y coordinate y pm , get the median point (x pm , y pm );
[0011] 3) Calculate the median value of all plane points in point set A (x pm , y pm ), and obtain the distance value set B;
[0012] 4) Sort all the distance values in the distance value set B in order from small to large (or from large to small), and take the distance value in the middle as the median distance value D m ;
[0013] 5) Set the threshold t. If the distance value in the distance value set B is greater than t × D m , then the matching point pairs corresponding to the distance value are eliminated; the threshold t is generally set to 2-3.
[0014] For grayscale images, the encoding information of the real-time image transmission video also includes the prediction weight value of the previous frame of all P frames; the prediction weight value includes the brightness weight value; the brightness of the image is adjusted by using the brightness weight value, and then the image sequence with adjusted brightness is projected onto the same plane using the overall transformation matrix; let the current frame prediction weight value extracted from the encoded video stream be W n , then the brightness adjustment value of the current frame image is: F Ln =F L(n-1) / W n Among them, F Ln is the brightness adjustment value of the nth frame image, F n-1 is the brightness adjustment value of the n-1th frame image.
[0015] Furthermore, for RGB format images, the encoding information of the real-time image transmission video also includes the prediction weight value of the previous frame of all P frames; the prediction weight value includes the brightness weight value and the chrominance weight value; the RGB format image is converted into a YUV format image, the brightness and chrominance of the image are adjusted using the brightness weight value and the chrominance weight value, and then the image sequence with adjusted brightness is projected onto the same plane using the overall transformation matrix, and finally converted back to RGB format; let the current frame prediction weight value extracted from the encoded video stream be W n , then the brightness adjustment value of the current frame image is: F Ln =F L(n-1) / W n Among them, F Ln is the brightness adjustment value of the nth frame image, F n-1 is the brightness adjustment value of the n-1th frame image; the chromaticity adjustment value of the current frame image is: F Sn =F S(n-1) / W n Among them, F Sn is the chromaticity adjustment value of the nth frame image, F S(n-1) is the chroma adjustment value of the n-1th frame image.
[0016] The image transmission video adopts AV1 encoding method, and the extracted encoding information of the real-time image transmission video includes global motion estimation data; the global motion estimation data is used to construct an affine transformation matrix as the overall transformation matrix.
[0017] The image transmission video adopts the VVC encoding method, and the extracted encoding information of the real-time image transmission video includes the sub-block division information of all P frames, the prediction mode of the sub-block, the reference frame ID and the motion vector; for any translation motion sub-block in the current frame image, its center point is used as the current point, and the center point of the corresponding sub-block in the previous frame image calculated according to its motion vector is used as the reference point, and the current point and the reference point are matched to form a matching point pair; for any affine motion sub-block in the current frame image, each of its control points and the corresponding control points of the corresponding sub-block in the previous frame image constitute a matching point pair, and the corresponding control points of the corresponding sub-block in the previous frame image are obtained by calculating the motion vector of the control point in the sub-block of the current frame image.
[0018] Beneficial effects: The present invention proposes a method for realizing fast stitching of aerial images in a strip area using the coding information of real-time image transmission video. The method uses the coding information of the image transmission video to directly construct matching point pairs of adjacent frames, avoiding the complex feature extraction and matching process, and can realize fast image registration, greatly improving the speed of panoramic stitching. At the same time, by using the weighted prediction weights of the P frame, the brightness and chromaticity of the image to be fused can be quickly adjusted to improve the image fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a flow chart of embodiment 1 of the present invention.
[0020] Figure 2 This is a schematic diagram of constructing matching point pairs in Example 1.
[0021] Figure 3 This is a schematic diagram of constructing control point matching point pairs in Example 3.
[0022] Figure 4 It is the image splicing effect diagram of the present invention. DETAILED DESCRIPTION
[0023] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention rather than to limit the present invention.
[0024] In the description of the embodiments, the terms "upper", "lower", "left", "right" and other directions or positional relationships are based on the directions or positional relationships shown in the drawings, and are only for the convenience of description and simplified operation, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are only used to distinguish in the description and have no special meaning.
[0025] Real-time image transmission uses compressed video streams for transmission. The current mainstream compression encoding methods are AVC (H264) and HEVC (H265). UAV aerial cameras all have stable gimbals to ensure the stability of the shooting angle, so that the overlapping areas of adjacent frame images have high consistency, ensuring the success rate of inter-frame prediction during encoding. As for encoding parameters, for real-time considerations, zerolatency mode is basically adopted. In this mode, the coded frames are only I frames (intra-frame coded frames, key frames) and P frames (forward reference frames), that is, inter-frame prediction has only one forward reference method. For preset (preset configuration) parameters, real-time image transmission basically uses ultrafast (fastest mode), superfast (ultra-fast mode) or veryfast (very fast mode). For these presets, the number of reference frames is limited to 1 frame, that is, only the previous frame is used as the reference frame. Even if other presets are used, the reference frames are 2 frames or more, but based on the encoding rules, the previous frame must be the frame that is referenced the most. Based on these facts, the present invention uses the motion vector data of the sub-blocks predicted between the current frame and the previous reference frame image in the overlapping area to construct a sufficient number of matching point pairs, which can achieve rapid registration of the current frame and the reference frame image. The accuracy of the motion vector is at the sub-pixel level, the brightness accuracy level is 1 / 4 pixel, and the chrominance accuracy level is 1 / 8 pixel, which can ensure the accuracy of the registration. The present invention only uses feature extraction and matching for the I frame and the last frame of the previous GOP (coded picture group), and can achieve the registration of all images, which greatly saves the amount of calculation.
[0026] Except for the ultrafast preset, all other presets use the current frame weighted prediction mode, which can use the frame prediction weight value to quickly adjust the brightness / chroma of two adjacent frame images to be fused, thereby improving the fusion effect.
[0027] Embodiment 1: Embodiment 1 is directed to AVC (H264) and HEVC (H265) encoding methods and VPx series encoders, such as Figure 1 As shown, the method of the present invention for realizing rapid stitching of aerial images using image transmission video coding information specifically includes the following steps:
[0028] 1. Extraction of coding information: Video coding frames are divided into I frames (intra-coded frames, key frames), P frames (forward reference frames) and B frames (bidirectional reference frames). Only I frames and P frames are used in image transmission video coding (usually I frames refer to the first frame of a GOP (group of coded pictures), and P frames refer to the subsequent frames). Extraction of coding information is to extract the sub-block division information of all P frames, the prediction mode of the sub-blocks, the reference frame ID and motion vector, and the prediction weight value of the reference frame.
[0029] There are two ways to extract the coding information. The first is to store / transmit the relevant information during coding. This method requires the encoder to be modified, but in unmanned equipment, hard coding is usually used, so this method is not applicable; the second is to extract directly from the encoded video stream. The extraction method needs to follow the decoding regulations of the corresponding video coding standard. This method is simple to implement, flexible and efficient. Therefore, the present invention adopts the second method to directly extract the sub-block division information of all P frames (forward reference frames), the prediction mode of the sub-block, the reference frame and the motion vector, and the prediction weight value of the reference frame from the encoded video stream.
[0030] 2. Construct matching point pairs: In the AVC, HEVC and VPx standards, motion estimation only considers translational motion. In the actual aerial photography process, in addition to translational motion, there are also small rotations and scaling between adjacent frame images. If the motion vectors of each sub-block in the current frame image that refers to the previous frame are used directly to calculate the overall transformation matrix, its rotation and scaling relationship cannot be included. Therefore, the best way is to construct a large number of matching point pairs between the current frame and the previous frame through the motion vectors of each sub-block in the current frame image that refers to the previous frame, and then calculate the overall transformation matrix through these matching point pairs, so as to accurately reflect its rotation and scaling relationship. The construction process of matching point pairs is as follows:
[0031] 1. For any sub-block of a P frame (forward reference frame), if its prediction mode is inter-frame prediction and its reference frame is the previous frame, the sub-block motion vector is used to construct a matching point pair. Figure 2 As shown in the figure, taking any sub-block in the current frame image (macroblock of AVC or PU (prediction unit) of HEVC) as an example, firstly, the center point of the sub-block is taken as the current point, and the center point of the corresponding sub-block in the previous frame image calculated according to its motion vector is taken as the reference point, and the current point and the reference point are matched to form a matching point pair.
[0032] 2. Matching point filtering: There may be local motion and repeated texture of objects in the image. The local motion of the object causes the motion relationship of its corresponding sub-block to be inconsistent with the motion relationship of the image as a whole. The existence of repeated textures and the influence of factors such as lighting and exposure time may cause the coding reference blocks of some sub-blocks to not be the best matching blocks that conform to the overall motion relationship of the image. In the above case, using these sub-blocks to construct matching point pairs forms mismatching points. In order to ensure the accuracy of the calculated overall motion transformation relationship, these mismatching point pairs need to be filtered out. The present invention uses the consistency of the motion vectors of each sub-block of the current frame used when constructing matching points to perform matching point filtering, and the filtering method is as follows:
[0033] 1) Convert the motion vectors of each sub-block in the current frame that refers to the previous frame into plane point coordinates; let the coordinates of the center point of the i-th sub-block be (x di ,y di), the reference point coordinates are (x ci , y ci ), i=1,2,…I; I is the number of sub-blocks in the current frame image that refer to the previous frame, then the plane point coordinates converted from the motion vector of the i-th sub-block in the current frame are (x ci -x di , y ci -y di ); add each plane point to point set A;
[0034] 2) Sort the x-coordinates and y-coordinates of all plane points in point set A, and take the x-coordinate x in the middle. pm and the y coordinate y pm , get the median point (x pm , y pm );
[0035] 3) Calculate the median value of all plane points in point set A (x pm , y pm ) to obtain the distance value set B;
[0036] 4) Sort all the distance values in the distance value set B in order from small to large (or from large to small), and take the median distance value as the median distance value D m ;
[0037] 5) Set the threshold t. If the distance value in the distance value set B is greater than t × D m , then the matching point pairs corresponding to the distance value are eliminated; the threshold t is generally set to 2-3.
[0038] 3. Projection transformation: Through the above step 2, a relatively accurate set of matching points can be obtained. This matching point set can be used to solve the perspective transformation matrix between adjacent frames as the overall transformation matrix. There are two main ways to solve it: using the least squares method, the overall transformation matrix between adjacent frames is solved by singular value decomposition; or using the random consensus algorithm (RANSAC) to solve the overall transformation matrix between adjacent frames; the method of solving the overall transformation matrix is a conventional technical means and will not be described in detail here. Finally, the conventional LM algorithm can be further used to optimize the overall transformation matrix to obtain the optimized overall transformation matrix.
[0039] 4. Brightness / chroma adjustment: Use the prediction weight values of each P frame (forward reference frame) extracted from the encoded video stream, including the brightness weight value and the chroma weight value, to adjust the brightness and chroma of the image. If it is a grayscale image, only the brightness adjustment is required. If it is a color image, you need to first convert the RGB format image into a YUV format image, then adjust the brightness and chroma respectively, and finally convert it back to RGB format. The calculation method of the brightness adjustment value and the chroma adjustment value is as follows:
[0040] Assume that the current frame prediction weight value extracted from the encoded video stream is W n , then the brightness adjustment value of the current frame image is: F Ln =F L(n-1) / W n Among them, F Ln is the brightness adjustment value of the nth frame image, F n-1 is the brightness adjustment value of the n-1th frame image, that is, the brightness adjustment value of the current frame image is the brightness adjustment value of the previous frame image divided by the prediction weight value of the current frame;
[0041] The chromaticity adjustment value of the current frame image is: F Sn =F S(n-1) / W n Among them, F Sn is the chromaticity adjustment value of the nth frame image, F S(n-1) is the chromaticity adjustment value of the n-1th frame image, that is, the chromaticity adjustment value of the current frame image is the chromaticity adjustment value of the previous frame image divided by the prediction weight value of the current frame image;
[0042] 5. Seam line search and fusion: For the image sequence with adjusted brightness (or brightness and chrominance), use the overall transformation matrix to project it to the same plane, and then search and fuse the best seam line between adjacent frames to achieve fast stitching of aerial images. Fusion algorithm, considering performance, you can use the fade-in and fade-out algorithm, considering the effect, you can use the pyramid fusion algorithm, both algorithms are conventional fusion algorithms.
[0043] Embodiment 2: This embodiment is for AV1 encoding method; the difference from Embodiment 1 is that only global motion estimation data needs to be extracted in step 1. Since the global motion estimation of AV1 takes into account the affine transformation relationship between adjacent frames, the global motion estimation data can be directly used to construct an affine transformation matrix as the overall transformation matrix without constructing matching point pairs.
[0044] Embodiment 3: This embodiment is directed to a coding standard including block affine motion compensation, such as VVC (H266). The difference from Embodiment 1 is that in step 2, the relative motion of the sub-block can be an affine transformation in addition to translation, and the affine motion of the sub-block is described by the motion vectors of two control points or three control points (four-parameter model or six-parameter model), such as Figure 3 shown.
[0045] The construction process of the matching point pair is different from the construction method of the matching point pair in step 1 of step 2 in that, for any affine motion sub-block in the current frame image, the matching point pair of the sub-block using affine motion compensation is constructed, and each control point in each sub-block using affine motion compensation in the current frame image and the corresponding control point of the corresponding sub-block in the previous frame image constitute a matching point pair, and the corresponding control point of the corresponding sub-block in the previous frame image is calculated by the motion vector of the control point in the sub-block of the current frame image. For any translation motion sub-block in the current frame image, the construction method of the matching point pair and the matching point filtering method are the same as those in Example 1.
Claims
1. A method for fast splicing of aerial images using image transmission video coding information, characterized in that The method is as follows: extracting the coding information of the real-time image transmission video and using the coding information to construct an overall transformation matrix; using the overall transformation matrix to project the image sequence onto the same plane, and then searching and fusing the best seam line between adjacent frames to achieve image stitching; the image transmission video adopts AVC, HEVC or VPx series coding methods; the extracted coding information of the real-time image transmission video includes sub-block division information of all P frames, prediction mode of the sub-block, reference frame ID and motion vector; the overall transformation matrix construction method is as follows: for any sub-block in the current frame image, its center point is used as the current point, and the center point of the corresponding sub-block in the previous frame image calculated according to its motion vector is used as the reference point, and the current point and the reference point are matched to form a matching point pair; the matching point set is used to solve the perspective transformation matrix between adjacent frames and use it as the overall transformation matrix.
2. The method for realizing rapid splicing of aerial images using image transmission video coding information according to claim 1, characterized in that Use the least squares method to solve the perspective transformation matrix between adjacent frames through singular value decomposition; or use the random consistency algorithm to solve the perspective transformation matrix between adjacent frames.
3. The method for realizing rapid splicing of aerial images using image transmission video coding information according to claim 1, characterized in that After matching the current point and the reference point to form matching point pairs, the mismatched point pairs are filtered out, and the remaining matching point pairs are used to calculate the perspective transformation matrix; the matching point pair filtering method is as follows: 1) Convert the motion vectors of each sub-block in the current frame with reference to the previous frame into plane point coordinates; let the coordinates of the center point of the i-th sub-block in the current frame be (x di ,y di ), the corresponding reference point coordinates are (x ci , y ci ), i=1,2,…I; I is the number of sub-blocks in the current frame image that refer to the previous frame, then the plane point coordinates converted from the motion vector of the i-th sub-block in the current frame are (x ci -x di ,y ci -y di ); add each plane point to point set A; 2) Sort the x-coordinates and y-coordinates of all plane points in point set A, and take the x-coordinate x in the middle. pm and the y coordinate y pm , get the median point (x pm , y pm ); 3) Calculate the median value of all plane points in point set A (x pm , y pm ), and obtain the distance value set B; 4) Sort all the distance values in the distance value set B from small to large or from large to small, and take the median distance value as the median distance value D m ; 5) Set the threshold t. If the distance value in the distance value set B is greater than t × D m , then the matching point pair corresponding to the distance value is eliminated; the threshold t is set to 2-3.
4. The method for realizing rapid splicing of aerial images using image transmission video coding information according to claim 1, characterized in that For grayscale images, the encoding information of the real-time image transmission video also includes the prediction weight value of the previous frame of all P frames; the prediction weight value includes the brightness weight value; the brightness of the image is adjusted by using the brightness weight value, and then the image sequence with adjusted brightness is projected onto the same plane using the overall transformation matrix; let the current frame prediction weight value extracted from the encoded video stream be W n , then the brightness adjustment value of the current frame image is: F Ln =F L(n-1) / W n ; Among them, F Ln is the brightness adjustment value of the nth frame image, F n-1 is the brightness adjustment value of the n-1th frame image.
5. The method for realizing rapid splicing of aerial images using image transmission video coding information according to claim 1, characterized in that For RGB format images, the encoding information of the real-time image transmission video also includes the prediction weight value of the previous frame of all P frames; the prediction weight value includes a brightness weight value and a chrominance weight value; Convert the RGB format image into a YUV format image, use the brightness weight value and chromaticity weight value to adjust the brightness and chromaticity of the image, and then use the overall transformation matrix to project the image sequence with adjusted brightness and chromaticity onto the same plane, and finally convert it back to RGB format; let the current frame prediction weight value extracted from the encoded video stream be W n , then the brightness adjustment value of the current frame image is: F Ln =F L(n-1) / W n Among them, F Ln is the brightness adjustment value of the nth frame image, F n-1 is the brightness adjustment value of the n-1th frame image; the chromaticity adjustment value of the current frame image is: F Sn =F S(n-1) / W n Among them, F Sn is the chromaticity adjustment value of the nth frame image, F S(n-1) is the chroma adjustment value of the n-1th frame image.
6. A method for fast splicing of aerial images using image transmission video coding information, characterized in that The method is as follows: extracting the coding information of the real-time image transmission video and using the coding information to construct an overall transformation matrix; using the overall transformation matrix to project the image sequence onto the same plane, and then searching and fusing the best stitching line between adjacent frames to realize image stitching; the image transmission video adopts the VVC coding method, and the extracted coding information of the real-time image transmission video includes the sub-block division information of all P frames, the prediction mode of the sub-block, the reference frame ID and the motion vector; for any translation motion sub-block in the current frame image, its center point is used as the current point, and the center point of the corresponding sub-block in the previous frame image calculated according to its motion vector is used as the reference point, and the current point and the reference point are matched to form a matching point pair; for any affine motion sub-block in the current frame image, each of its control points and the corresponding control point of the corresponding sub-block in the previous frame image form a matching point pair, and the corresponding control point of the corresponding sub-block in the previous frame image is obtained by calculating the motion vector of the control point in the sub-block of the current frame image.
