A method for automatically identifying the direction of video images and performing panoramic image stitching

Abstract

The present invention discloses a method for automatically identifying the direction of video images and performing panoramic image stitching. A drone cruises at a fixed altitude in the sky, captures a ground-based video, filters blurred frames in the video using the Tenengrad gradient method, grayscales the resulting clear frame images, outputs a difference binarization matrix, selects key frames suitable for stitching using the Frobenius norm of the difference binarization matrix, then determines the direction of video motion by judging the frame slip direction, and finally stitches the selected key frames using the SURF algorithm to obtain a panoramic image. The method of the present invention proposes an intelligent panoramic image stitching algorithm that can automatically process videos to convert them into clear image resources suitable for stitching while saving computing resources. Furthermore, for a large number of video sets with uncertain headings, the method can automatically determine the direction of motion of the video images, saving a large amount of human resources and solving the existing problem of being unable to stitch due to different headings and poor video quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of image data processing, and in particular relates to a method for automatically identifying the direction of video images and performing panoramic image stitching. Background Art

[0002] During aerial photography, drones typically capture ground conditions in the form of video. However, due to the vibrations and shaking associated with drone flight, a large number of blurred frames appear in the video. Therefore, these blurred frames must be removed before selecting keyframes. Current algorithms focus on stitching, relying heavily on manual resource selection. Algorithms for automatically selecting suitable resources are still insufficiently developed. Repeated keyframe stitching requires determining the direction of motion of the video images. For large video collections with uncertain headings, selecting suitable keyframes for stitching and determining the direction of motion consumes significant manual effort. Therefore, an algorithm is needed to transform videos into clear images suitable for stitching while also automatically determining the direction of motion for subsequent stitching. Existing image stitching algorithms utilize the SIFT algorithm for stitching, which leaves room for improvement in computational efficiency. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides a method for automatically identifying the direction of video images and performing panoramic image stitching, and proposes an intelligent panoramic image stitching algorithm to solve the existing problem of being unable to stitch due to different headings and poor video quality.

[0004] The technical solution adopted by the present invention is: a method for automatically identifying the direction of video images and performing panoramic image stitching, the specific steps are as follows:

[0005] S1. Use a drone to cruise at a fixed altitude in the sky and capture a ground-based video.

[0006] S2, filtering the blurred frames in the video using the Tenengrad gradient method;

[0007] S3, grayscale processing is performed on the clear frame image obtained in step S2, and a difference binary matrix is ​​output;

[0008] S4, selecting key frames suitable for splicing through the Frobenius norm of the difference binarization matrix;

[0009] S5. Determine the video motion direction by determining the frame slip direction;

[0010] S6. Use the SURF algorithm to stitch the selected key frames together to obtain a panoramic image.

[0011] Furthermore, the step S2 is specifically as follows:

[0012] First, the clarity of each frame in the video is determined by the Tenengrad gradient method, and then the Sobel operator is used to calculate the gradients in the horizontal and vertical directions respectively.

[0013] Finally, a threshold of 1.2 is set. Images smaller than this threshold are blurred frames. Blurred frames are filtered out, and the remaining images are clear frames.

[0014] Furthermore, the step S3 is specifically as follows:

[0015] First, the clear frame image obtained in step S2 is grayscaled. In the RGB model, if R=G=B, the color represents a grayscale color, where the value of R=G=B is called the grayscale value. The difference in the pixel grayscale values ​​of the previous and next clear frames is stored in a matrix. Then, the grayscale value difference matrix of the previous and next two frames is binarized. That is, the value in the difference matrix is ​​marked as 1 if it is higher than 20, and as 0 if it is lower than or equal to 20. The binary difference matrix is ​​output.

[0016] Furthermore, the step S4 is specifically as follows:

[0017] Based on step S3, the Frobenius norm of the binary difference matrix is ​​calculated, and a threshold value is set in the range of 0.4 to 0.6. If the Frobenius norm of the matrix exceeds the threshold, it indicates that the difference between the front and back images is large enough, and there will be no waste of computing resources. The back image is selected as the key frame and used as the new front clear frame for matrix difference operation. If the Frobenius norm of the matrix does not exceed the threshold, it indicates that the difference between the front and back images is not large enough, and there will be a waste of computing resources. The next clear frame is used as the next clear frame for matrix difference operation.

[0018] Then determine whether the new previous clear frame is a ghost. If it is a ghost, the Frobenius norm of the difference binary matrix with the next frame will be greater than 0.1. Then perform another test to determine whether the Frobenius norm of the difference binary matrix has a large jump. If so, remove the new previous clear frame and filter out key frames that can be used for splicing and do not occupy a large amount of computing resources.

[0019] Furthermore, the step S5 is specifically as follows:

[0020] Take the first key frame and the second key frame of a video output, and take the sub-matrices of length*(0.1*height) on the top and bottom and the sub-matrices of (0.1*length)*height on the left and right of the binarized matrices of the two key frames respectively. The sub-matrix of the first key frame remains unchanged, and the sub-matrix of the second key frame slides 1 to 5 pixels each time. The number of pixels to slide is 0.1*height on the top and bottom and 0.1*length on the left and right. The difference sub-matrix sets of the two key frames in the four directions correspond to the four Frobenius norm sets. The direction with the smallest value of the Frobenius norm set is the sliding direction of the video.

[0021] The higher the pixel value of each slide, the faster the speed of judging the direction of video motion is, and the greater the probability of misjudgment.

[0022] Furthermore, the step S6 is specifically as follows:

[0023] S61, key point detection;

[0024] The approximate Hessian matrix determinant is obtained through a square filter to represent the spot response value at a certain point x in the image to be spliced. All pixel points in the image are traversed to form a response image for key point detection at a certain scale.

[0025] Using different template sizes, a pyramid image of multi-scale key point responses is formed. This pyramid image is used to search for extreme points of key point responses, and the determinant value of the key point Hessian matrix is ​​used to interpolate neighboring data to locate the key point.

[0026] S62, feature proximity description;

[0027] A descriptor is assigned to the key point, and a square box is drawn around the key point. The side length of the box is 20s, where s represents the scale at which the key point is detected. The box also has a direction, which is the main direction detected in step S5.

[0028] The frame is then divided into 16 sub-regions, and the Haar wavelet features of 25 pixels in the horizontal and vertical directions are counted in each sub-region.

[0029] Among them, the horizontal and vertical directions of each sub-region are relative to the main direction, and the Haar wavelet feature is the sum of the horizontal direction value, the sum of the horizontal direction absolute value, the sum of the vertical direction, and the sum of the vertical direction absolute value. These four values ​​are used to describe the key points. The 16 sub-regions generate a total of 4×4×4=64-dimensional feature vectors.

[0030] S63, descriptor pairing;

[0031] After the feature vectors of the key points of the two images are generated, the Euclidean distance of the key point feature vectors is used as the similarity determination metric for the key points in the two images.

[0032] Take a keypoint in the previous keyframe and find the two keypoints with the closest Euclidean distance to it in the next frame. Set a ratio threshold of 0.15 between these two keypoints. If the closest distance divided by the next closest distance is less than the set ratio threshold, the pair of matching points is accepted. Lowering this ratio threshold will reduce the number of SURF matches, but will also increase stability.

[0033] S64, after the first set of key frames are matched, the next key frame is selected and stitched again with the stitched image, and the cycle is repeated until all key frames are stitched together to obtain a panoramic image;

[0034] Among them, the SURF algorithm automatically detects the number of key frames to complete the splicing.

[0035] Beneficial effects of the present invention: The method of the present invention first uses a drone to cruise at a fixed altitude in the sky to capture a ground-based video, then filters the blurred frames in the video using the Tenengrad gradient method, performs grayscale processing on the obtained clear frame images, outputs a difference binary matrix, selects key frames suitable for splicing through the Frobenius norm of the difference binary matrix, then determines the direction of video motion by judging the frame slip direction, and finally splices the filtered key frames through the SURF algorithm to obtain a panoramic image. The method of the present invention proposes an intelligent panoramic image stitching algorithm that can automatically process videos to turn them into clear image resources suitable for stitching while saving computing resources. Furthermore, for a large number of video sets with uncertain headings, the movement direction of the video image can be automatically determined, saving a lot of human resources and solving the existing problem of being unable to stitch due to different headings and poor video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for automatically identifying the direction of video images and performing panoramic image stitching according to the present invention. DETAILED DESCRIPTION

[0037] The method of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0038] like Figure 1 As shown in FIG, a flow chart of a method for automatically identifying the direction of a video screen and performing panoramic image stitching of the present invention, the specific steps are as follows:

[0039] S1. Use a drone to cruise at a fixed altitude in the sky and capture a ground-based video.

[0040] S2, filtering the blurred frames in the video using the Tenengrad gradient method;

[0041] S3, grayscale processing is performed on the clear frame image obtained in step S2, and a difference binary matrix is ​​output;

[0042] S4, selecting key frames suitable for splicing through the Frobenius norm of the difference binarization matrix;

[0043] S5. Determine the video motion direction by determining the frame slip direction;

[0044] S6. Use the SURF algorithm to stitch the selected key frames together to obtain a panoramic image.

[0045] A drone is used to cruise at a fixed altitude in the sky to capture a ground-viewing video, i.e., a ground observation video. The intelligent panoramic image stitching algorithm proposed by the method of the present invention has an MFC (Microsoft Foundation Classes) interface. The user can select the corresponding video as usual in operating the Windows system. The MFC can automatically read the video path and input it into the intelligent panoramic image stitching algorithm.

[0046] In this embodiment, step S2 is specifically as follows:

[0047] First, the clarity of each frame in the video is determined by the Tenengrad gradient method, and then the Sobel operator is used to calculate the gradients in the horizontal and vertical directions respectively.

[0048] Among them, in the same scene, the higher the gradient value, the clearer the image, that is, the average gray value of the image after being processed by the Sobel operator, the larger the value, the clearer the image.

[0049] Finally, this embodiment sets a threshold value of 1.2. The larger the threshold value, the higher the definition requirement for the clear frame. Images smaller than this threshold value are blurred frames. Blurred frames are filtered out, and the remaining images are clear frames.

[0050] Next, key frames are extracted. The total number of frames in the video = frame rate × time. Due to the advancement of modern photography technology, the frame rate of general cameras can reach 30fps or even 60fps. If all pictures are stitched together, it will undoubtedly cause a lot of computing resources to be wasted. Therefore, it is necessary to select pictures as key frames when the image difference reaches a certain level. Since the movement of the drone is not uniform and the removal of blurred frames also makes it impossible to use the timed frame method, the key frames are extracted based on the method of steps S3-S4.

[0051] In this embodiment, step S3 is specifically as follows:

[0052] First, the clear frame image obtained in step S2 is grayscaled. In the RGB model, if R=G=B, the color represents a grayscale color, where the value of R=G=B is called the grayscale value. Compared with color images, grayscale images occupy less memory and run faster. The difference in the pixel grayscale values ​​of the previous and next clear frames is stored in a matrix, and then the grayscale value difference matrix of the previous and next two frames is binarized. That is, the value in the difference matrix is ​​marked as 1 if it is higher than 20, and as 0 if it is lower than or equal to 20. The binary difference matrix is ​​output.

[0053] In this embodiment, step S4 is specifically as follows:

[0054] Based on step S3, the Frobenius norm of the difference binary matrix is ​​calculated, and a threshold value between 0 and 1 is set. If the Frobenius norm of the matrix exceeds the threshold, it means that the difference between the front and back images is large enough. The larger the threshold value is set, the less computing resources are required. However, if it is too large, it will affect the subsequent stitching effect. Therefore, the threshold range in this implementation is: 0.4 to 0.6. It will not cause a waste of computing resources, so the back image is selected as the key frame and used as the new front clear frame for matrix difference operation. If the Frobenius norm of the matrix does not exceed the threshold, it means that the difference between the front and back images is not large enough, which will cause a waste of computing resources. Therefore, the next clear frame is used as the back clear frame for matrix difference operation.

[0055] High-speed camera movement can cause ghosting. This means that the image clarity is high, but the image is torn or fused. It cannot be stitched together as a keyframe, but it will pass the keyframe screening. Therefore, it is necessary to determine whether the new previous clear frame is a ghosting image:

[0056] Under normal circumstances, the Frobenius norm of the next frame's difference binarization matrix will be small because the displacement between the two frames is small, so the Frobenius norm of the difference binarization matrix will naturally be small. However, if there is ghosting, the Frobenius norm of the difference binarization matrix of the next frame will be greater than 0.1, so the next step is to detect whether the Frobenius norm of the difference binarization matrix has changed significantly. If so, the previous frame is removed, so that key frames that can be used for stitching can be selected without occupying a large amount of computing resources.

[0057] In this embodiment, step S5 is specifically as follows:

[0058] Take the first key frame and the second key frame of a video output, and take the sub-matrices of length*(0.1*height) on the top and bottom and the sub-matrices of (0.1*length)*height on the left and right of the binarized matrices of the two key frames respectively. The sub-matrix of the first key frame remains unchanged, and the sub-matrix of the second key frame slides 1 to 5 pixels each time. The number of pixels to slide is 0.1*height on the top and bottom and 0.1*length on the left and right. The difference sub-matrix sets of the four directions of the two key frames correspond to four Frobenius norm sets. The direction with the smallest value of the Frobenius norm set is the sliding direction of the video. The higher the pixel value of each slide, the faster the speed of judging the direction of video movement, but the greater the probability of misjudgment.

[0059] In this embodiment, step S6 is specifically as follows:

[0060] The method of the present invention adopts the SURF (Speeded Up Robust Features) algorithm in the splicing part. The SURF algorithm is superior to the existing SIFT algorithm in execution efficiency because the SURF algorithm has two winning magic weapons - one is the use of the integral image on the Hessian (Hessian matrix), and the other is the use of the dimensionality reduction feature descriptor.

[0061] The filtered key frame materials are spliced ​​using the SURF algorithm, which includes: key point detection, feature proximity description, and descriptor pairing.

[0062] S61, key point detection;

[0063] The approximate Hessian matrix determinant is obtained through a square filter to represent the spot response value at a certain point x in the image to be spliced. All pixel points in the image are traversed to form a response image for key point detection at a certain scale.

[0064] Using different template sizes, a pyramid image of multi-scale key point responses is formed. This pyramid image is used to search for extreme points of key point responses, and the determinant value of the key point Hessian matrix is ​​used to interpolate neighboring data to locate the key point.

[0065] To locate points of interest in images of different sizes, the SURF algorithm in this embodiment uses 3×3×3 neighborhood non-maximum suppression: all values ​​less than a preset extreme value are discarded, and increasing the extreme value reduces the number of key points detected. Ultimately, only a few points with the strongest features will be detected.

[0066] S62, feature proximity description;

[0067] In order to make the key points have the property of rotation invariance, a descriptor is given to the key points so that the key points can retain their invariance and can be easily distinguished.

[0068] Most descriptors are constructed by describing the changes between keypoints and their adjacent pixels, so they are often regional. Descriptor dimensionality is also a key consideration. A descriptor with insufficient dimensionality may make keypoints difficult to distinguish, while a descriptor with excessive dimensionality is computationally more complex. SURF's descriptor uses the Haar wavelet transform concept and utilizes integral graphs to simplify descriptor computation.

[0069] Take a square frame around the key point, and the side length of the frame is 20s (s represents the scale of the key point detected). The frame has a direction, and the direction is the main direction detected in step S5. Then divide the frame into 16 sub-areas, and count the Haar wavelet features of 25 pixels in the horizontal and vertical directions of each sub-area. The horizontal and vertical directions here are relative to the main direction. The Haar wavelet feature is the sum of the horizontal direction value, the sum of the horizontal direction absolute value, the sum of the vertical direction, and the sum of the vertical direction absolute value. Use these four values ​​to describe the key point, then the 16 sub-areas can generate a total of 4×4×4=64-dimensional feature vectors.

[0070] S63, descriptor pairing;

[0071] After the feature vectors of the key points of the two images are generated, the next step is to use the Euclidean distance of the key point feature vectors as the similarity determination metric for the key points in the two images.

[0072] Take a keypoint in the previous keyframe and find the two keypoints with the closest Euclidean distance to it in the next frame. Set a ratio threshold of 0.15 between these two keypoints. If the closest distance divided by the next closest distance is less than the set ratio threshold, the pair of matching points is accepted. Lowering this ratio threshold will reduce the number of SURF matches, but will also increase stability.

[0073] S64, after the first set of key frames are matched, the next key frame is selected and stitched again with the stitched image, and the cycle is repeated until all key frames are stitched together to obtain a panoramic image;

[0074] Among them, the SURF algorithm can automatically detect the number of key frames to complete the splicing.

[0075] In summary, the method of the present invention proposes an intelligent panoramic image stitching algorithm, which can automatically process the video and turn it into a clear image resource suitable for stitching, while saving computing resources. Moreover, for a large number of video sets with uncertain headings, it can automatically determine the direction of motion of the video image, saving a lot of human resources and solving the existing problem of being unable to stitch due to different headings and poor video quality.

[0076] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A method for automatically identifying the direction of a video screen and stitching a panoramic image, the specific steps are as follows: S1. Use a drone to cruise at a fixed altitude in the sky and capture a ground-based video. S2, filtering the blurred frames in the video using the Tenengrad gradient method; S3, grayscale processing is performed on the clear frame image obtained in step S2, and a difference binary matrix is ​​output; S4, selecting key frames suitable for splicing through the Frobenius norm of the difference binarization matrix; S5. Determine the video motion direction by determining the frame slip direction; The step S5 is specifically as follows: Take the first and second key frames of a video output, and take sub-matrices of length*(0.1*height) on the top and bottom, and sub-matrices of (0.1*length)*height on the left and right of the binarized matrices of the two key frames. The sub-matrix of the first key frame remains unchanged, and the sub-matrix of the second key frame slides 1 to 5 pixels each time. The number of pixels to slide is 0.1*height on the top and bottom, and 0.1*length on the left and right. Record the difference sub-matrices of the two key frames in four directions, corresponding to the four Frobenius norm sets. The direction with the smallest value of the Frobenius norm set is the sliding direction of the video. The higher the pixel value of each slide, the faster the speed of judging the direction of video motion is, and the greater the probability of misjudgment. S6. Splice the selected key frames using the SURF algorithm to obtain a panoramic image; The step S6 is specifically as follows: S61, key point detection; The approximate Hessian matrix determinant is obtained through a square filter to represent the spot response value at a certain point x in the image to be stitched, and all the pixels in the image are traversed to form a response image for key point detection at a certain scale; Use different template sizes to form a pyramid image of multi-scale key point responses. Use this pyramid image to search for the extreme points of key point responses, and use the determinant value of the key point Hessian matrix to interpolate neighboring data to locate the key point. S62, feature proximity description; Assign a descriptor to the key point and draw a square box around the key point. The side length of the box is 20s, where s represents the scale of the key point detected. The box also has a direction, which is the main direction detected in step S5. Then the frame is divided into 16 sub-regions, and the horizontal and vertical Haar wavelet features of 25 pixels in each sub-region are counted; Among them, the horizontal and vertical directions of each sub-region are relative to the main direction, so the Haar wavelet feature is the sum of the horizontal direction value, the sum of the horizontal direction absolute value, the sum of the vertical direction, and the sum of the vertical direction absolute value. These four values ​​are used to describe the key points. The 16 sub-regions generate a total of 4×4×4=64-dimensional feature vectors; S63, descriptor pairing; After the feature vectors of the key points of the two images are generated, the Euclidean distance of the key point feature vectors is used as the similarity determination metric of the key points in the two images; Take a keypoint in the previous keyframe and find the first two keypoints with the closest Euclidean distance to it in the next frame. Set a ratio threshold of 0.15 between these two keypoints. If the closest distance divided by the next closest distance is less than the set ratio threshold, then accept this pair of matching points. Lowering this ratio threshold will reduce the number of SURF matching points, but will make it more stable. S64, after the first set of key frames are matched, the next key frame is selected and stitched again with the stitched image, and the cycle is repeated until all key frames are stitched together to obtain a panoramic image; Among them, the SURF algorithm automatically detects the number of key frames to complete the splicing.

2. The method for automatically identifying the direction of a video screen and performing panoramic image stitching according to claim 1, characterized in that: The step S2 is specifically as follows: First, the clarity of each frame in the video is determined by the Tenengrad gradient method, and then the Sobel operator is used to calculate the gradients in the horizontal and vertical directions respectively; Finally, a threshold of 1.2 is set. Images smaller than this threshold are blurred frames. Blurred frames are filtered out, and the remaining images are clear frames.

3. The method for automatically identifying the direction of a video image and performing panoramic image stitching according to claim 1, characterized in that: The step S3 is specifically as follows: First, the clear frame image obtained in step S2 is grayscaled. In the RGB model, if R=G=B, the color represents a grayscale color, where the value of R=G=B is called the grayscale value. The difference in the pixel grayscale values ​​of the previous and next clear frames is stored in a matrix. Then, the grayscale value difference matrix of the previous and next two frames is binarized. That is, the value in the difference matrix is ​​marked as 1 if it is higher than 20, and as 0 if it is lower than or equal to 20. The binary difference matrix is ​​output.

4. The method for automatically identifying the direction of a video image and performing panoramic image stitching according to claim 1, characterized in that: The step S4 is specifically as follows: Based on step S3, the Frobenius norm of the difference binarization matrix is ​​calculated, and a threshold value in the range of 0.4 to 0.6 is set. If the Frobenius norm of the matrix exceeds the threshold, it indicates that the difference between the front and back images is large enough, and no waste of computing resources will be caused. The back image is selected as the key frame and used as the new front clear frame for matrix difference operation. If the Frobenius norm of the matrix does not exceed the threshold, it indicates that the difference between the front and back images is not large enough, which will cause a waste of computing resources. The next clear frame is used as the next clear frame for matrix difference operation. Then determine whether the new previous clear frame is a ghost. If it is a ghost, the Frobenius norm of the difference binary matrix with the next frame will be greater than 0.

1. Then perform another test to determine whether the Frobenius norm of the difference binary matrix has a large jump. If so, remove the new previous clear frame and filter out key frames that can be used for splicing and do not occupy a large amount of computing resources.

Images