Video stitching method based on feature point homogenization and suture line iteration

Through the uniformization of feature points and the iterative strategies of suture lines, the problems of uneven distribution of feature points and high complexity of suture lines in the existing technology are solved, and more efficient and more accurate video stitching effect is achieved, suitable for large field of view and panoramic views Real-time video requirements.

CN119012025APending Publication Date: 2024-11-22FUZHOU UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411108666.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing video stitching technology has feature point detection and matching for the whole image processing, and does not consider the impact of local smooth areas, resulting in uneven distribution of feature points, affecting the accuracy and efficiency of the estimation homography matrix, and the best suture update algorithm High complexity and not suitable for real-time scenarios.

Method used

Feature point uniformization and suture iteration strategies are adopted to calculate image information entropy by blocks, introduce local image enhancement algorithm and quadtree algorithm to achieve feature point uniformization, and detect suture neighborhood differences in frame-by-frame fusion stage, and perform Local marking and iteration, optimize sutures to improve the accuracy and real-timeness of video stitching.

Benefits of technology

Improve the accuracy and efficiency of first frame registration, obtain the optimal suture that is more in line with the real standards, reduce complex suture constraint terms, reduce the delay of moving object detection algorithm, and achieve more efficient video stitching effect. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119012025A_ABST
    Figure CN119012025A_ABST
Patent Text Reader

Abstract

The invention provides a video stitching method based on feature point homogenization and suture line iteration. The video stitching method comprises two stages of first frame registration and frame-by-frame fusion. In the first frame registration stage, firstly, a pre-processed image rough matching overlapping region is partitioned, image information entropy is calculated, a local image enhancement algorithm is adopted in a low entropy region to improve the image information entropy, image feature points are extracted through a feature detection algorithm, then a quadtree algorithm is adopted to further subdivide the feature points, and the image information entropy is calculated; the feature points are homogenized; in the frame-by-frame fusion stage, the suture line and the neighborhood of the suture line are guided to be in a smooth region by detecting the difference index of the neighborhood of the suture line and carrying out local marking and iteration on the region with large difference, so that the accuracy and the real-time performance of video stitching are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of video splicing processing, and in particular relates to a video splicing method based on feature point homogenization and seam line iteration. Background Art

[0002] In recent years, smart cameras have developed rapidly, not only achieving ultra-high-definition visual effects, but also bringing a variety of intelligent functions. However, the field of view that most cameras can obtain is small, which cannot meet people's needs for large field of view and panoramic videos. Video stitching technology has thus developed. Video stitching technology is to synthesize two or more videos with a certain overlapping area into a large field of view or panoramic video. It is widely used in intelligent driving, map remote sensing, virtual reality and other fields. Compared with image stitching technology, video stitching technology needs to take into account both stitching effect and real-time performance, and cannot use methods that are too complex.

[0003] In the prior art, video stitching technology consists of two stages: image registration and image fusion. In order to obtain good shooting effects, the positions of multiple cameras are usually fixed. Therefore, image registration only needs to calculate accurate transformation matrix parameters at the beginning of stitching and keep them unchanged in subsequent frames, while image fusion needs to be processed frame by frame, and the fusion efficiency affects the real-time performance of video stitching. The mainstream method of video stitching is based on feature point matching and optimal seam line update algorithm, but this scheme has the following problems: (1) Feature point detection and matching are processed for the entire image, without considering the influence of local smooth areas; (2) Feature points are unevenly distributed, and local dense feature points affect the accuracy and efficiency of estimating the homography matrix; (3) The optimal seam line update and moving object detection algorithms are too complex and are not suitable for actual real-time scenarios.

[0004] The existing technology usually establishes a stitching template from the first frame image, and uses a stitching line-based image fusion and stitching line update method to process the subsequent image fusion part. However, when the feature points of the first frame image are unevenly distributed, this method will cause a large error in the established stitching template, thereby affecting the subsequent stitching quality. In addition, the traditional optimal stitching line often performs poorly in some parts, with obvious artifacts and splitting.

[0005] For example, the existing technical solution "A real-time video stitching method based on the best seam line update": first, the fisheye lens is preprocessed and corrected, and then the first frame image is subjected to the SURF (Speeded Up Robust Features) algorithm to extract feature points, and the RANSAC (Random Sample Consensus) algorithm is used to screen feature points and match feature points, thereby estimating the homography matrix, generating the corresponding projection transformation mapping table, and mapping the subsequent frame images according to the mapping table. After the image is mapped, the background of the overlapping area is eliminated based on the Gaussian mixture model and the foreground moving object detection is performed, the foreground significance item is set, and the best seam line is calculated by the dynamic programming method. The loss function of the best seam line consists of a spatial coherence item and a significance item. When the foreground significance item of the image is greater than the threshold, the seam line is updated, otherwise the seam line is maintained unchanged, and finally the smooth fusion of the image is achieved through the MVC (Mean Value Coordinate) interpolation film algorithm based on the convolutional pyramid.

[0006] The existing technical solution "Video stitching method and system based on grid transformation": first, feature points are extracted through the SIFT (Scale-invariant Feature Transform) algorithm, and adaptive feature point screening and matching are performed based on the distance information of the feature descriptor. The matching results are passed through the grid deformation constraint algorithm, combined with global and local constraints, to perform precise image transformation and overlap area alignment. Finally, the object movement trajectory is detected through the target detection algorithm, and the moving objects that appear in the overlapping area are added to the artifact candidate set. The artifact candidate set and the background elimination algorithm are combined to determine whether the current stitching line needs to be updated, and the image result of the seam fusion is output. Summary of the invention

[0007] It is of great significance to consider proposing a video stitching method with low complexity, efficient registration and real-time fusion. The present invention aims to provide a video stitching method based on feature point homogenization and seam line iteration to overcome the problems of poor local registration and local error of seam lines in the prior art, and provide a real-time video effect with a larger field of view.

[0008] Therefore, the present invention proposes a video stitching method based on feature point homogenization and seam line iteration. By homogenizing feature points, the distribution of feature points in the image is controlled to enhance the alignment effect of the overlapping area; the seam line iteration strategy is introduced to iterate the seams after marking the seams with local differences, reducing the complex seam line constraints and solving the problem that the first frame registration of video stitching ignores the local feature differences, resulting in poor registration and failure to optimize the local error area of ​​the seam line.

[0009] The video stitching in the scheme of the present invention consists of two parts: first frame registration and frame-by-frame fusion. First frame registration is the process of aligning two images in the overlapping area, but errors often occur when feature points are unevenly distributed. In order to reduce the first frame registration error, the overlapping area of ​​the pre-processed image is firstly divided into blocks and the image information entropy is calculated. The local image enhancement algorithm is introduced in the low entropy area to improve the image information entropy, and the image feature points are extracted by the feature detection algorithm. Then, the quadtree algorithm is introduced to further subdivide the feature points, thereby achieving feature point homogenization and ensuring accurate estimation of the registration matrix; in the frame-by-frame fusion stage, the traditional idea of ​​detecting moving objects and updating the seam line is converted into a difference index for detecting the neighborhood of the seam line, and the areas with large differences are locally marked and iterated to guide the seam line and its neighborhood to be in a smooth area, thereby ensuring the accuracy and real-time performance of video stitching.

[0010] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0011] A video stitching method based on feature point homogenization and seam line iteration, video stitching includes two stages: first frame registration and frame-by-frame fusion. In the first frame registration stage, the overlapping areas of the preprocessed image are firstly divided into blocks and the image information entropy is calculated. In the low entropy area, the local image enhancement algorithm is used to improve the image information entropy, and the image feature points are extracted by the feature detection algorithm. Then, the feature points are further subdivided by the quadtree algorithm, so as to realize the homogenization of the feature points. In the frame-by-frame fusion stage, the difference index of the seam line neighborhood is detected, and the areas with large differences are locally marked and iterated, so as to guide the seam line and its neighborhood to be in a smooth area, so as to ensure the accuracy and real-time performance of video stitching.

[0012] Furthermore, the first frame registration stage includes: preprocessing, feature point detection and homogenization, feature point matching, and estimation of the registration matrix; after preprocessing rough matching, the feature point detection and homogenization are to extract feature points as much as possible in the overlapping area and disperse the feature points, and then the feature points of the two images are matched with the nearest neighbor feature points to obtain matching point pairs, and the matching point pairs are purified by the RANSAC algorithm, and finally the registration matrix is ​​estimated to transform the image to be registered into the target image plane.

[0013] Furthermore, the video frame-by-frame fusion stage includes: seam line calculation, seam line evaluation and iteration, seam line-based weighted fusion and generation of stitched images;

[0014] The stitching line calculation is to calculate the energy difference between corresponding pixels based on the overlapping areas of the two images, and the energy function is composed of structural difference and color difference; the error analysis is performed between the stitching line neighborhoods through stitching line evaluation, and the local area with larger error in the error analysis curve is iterated through stitching line iteration to obtain the optimal stitching line, and finally the stitching line neighboring area is weightedly fused, and other areas are directly mapped to obtain a stitched image, and then the stitching video is obtained by stitching frame by frame.

[0015] As a preferred solution, GPU can be used to accelerate the stitching and fusion process at this stage, and parallel processing of two video images can be achieved by writing CUDA kernel functions.

[0016] Furthermore, image preprocessing includes performing template matching on the video input image: Let the image to be registered be I r (x,y), the target image is I t (x, y), the image size is w×h, take the 1 / 5 pixel block from the left edge of the target image to the right edge of the image, record it as T(x, y), perform sliding matching in the image to be registered, find the best matching pixel block and perform coordinate compensation, thereby obtaining the coarse matching overlap area, which are I r ″(x,y) and I t ″(x,y); the correlation coefficient formula for template matching is as follows;

[0017]

[0018] Where R(x,y) represents the correlation and its value range is (0,1), x′ and y′ represent all coordinate values ​​of the traversed template pixel block, T′(x′,y′) represents the pixel value of the template pixel block, and I′(x+x′,y+y′) represents the pixel value of the matching image at the position (x+x′,y+y′).

[0019] Furthermore, the feature point detection and homogenization includes: before the feature point detection, the coarse matching overlapping area is divided into N blocks by grid partitioning, and the image information entropy is calculated for each block. The entropy value is calculated as shown in the following formula:

[0020]

[0021] where p ij is the probability of occurrence of a feature binary consisting of the grayscale mean of the image neighborhood and the grayscale pixel of the image (i, j), and H is the two-dimensional information entropy of the image. The entropy information of each grid block is judged according to the image information entropy. The entropy information is used as a quantitative standard of image quality. When the entropy value exceeds 10, it means that it is in a non-low entropy area, otherwise it is in a low entropy area. Therefore, each grid block of the image is divided into a low entropy domain and a non-low entropy domain.

[0022] For low entropy domain, a local adaptive histogram equalization algorithm is used to balance image brightness, chroma and saturation to improve image information entropy without increasing noise.

[0023] SURF feature point detection is performed on the processed coarse matching overlapping area, and then the image is subjected to quadtree homogenization processing. The specific process is as follows: set the maximum number of retained feature points M, the initial area is S, if the number of feature points in S is greater than 1, then S is split into 4 sub-areas, if the number of sub-areas is less than M, each sub-area continues to be split into four, when the number of feature points in the sub-area obtained after the split is zero, the area is deleted; in this way, the number of sub-areas with feature points S is split n If it is greater than M or does not meet the splitting condition, the splitting is stopped, and then the feature point with the highest Hessian matrix response value is selected from each sub-region; finally, the feature vector is constructed by calculating the Haar wavelet response value of the neighboring area of ​​the feature point, and the feature descriptor is obtained after the feature vector is normalized.

[0024] Furthermore, the feature point matching adopts the nearest neighbor matching algorithm to calculate the Euclidean distance between the feature points of the two images, compare the ratio of the nearest neighbor Euclidean distance and the next nearest neighbor Euclidean distance, and when the ratio is less than a set threshold, it is identified as a matching point.

[0025] Furthermore, the process of calculating and searching for the optimal seam line includes: converting the image to be registered to the same plane as the target image through the homography matrix to ensure the alignment of the two images in the overlapping area, and calculating the optimal seam line according to the overlapping area. The calculation method is to search for a path with the lowest energy function in the overlapping area. The search method is as follows: Calculate the energy value of each pixel in the entire overlapping area to measure the similarity of pixel intensity, gradient and texture energy in the overlapping area. The smaller the energy value, the smaller the difference between the two images at the pixel point. Then, the optimal seam line is found by dynamic programming. The energy value is calculated as follows:

[0026] E(x,y)=ωP(x,y)+υG(x,y) 2 +λT(x,y) (3)

[0027] Where E(x,y) is the energy difference of the pixels with the same coordinates in the overlapping area, P(x,y) is the intensity difference, and G(x,y) is 2 is the gradient difference, T(x,y) is the texture energy difference, ω, υ and λ are weight coefficients.

[0028] Furthermore, in the process of suture evaluation and iteration: the difference of the suture neighborhood is evaluated using the suture evaluation function, and the evaluation function is generated; after the evaluation, a suture difference curve is generated, and if the difference of the suture neighborhood is lower than the threshold, it remains unchanged; if the difference of the suture neighborhood is higher than the threshold, the suture and its neighborhood are marked as maximum values, and the suture is iterated until the suture meets the evaluation criteria;

[0029] The calculation of the seam evaluation function is as follows:

[0030]

[0031] Among them, X s To synthesize similarity deviations, including error comparisons of brightness, contrast, structure, and gradient, is the average value of the pixels in the left neighborhood of the seam line, is the average value of the right neighborhood, is the variance of the left neighbor image pixels, is the right neighborhood variance, is the covariance between the left and right neighbor images, is the average gradient amplitude of the left neighborhood image pixels, is the average gradient amplitude of the right neighborhood, and is the pixel value dynamic equilibrium constant.

[0032] Furthermore, based on the weighted fusion of the seam line, after the seam line is obtained, the overlapping area is divided into a seam line area and a non-seam line area. The seam line area is the seam line and its neighborhood pixels, and the non-seam line area is other overlapping areas and non-overlapping areas. The two sides of the optimal seam line are processed according to the fade-in and fade-out weighted fusion algorithm, and the weight value is adjusted according to the Euclidean distance from the optimal seam line to the neighborhood boundary.

[0033] And, a video splicing system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the video splicing method as described above when executing the program.

[0034] Compared with the prior art, the solution of the present invention mainly provides the following innovative designs:

[0035] (I) Overall architecture of video splicing

[0036] The video stitching method provided by the present invention includes video first frame registration and video frame-by-frame fusion, and can provide good real-time stitching effect for videos with overlapping areas.

[0037] Video first frame registration: includes four steps: preprocessing, feature point detection and homogenization, feature point matching, and estimation of registration matrix. Preprocessing is to roughly match the image, improve the efficiency of subsequent feature point detection and reduce the feature point mismatch rate. The purpose of feature point detection and homogenization is to extract feature points as much as possible in the overlapping area and distribute them in a dispersed manner to avoid matching errors caused by too dense. Then, the feature points of the two images are matched by the nearest neighbor feature points to obtain matching point pairs, and the matching point pairs are purified by the RANSAC algorithm. Finally, the homography matrix, i.e., the first frame registration matrix, is estimated to transform the image to be registered into the target image plane.

[0038] Video frame-by-frame fusion: It includes four steps: seam calculation, seam evaluation and iteration, seam-based weighted fusion, and generation of stitched images. The seam calculation is to calculate the energy difference between corresponding pixels based on the overlapping areas of the two images. The energy function is composed of structural difference and color difference. The seam iteration performs error analysis between the seam neighborhoods. The seam iteration iterates the local areas with large errors in the error analysis curve to obtain the optimal seam. Finally, the seam neighboring areas are weighted fused, and other areas are directly mapped to obtain a stitched image, which is then stitched frame by frame to obtain a stitched video.

[0039] (II) Feature Point Homogenization Scheme

[0040] In this scheme, feature point homogenization can be divided into entropy detection, local processing and homogenization. First, entropy detection is performed to divide the image of the coarse matching overlapping area into multiple grids. The image information entropy is calculated in parallel for each grid. Generally speaking, the higher the entropy value, the greater the amount of information corresponding to the grid, the more complex the texture features, and the more extractable feature points, while the opposite is true for low entropy values. The image is divided into low entropy domain and non-low entropy domain according to the grid. Local processing is to use an adaptive histogram equalization algorithm in the low entropy domain to redistribute the image brightness and contrast to improve the local feature performance, which not only increases the number of local feature points in the image, but also avoids introducing noise into the global image. Homogenization combines the idea of ​​the quadtree algorithm to split the image into multiple sub-regions according to the image size. Each sub-region only retains one feature point to prevent the feature points from being too dense and affecting the registration effect.

[0041] 3. Suture Line Evaluation and Iteration Scheme

[0042] In this scheme, the seam processing is divided into evaluation and iteration. The seam evaluation aims to evaluate the error of the seam calculated by the first energy. The evaluation method is to calculate the brightness, contrast, structure and gradient comprehensive similarity deviation of the neighborhood images on both sides of the seam. The larger the value, the worse the stitching effect. The evaluation curve obtained marks the area with large seam evaluation error, and finds the optimal seam of the marked image. In this way, the seam is iterated. Each frame of the image only needs to calculate the seam evaluation function. After the local error mark exists, the seam is iterated until the evaluation criteria are met. This method saves the delay of the algorithm for detecting moving objects, prevents the discontinuity of the picture caused by updating the seam multiple times in a short period of time, and achieves frame-by-frame seam processing.

[0043] The key points of the above design include but are not limited to:

[0044] (1) Uniform design of feature points. The first frame registration task is subdivided into grid blocks, local enhancement is performed based on the grid block image information entropy, and the feature points in the grid area are distributed as evenly as possible through the quadtree algorithm to reduce the registration error.

[0045] (2) Seam evaluation and iteration method: A comprehensive evaluation is performed based on the differences in the seam neighborhood, and an error analysis curve is obtained. The areas with high differences are locally marked and iterated to obtain the optimal seam.

[0046] (3) Overall stitching framework design. Unnecessary moving object detection and seam line update were optimized. Image information evaluation and seam line error evaluation were added to the registration and fusion stages respectively. Corresponding optimization measures were taken based on the evaluation results.

[0047] Compared with the prior art, the beneficial effects of the present invention and its preferred embodiments include at least:

[0048] (1) The present invention greatly improves the accuracy and efficiency of first frame registration alignment. The prior art usually only considers feature point detection and matching under the global image. When the feature points in the overlapping area are concentrated in the local area or there are too few feature points in some areas, obvious registration errors will occur. While enhancing the detection of local feature points, the present invention uses a feature point homogenization method to make the feature points as evenly distributed as possible, reducing the calculation of a large number of homogeneous feature points, and improving the efficiency and accuracy of first frame registration.

[0049] (2) The optimal stitching line obtained by the present invention is more in line with the real standard and takes less time. The stitching line obtained by the prior art is not necessarily the optimal stitching line, and the moving object detection algorithm in the stitching line update will consume a lot of time. Multiple updates of the stitching line in a short period of time will greatly reduce the splicing effect. The present invention adopts the stitching line evaluation and iteration method, and uses the error analysis curve of the stitching line after evaluation to perform local marking and iterative optimization of the stitching line, which reduces the complex moving object detection algorithm, and the stitching line obtained is of higher quality, takes less time, and can achieve frame-by-frame fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0051] Figure 1 is a system working flow diagram of an embodiment of the present invention;

[0052] Figure 2 This is a flowchart of the first frame registration process according to an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of stitching line iteration and evaluation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:

[0055] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0056] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0057] like Figure 1 The figure shows the video stitching method flow based on feature point homogenization and stitching line iteration provided by an embodiment of the present invention. In the actual processing process, the two videos to be stitched are processed in parallel with the same process. Steps 1 to 4 are first frame registration. The processing flow is as follows: Figure 2 As shown in the figure; Steps 5 to 8 are frame-by-frame fusion. The overall video splicing process is as follows:

[0058] Step 1: Image preprocessing. Grayscale the video input image and use template matching to reduce errors while ensuring that the image is clear. r (x,y), the target image is I t (x, y), the image size is w×h. In order to reduce the time of template matching, a 1 / 5 pixel block from the left edge of the target image to the right edge of the image is taken, denoted as T(x, y), and sliding matching is performed in the image to be registered. After finding the best matching pixel block, coordinate compensation is performed to obtain the coarse matching overlapping area, which is I r ″(x,y) and I t ″(x,y). The correlation coefficient formula for template matching is shown in Formula 1.

[0059]

[0060] Where R(x,y) represents the correlation and its value range is (0,1), x′ and y′ represent all coordinate values ​​of the traversed template pixel block, T′(x′,y′) represents the pixel value of the template pixel block, and I′(x+x′,y+y′) represents the pixel value of the matching image at the position (x+x′,y+y′).

[0061] Step 2: Feature point detection and homogenization. Before feature point detection, this embodiment divides the coarse matching overlapping area into N blocks by grid partitioning, and calculates the image information entropy for each block. Generally, the larger the information entropy, the more feature points can be detected. The entropy value calculation is shown in Formula 2:

[0062]

[0063] where p ij is the probability of occurrence of a feature binary consisting of the grayscale mean of the image neighborhood and the grayscale pixel of the image (i, j), and H is the two-dimensional information entropy of the image. The entropy value information of each grid block is judged according to the image information entropy, and thus it is divided into a low entropy domain and a non-low entropy domain. A local adaptive histogram equalization algorithm is introduced in the low entropy domain to improve the image information entropy without increasing noise by balancing the image brightness, chroma and saturation.

[0064] SURF feature point detection is performed on the processed coarse matching overlapping area, and then the image is subjected to quadtree homogenization processing. The specific process is as follows: set the maximum number of retained feature points M, the initial area is S, if the number of feature points in S is greater than 1, then S is split into 4 sub-areas, if the number of sub-areas is less than M, each sub-area continues to be split into four, and when the number of feature points in the sub-area obtained after the split is zero, the area is deleted. In this way, the number of sub-areas with feature points S is split nIf it is greater than M or does not meet the splitting condition, the splitting is stopped, and then the feature point with the highest Hessian matrix response value is selected from each sub-region. Finally, the feature vector is constructed by calculating the Haar wavelet response value of the neighboring area of ​​the feature point, and the feature vector is normalized to obtain the feature descriptor.

[0065] Step 3: Feature point matching: The nearest neighbor matching algorithm is used to calculate the Euclidean distance between the feature points of the two images, and the ratio of the nearest neighbor Euclidean distance to the next nearest neighbor Euclidean distance is compared. When the ratio is less than the set threshold, it is identified as a matching point.

[0066] Step 4: Estimate the registration matrix. Purify the obtained matching point pairs, remove the wrong matching point pairs through the RANSAC algorithm, and finally estimate the homography matrix, that is, the registration matrix.

[0067] Step 5: Calculate and search for the optimal seam line. The image to be registered is transformed into the same plane as the target image through the homography matrix to ensure the alignment of the two images in the overlapping area. The optimal seam line is calculated based on the overlapping area. The calculation method is to search for a path with the lowest energy function in the overlapping area. The search method is as follows: Calculate the energy value of each pixel in the entire overlapping area. The energy function is shown in Formula 3, which is used to measure the similarity of pixel intensity, gradient and texture energy in the overlapping area. The smaller the energy value, the smaller the difference between the two images at this pixel point. Then, the optimal seam line is found through dynamic programming.

[0068] E(x,y)=ωP(x,y)+υG(x,y) 2 +λT(x,y) (3)

[0069] Where E(x,y) is the energy difference of the pixel points with the same coordinates in the overlapping area, P(x,y) is the intensity difference of the pixel points with the same coordinates, and G(x,y) 2 is the gradient difference, T(x, y) is the texture energy difference, ω, υ and λ are the weight ratios of the three respectively, and ω+υ+λ=1. Usually, the gradient change is more sensitive, so υ is taken as 0.6, ω and λ are each taken as 0.2.

[0070] Step 6: Suture evaluation and iteration. Considering that the sutures found by the dynamic programming method have local errors, in order to make the sutures globally in a smooth and stable area, a suture evaluation function is introduced to evaluate the difference of the suture neighborhood. The evaluation function is shown in Formula 4. After evaluation, a suture difference curve is generated. If the difference of the suture neighborhood is lower than the threshold, it remains unchanged; if the difference of the suture neighborhood is higher than the threshold, the suture and its neighborhood are marked as the maximum value, and the suture is iterated until the suture meets the evaluation criteria. The schematic diagram of suture iteration and evaluation is shown in Figure 3As shown, the left side is the iterative method, and the right side is the corresponding difference curve change. The neighborhood pixel width is usually 40 unit pixels.

[0071]

[0072] Among them, X s To synthesize similarity deviations, including error comparisons of brightness, contrast, structure, and gradient, is the average value of the pixels in the left neighborhood of the seam line, is the average value of the right neighborhood, is the variance of the left neighbor image pixels, is the right neighborhood variance, is the covariance between the left and right neighbor images, is the average gradient amplitude of the left neighborhood image pixels, is the average gradient amplitude of the right neighborhood, and is the dynamic equilibrium constant of pixel value, usually 6.5, 26 and 58.5.

[0073] Step 7: Weighted fusion based on seam lines. After obtaining the seam lines, the overlapping areas are divided into seam line areas and non-seam line areas. The seam line areas are the seam lines and their neighborhood pixels, and the non-seam line areas are other overlapping areas and non-overlapping areas. The two sides of the optimal seam line are processed according to the fade-in and fade-out weighted fusion algorithm. The weight value is adjusted according to the Euclidean distance from the optimal seam line to the neighborhood boundary, and finally the brightness and blur on both sides of the seam line are eliminated.

[0074] Step 8: Generate a stitched image. According to the above steps, the two original images are registered and fused, and the stitched image is generated frame by frame, and finally a stitched video is obtained.

[0075] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0076] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0077] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0078] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0079] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0080] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0081] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of a video stitching method based on feature point homogenization and seam line iteration under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. A video stitching method based on feature point homogenization and seam line iteration, Features: Video stitching includes two stages: first frame registration and frame-by-frame fusion. In the first frame registration stage, the overlapping areas of the preprocessed image are firstly divided into blocks and the image information entropy is calculated. In the low entropy area, the local image enhancement algorithm is used to improve the image information entropy, and the image feature points are extracted through the feature detection algorithm. Then, the quadtree algorithm is used to further subdivide the feature points, so as to achieve the homogenization of the feature points. In the frame-by-frame fusion stage, the stitching line neighborhood differentiation index is detected, and the areas with large differences are locally marked and iterated, so as to guide the stitching line and its neighborhood to be in the smooth area, so as to ensure the accuracy and real-time performance of video stitching.

2. The video stitching method based on feature point homogenization and stitching line iteration according to claim 1, Features: The first frame registration stage includes: preprocessing, feature point detection and homogenization, feature point matching, and estimation of the registration matrix; after preprocessing rough matching, the feature point detection and homogenization are to extract feature points as much as possible in the overlapping area and disperse the feature points, then perform nearest neighbor feature point matching on the feature points of the two images to obtain matching point pairs, and purify the matching point pairs through the RANSAC algorithm, and finally estimate the registration matrix to transform the image to be registered into the target image plane.

3. The video stitching method based on feature point homogenization and stitching line iteration according to claim 1, Features: The video frame-by-frame fusion stage includes: seam line calculation, seam line evaluation and iteration, seam line-based weighted fusion and generation of stitched images; The stitching line calculation is to calculate the energy difference between corresponding pixels based on the overlapping areas of the two images, and the energy function is composed of structural difference and color difference; the error analysis is performed between the stitching line neighborhoods through stitching line evaluation, and the local area with larger error in the error analysis curve is iterated through stitching line iteration to obtain the optimal stitching line, and finally the stitching line neighboring area is weightedly fused, and other areas are directly mapped to obtain a stitched image, and then the stitching video is obtained by stitching frame by frame.

4. The video stitching method based on feature point homogenization and stitching line iteration according to claim 1, Features: Image preprocessing includes template matching of the video input image: Let the image to be registered be I r (x,y), the target image is I t (x, y), the image size is w×h, take the 1 / 5 pixel block from the left edge of the target image to the right edge of the image, record it as T(x, y), perform sliding matching in the image to be registered, find the best matching pixel block and perform coordinate compensation, thereby obtaining the coarse matching overlap area, which are I r ″(x,y) and I t ″(x,y); the correlation coefficient formula for template matching is as follows; Where R(x,y) represents the correlation and its value range is (0,1), x′ and y′ represent all coordinate values ​​of the traversed template pixel block, T′(x′,y′) represents the pixel value of the template pixel block, and I′(x+x′,y+y′) represents the pixel value of the matching image at the position (x+x′,y+y′).

5. The video stitching method based on feature point homogenization and stitching line iteration according to claim 2, Features: The feature point detection and homogenization include: before the feature point detection, the coarse matching overlapping area is divided into N blocks by grid partitioning, and the image information entropy is calculated for each block. The entropy value is calculated as shown in the following formula: where p ij is the probability of occurrence of a feature binary consisting of the grayscale mean of the image neighborhood and the grayscale pixel of the image (i, j), and H is the two-dimensional information entropy of the image; the entropy value information of each grid block is judged according to the image information entropy. When the entropy value exceeds 10, it means that it is in a non-low entropy value area, otherwise it is in a low entropy value area, thereby distinguishing each grid block of the image into a low entropy domain and a non-low entropy domain; For low entropy domain, a local adaptive histogram equalization algorithm is used to balance image brightness, chroma and saturation to improve image information entropy without increasing noise. SURF feature point detection is performed on the processed coarse matching overlapping area, and then the image is subjected to quadtree homogenization processing. The specific process is as follows: set the maximum number of retained feature points M, the initial area is S, if the number of feature points in S is greater than 1, then S is split into 4 sub-areas, if the number of sub-areas is less than M, each sub-area continues to be split into four, when the number of feature points in the sub-area obtained after the split is zero, the area is deleted; in this way, the number of sub-areas with feature points S is split n If it is greater than M or does not meet the splitting condition, the splitting is stopped, and then the feature point with the highest Hessian matrix response value is selected from each sub-region; finally, the feature vector is constructed by calculating the Haar wavelet response value of the neighboring area of ​​the feature point, and the feature descriptor is obtained after the feature vector is normalized.

6. The video stitching method based on feature point homogenization and stitching line iteration according to claim 2, Features: The feature point matching adopts the nearest neighbor matching algorithm to calculate the Euclidean distance between the feature points of the two images, and compare the ratio of the nearest neighbor Euclidean distance and the next nearest neighbor Euclidean distance. When the ratio is less than a set threshold, it is identified as a matching point.

7. The video stitching method based on feature point homogenization and stitching line iteration according to claim 3, Features: The process of calculating and searching for the optimal seam line includes: converting the image to be registered to the same plane as the target image through the homography matrix to ensure the alignment of the two images in the overlapping area, and calculating the optimal seam line based on the overlapping area. The calculation method is to search for a path with the lowest energy function in the overlapping area. The search method is as follows: Calculate the energy value of each pixel in the entire overlapping area to measure the similarity of pixel intensity, gradient and texture energy in the overlapping area. The smaller the energy value, the smaller the difference between the two images at the pixel point. Then, the optimal seam line is found by dynamic programming. The energy value is calculated as follows: E(x,y)=ωP(x,y)+υG(x,y) 2 +λT(x,y) (3) Where E(x,y) is the energy difference of the pixels with the same coordinates in the overlapping area, P(x,y) is the intensity difference, and G(x,y) is 2 is the gradient difference, T(x,y) is the texture energy difference, ω, υ and λ are weight coefficients.

8. The video stitching method based on feature point homogenization and stitching line iteration according to claim 3, Features: During the suture evaluation and iteration process: the suture evaluation function is used to evaluate the difference of the suture neighborhood, and the evaluation function is used; after the evaluation, a suture difference curve is generated. If the difference of the suture neighborhood is lower than the threshold, it remains unchanged; if the difference of the suture neighborhood is higher than the threshold, the suture and its neighborhood are marked as the maximum value, and the suture is iterated until the suture meets the evaluation criteria; The calculation of the seam evaluation function is as follows: Among them, X s To synthesize similarity deviations, including error comparisons of brightness, contrast, structure, and gradient, is the average value of the pixels in the left neighborhood of the seam line, is the average value of the right neighborhood, is the variance of the left neighbor image pixels, is the right neighborhood variance, is the covariance between the left and right neighbor images, is the average gradient amplitude of the left neighborhood image pixels, is the average gradient amplitude of the right neighborhood, and is the pixel value dynamic equilibrium constant.

9. The video stitching method based on feature point homogenization and stitching line iteration according to claim 3, Features: Weighted fusion based on seam lines; after obtaining the seam lines, the overlapping areas are divided into seam line areas and non-seam line areas. The seam line areas are the seam lines and their neighborhood pixels, and the non-seam line areas are other overlapping areas and non-overlapping areas. The two sides of the optimal seam line are processed according to the fade-in and fade-out weighted fusion algorithm, and the weight value is adjusted according to the Euclidean distance from the optimal seam line to the neighborhood boundary.

10. A video splicing system, Features: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the video stitching method according to any one of claims 1 to 9 when executing the program.

Citation Information

Cited By

  • Video stitching method and device based on embedded platform

    CN120050373A

  • Depth image alignment method based on quadtree grid optimization

    CN120070527A