A video image stitching method and system based on feature matching

Abstract

The present invention relates to a method and system for video image splicing based on feature matching, and belongs to the field of digital video technology. The method comprises: obtaining a real-time video stream and a reference image and performing multi-scale detection to obtain feature points of the video stream image and the reference image; performing feature matching on the feature points to obtain feature matching points; calculating transformation model parameters according to the feature matching points, and projecting and changing the video stream image through the transformation model to obtain a registered image; planning stitching lines in overlapping areas through energy values ​​of the registered images; updating stitching lines in video images by detecting moving foreground objects; fusing images within a fixed width range of stitching lines, and adaptively adjusting the fusion area through color difference. The optimization accelerates the feature point detection and matching process, and solves the problem of artifacts in the image splicing process.

Description

Technical Field

[0001] The invention belongs to the field of digital video technology, and in particular relates to a video image splicing method and system based on feature matching. Background Art

[0002] Video stitching technology needs to process real-time video streams, but traditional algorithms such as SIFT and SURF have a large amount of computation in the process of feature point detection and matching, resulting in poor real-time performance and affecting video fluency. When processing scenes with low texture, high contrast or motion blur, feature matching algorithms are prone to mismatching, resulting in poor stitching effects. During the stitching process, due to inaccurate feature point matching, stitching cracks are easily generated, affecting the visual effect. Feature matching algorithms usually require a lot of computing resources, especially when processing large-scale images or video streams, the computational complexity is high and it is difficult to meet real-time requirements. In areas with a single background or unclear texture, such as the sky and the sea, it is difficult to extract feature points, resulting in registration failure; however, video stitching cannot be simply considered as image stitching frame by frame. If the image registration process is completed for each video frame, it will be very time-consuming and cannot meet real-time requirements. On the other hand, considering the changes in video content over time, when a moving foreground object passes through a seam, its structure will easily break, resulting in artifacts around the moving foreground object. Summary of the invention

[0003] In order to solve the above problems existing in the prior art, the present invention provides a video image stitching method and system based on feature matching.

[0004] The purpose of the present invention can be achieved through the following technical solutions:

[0005] A video image stitching method based on feature matching, comprising:

[0006] S1: Acquire a real-time video stream and a reference image, and perform multi-scale detection on the image of the real-time video stream and the reference image to obtain feature points of the video stream image and feature points of the reference image;

[0007] Performing feature matching on the feature points to obtain feature matching points;

[0008] S2: Calculate the transformation model parameters according to the feature matching points, project the video stream image through the transformation model to obtain a registered image, and store the frame number mapping data of the registered image in an index table; use the foreground area item, grayscale difference item, edge item, and color difference item of the registered image as the energy value of the pixel point of the registered image, and plan the stitching line in the overlapping area of ​​the registered image according to the energy value;

[0009] S3: Initialize sampling of the background model of the first frame of the video to obtain background sample points, preset a background difference value and a background judgment threshold, if the number of sample points of the pixel point of the video stream image in the pixel set composed of the background difference value and the number of common sample points of the background sample point are greater than the background judgment threshold, then the pixel point is classified as a background pixel point, otherwise it is set as a foreground pixel point; for the subsequent frame images of the video stream image, the subsequent frames of the video are directly input, and the values ​​in the index table are directly searched for to perform image transformation, and if the moving object passes through the stitching line, the stitching line is updated;

[0010] S4: fusing the images within the fixed width range of the stitching line, and adaptively adjusting the fusion area by color difference.

[0011] Specifically, the multi-scale detection method is: by using a Gaussian function as a convolution kernel smoothing function, convolving the image to obtain a scale space, and using the extreme points of the scale space as image feature points.

[0012] Specifically, the feature matching method is: presetting a feature distance threshold, if the ratio of the Euclidean distance from the nearest neighbor feature point to the next nearest neighbor feature point to the feature point is greater than the feature distance threshold, then including the matching point pair in the initial matching point pair set;

[0013] Select 4 pairs of matching points from the initial matching point pair set, any three of the matching points are not collinear, and calculate the transformation matrix corresponding to the matching points through the projection transformation matrix; calculate the projection error through the transformation matrix, if the projection error is less than a preset deviation threshold, add the corresponding matching point pair to the matching point pair set, if the point pair scale of the matching point pair set reaches a preset recognition upper limit, use the matching point pair set as the feature matching result.

[0014] Specifically, the initialization sampling: preset sampling times, randomly select a pixel point in the image, select the pixel point and one of the eight adjacent pixels around it and put it into the background model, and continue to select sample points according to the sampling times to form a set of background sample points.

[0015] Specifically, the index table is used to map each pixel of the video frame image to be spliced ​​to a reference reference coordinate using a bilinear interpolation method.

[0016] Specifically, the color difference adaptive adjustment method is:

[0017] Convert the image from RGB space to HIS space, and calculate the saturation component in the HIS space;

[0018] The width of the suture line is calculated according to the saturation component, and the calculation formula is:

[0019] ,

[0020] Among them, (i, j) is the coordinate of the pixel point on the stitching line, L represents the width, b is the initial value of the fusion width, λ is the color difference weight, S M , S R is the color saturation component of the two fused images;

[0021] The two images in the fusion area are fused according to the width.

[0022] A video image stitching system based on feature matching, comprising: an image initialization module, a projection transformation module, a stitching line update module, and a video image fusion module;

[0023] The image initialization module is used to obtain a real-time video stream and a reference image, perform multi-scale detection on the image of the real-time video stream and the reference image to obtain feature points of the video stream image and feature points of the reference image; perform feature matching on the feature points to obtain feature matching points;

[0024] The projection transformation module is used to calculate the transformation model parameters according to the feature matching points, project the video stream image through the transformation model to obtain the registered image, and store the frame number mapping data of the registered image in the index table; the foreground area item, grayscale difference item, edge item, and color difference item of the registered image are used as the energy value of the pixel point of the registered image, and the stitching line in the overlapping area of ​​the registered image is planned according to the energy value;

[0025] The stitching line update module is used to perform initialization sampling of the background model on the first frame image of the video to obtain background sample points, preset background difference value and background judgment threshold, if the number of sample points of the pixel point of the video stream image in the pixel set composed of the background difference value and the number of common sample points of the background sample point are greater than the background judgment threshold, then the pixel point is classified as a background pixel point, otherwise it is set as a foreground pixel point; for the subsequent frame image of the video stream image, the subsequent frame of the video is directly input, and the value in the index table is directly searched for image transformation, and if the moving object passes through the stitching line, the stitching line is updated;

[0026] The video image fusion module is used to fuse images within a fixed width range of the stitching line and to adaptively adjust the fusion area through color difference.

[0027] The beneficial effects of the present invention are:

[0028] The algorithm speed is significantly improved by optimizing and accelerating the feature point detection and matching process. The improved feature detection algorithm is used to remove mismatched points through multi-scale spatial detection to improve the image stitching accuracy and robustness. The best stitching line is found by using a dynamic programming algorithm to effectively suppress the stitching cracks and "GHOST" phenomenon. The feature-based registration method is used to improve the registration accuracy. In the image fusion process, the energy-minimizing dynamic programming technology is used to find the best stitching line, and the best stitching line is updated according to the moving target recognition. The fusion area is adaptively adjusted according to the color saturation difference to reduce the appearance of stitching seams and solve the parallax artifact problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0030] Figure 1 The figure is a flow chart of a method for video image stitching based on feature matching according to the present invention. DETAILED DESCRIPTION

[0031] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0032] See also Figure 1 , a video image stitching method based on feature matching, comprising:

[0033] S1: Acquire a real-time video stream and a reference image, and perform multi-scale detection on the image of the real-time video stream and the reference image to obtain feature points of the video stream image and feature points of the reference image;

[0034] Performing feature matching on the feature points to obtain feature matching points;

[0035] S2: Calculate the transformation model parameters according to the feature matching points, project the video stream image through the transformation model to obtain a registered image, and store the frame number mapping data of the registered image in an index table; use the foreground area item, grayscale difference item, edge item, and color difference item of the registered image as the energy value of the pixel point of the registered image, and plan the stitching line in the overlapping area of ​​the registered image according to the energy value;

[0036] S3: Initialize sampling of the background model of the first frame of the video to obtain background sample points, preset a background difference value and a background judgment threshold, if the number of sample points of the pixel point of the video stream image in the pixel set composed of the background difference value and the number of common sample points of the background sample point are greater than the background judgment threshold, then the pixel point is classified as a background pixel point, otherwise it is set as a foreground pixel point; for the subsequent frame images of the video stream image, the subsequent frames of the video are directly input, and the values ​​in the index table are directly searched for to perform image transformation, and if the moving object passes through the stitching line, the stitching line is updated;

[0037] S4: fusing the images within the fixed width range of the stitching line, and adaptively adjusting the fusion area by color difference.

[0038] Specifically, the multi-scale detection method is: by using a Gaussian function as a convolution kernel smoothing function, convolving the image to obtain a scale space, and using the extreme points of the scale space as image feature points.

[0039] In this embodiment, FAST operator detection is adopted, and FAST feature detection is performed using segmentation detection criteria. An image pyramid is established to introduce scale characteristics. The direction of the top feature point is then removed by the grayscale centroid method, and the BRIEF operator is extracted based on this direction. The BRIEF algorithm first randomly selects a number of point pairs near the extracted feature point to form an image block, and then binarizes the image block to form a binary string, which is used as the feature operator of the feature point. Finally, two randomly selected binary points are compared to obtain one bit of the BRIEF operator. During feature detection, an integer is set to control the number of feature points to be extracted and the minimum spacing between feature points is set to obtain a uniformly distributed feature set.

[0040] Specifically, the feature matching method is: presetting a feature distance threshold, if the ratio of the Euclidean distance from the nearest neighbor feature point to the next nearest neighbor feature point to the feature point is greater than the feature distance threshold, then including the matching point pair in the initial matching point pair set;

[0041] Select 4 pairs of matching points from the initial matching point pair set, any three of the matching points are not collinear, and calculate the transformation matrix corresponding to the matching points through the projection transformation matrix; calculate the projection error through the transformation matrix, if the projection error is less than a preset deviation threshold, add the corresponding matching point pair to the matching point pair set, if the point pair scale of the matching point pair set reaches a preset recognition upper limit, use the matching point pair set as the feature matching result.

[0042] Specifically, the initialization sampling: preset sampling times, randomly select a pixel point in the image, select the pixel point and one of the eight adjacent pixels around it and put it into the background model, and continue to select sample points according to the sampling times to form a set of background sample points.

[0043] Specifically, the index table is used to map each pixel of the video frame image to be spliced ​​to a reference reference coordinate using a bilinear interpolation method.

[0044] Specifically, the color difference adaptive adjustment method is:

[0045] Convert the image from RGB space to HIS space, and calculate the saturation component in the HIS space;

[0046] The width of the suture line is calculated according to the saturation component, and the calculation formula is:

[0047] ,

[0048] Among them, (i, j) is the coordinate of the pixel point on the stitching line, L represents the width, b is the initial value of the fusion width, λ is the color difference weight, S M , S R is the color saturation component of the two fused images;

[0049] The two images in the fusion area are fused according to the width.

[0050] A video image stitching system based on feature matching, comprising: an image initialization module, a projection transformation module, a stitching line update module, and a video image fusion module;

[0051] The image initialization module is used to obtain a real-time video stream and a reference image, perform multi-scale detection on the image of the real-time video stream and the reference image to obtain feature points of the video stream image and feature points of the reference image; perform feature matching on the feature points to obtain feature matching points;

[0052] The projection transformation module is used to calculate the transformation model parameters according to the feature matching points, project the video stream image through the transformation model to obtain the registered image, and store the frame number mapping data of the registered image in the index table; the foreground area item, grayscale difference item, edge item, and color difference item of the registered image are used as the energy value of the pixel point of the registered image, and the stitching line in the overlapping area of ​​the registered image is planned according to the energy value;

[0053] The stitching line update module is used to perform initialization sampling of the background model on the first frame image of the video to obtain background sample points, preset background difference value and background judgment threshold, if the number of sample points of the pixel point of the video stream image in the pixel set composed of the background difference value and the number of common sample points of the background sample point are greater than the background judgment threshold, then the pixel point is classified as a background pixel point, otherwise it is set as a foreground pixel point; for the subsequent frame image of the video stream image, the subsequent frame of the video is directly input, and the value in the index table is directly searched for image transformation, and if the moving object passes through the stitching line, the stitching line is updated;

[0054] The video image fusion module is used to fuse images within a fixed width range of the stitching line and to adaptively adjust the fusion area through color difference.

[0055] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0056] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0057] The program code included on the computer readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages-such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0058] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A video image stitching method based on feature matching, characterized in that: include: S1: Acquire a real-time video stream and a reference image, perform multi-scale detection on the image of the real-time video stream and the reference image to obtain feature points of the video stream image and feature points of the reference image; perform feature matching on the feature points to obtain feature matching points; The feature matching method is as follows: a feature distance threshold is preset, and if the ratio of the Euclidean distance from the nearest neighbor feature point to the next nearest neighbor feature point to the feature point is greater than the feature distance threshold, the matching point pair is included in the initial matching point pair set; 4 pairs of matching points are selected from the initial matching point pair set, and any three of the matching points are not collinear, and the transformation matrix corresponding to the matching points is calculated by a projection transformation matrix; the projection error is calculated by the transformation matrix, and if the projection error is less than a preset deviation threshold, the corresponding matching point pair is added to the matching point pair set, and if the point pair scale of the matching point pair set reaches a preset recognition upper limit, the matching point pair set is used as a feature matching result; S2: Calculate the transformation model parameters according to the feature matching points, project the video stream image through the transformation model to obtain a registered image, and store the frame number mapping data of the registered image in an index table; use the foreground area item, grayscale difference item, edge item, and color difference item of the registered image as the energy value of the pixel point of the registered image, and plan the stitching line in the overlapping area of ​​the registered image according to the energy value; S3: Initialize sampling of the background model of the first frame of the video to obtain background sample points, preset a background difference value and a background judgment threshold, if the number of sample points of the pixel point of the video stream image in the pixel set composed of the background difference value and the number of common sample points of the background sample point are greater than the background judgment threshold, then the pixel point is classified as a background pixel point, otherwise it is set as a foreground pixel point; for the subsequent frame images of the video stream image, the subsequent frames of the video are directly input, and the values ​​in the index table are directly searched for to perform image transformation, and if the moving object passes through the stitching line, the stitching line is updated; S4: fusing the images within the fixed width range of the stitching line, and adaptively adjusting the fusion area by color difference; the color difference adaptive adjustment method is: Convert the image from RGB space to HIS space, and calculate the saturation component in the HIS space; The width of the suture line is calculated according to the saturation component, and the calculation formula is: ， Among them, (i, j) is the coordinate of the pixel point on the stitching line, L represents the width, b is the initial value of the fusion width, λ is the color difference weight, S M , S R is the color saturation component of the two fused images; The two images in the fusion area are fused according to the width.

2. The method according to claim 1, characterized in that The multi-scale detection method is: by using a Gaussian function as a convolution kernel smoothing function, convolving the image to obtain a scale space, and using the extreme value points of the scale space as image feature points.

3. The method according to claim 1, characterized in that The initialization sampling: preset the sampling times, randomly select a pixel point in the image, select the pixel point and one of the eight adjacent pixels around it and put it into the background model, and continue to select sample points according to the sampling times to form a set of background sample points.

4. The method according to claim 1, characterized in that The index table is used to map each pixel of the video frame image to be spliced ​​to a reference reference coordinate using a bilinear interpolation method.

5. A system using the video image stitching method based on feature matching according to claim 1, characterized in that: include: Image initialization module, projection transformation module, seam line update module, video image fusion module; The image initialization module is used to obtain a real-time video stream and a reference image, perform multi-scale detection on the image of the real-time video stream and the reference image to obtain feature points of the video stream image and feature points of the reference image; perform feature matching on the feature points to obtain feature matching points; The projection transformation module is used to calculate the transformation model parameters according to the feature matching points, project the video stream image through the transformation model to obtain the registered image, and store the frame number mapping data of the registered image in the index table; the foreground area item, grayscale difference item, edge item, and color difference item of the registered image are used as the energy value of the pixel point of the registered image, and the stitching line in the overlapping area of ​​the registered image is planned according to the energy value; The stitching line update module is used to perform initialization sampling of the background model on the first frame image of the video to obtain background sample points, preset background difference value and background judgment threshold, if the number of sample points of the pixel point of the video stream image in the pixel set composed of the background difference value and the number of common sample points of the background sample point are greater than the background judgment threshold, then the pixel point is classified as a background pixel point, otherwise it is set as a foreground pixel point; for the subsequent frame image of the video stream image, the subsequent frame of the video is directly input, and the value in the index table is directly searched for image transformation, and if the moving object passes through the stitching line, the stitching line is updated; The video image fusion module is used to fuse images within a fixed width range of the stitching line and to adaptively adjust the fusion area through color difference.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the video image stitching method based on feature matching as described in any one of claims 1 to 4 is implemented.

7. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the video image stitching method based on feature matching as described in any one of claims 1 to 4 when executed by a computer processor.

Images