A panoramic video real-time stitching method and system

Abstract

The present invention provides a panoramic video real-time stitching method and system, and relates to the field of panoramic video technology. The method comprises: S1, obtaining a picture from each camera, and performing multiple iterations to calculate stitching parameters; S2, from the stitching parameters, according to the clarity of the overlapping area of ​​the panoramic image, selecting the stitching parameters with the highest clarity as the default stitching parameters; S3, reading the default stitching parameters, fusing and generating a complete stitching result, calculating various parameters in the stitching algorithm through cyclic iteration, generating multiple possible results, and calculating the clarity of the panoramic image, selecting the result with the highest clarity as the default stitching parameter for real-time stitching, so as to quickly and conveniently obtain the optimal stitching result of the panoramic video.

Description

Technical Field

[0001] The present invention relates to the field of panoramic video technology, and in particular to a panoramic video real-time stitching method and system. Background Art

[0002] Panoramic video refers to wide-angle video content that exceeds the general field of view. Usually, what we call circular / cylindrical 360-degree video provides a 360-degree field of view in the horizontal direction, and the spherical panorama can reach 180 degrees in the vertical direction, thus covering the entire three-dimensional space where the user is located. Panoramic video can give users a more immersive experience and can be applied in many fields, such as virtual reality, security, monitoring, film, video communication, education, tourism, etc.

[0003] A common method of acquiring panoramic videos is to use multiple cameras pointing in different directions and stitching several videos together using software or hardware. However, since panoramic video stitching is very complex and requires a lot of computation, it takes a long time to stitch panoramic videos together. In addition, due to the influence of stitching algorithms, parameters, and image noise, the same set of images will have greatly different stitching results each time, making it difficult to select an optimal result. Summary of the invention

[0004] In order to overcome the deficiencies of the prior art, the present invention provides a panoramic video real-time stitching method and system to solve the above-mentioned technical problems.

[0005] The technical method adopted by the present invention to solve its technical problem is: a real-time panoramic video stitching method, the improvement of which is: comprising the following steps: S1, obtaining a picture from each camera, and performing multiple iterations to calculate the stitching parameters; S2, from the stitching parameters, according to the clarity of the overlapping area of ​​the panoramic image, selecting the stitching parameters with the highest clarity as the default stitching parameters; S3, reading the default stitching parameters, and fusing them to generate a complete stitching result.

[0006] In the above method, the step S1 performs multiple iterative calculations, and before each iteration, a corresponding coefficient may be selected, which is an image feature extractor, image feature matching and / or bundle adjustment.

[0007] In the above method, step S2 comprises the following steps:

[0008] S21, intercepting a corresponding number of overlapping area pictures, where the corresponding number is the number of cameras minus one;

[0009] S22, obtaining a definition value of the image;

[0010] S23, calculating the cumulative sum of the sharpness of each overlapping area of ​​each panoramic image, and sorting them according to this, and selecting the panoramic image parameter with the largest sharpness value as the default stitching parameter.

[0011] In the above method, step S22 comprises the following steps:

[0012] S221, converting the overlapping area image from a color image to a grayscale image, extracting features, and obtaining a feature map;

[0013] S222. Calculate the average value of the feature map to obtain a clarity value.

[0014] In the above method, in step S221, feature extraction is performed by using a Sobel operator or a Laplacian operator.

[0015] In the above method, step S22 can also be implemented by the following steps:

[0016] Convert the overlapping area images from color to grayscale;

[0017] Then the standard deviation of the image is calculated, and the obtained standard deviation is the clarity value.

[0018] In the above method, step S3 comprises the following steps:

[0019] S31, read the default splicing parameters, and obtain the feature extractor and matching parameters therein;

[0020] S32, read all video stream addresses, and independently decode each stream to obtain a video frame;

[0021] S33, the images acquired by each stream are perspective transformed into the same plane coordinate system through a homography matrix algorithm;

[0022] S34, using the feature extractor in the default stitching parameters to extract features from the two stitched pictures, and determine matching points;

[0023] S35, matching and fusing the determined matching points according to the matching parameters to generate a spliced ​​image, and repeating the process until all cameras are fused, and then jumping to step S36;

[0024] S36, generating a complete splicing result.

[0025] The present invention also provides a panoramic video real-time stitching system, comprising an offline module and an online module, wherein the offline module is used to obtain a picture from each camera, and perform multiple iterations to calculate stitching parameters; and from the stitching parameters, according to the clarity of the overlapping area of ​​the panoramic image, the stitching parameters with the highest clarity are selected as default stitching parameters;

[0026] The online module is used to read the default splicing parameters and fuse them to generate a complete splicing result.

[0027] The beneficial effects of the present invention are as follows: by cyclically iterating and calculating various parameters in the stitching algorithm, a variety of possible results are generated, and by calculating the clarity of the panoramic image, the result with the highest clarity is selected as the default stitching parameter for real-time stitching, so that the optimal stitching result of the panoramic video can be obtained quickly and conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Attached Figure 1 The present invention is a flowchart of a panoramic video real-time stitching method. DETAILED DESCRIPTION

[0029] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0030] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0031] Reference Figure 1 As shown, the present invention provides a panoramic video real-time stitching method, comprising the following steps S1-S3:

[0032] S1, respectively obtain a picture from each road camera, carry out multiple iterations to calculate splicing parameters, before each iteration, can select different coefficients according to actual conditions, this coefficient is image feature extractor (comprising characteristic points such as orb, akaze, surf, sift), image feature matching and / or bundle adjustment;

[0033] S2. From among the plurality of stitching parameters, according to the clarity of the overlapping area of ​​the panoramic image, select the stitching parameter with the highest clarity as the default stitching parameter for actual online stitching;

[0034] Specifically, the step S2 includes the following steps:

[0035] S21, intercepting a corresponding number of overlapping area pictures, where the corresponding number is the number of cameras minus one. For example, if 5 cameras are stitched together into a panoramic image, 4 overlapping area pictures can be intercepted;

[0036] S22, obtaining a definition value of the image;

[0037] Specifically, the step S22 includes the following steps:

[0038] S221, converting the overlapping area image from a color image to a grayscale image, extracting features, and obtaining a feature map;

[0039] S222. Calculate the average value of the feature map to obtain a clarity value.

[0040] Furthermore, in step S221, feature extraction is performed by using a Sobel operator (ie, Sobel operator) or a Laplacian operator (ie, Laplacian operator, which is a second-order differential operator in n-dimensional Euclidean space).

[0041] Furthermore, obtaining the clarity value of the image can also be achieved through the following steps:

[0042] Convert the overlapping area images from color to grayscale;

[0043] Then the standard deviation of the image is calculated, and the obtained standard deviation is the clarity value.

[0044] S23, calculating the cumulative sum of the sharpness of each overlapping area of ​​each panoramic image, and sorting them according to this, and selecting the panoramic image parameter with the largest sharpness value as the default stitching parameter.

[0045] S3, reading the default splicing parameters, and fusing them to generate a complete splicing result,

[0046] Specifically, step S3 includes the following steps:

[0047] S31, read the default splicing parameters, and obtain the feature extractor and matching parameters therein;

[0048] S32, read all video stream addresses, and independently decode each stream to obtain video frames; for different hardware devices, such as Nvidia series graphics cards, hardware decoding can also be supported to improve performance;

[0049] S33, the images acquired by each stream are perspective transformed into the same plane coordinate system through the homography matrix algorithm to solve the distortion and tilt problems that each camera has;

[0050] S34, using the feature extractor in the default stitching parameters to extract features from the two stitched pictures, and determine matching points;

[0051] S35, matching and fusing the determined matching points according to the matching parameters to generate a spliced ​​image, and repeating the process until all cameras are fused, and then jumping to step S36;

[0052] S36, generating a complete splicing result.

[0053] The present invention also provides a panoramic video real-time stitching system, comprising an offline module and an online module, wherein the offline module is used to obtain a picture from each camera, and perform multiple iterations to calculate stitching parameters; and from the stitching parameters, according to the clarity of the overlapping area of ​​the panoramic image, the stitching parameters with the highest clarity are selected as default stitching parameters;

[0054] The online module is used to read the default splicing parameters and fuse them to generate a complete splicing result.

[0055] The present invention generates multiple possible results by cyclically iteratively calculating various parameters in the stitching algorithm, and by calculating the clarity of the panoramic image, selecting the result with the highest clarity as the default stitching parameter for real-time stitching, so as to quickly and conveniently obtain the optimal stitching result of the panoramic video.

[0056] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A panoramic video real-time stitching method, characterized by: The steps include: S1, obtain a picture from each camera, and perform multiple iterations to calculate the stitching parameters; S2. From the stitching parameters, according to the clarity of the overlapping area of ​​the panoramic image, select the stitching parameter with the highest clarity as the default stitching parameter; S3, reading the default splicing parameters, and fusing them to generate a complete splicing result; The step S2 comprises the following steps: S21, intercepting a corresponding number of overlapping area pictures, where the corresponding number is the number of cameras minus one; S22, obtaining a definition value of the image; S23, calculating the cumulative sum of the sharpness of each overlapping area of ​​each panoramic image, and sorting them according to this, and selecting the panoramic image parameter with the largest sharpness value as the default stitching parameter; The step S3 comprises the following steps: S31, read the default splicing parameters, and obtain the feature extractor and matching parameters therein; S32, read all video stream addresses, and independently decode each stream to obtain a video frame; S33, the images acquired by each stream are perspective transformed into the same plane coordinate system through a homography matrix algorithm; S34, using the feature extractor in the default stitching parameters to extract features from the two stitched pictures, and determine matching points; S35, matching and fusing the determined matching points according to the matching parameters to generate a spliced ​​image, and repeating the process until all cameras are fused, and then jumping to step S36; S36, generating a complete splicing result.

2. A panoramic video real-time stitching method as claimed in claim 1, characterized in that: In the step S1, multiple iterative calculations are performed, and before each iteration, corresponding coefficients are selected, which are image feature extractors, image feature matching and / or bundle adjustment.

3. A panoramic video real-time stitching method as claimed in claim 1, characterized in that: The step S22 comprises the following steps: S221, converting the overlapping area image from a color image to a grayscale image, extracting features, and obtaining a feature map; S222. Calculate the average value of the feature map to obtain a clarity value.

4. A panoramic video real-time stitching method as claimed in claim 3, characterized in that: In the step S221, features are extracted by using a Sobel operator or a Laplacian operator.

5. A panoramic video real-time stitching method as claimed in claim 1, characterized in that: The step S22 is also implemented by the following steps: Convert the overlapping area images from color to grayscale; Then the standard deviation of the image is calculated, and the obtained standard deviation is the clarity value.

6. A panoramic video real-time stitching system, characterized by: It includes an offline module and an online module. The offline module is used to obtain a picture from each camera, and perform multiple iterations to calculate stitching parameters. From the stitching parameters, according to the clarity of the overlapping area of ​​the panoramic image, the stitching parameters with the highest clarity are selected as the default stitching parameters. The online module is used to read the default splicing parameters and fuse them to generate a complete splicing result; The offline module is also used to capture a corresponding number of overlapping area images, where the corresponding number is the number of cameras minus one; Obtaining a clarity value of the image; Calculate the cumulative sum of the clarity of each overlapping area of ​​each panoramic image, sort them according to this, and select the panoramic image parameter with the largest clarity value as the default stitching parameter; The online module is also used to read the default splicing parameters and obtain the feature extractor and matching parameters therein; Read all video stream addresses and decode each stream independently to obtain video frames; The images acquired by each stream are perspective transformed to the same plane coordinate system through the homography matrix algorithm; Use the feature extractor in the default stitching parameters to extract features from the two stitched images and determine matching points; The determined matching points are matched and fused according to the matching parameters to generate a stitched image, and the process is repeated until all cameras are fused to generate a complete stitching result.

Images