Method and device for splicing multiple videos in real time

Abstract

The invention relates to a multi-view video real-time splicing method and device, belongs to the technical field of multi-view video splicing, and solves the problems of insufficient view fields in the vertical direction and the horizontal direction during three-view or four-view video splicing, insufficient real-time performance during multi-view video splicing, low splicing quality and high hardware cost. The method comprises the following steps: acquiring frame images respectively shot by a multi-view camera in real time; wherein the multi-view cameras are uniformly distributed on a 180-degree arc on the same horizontal plane, and each optical axis is perpendicular to the horizontal plane; based on the image remapping parameter of each camera, mapping pixel points of each frame of image to a target coordinate system, and generating a seam mask between adjacent images; wherein the adjacent images are frame images shot by two adjacent cameras; and inputting the frames of images unified to the target coordinate system and the seam masks into a fusion device to obtain a spliced panoramic image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-view video stitching, in particular to a multi-view video real-time stitching method and device. BACKGROUND

[0002] Multi-view video stitching is a technology of stitching video images captured by multiple cameras into a panoramic image, which has important application value in virtual reality, security monitoring, intelligent driving, video conference, unmanned aerial vehicle aerial photography and other fields.

[0003] The existing multi-view video stitching technology mainly focuses on three or four multi-view video stitching. However, since most of the existing cameras have a horizontal field of view greater than a vertical field of view, if the three or four multi-view camera video stitching process is placed horizontally, the stitched image will have insufficient vertical direction view coverage. If the three or four multi-view camera video stitching process is placed vertically, a larger horizontal field of view cannot be obtained. Moreover, the existing stitching technology mostly uses offline processing for image stitching or uses fewer cameras for real-time video stitching. Although offline image stitching can stitch many images, it cannot be used for real-time video stitching. Fewer cameras such as three or four multi-view video stitching can be used for real-time video stitching, but cannot obtain a larger field of view. At the same time, fewer cameras such as three or four multi-view video stitching can obtain a larger field of view, but the overlapping area between the images output by adjacent cameras is small, the number of feature points that can be matched is reduced, and inaccurate image registration is likely to occur, which can cause obvious visual discontinuity at the joint. Using more cameras can solve the problem of field of view, but increasing the number of cameras will lead to high computational complexity, insufficient image stitching quality, and high cost of high-performance GPUs. SUMMARY

[0004] In view of the above analysis, the embodiments of the present application aim to provide a multi-view video real-time stitching method and device to solve the problems of insufficient vertical and horizontal field of view during three or four multi-view video stitching, insufficient real-time performance during multi-view video stitching, low stitching quality, and high hardware cost.

[0005] The main purpose of the present application is achieved by the following technical solutions:

[0006] The present application provides a multi-view video real-time stitching method, the multi-view is five or more, and the method comprises the following steps:

[0007] Real-time acquisition of frame images captured by a multi-view camera; wherein the multi-view cameras are uniformly distributed on a 180-degree circular arc of the same horizontal plane, and the optical axes of the cameras are perpendicular to the horizontal plane;

[0008] Map the pixel points of each frame image to a target coordinate system based on image remapping parameters of each camera, and generate a seam mask between adjacent images; wherein the adjacent images are frame images captured by two adjacent cameras;

[0009] Input each frame image unified to the target coordinate system and each seam mask into a fusion device to obtain a spliced panoramic image.

[0010] Further, based on the camera extrinsic and intrinsic parameters of each camera and the size of the spliced image, the image remapping parameters of each camera are obtained using a spherical projection model.

[0011] Further, the camera extrinsic parameters of each camera are obtained, including:

[0012] Obtain calibration images captured simultaneously by each camera;

[0013] Use the LET-NET model to obtain feature points and their corresponding descriptors of each calibration image, and perform feature point matching on each adjacent calibration image to obtain a feature point matching result of each adjacent calibration image; wherein each adjacent calibration image refers to the calibration image corresponding to each adjacent camera;

[0014] Based on the feature point matching result of each adjacent calibration image, obtain a geometric transformation matrix of each adjacent camera;

[0015] Randomly select one camera as a reference camera, and based on the geometric transformation matrix of each adjacent camera, obtain the camera extrinsic parameters of each camera relative to the reference camera; wherein the extrinsic parameters of the reference camera are set to a unit matrix.

[0016] Further, the feature point matching of each adjacent calibration image to obtain the feature point matching result of each adjacent calibration image comprises:

[0017] Based on the descriptors of each feature point in each calibration image, calculate the cosine similarity of each feature point in each adjacent calibration image to obtain a preliminary matching feature point pair of each adjacent calibration image;

[0018] Based on the distance between the preliminary matching feature point pairs of each adjacent calibration image, obtain a matching feature point pair of each adjacent calibration image;

[0019] Perform bidirectional consistency detection on the matching feature point pairs of each adjacent calibration image to obtain the feature point matching result of each adjacent calibration image.

[0020] Further, the geometric transformation matrix is a homography matrix;

[0021] Based on the feature point matching result of each adjacent calibration image, use a random sample consensus algorithm to obtain the homography matrix of each adjacent camera;

[0022] Based on the homography matrix of each adjacent camera, the camera extrinsic parameters of each camera relative to the reference camera are obtained by using a homography matrix decomposition function and through chain multiplication.

[0023] Further, the geometric transformation matrix can also be an affine transformation matrix.

[0024] Based on the feature point matching results of each adjacent calibration image, the affine transformation matrix of each adjacent camera is obtained by using a least square method.

[0025] Based on the affine transformation matrix of each adjacent camera, the camera extrinsic parameters of each camera relative to the reference camera are obtained through chain multiplication.

[0026] Further, the generation of the seam mask of each adjacent image comprises:

[0027] Based on the positions of each pixel point of each adjacent image in the target coordinate system, the overlapping region between each adjacent image is obtained.

[0028] The gradient information of each overlapping region is calculated, and the path with the smallest gradient is selected as the seam position of each adjacent image.

[0029] Based on the seam position of each adjacent image, the corresponding seam mask is generated; wherein the mask value of the seam region is 1; and the mask value of the non-seam region is 0.

[0030] Further, the generation of the seam mask of each adjacent image comprises:

[0031] Based on the corner point coordinates and image size of each frame image after exposure error compensation, the blender is initialized.

[0032] Based on each seam mask, the pixel points of each frame image after exposure compensation at the seam are fused in the blender, so as to obtain the spliced panoramic image.

[0033] Further, in the image splicing process, a mutex is used to avoid thread conflict.

[0034] On the other hand, the application provides a device for real-time splicing of multi-view video, which comprises a multi-view camera, an image processor, an image storage and a display; the multi-view camera is five or more than five cameras.

[0035] The multi-view camera is five or more than five same cameras which are uniformly and vertically distributed on a horizontal fan, the included angle between adjacent cameras is equal, and the camera covers a view angle of more than 180 degrees, so as to obtain multi-angle video images in the same plane.

[0036] The image processor is used for receiving frame images respectively shot by the multi-view camera in real time, and performing image splicing.

[0037] The image memory is used for storing the spliced video images.

[0038] The display is used for displaying the spliced video images.

[0039] Compared with the prior art, the present application can achieve at least one of the following beneficial effects:

[0040] 1. The present application uses multiple cameras and places all cameras vertically and uniformly in a 180-degree sector plane, significantly improves the vertical field of view of the image, uses a limited number of cameras for real-time splicing of more than 180 degrees, simplifies the system structure, reduces the hardware cost, and makes the deployment and maintenance easier.

[0041] 2. The present application performs accurate parameter estimation on the multi-camera system in the offline stage and obtains the remapping parameters of each camera, which can avoid repeated calculation in real-time splicing process, significantly improve the splicing efficiency, improve the stability and reliability of the system, and ensure the consistency of the splicing result.

[0042] 3. The present application uses a lightweight convolutional neural network (LET-NET) for feature point detection and matching, improves the accuracy of image registration, at the same time, adopts a spherical projection method to reduce image distortion, effectively improves the quality of video splicing, reduces the splicing gap and visual fault, and improves the overall visual effect of panoramic video.

[0043] In the present application, the above technical solutions can be combined with each other to realize more preferred combination solutions. Other features and advantages of the present application will be described in the subsequent specification, and some advantages will become apparent from the specification or be understood by implementing the present application. The purpose and other advantages of the present application can be realized and obtained through the contents specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings are included to provide a further understanding of the embodiments, and are incorporated in and constitute a part of this specification, illustrate embodiments of the application, and together with the description serve to explain the principles of the application. In the drawings:

[0045] Figure 1 A flowchart of a method for real-time multi-view video splicing in an embodiment of the present application;

[0046] Figure 2 A structural diagram of a multi-view camera connection in an embodiment of the present application. DETAILED DESCRIPTION

[0047] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0048] One specific embodiment of the present invention discloses a method for real-time stitching of multi-view videos, wherein "multi-view" refers to five or more views, such as... Figure 1 As shown, the method includes the following steps S1-S3:

[0049] Step S1: Real-time acquisition of frame images captured by the multi-camera system; wherein the multi-camera system is evenly distributed on a 180-degree arc on the same horizontal plane, and each optical axis is perpendicular to the horizontal plane.

[0050] Specifically, such as Figure 2 As shown, this embodiment uses a six-lens camera for illustration. To achieve over 180-degree panoramic video stitching, the multiple cameras are evenly distributed on a 180-degree fan-shaped structure on the same horizontal plane, forming a fan-shaped structure. This allows the viewing angles of each camera to overlap, covering a viewing range greater than 180 degrees. The included angles between each camera are equal, ensuring uniform viewing angle coverage and providing a better foundation for subsequent image stitching. For example, the included angles between adjacent cameras can be as follows: Figure 2 The angle shown is 180° / 5 = 36°. Since the field of view of a camera is usually larger horizontally than vertically, in this embodiment, each camera is placed vertically, that is, the optical axis of the camera is perpendicular to the horizontal plane, to provide a larger vertical field of view and compensate for the problem of insufficient vertical field of view coverage when the camera is usually placed horizontally. It should be noted that the resolution of each camera in this embodiment is 1920×1080 pixels.

[0051] Step S2: Based on the image remapping parameters of each camera, map the pixels of each frame image to the target coordinate system and generate a seam mask between adjacent images; wherein, the adjacent images are frame images captured by two adjacent cameras.

[0052] Specifically, the image remapping parameters are parameters describing the mapping relationship between image pixels from the original coordinate system (source image) to the target coordinate system (stitched image). They typically include a horizontal mapping matrix xmap and a vertical mapping matrix ymap. Each element in the horizontal mapping matrix corresponds to a pixel in the source image, and its value is the horizontal coordinate value of that pixel in the target image. Similarly, each element in the vertical mapping matrix corresponds to a pixel in the source image, and its value is the vertical coordinate value of that pixel in the target image.

[0053] Furthermore, based on the extrinsic and intrinsic parameters of each camera and the size of the stitched image, a spherical projection model is used to obtain the image remapping parameters for each camera.

[0054] Specifically, in order to accurately map the images captured by different cameras into a unified panoramic coordinate system, i.e. to obtain the image remapping parameters of each camera, the image plane of each camera needs to be mapped onto a virtual sphere using a spherical projection model according to the intrinsic and extrinsic parameters of each camera and the size of the final stitched image, so as to calculate the new position of each pixel in the stitched image and generate the corresponding image remapping parameters.

[0055] For example, the SphericalWarper is used to rewrite the cv: :detail: :RotationWarper: :buildMaps() method of OpenCV, and the spherical projection model is applied therein to generate the image remapping parameters.

[0056] More specifically, in the present embodiment, the positions of the multi-camera are relatively fixed, so the camera parameters and image remapping parameters of each camera are pre-calculated and stored using offline estimation to obtain accurate parameters and reduce the burden of real-time processing. This can quickly and accurately align and fuse video streams from different cameras in the real-time video stitching process, not only improving the stitching efficiency, but also ensuring the stitching quality.

[0057] Further, the camera extrinsic parameters of each camera are obtained, including steps S21-S23:

[0058] Step S21, obtain calibration images captured by each camera at the same time.

[0059] Specifically, a multi-camera is used to capture an image at the same time at its corresponding fixed position as a calibration image.

[0060] Step S22, use the LET-NET model to obtain feature points and their corresponding descriptors of each calibration image and perform feature point matching on each adjacent calibration image to obtain the feature point matching results of each adjacent calibration image; wherein each adjacent calibration image refers to the calibration image corresponding to each adjacent camera.

[0061] Specifically, the LET-NET model is a very lightweight convolutional neural network specially designed for feature point extraction and image consistency calculation, which can extract the coordinates and descriptors of feature points from images and give a feature score; wherein the descriptor of the feature point is a quantitative representation of the local image content around the feature point, which can capture the texture, color and shape information of the image near the point, and is used to match similar feature points in different images; more specifically, the LET-NET model also outputs a score map indicating the reliability of each key point, which is used for subsequent screening of high-reliability key points.

[0062] The LET-NET model adopts a deep learning method to achieve efficient focus detection, and is deployed through the ncnn library specially designed for speed optimization. While maintaining high efficiency, it can effectively process images in complex environments such as changes in lighting, outdoor sunrise, or indoor light switching. The LET-NET model can more accurately track feature points in these environments.

[0063] More specifically, in this embodiment, the LET-NET model is trained using the following method to obtain a trained LET-NET model for obtaining feature points of each calibration image:

[0064] Obtain a public training dataset, such as the MNIST dataset, wherein the training dataset includes diverse scenes to adapt to different lighting conditions, angles, and dynamic changes.

[0065] Preprocess and label the public training data; wherein the preprocessing includes but is not limited to adjusting the image size, rotating, flipping and enhancing the image to increase the diversity of the dataset and improve the generalization ability of the model; labeling each image in the dataset includes identifying and labeling key points in the image for detecting feature points during the training process.

[0066] Divide the preprocessed and labeled dataset into a training dataset, a validation dataset, and a test dataset for training the model, adjusting the parameters, and evaluating the performance of the model.

[0067] Load the training dataset, train the LET-NET model using the mean square error loss function, update the model parameters using gradient backpropagation, and end the training when the loss function value converges to obtain the trained LET-NET model.

[0068] Further, in step S22, the feature point matching of each adjacent calibration image is performed to obtain the feature point matching result of each adjacent calibration image, including steps S221-S223:

[0069] Step S221, based on the descriptors of each feature point in each calibration image, calculate the cosine similarity of each feature point in each adjacent calibration image to obtain the preliminary matching feature point pairs of each adjacent calibration image.

[0070] Specifically, in multi-view video stitching, it is necessary to accurately align and fuse each frame of image captured by adjacent cameras. In order to effectively stitch images, it is necessary to determine the corresponding feature points in the images captured by adjacent cameras, so using feature point matching can help the system understand the position of the same object in different camera images, while quickly narrowing down the matching candidate range and only retaining those most likely matching feature point pairs, laying the foundation for subsequent accurate matching.

[0071] More specifically, the cosine similarity is an index for measuring the degree of similarity between two vectors, the value is between -1 and 1, the value is closer to 1, the more similar, based on the preset similarity threshold, the feature point pairs with similarity higher than the threshold are selected as the preliminary matched feature point pairs, by calculating the cosine similarity, the most similar feature point pairs of the descriptors in the two images can be found, which may be different perspectives of the same physical point. Illustratively, the similarity threshold can be set to 0.7.

[0072] Step S222, based on the distance between the preliminary matched feature point pairs of each adjacent calibration image, the matched feature point pairs of each adjacent calibration are obtained.

[0073] Specifically, according to the distance between the preliminary matched feature point pairs, the point pairs with smaller distance are selected as the final matched point pairs, which can further exclude the mismatch caused by image content or perspective difference, and ensure the accuracy and reliability of the matched point pairs. Illustratively, the Euclidean distance can be used to measure the distance between each preliminary matched feature point pair.

[0074] Step S223, bidirectional consistency detection is performed on the matched feature point pairs of each adjacent calibration image to obtain the feature point matching result of each adjacent calibration image.

[0075] Specifically, the bidirectional consistency detection is a feature point matching strategy, that is, the point pairs matched in one image pair should also be matched in another image pair, which is used to ensure that the matched feature point pairs are effective in both directions, can significantly improve the accuracy and robustness of feature matching, and thus improve the quality of image stitching.

[0076] Step S23, based on the feature point matching result of each adjacent calibration image, the geometric transformation matrix of the corresponding adjacent camera is obtained.

[0077] Specifically, the geometric transformation matrix is a tool for describing the transformation of a pixel point in an image from one coordinate system to another coordinate system. In the multi-view video stitching process, the geometric transformation matrix of each adjacent camera is obtained, which is used to align the images taken by different cameras to a common target coordinate system, thereby performing image stitching.

[0078] Further, the geometric transformation matrix can be a homography matrix or an affine transformation matrix; wherein,

[0079] Based on the feature point matching result of each adjacent calibration image, a random sample consensus algorithm is used to obtain the homography matrix of each adjacent camera;

[0080] Or, based on the feature point matching result of each adjacent calibration image, a least squares method is used to obtain the affine transformation matrix of each adjacent camera.

[0081] Specifically, when the number of matched feature points of each adjacent camera is greater than a preset threshold and is uniformly distributed, the homography matrix can be used to more fully utilize the feature point information, improve the accuracy and robustness of geometric transformation estimation, and thus optimize the quality of image stitching; when the number of feature points is insufficient or the distribution is uneven, the affine transformation matrix can be used to simplify the calculation, improve the robustness, avoid overfitting, and meet the real-time processing requirements. The homography matrix or the affine transformation matrix can be selected as the geometric transformation matrix according to actual conditions.

[0082] More specifically, the homography matrix includes an affine transformation and an additional perspective scaling, and is usually a 3x3 matrix; the affine transformation matrix includes six degrees of freedom, which are rotation, translation, scaling, and shearing, and is usually a 2x3 matrix.

[0083] It should be noted that the random sample consensus algorithm (RANSAC) is used to robustly fit the matched points to obtain the homography matrix. For example, the findHomography function of OpenCV can directly calculate the homography matrix from the matched points.

[0084] The least squares method is used to robustly fit the matched points to obtain the affine transformation matrix. For example, the estimateAffine2D or estimateAffine3D function of OpenCV can directly calculate the affine transformation matrix from the matched points.

[0085] Step S24, a camera is randomly selected as a reference camera, and the camera extrinsic parameters of each camera relative to the reference camera are obtained based on the geometric transformation matrix of each adjacent camera; wherein the extrinsic parameters of the reference camera are set to a unit matrix.

[0086] Specifically, since the geometric transformation matrix of each adjacent camera has been obtained in the foregoing step, one camera in the multi-view camera is randomly selected as a reference camera, and the position and direction of each camera relative to the reference camera can be obtained based on the geometric transformation matrix of each adjacent camera through chain multiplication. More specifically, the extrinsic parameters of the reference camera are set to a unit matrix, i.e., the reference camera is located at the origin and faces forward without rotation and translation.

[0087] Further, the homography matrix decomposition function is used and chain multiplication is used to obtain the camera extrinsic parameters of each camera relative to the reference camera based on the homography matrix of each adjacent camera.

[0088] Alternatively, the camera extrinsic parameters of each camera relative to the reference camera are obtained through chain multiplication based on the affine transformation matrix of each adjacent camera.

[0089] Specifically, the camera extrinsic parameter includes a rotation matrix R and a translation vector t; wherein the homography matrix is a 3x3 matrix, each homography matrix is decomposed into a rotation matrix R and a translation vector t using a homography matrix decomposition function, and the homography matrix decomposition function can be a decomposeHomographyMat function in OpenCV; the affine transformation matrix is a 2x3 matrix, the first two rows of the matrix represent the rotation matrix R, and the third row represents the translation vector t.

[0090] More specifically, the chain multiplication is used to accumulate a plurality of transformation matrices to obtain a total transformation from an initial coordinate system to a final coordinate system.

[0091] For the frame images respectively photographed by the multi-view camera obtained in real time, the image remapping parameters (xmap, ymap) corresponding to each camera are used to map to the target coordinate system of the reference camera for subsequent splicing processing, and preferably, the remap() function of OpenCV is used to perform warp transformation based on the image remapping parameters on the frame images respectively photographed by the multi-view camera obtained in real time

[0092] Further, the generation of the seam mask of each adjacent image includes:

[0093] Based on the positions of each pixel point of each adjacent image in the target coordinate system, an overlapping region between each adjacent image is obtained;

[0094] Gradient information of each of the overlapping regions is calculated, and a path with the smallest gradient is selected as the seam position of each adjacent image;

[0095] Based on the seam position of each adjacent image, a corresponding seam mask is generated; wherein the mask value of the seam region is 1; and the mask value of the non-seam region is 0.

[0096] Specifically, when two images have been aligned, there is an "overlapping region" between the images, which may cause ghosting, sudden change in brightness, and obvious seam line of the images. Therefore, a line is found as a seam, and smoothing fusion is performed near the line, so that the fused image is more natural.

[0097] In the overlapping region between each adjacent image, the gradient amplitude of each pixel point is calculated using Sobel, Scharr, or the like. The greater the gradient, the more intense the color change of the pixel point. A path from one end of the overlapping region to the other end is found, so that the "gradient sum" on the path is the smallest, that is, the area that is least likely to be perceived by the human eye. Exemplarily, a dynamic programming (DP) or graph-cut (Graph-Cut) algorithm can be used to find the minimum cost path.

[0098] More specifically, a strip-shaped region is formed by expanding several pixels to the left and right of the seam line, and in this strip-shaped region, a mask value of 1 is set to indicate that fusion is needed, and the rest of the region is set to 0 to indicate that fusion is not needed.

[0099] In step S3, the images of each frame unified to the target coordinate system and the seam masks are input into a blender to obtain a stitched panoramic image.

[0100] Specifically, the target coordinate system is the coordinate system of the stitched panoramic image, and in this embodiment, the target coordinate system is the coordinate system of the reference camera.

[0101] After the images captured by different cameras are unified to the target coordinate system, the images of different perspectives can be integrated into a continuous and seamless panoramic view.

[0102] For the images of each frame unified to the target coordinate system, due to different shooting conditions (such as exposure, light changes, and different camera settings), the brightness, contrast, and other properties of different images may be different. Such exposure differences will affect the stitching effect, resulting in an uneven image edge and obvious seams. Therefore, before image stitching, exposure error compensation is performed on the images to be stitched to reduce or eliminate these differences, thereby making the stitching result more natural.

[0103] For example, the cv: :detail: :ExposureCompensator exposure error compensation class of OpenCV is used to analyze the brightness, contrast, and other properties of each image through different compensation strategies, and then estimate the exposure parameters that need to be adjusted. When the required adjustment is calculated, the exposure error compensation applies these compensation values to the image to reduce the exposure differences between images.

[0104] Further, the obtained stitched panoramic image includes:

[0105] Based on the corner coordinates and image sizes of the images after exposure error compensation, the blender is initialized.

[0106] Based on the seam masks, the pixel points of the corresponding images after exposure compensation at the seams are fused in the blender to obtain a stitched panoramic image.

[0107] In this embodiment, a multi-band fusion algorithm is used to eliminate stitching gaps and complete seamless stitching. For example, the image blender (Blender) in OpenCV can be used to merge multiple input images to obtain a stitched image.

[0108] Specifically, before starting the stitching, the state of the image blender object is initialized and reset using the prepare() function of the image blender, by accepting the corner point coordinates of the image to be stitched and the size information of the image to be stitched, wherein the corner point coordinates are the upper left corner coordinates of the image to be stitched, used to indicate the position of the image in the stitched image.

[0109] Further, in the image stitching process, a mutex is used to avoid thread conflicts.

[0110] Specifically, since the image stitching process is parallel, multiple threads may access the image blender at the same time. In order to ensure thread safety, std::mutex is used to prevent multiple threads from modifying the state of the image blender at the same time; wherein std::lock_guard <std::mutex>A RAII (Resource Acquisition Is Initialization) mechanism that automatically releases a lock when the scope ends, simplifying lock management.

[0111] More specifically, each exposure compensated image and its corresponding seam mask and position corner points are passed to the image blender using the feed() method. Again, since blender->feed() can be called by multiple threads simultaneously, it needs to be wrapped in std::lock_guard <std::mutex>In this way, it is ensured that no race condition occurs when accessing the image blender each time.

[0112] Finally, when all the images to be stitched are transmitted into the image blender, the blend() method is called to stitch the images, in which all the images to be stitched are blended into a seamless panoramic image according to the seam information calculated in the foregoing manner and output.

[0113] Another embodiment of the present application discloses a multi-view video real-time stitching device, comprising a multi-view camera, an image processor, an image storage and a display.

[0114] The multi-view camera is five or more than five same cameras which are evenly and vertically distributed on a horizontal sector, the adjacent cameras have equal angles, and cover a view angle of more than 180 degrees, and is used for obtaining multi-angle video images on the same plane.

[0115] The image processor is used for receiving frame images respectively shot by the multi-view camera in real time and performing image stitching.

[0116] The image storage is used for storing the frame video images stitched by the image processor.

[0117] The display is used for displaying the frame video images stitched.

[0118] The device embodiment and the method embodiment described above are based on the same principle, can be mutually referenced and can achieve the same technical effects.

[0119] In summary, the multi-view video real-time stitching method and device have the following beneficial effects:

[0120] 1. The present application uses the multi-view camera and vertically and evenly places all the cameras on a 180-degree sector plane, significantly improves the vertical field of view of the image, uses a limited number of cameras to perform real-time stitching of more than 180 degrees, simplifies the system structure, reduces the hardware cost, and makes the deployment and maintenance easier.

[0121] 2. The present application performs accurate parameter estimation on the multi-view camera system in the offline stage and obtains the remapping parameters of each camera, can avoid repeated calculation in the real-time stitching process, significantly improves the stitching efficiency, improves the stability and reliability of the system, and ensures the consistency of the stitching result.

[0122] 3、The scheme uses a lightweight convolutional neural network (LET-NET) for feature point detection and matching, improves the accuracy of image registration, at the same time, adopts a spherical projection method to reduce image distortion, effectively improves the quality of video splicing, reduces splicing gaps and visual faults, and improves the overall visual effect of panoramic video.

[0123] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.< / std::mutex> < / std::mutex>

Claims

1. A method for multi-view video real-time stitching, characterized in that, The multi-view is five or more, and the method comprises the following steps: Real-time acquisition of frame images respectively photographed by a plurality of multi-view cameras; wherein the plurality of multi-view cameras are uniformly distributed on a 180-degree circular arc of a same horizontal plane, and each optical axis is perpendicular to the horizontal plane; Mapping of pixel points of each frame image to a target coordinate system based on image remapping parameters of each camera, and generation of a seam mask between each adjacent image; wherein the adjacent image is a frame image photographed by two adjacent cameras; Input of each frame image unified to the target coordinate system and each seam mask to a fusion device to obtain a spliced panoramic image.

2. The method of claim 1, wherein, Based on camera extrinsic and intrinsic parameters of each camera and a size of the spliced image, image remapping parameters of each camera are obtained using a spherical projection model.

3. The method of claim 2, wherein, Camera extrinsic parameters of each camera are obtained, including: Obtaining calibration images simultaneously photographed by each camera; Obtaining feature points and corresponding descriptors of each calibration image using a LET-NET model, and performing feature point matching on each adjacent calibration image to obtain a feature point matching result of each adjacent calibration image; wherein each adjacent calibration image refers to a calibration image corresponding to each adjacent camera; Based on the feature point matching result of each adjacent calibration image, a geometric transformation matrix of each adjacent camera is obtained; Randomly selecting one camera as a reference camera, and based on the geometric transformation matrix of each adjacent camera, camera extrinsic parameters of each camera relative to the reference camera are obtained; wherein the extrinsic parameters of the reference camera are set as a unit matrix.

4. The method of claim 3, wherein, The feature point matching on each adjacent calibration image to obtain the feature point matching result of each adjacent calibration image comprises: Based on the descriptors of each feature point in each calibration image, a cosine similarity of each feature point in each adjacent calibration image is calculated to obtain a preliminary matching feature point pair of each adjacent calibration image; Based on a distance between the preliminary matching feature point pairs of each adjacent calibration image, a matching feature point pair of each adjacent calibration image is obtained; Performing bidirectional consistency detection on the matching feature point pair of each adjacent calibration image to obtain the feature point matching result of each adjacent calibration image.

5. The method of claim 4, wherein, The geometric transformation matrix is a homography matrix; Based on the feature point matching result of each adjacent calibration image, the random sample consensus algorithm is used to obtain the homography matrix of each adjacent camera; Based on the homography matrix of each adjacent camera, the homography matrix decomposition function is used and the camera extrinsic parameters of each camera relative to the reference camera are obtained through chain multiplication.

6. The method of claim 4, wherein, The geometric transformation matrix can also be an affine transformation matrix; Based on the feature point matching result of each adjacent calibration image, the least squares method is used to obtain the affine transformation matrix of each adjacent camera; Based on the affine transformation matrix of each adjacent camera, the camera extrinsic parameters of each camera relative to the reference camera are obtained through chain multiplication.

7. The method of claim 1 wherein, The generation of the seam mask of each adjacent image comprises: Based on positions of each pixel point of each adjacent image in the target coordinate system, an overlapping area between each adjacent image is obtained; Gradient information of each overlapping area is calculated, and a path with the smallest gradient is selected as a seam position of each adjacent image; Based on the seam position of each adjacent image, a corresponding seam mask is generated; wherein the mask value of the seam area is 1; the mask value of the non-seam area is 0.

8. The method of claim 7, wherein, The obtained spliced panoramic image comprises: Based on the corner point coordinates and image size of each frame image after exposure error compensation, the fuser is initialized; Based on each seam mask, the pixel points of the corresponding frame images after exposure compensation at the seam are fused in the fuser to obtain the spliced panoramic image.

9. The method of claim 8, wherein, In the image splicing process, a mutex is used to avoid thread conflicts.

10. An apparatus for multi-view video real-time stitching, the apparatus comprising: The device comprises: A multi-view camera, an image processor, an image memory and a display; the multi-view refers to five or more views; The multi-view camera is five or more identical cameras uniformly and vertically distributed on a horizontal fan, the included angle between adjacent cameras is equal, and it covers a view angle of more than 180 degrees, which is used to obtain multi-angle video images on the same plane; The image processor is used to receive frame images respectively shot by the multi-view camera in real time and perform image splicing; The image memory is used to store the frame video images spliced by the image processor; The display is used to display the spliced frame video images.

Images