Multi-camera synchronous electronic image stabilization method and device based on virtual camera imaging

Abstract

The invention discloses a multi-camera synchronous electronic image stabilization method and device based on virtual camera imaging, and the method comprises the steps: obtaining and splicing images captured by all cameras in a multi-camera cooperative imaging system, and constructing a virtual camera image; calculating the translation offset of the virtual camera according to the perspective transformation matrix between the adjacent cameras, and calculating the internal reference of the virtual camera; converting the pixel coordinate system into a to-be-stabilized virtual camera coordinate system according to the internal reference of the virtual camera; converting the compensation amount to a virtual camera coordinate system to be stabilized; calculating an image stabilization rotation matrix for converting the coordinate system of the virtual camera to be stabilized into the coordinate system of the stable virtual camera; and converting the to-be-stabilized image from the stable virtual camera coordinate system to the pixel coordinate system according to the image stabilization rotation matrix and the internal reference of the virtual camera to obtain a stabilized image. The method can be applied to multi-camera synchronous electronic image stabilization in a multi-camera collaborative imaging system, and high-quality anti-shake video real-time output is realized.

Description

Technical Field

[0001] The present invention relates to a multi-camera collaborative imaging system, and particularly to a multi-camera synchronous electronic image stabilization method and device based on virtual camera imaging. Background Art

[0002] The viewing angle range of an ordinary camera is limited, and users can only see a partial picture of a certain scene. A multi-camera collaborative imaging system consists of multiple cameras. Each camera independently captures images, and then through image processing algorithms, these images are stitched, fused, etc. to obtain more comprehensive and higher-quality image information. Taking a panoramic camera composed of multiple cameras as an example, the panoramic camera stitches the videos of multiple cameras horizontally or vertically to form a panoramic video with a larger field of view, even a 360-degree panoramic video, enabling users to see video pictures that cannot be captured by ordinary cameras. For panoramic image stitching, in the prior art, usually multiple sensors and cameras with exactly the same specifications and parameters are fixed on the same bracket according to certain rules. The processor simultaneously acquires the original videos of multiple cameras, and through a stitching algorithm, each frame of the multiple original videos is fused into a frame of panoramic image.

[0003] During the use of a camera, jitter will inevitably occur, resulting in unstable video images, and further leading to a serious decline in the recognition accuracy of video intelligent analysis algorithms. Therefore, necessary anti-shake technologies need to be adopted. Regarding camera anti-shake technologies, in the prior art, mechanical anti-shake, optical anti-shake, electronic anti-shake, and digital anti-shake methods are usually used. Among them, mechanical anti-shake stabilizes the camera through a physical structure, such as a stabilizer or a three-axis gimbal. This type of method requires additional hardware devices, with high costs and large volumes, and is not convenient to carry; optical anti-shake eliminates jitter through physical movement inside the lens or sensor, but it also increases the complexity and cost of the device; digital anti-shake reduces the impact of jitter through software algorithms in post-image processing. This type of method relies on algorithms, may cause loss of some image quality, and has limited usage environments. For example, it is only applicable to scenes with rich image features and good lighting conditions, and is not suitable for weak texture scenes; while the electronic anti-shake method uses an IMU (Inertial Measurement Unit) to detect the motion state of the device in real time and compensates for jitter through calculation, which can flexibly adapt to various dynamic environments. However, in the prior art, the electronic anti-shake method is usually for a single camera. Since there are multiple cameras in a multi-camera collaborative imaging system, there will be a fixed positional relationship between each camera. The traditional electronic anti-shake method for a single camera is not applicable to a multi-camera collaborative imaging system for multiple cameras, and compared with single-camera anti-shake, it is more difficult to achieve anti-shake for multiple cameras. Summary of the Invention

[0004] The technical problem to be solved by the present invention lies in: aiming at the technical problems existing in the prior art, the present invention provides a multi-camera synchronous electronic image stabilization method and device based on virtual camera imaging, which has a simple implementation method, low cost, high execution efficiency and wide application range, and can be applied to multi-camera collaborative imaging systems for multi-camera synchronous electronic image stabilization, so as to realize the real-time output of high-quality anti-shake videos.

[0005] To solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A multi-camera synchronous electronic image stabilization method based on virtual camera imaging, the steps include:

[0007] Obtain the images captured by each camera in the multi-camera collaborative imaging system and splice them to construct a virtual camera image; the multi-camera collaborative imaging system includes multiple cameras, and each camera collects images from different perspectives;

[0008] Calculate the translational offset of the virtual camera according to the perspective transformation matrix between adjacent cameras, and calculate the internal parameters of the virtual camera according to the translational offset, and the perspective transformation matrix is calculated according to the geometric position relationship between adjacent cameras;

[0009] Transform the pixel coordinate system to the virtual camera coordinate system to be stabilized according to the internal parameters of the virtual camera;

[0010] Convert the compensation amount calculated according to the gyroscope data to obtain the three-axis rotation angle in the virtual camera coordinate system to be stabilized;

[0011] Calculate the rotation matrix for converting the virtual camera coordinate system to be stabilized to the stable virtual camera coordinate system according to the three-axis rotation angle to obtain the image stabilization rotation matrix;

[0012] Convert all pixel points in the image to be stabilized from the stable virtual camera coordinate system back to the pixel coordinate system according to the image stabilization rotation matrix and the internal parameters of the virtual camera to obtain the stabilized image output.

[0013] Further, there is a view overlap area between every two cameras in the multi-camera collaborative imaging system. When obtaining the images captured by each camera in the multi-camera collaborative imaging system and splicing them, it also includes smoothing the overlap area, and calculating the pixel value corresponding to the virtual camera according to the distance from each pixel point in the overlap area to the boundaries of the left and right camera images in the overlap area and the pixel values of each pixel point in the left and right images.

[0014] Further, the weighted average method is used to smooth the overlap area, and weights are assigned according to the distance of the pixel point p(x, y) in the overlap area from the boundaries of the left and right camera images. After smoothing, the pixel point T in the overlap area virtual_valueThe calculation expression is:

[0015]

[0016] where d l and d r are the distances from the pixel point p(x, y) in the overlapping area to the boundaries of the left matching image and the right matching image respectively; T left_value and T right_value are the pixel values of the pixel point p(x, y) in the left matching image and the right matching image respectively; T virtual_value is the pixel value of the pixel point p(x, y) in the virtual camera imaging image.

[0017] Furthermore, calculating the translation offset of the virtual camera according to the perspective transformation matrix between adjacent cameras, and calculating the internal parameters of the virtual camera according to the translation offset includes:

[0018] Select one camera in the multi-camera collaborative imaging system as the target camera and obtain the internal parameter K select ;

[0019] Calculate the first perspective transformation matrix between the target camera and the leftmost camera and the second perspective transformation matrix between the target camera and the rightmost camera according to the perspective transformation matrix between adjacent cameras;

[0020] Calculate the maximum coordinates and minimum coordinates of the vertices in the images of the leftmost camera and the rightmost camera respectively according to the first perspective transformation matrix and the second perspective transformation matrix;

[0021] Determine the translation offset of the virtual camera according to the maximum coordinates and minimum coordinates of the vertices in the images of the leftmost camera and the rightmost camera, and construct a translation matrix between the camera images according to the translation offset;

[0022] Calculate the internal parameter K select of the virtual camera according to the translation matrix and the internal parameter K virtual of the target camera.

[0023] Furthermore, the first perspective transformation matrix is H L = H Ln …H L2 ·H L1 , and the second perspective transformation matrix is H R = H Rn …H R2 ·H R1 , where H Ln is the perspective matrix between the nth camera on the left side of the target camera and the (n - 1)th camera, H RnThe perspective matrix between the nth camera on the right side of the target camera and the (n - 1)th camera, using the first perspective transformation matrix H L , the second perspective transformation matrix H R transforms the vertex coordinates of the image, and the minimum coordinates of the pixel coordinates of the rightmost camera image are obtained as (x Rmin , y Rmin ), the maximum coordinates are (x Rmax , y Rmax ), the minimum coordinates of the pixel coordinates of the leftmost camera image are (x Lmin , y Lmin ), the maximum coordinates are (x Lmax , y Lmax ). It is determined that the second image width for image transformation and translation is width R = x Rmax , the second image height is height R = y Rmax + |y Rmin |, and the first image height is height L = y Lmax + |y Lmin |, the first image width is width L = |x Lmin | + w, where w is the width of the selected camera image;

[0024] Set the image width of the virtual camera to x Rmax + |x Lmin |, the horizontal translation offset is offsetX = |x Lmin |. If height L > height R , then set the image height of the virtual camera to height L , the vertical offset is offsetY = |y Lmin |, otherwise set the image height of the virtual camera to height R , the vertical translation offset is offsetY = |y Rmin |;

[0025] Construct a translation matrix between camera images according to the translation offset

[0026] Furthermore, the construction steps of the perspective transformation matrix between adjacent cameras include:

[0027] After obtaining the images captured by adjacent cameras and converting them into grayscale images, extract the feature points in the images;

[0028] Perform feature point matching on the extracted feature points to obtain matching feature point pairs;

[0029] Construct a system of linear equations based on the paired matching feature points, and obtain the perspective transformation matrix between the current adjacent cameras after solving.

[0030] Furthermore, it also includes spatio-temporal synchronization calibration for the gyroscope and the target camera in the multi-camera collaborative imaging system. The spatio-temporal synchronization calibration includes time synchronization and coordinate system alignment. The time synchronization includes:

[0031] Use the target camera to capture a video of a fixed scene and synchronously record the gyroscope data;

[0032] Extract the feature points between adjacent video frames and calculate the optical flow trajectories of the feature points;

[0033] Construct the transformation relationship of the feature points between adjacent frames and estimate the change in the rotation angle between frames;

[0034] Compare and analyze the calculated visual rotation information with the angle obtained by integrating the gyroscope angular velocity to find the optimal time offset Δt to align the angle change curves;

[0035] Find the time delay parameter t for the optimal alignment of the visual motion trajectory and the gyroscope integration trajectory delay ;

[0036] Use the found time delay parameter t delay Calibrate the time synchronization deviation between the gyroscope data and the target camera images.

[0037] Furthermore, the step of converting all pixel points in the image to be stabilized from the stabilized virtual camera coordinate system back to the pixel coordinate system according to the image stabilization rotation matrix and the internal parameters of the virtual camera to obtain the stabilized image includes:

[0038] According to the image stabilization rotation matrix R and the internal parameters K of the virtual camera virtual Calculate the motion matrix

[0039] Use the motion matrix P to perform perspective transformation on all pixel points of the image to be stabilized to obtain the stabilized virtual camera image.

[0040] An electronic device includes a processor and a memory. The memory is used to store a computer program, and the processor is used to execute the computer program to perform the method as described above.

[0041] A computer-readable storage medium storing a computer program, where the computer program, when executed by a processor, implements the method as described above.

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] 1. The present invention aims at multi-camera synchronous electronic image stabilization in a multi-camera collaborative imaging system. By constructing a virtual camera imaging model based on multi-camera perspective transformation, it can effectively integrate the perspective information of multiple cameras, construct a unified virtual camera viewpoint, thereby obtaining seamlessly connected and perspective-consistent images. By constructing a virtual camera to obtain a large field of view angle and simultaneously implementing anti-shake for the virtual camera, it can better obtain stable video output and solve the problem of unstable images caused by motion jitter when the virtual camera has a large field of view angle.

[0044] 2. By constructing a virtual camera to achieve multi-camera synchronous electronic image stabilization, the present invention can also simplify the calculation process, reduce the calculation amount, minimize the loss of the original images of multiple cameras, and avoid the influence of the black edges of monocular camera anti-shake on the stitched images while ensuring the image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic flow chart of the implementation of the multi-camera synchronous electronic image stabilization method based on virtual camera imaging in this embodiment.

[0046] Figure 2 is a schematic principle diagram of implementing multi-camera synchronous electronic image stabilization in this embodiment.

[0047] Figure 3 is a schematic principle diagram of coordinate system synchronization in this embodiment.

[0048] Figure 4 is a schematic principle diagram of adjacent camera image acquisition in the multi-camera collaborative imaging system in this embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The present invention will be further described below in conjunction with the accompanying drawings of the specification and specific preferred embodiments, but this does not limit the protection scope of the present invention.

[0050] As disclosed in the present invention, unless the context clearly indicates an exceptional situation, words such as "a", "one", "kind" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "include" or "comprise" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects.

[0051] For ease of understanding, first, the relevant technical background related to the present invention will be introduced by way of example.

[0052] Regarding multi-camera synchronous anti-shake, some existing techniques use multi-camera image stitching and then apply digital anti-shake to the stitched image for image stabilization, that is, first perform multi-camera stitching and then use digital anti-shake. Although this type of method can achieve multi-camera synchronous anti-shake, its real-time performance is poor, and it still relies on image features when using the digital anti-shake method. Therefore, it has high requirements for the applicable environment and is not suitable for weak texture scenes, etc.; there are also some methods where a single camera uses gyroscopic electronic anti-shake and then stitches the images after image stabilization for each camera. However, there will be black edges in the image stabilized by single-camera electronic anti-shake: the anti-shake algorithm will detect the minute displacement of the camera based on gyroscopic data and perform translational compensation. If the camera undergoes displacement, the image will be translated to maintain a stable viewing angle. The translation process may cause a part of the image area to exceed the visible range of the original image, thus exposing blank areas. These blank areas are usually black, forming the so-called "black edges". Therefore, it is necessary to crop the black edges before stitching, which will result in the loss of part of the field of view. Moreover, the size of the black edges and the anti-shake effect are in an opposing relationship, that is, the size of the black edges is opposite to the anti-shake effect. That is, the better the anti-shake effect, the more the image is translated and compensated, exceeding the original image by a larger range, resulting in larger black edges. In order to facilitate stitching, it is necessary to make the black edges of the image stabilized as small as possible, but the corresponding anti-shake effect will deteriorate. Therefore, when panoramically stitching the images after anti-shake of multiple cameras, it will be affected by the black edges of each camera's image.

[0053] In a multi-camera collaborative imaging system, when multiple cameras collect images from different perspectives, they often face the problem of viewpoint differences. Such differences often stem from the relative positions, optical parameters, and imaging conditions of each camera. The present invention aims at multi-camera synchronous electronic image stabilization in a multi-camera collaborative imaging system. By constructing a virtual camera imaging model based on multi-camera perspective transformation, it can effectively integrate the perspective information of multiple cameras, construct a unified virtual camera viewpoint, and thus obtain images with seamless connection and consistent perspectives.

[0054] Specifically, by stitching and fusing the images of multiple cameras, a virtual camera image is constructed. Based on the perspective transformation matrix between adjacent cameras, the internal parameters of the virtual camera are calculated to construct a virtual camera imaging model. Then, using the internal parameters of the virtual camera, the virtual camera image is transformed from the pixel coordinate system to the virtual camera coordinate system to be stabilized. According to the gyroscope data in the multi-camera collaborative imaging system, the attitude of the virtual camera to be stabilized is obtained. Then, by filtering the gyroscope data to calculate the compensation amount, a stabilized image rotation matrix is obtained, and the attitude of the virtual camera to be stabilized is converted into the attitude of the stabilized virtual camera. After transforming back to the pixel coordinate system, the finally stabilized virtual camera image is obtained, realizing anti-shake for the virtual camera. By constructing a virtual camera, a large field of view can be obtained. Moreover, since the resolution of the stitched virtual camera is larger, the pixel movement caused by a slight shake is visually smaller compared to that on a single camera. On this basis, gyroscopic electronic anti-shake can better solve the problem of unstable images caused by motion jitter when the virtual camera has a large field of view, and can realize real-time output of high-quality anti-shake videos for the virtual camera. On the other hand, by constructing a virtual camera to achieve multi-camera synchronous electronic image stabilization, the present invention can also simplify the calculation process, reduce the calculation amount, reduce the loss of the original images of multiple cameras, and avoid the influence of the black edges of monocular camera anti-shake on the stitched images while ensuring the image quality.

[0055] The present invention will be further described below in conjunction with specific embodiments.

[0056] As Figure 1 、 Figure 2 shown, the steps of the multi-camera synchronous electronic image stabilization method based on virtual camera imaging in this embodiment include:

[0057] Step S01. Obtain the images captured by each camera in the multi-camera collaborative imaging system for stitching to construct a virtual camera image. The multi-camera collaborative imaging system includes multiple cameras, and each camera captures images from different perspectives.

[0058] In this embodiment, based on the fixed geometric position relationship of multiple cameras, the perspective transformation matrix between adjacent cameras is calculated. Using the perspective transformation matrix, a unified virtual camera model is established and the internal parameters of the virtual camera are calculated to obtain a larger camera field of view. The virtual camera imaging model is based on the principle of perspective transformation. Through the combination of camera internal parameters and external parameters, and in combination with the mapping relationship between each camera viewpoint and the virtual camera viewpoint, the images captured by multiple cameras are mapped onto the projection plane of the virtual camera.

[0059] After the relative positions of the cameras in the multi-camera collaborative imaging system are fixed, multiple cameras capture images respectively. There is an overlapping area of viewpoints between every two cameras. The overlapping part between two cameras can be determined through the image markers of every two cameras. The minimum value of the overlapping parts between each pair of cameras is taken as the overlapping area. Assuming the pixel width is d, as Figure 3As shown in the figure. In order to generate a seamlessly stitched virtual camera image, when this embodiment obtains the images captured by each camera in the multi-camera collaborative imaging system for stitching, it also includes smoothing the overlapping area to achieve a smooth transition, eliminating the illumination differences and brightness inconsistencies between different viewpoints, and thus achieving efficient image stitching and fusion. To achieve smoothing, specifically, the pixel value corresponding to the virtual camera can be calculated according to the distance of each pixel point in the overlapping area to the boundaries of the left and right camera images in the overlapping area and the pixel values of each pixel point in the left and right images.

[0060] As an alternative embodiment, a weighted average method can be used to smooth the overlapping area, and weights are assigned according to the distances of the pixel points p(x, y) in the overlapping area to the boundaries of the left and right camera images. For example, the calculation expression of the pixel point T in the overlapping area after smoothing can be expressed as: virtual_value The calculation expression can be expressed as:

[0061]

[0062] where d l and d r are the distances between the pixel point p(x, y) in the overlapping area and the boundaries of the left matching image and the right matching image respectively; T left_value and T right_value are the pixel values of the pixel point p(x, y) in the left matching image and the right matching image respectively; T virtual_value is the pixel value of the pixel point p(x, y) in the virtual camera imaging image. After calculating the T virtual_value of all pixel points in the overlapping area, the virtual camera imaging image can be obtained.

[0063] In this embodiment, by using the weighted average value to smooth the seam in the overlapping area and assigning weights according to the distances of the pixel point p(x, y) to the boundaries of the left and right camera images, a smooth transition is achieved, the illumination differences and brightness inconsistencies between different viewpoints are eliminated, and since the calculation only involves linear interpolation, the implementation is simple, the processing efficiency can be improved, and it is especially suitable for real-time processing scenarios.

[0064] Considering that in a multi-sensor fusion system, the sampling frequency of the gyroscope of the inertial measurement unit is usually in the microsecond level, while the frame rate of the video acquisition device is generally in the millisecond level, and the sampling frequencies of the two do not match, resulting in a non-negligible delay deviation between the timestamps. This timing error may cause the angular velocity information corresponding to the camera frame to be inaccurate, thus affecting the video stabilization based on the gyroscope. Further, as Figure 2As shown, after collecting gyroscope data, this embodiment further includes performing time synchronization and coordinate system synchronization (aligning the coordinate system) based on the camera motion trajectory and the angular velocity data of the gyroscope, that is, performing spatio-temporal synchronization calibration on the gyroscope and the target camera in the multi-camera collaborative imaging system.

[0065] As an alternative implementation, the time synchronization can be performed using the following steps:

[0066] Step S101. Use the target camera to capture a video of a fixed scene and synchronously record the gyroscope data;

[0067] Step S102. Extract the feature points between adjacent video frames and calculate the optical flow trajectory of the feature points;

[0068] Step S103. Construct the transformation relationship of the feature points between adjacent frames and estimate the change in the rotation angle between frames;

[0069] Step S104. Compare and analyze the calculated visual rotation information with the angle obtained by integrating the gyroscope angular velocity to find the optimal time offset Δt to align the angle change curve;

[0070] Step S105. Find the time delay parameter t at which the visual motion trajectory and the gyroscope integration trajectory are optimally aligned delay ;

[0071] Step S106. Use the found time delay parameter t delay to calibrate the time synchronization deviation between the gyroscope data and the target camera image.

[0072] To more accurately estimate the time delay t between the gyroscope and the camera delay , specifically, the optical flow method can be used to analyze the motion information between video frames, thereby calibrating the time synchronization deviation. Specifically, the following steps can be adopted:

[0073] First, perform data acquisition: Use the target camera to capture a video of a specified duration of a fixed scene and synchronously record the gyroscope data;

[0074] Then, perform feature point matching: Use feature detection algorithms such as Scale-Invariant Feature Transform to extract the feature points between adjacent video frames;

[0075] Next, calculate the optical flow trajectory of the feature points, and algorithms such as Random Sample Consensus can also be used to remove abnormal matching points to improve robustness.

[0076] Then perform motion modeling: Based on the pinhole camera model, etc., construct the transformation relationship of feature points between adjacent frames, estimate the change in the rotation angle between frames, compare and analyze the calculated visual rotation information with the angle obtained by integrating the angular velocity of the gyroscope, and find the optimal time offset Δt to align the angle change curves of the two.

[0077] Finally, perform time offset optimization: Use algorithms such as Least Squares Optimization to find the time delay parameter t for the optimal alignment of the visual motion trajectory and the gyroscope integration trajectory. delay 。

[0078] To accurately align the gyroscope and camera coordinate systems to ensure that data is calculated in the same coordinate system, as an alternative implementation, the following steps can be adopted:

[0079] As Figure 4 shown, given the x, y, and z axis directions of the camera coordinate system and the gyroscope, keeping the x-axis and y-axis of the camera coordinate system parallel and in the same direction as the x-axis and z-axis of the gyroscope coordinate system respectively, then the z-axis of the camera coordinate system is parallel to the y-axis of the gyroscope coordinate system and in the opposite direction, that is:

[0080]

[0081]

[0082] wherein, R I->C is the rotation matrix for transforming the gyroscope coordinate system to the camera coordinate system. Using this rotation matrix R I->C to align the gyroscope and the camera coordinate system.

[0083] Step S02. Calculate the translation offset of the virtual camera according to the perspective transformation matrix between adjacent cameras, calculate the internal parameters of the virtual camera according to the translation offset, and the perspective transformation matrix is calculated according to the geometric position relationship between adjacent cameras; transform the virtual camera image from the pixel coordinate system to the virtual camera coordinate system according to the internal parameters of the virtual camera, and obtain the attitude of the virtual camera according to the gyroscope data in the multi-camera collaborative imaging system.

[0084] To construct the virtual camera model, in this embodiment, the perspective matrix of each camera is first calibrated, and image alignment is achieved through the calibration of the perspective matrix to ensure that images taken from different perspectives can be correctly transformed to the same coordinate system, thereby achieving seamless stitching.

[0085] As an alternative implementation, the perspective transformation matrix between adjacent cameras can be constructed according to the following steps. The steps include:

[0086] Step S201. After acquiring the images captured by adjacent cameras and converting them into grayscale images, extract the feature points in the images;

[0087] Step S202. Perform feature point matching on the extracted feature points to obtain matching feature point pairs;

[0088] Step S203. Construct a system of linear equations based on the matching feature point pairs, and obtain the perspective transformation matrix between the current adjacent cameras after solving.

[0089] Specifically, when calibrating the perspective transformation matrix of two adjacent cameras, images of the two cameras from different perspectives in the same scene can be collected first, and the diversity of different angles and perspectives is ensured; then image preprocessing is performed, converting the collected images into grayscale images to reduce the amount of calculation, and performing Gaussian filtering to remove noise and improve the stability of feature point matching; then key point detection is performed. For example, methods such as Scale-Invariant Feature Transform (SIFT) can be used to extract the feature points in the images to ensure that the feature points are invariant to scale, rotation, and illumination; finally, the K-nearest neighbor matching algorithm or the like is used for feature point matching.

[0090] As an alternative implementation, the detailed steps of using the Scale-Invariant Feature Transform method for key point detection can be:

[0091] Construct a Gaussian pyramid: Perform Gaussian blur and smoothing on the image to generate image pyramids of multiple scales.

[0092] Calculate the Difference of Gaussian pyramid: Obtain the Difference of Gaussian pyramid by subtracting Gaussian blurred images of different scales.

[0093] Extreme value detection: To ensure that local extreme points can be detected in both the two-dimensional plane space and the scale space, search for local extreme points among all sampling points on the central plane of the scale space and 26 pixel points in the 3×3×3 cubic neighborhood centered on the sampling point, and preliminarily screen out potential feature points.

[0094] Key point direction assignment: Assign a direction to each determined feature point, and endow the feature point with rotation invariance by using the gradient direction distribution characteristics of its neighboring pixels.

[0095] Generate SIFT descriptors: Based on the gradient information around the feature points, construct a 128-dimensional SIFT descriptor vector.

[0096] As an alternative implementation, the detailed steps of using the K-nearest neighbor matching algorithm for feature point matching are:

[0097] Calculate the Euclidean distance between feature points and find the K nearest neighbors of each feature point.

[0098] Use the ratio test to filter the matching points so that only the matching points with the ratio of the distance between the nearest neighbor and the second nearest neighbor less than the set threshold are retained.

[0099] Use Random Sample Consensus (RANSAC) to further eliminate the wrong matching points to improve the robustness of the matching.

[0100] To calculate the perspective transformation matrix H T , after obtaining a sufficient number of matching points, the Direct Linear Transform (DLT) method can be used. By constructing a system of linear equations and then using singular value decomposition to calculate the perspective matrix.

[0101] For example, given 4 or more pairs of feature matching points: (x i , y i ) <-> (x i ′, y i ′), the corresponding relationship is:

[0102]

[0103] Expand and construct the system of linear equations as:

[0104]

[0105] And then organize it into matrix form as:

[0106]

[0107] For n pairs of matching points, the following matrix can be constructed:

[0108] A·h = 0 (7)

[0109]

[0110] where A is a 2n×9 matrix and h is a 9-dimensional column vector. Further calculate the singular value decomposition of matrix A:

[0111] A = U∑V T (9)

[0112] Take the right singular vector V corresponding to the smallest singular value min , that is, the last column of A, and this vector is the expanded form of the perspective transformation matrix H T .

[0113] The internal parameters of the virtual camera affect the internal optical characteristics of the image projection process, including the focal length, the position of the principal point, etc. In this embodiment, a target camera is first selected, and then the internal parameters of the virtual camera are determined based on the internal parameters of the target camera and the translation matrix between the cameras. The translation matrix can describe the relative displacement relationship between the images of the cameras.

[0114] As an alternative implementation, in this embodiment, the translation offset of the virtual camera is calculated according to the perspective transformation matrix between adjacent cameras. The steps for calculating the internal parameters of the virtual camera according to the translation offset can be as follows:

[0115] Step S211. Select a camera in the multi-camera collaborative imaging system as the target camera and obtain the internal parameter K of the target camera select ;

[0116] Step S212. Calculate the first perspective transformation matrix between the target camera and the leftmost camera and the second perspective transformation matrix between the target camera and the rightmost camera according to the perspective transformation matrix between adjacent cameras;

[0117] Step S213. Calculate the maximum coordinates and minimum coordinates of the vertices in the images of the leftmost camera and the rightmost camera respectively according to the first perspective transformation matrix and the second perspective transformation matrix;

[0118] Step S214. Determine the translation offset of the virtual camera according to the maximum coordinates and minimum coordinates of the vertices in the images of the leftmost camera and the rightmost camera, and construct the translation matrix between the camera images according to the translation offset;

[0119] Step S215. Calculate the internal parameter K of the virtual camera according to the translation matrix and the internal parameter K of the target camera select to obtain the internal parameter K of the virtual camera virtual .

[0120] Specifically, when constructing a virtual camera by multi-camera stitching and fusion, one of the cameras is selected as the target camera, and the internal parameter K can be calibrated by methods such as the checkerboard method select . The first camera on the left of the target camera determines the perspective matrix H between two cameras through the above perspective transformation calibration method L1 , the perspective matrix between the second camera on the left and the first camera is H L2 , the perspective matrix between the nth camera on the left and the (n - 1)th camera is H Ln , then the perspective matrix H between the leftmost camera and the selected camera L = H Ln …H L2 ·H L1 ; Similarly, the first camera on the right determines the perspective matrix H between two cameras through the second-step perspective transformation calibration R1, the perspective matrix between the second camera on the right and the first camera is H R2 , the perspective matrix between the nth camera on the right and the (n - 1)th camera is H Rn , then the perspective matrix H R = H Rn …H R2 ·H R1 , that is, the first perspective transformation matrix is obtained as H L = H Ln …H L2 ·H L1 , the second perspective transformation matrix is H R = H Rn …H R2 ·H R1 .

[0121] Then use the first perspective transformation matrix H L , the second perspective transformation matrix H R to transform the vertex coordinates of the image, and the minimum coordinates of the rightmost camera are obtained as (x Rmin , y Rmin ), the maximum coordinates are (x Rmax , y Rmax ), the minimum coordinates of the leftmost camera are (x Lmin , y Lmin ), the maximum coordinates are (x Lmax , y Lmax ). For example, the following steps can be used to achieve this:

[0122] Assume that the width and height of the camera image are w and h; the four vertex coordinates of the image are (0, 0), (0, w), (h, 0), (h, w) respectively. Apply a point p = (x, y) to the perspective matrices H L and H R for transformation:

[0123]

[0124] where (x ∧ , y ∧ ) are the transformed point coordinates, and x ∧ and y ∧ are normalized by z ∧ :

[0125]

[0126] Transform the four vertices (x i , y i ) of the image using the above formula to obtain the transformed coordinates Then calculate the minimum and maximum coordinates after transformation:

[0127]

[0128] That is, the four vertices utilize the perspective matrix H R The minimum coordinates of the rightmost camera are calculated as (x Rmin , y Rmin ), and the maximum coordinates are (x Rmax , y Rmax ). Using the perspective matrix H L The minimum coordinates of the leftmost camera are calculated as (x Lmin , y Lmin ), and the maximum coordinates are (x Lmax , y Lmax ). To ensure that the transformed image does not go beyond the boundaries, the image is translated after transformation. Based on the determined maximum and minimum coordinates, the width, height, and translation offset of the image can be further determined after image transformation and translation.

[0129] Specifically, the second image width for image transformation and translation can be determined as width R = x Rmax , the second image height is height R = y Rmax + |y Rmin |, and the first image height is height L = y Lmax + |y Lmin |, the first image width is width L = |x Lmin | + w. Set the image width of the virtual camera as x Rmax + |x Lmin |, the horizontal translation offset offsetX = |x Lmin |. If height L > height R , then set the image height of the virtual camera as height L , and the vertical offset offsetY = |y Lmin |. Otherwise, set the image height of the virtual camera as height R , and the vertical translation offset offsetY = |y Rmin |.

[0130] Furthermore, the translation matrix between camera images can be constructed based on the translation offset If part of the image falls in the negative coordinate interval, it is translated according to the translation matrix to obtain the entire field of view.

[0131] Finally, the internal parameters of the virtual camera model can be expressed as:

[0132] K virtual = T·K select (13)

[0133] So far, the virtual camera model is constructed. The virtual camera can obtain the stitched image and calculate the internal parameters of the virtual camera and the size of the image resolution.

[0134] Step S03. Transform the pixel coordinate system to the virtual camera coordinate system to be stabilized according to the internal parameters of the virtual camera.

[0135] Specifically, it can be achieved by the inverse of the internal parameter matrix of the virtual camera to transform the pixel coordinate system to the virtual camera coordinate system to be stabilized.

[0136] Step S04. Convert the compensation amount calculated according to the gyroscope data to the three-axis rotation angles in the virtual camera coordinate system to be stabilized.

[0137] Specifically, the Kalman filter can be used to process the gyroscope data to obtain the compensation amount diff(k), and then according to the relationship between the virtual camera coordinate system to be stabilized and the gyroscope coordinate system, the compensation amount diff(k) is converted into the three-axis rotation angles in the virtual camera coordinate system to be stabilized

[0138] As an important sensor in the inertial measurement unit, the gyroscope calculates the rotation information of an object by measuring the angular velocity. In this embodiment, the Euler integration of the gyroscope is used to calculate the cumulative absolute angle, that is, the rotation angle is obtained by multiplying the angular velocity by the time interval:

[0139] θ(t) = ∫ω(t - t delay )dt = ω·Δt (14)

[0140] where θ(t) represents the rotation angle at the current time t, ω(t - t delay ) is the angular velocity output by the gyroscope, t delay is the time delay parameter, and Δt is the gyroscope sampling time interval.

[0141] Considering that the measurement accuracy of the gyroscope is closely related to the sampling frequency, in this embodiment, the gyroscope angle is further interpolated to ensure that its sampling frequency reaches 1 ms, so as to obtain a more refined angle estimate and then calculate the corresponding motion trajectory.

[0142] In the actual motion estimation process, the angular cumulative error and noise of the gyroscope will cause deviations in trajectory estimation. To improve the estimation accuracy and eliminate the influence of noise, in this embodiment, a Kalman filter is further used to smooth the motion trajectory of the camera to remove high-frequency noise and obtain the smoothed data after smoothing processing. The Kalman filter is a recursive filter based on Bayesian estimation theory. It can utilize the process model and observation model of the system, combine the process noise covariance matrix and the observation noise covariance matrix, predict the state at the current moment based on the state estimation value at the previous moment and the motion model, and then combine the observation value of the sensor to recursively update the optimal state estimation value and output the optimal state value.

[0143] Specifically, the input of the Kalman filter is the angular information obtained by gyroscope integration, and the output is the smoothed motion trajectory estimation. By setting appropriate process noise covariance matrix and observation noise covariance matrix, the noise from different sources can be effectively suppressed, and the estimation accuracy and stability can be improved. In this embodiment, the difference between the optimal state value E(k) output by the Kalman filter and the original motion amount z(k) of the camera is calculated to obtain the compensation amount diff(k) of the current frame jitter:

[0144] diff(k) = E(k) - z(k) (15)

[0145] The compensation amount diff(k) corresponds to the compensation in three directions, namely the x direction, the y direction, and the rotation angle. According to the compensation amount diff(k), the current frame image is compensated to the trajectory of the active motion of the camera, thereby outputting a stable video image sequence.

[0146] Furthermore, the compensation amount diff(k) is converted into the rotation angle in the camera coordinate system through the relationship between the camera coordinate system and the gyroscope coordinate system. Specifically, the compensation amount diff(k) can be converted into the rotation angle in the camera coordinate system according to the following formula, where the relationship between the gyroscope coordinate system and the camera coordinate system is known through the space-time synchronization calibration step.

[0147]

[0148] Step S05. Calculate the rotation matrix for converting from the virtual camera coordinate system to be stabilized to the stabilized virtual camera coordinate system according to the three-axis rotation angles, that is, the image stabilization rotation matrix

[0149] Specifically, the rotation matrix can be calculated according to the Rodrigues rotation matrix formula :

[0150]

[0151]

[0152] Step S06. According to the image stabilization rotation matrix and the internal parameter matrix K of the virtual camera virtual , convert all pixel points of the image to be stabilized back to the pixel coordinate system from the stabilized virtual camera coordinate system, and obtain the finally stabilized image.

[0153] Specifically, use the motion matrix P to perform perspective transformation on all pixel points of the image to be stabilized to obtain the stabilized virtual camera image.

[0154] First, the motion matrix can be calculated according to the image stabilization rotation matrix R and the internal parameter K of the virtual camera virtual Calculate the motion matrix For example, the calculation expression of the motion matrix is:

[0155]

[0156] After obtaining the motion matrix , multiply the coordinates (x ij , y ij ) of the pixel by this P matrix to obtain the coordinate value (x i ′ j , y i ′ j ) of the anti-shake pixel, that is By performing the above coordinate position transformation on each pixel point of the virtual camera image, the anti-shake and image stabilization image can be obtained.

[0157] In this embodiment, the motion of the virtual camera is compensated by combining the internal parameters of the virtual camera. By adjusting the geometric transformation of the image, the instability caused by camera jitter and rotation can be eliminated, and the image jitter caused by camera motion can be accurately compensated to achieve a better electronic image stabilization effect.

[0158] Since some image regions will be mapped outside the image after the image stabilization transformation, resulting in undefined pixel values, black edges may appear in the edge region of the image stabilization image, that is, undefined pixel regions. To eliminate these black edges and ensure the quality of the image output, a cropping technique can be further adopted, that is, by removing the undefined region (i.e., the black edge part) in the image through a cropping operation, retaining the valid region, and outputting a stable image sequence. Specifically, the cropping process can be achieved through the following steps:

[0159] ① Calculate the boundary of the transformed image, that is, the minimum rectangular range of the new image.

[0160] ② According to this rectangular range, crop out the valid image region and remove the black region at the edge.

[0161] The cropped image can not only remove the invalid pixel region, but also reduce the distortion phenomenon caused by the transformation while maintaining the image quality.

[0162] In summary, the present invention constructs a virtual camera by stitching and fusing multiple cameras, calculates the internal parameters of the virtual camera using the perspective transformation matrix between the cameras, and then stabilizes the constructed virtual camera, which can efficiently achieve multi-camera synchronous electronic image stabilization in a multi-camera collaborative imaging system, avoid the influence of the anti-shake black edge of a single camera on stitching, and since a linear processing method is adopted, there is no need to approximate the result through iterative or numerical methods, that is, the global optimal solution can be quickly obtained, which can reduce the amount of calculation, improve the calculation efficiency, quickly obtain the result when processing large-scale data, and improve the real-time performance.

[0163] This embodiment further provides an electronic device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to execute the computer program to execute the method as described above.

[0164] It can be understood that the above method of this embodiment can be executed by a single device, such as a computer or a server, etc., or can also be applied to a distributed scenario where multiple devices cooperate with each other to complete. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps of the above method of this embodiment, and the multiple devices interact with each other to complete the above method. The processor can be implemented in ways such as a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., and is used to execute relevant programs to implement the above method of this embodiment. The memory can be implemented in forms such as a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When implementing the above method of this embodiment through software or firmware, the relevant program codes are stored in the memory and are called and executed by the processor.

[0165] This embodiment further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method as described above is implemented.

[0166] Those skilled in the art should understand that the above embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in the process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks

[0167] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Although the present invention has been disclosed above with the preferred embodiments, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change, and modification made to the above embodiments according to the technical essence of the present invention without departing from the technical solution of the present invention shall fall within the scope of the technical solution of the present invention

Claims

1. A multi-camera synchronous electronic image stabilization method based on virtual camera imaging, characterized in that the steps include: Obtain images captured by each camera in a multi-camera collaborative imaging system, stitch them together, and construct a virtual camera image; The multi-camera collaborative imaging system includes multiple cameras, each camera capturing images from a different perspective; Calculating a translation offset of a virtual camera according to a perspective transformation matrix between adjacent cameras, and obtaining an intrinsic parameter of the virtual camera according to the translation offset, wherein the perspective transformation matrix is ​​calculated according to a geometric position relationship between adjacent cameras; Transforming the pixel coordinate system to the virtual camera coordinate system to be stabilized according to the intrinsic parameters of the virtual camera; The compensation amount calculated according to the gyroscope data is converted into the coordinate system of the virtual camera to be stabilized to obtain the three-axis rotation angle; Calculate the rotation matrix for converting the virtual camera coordinate system to be stabilized into the stable virtual camera coordinate system according to the three-axis rotation angles to obtain the image stabilization rotation matrix; According to the image stabilization rotation matrix and the internal parameters of the virtual camera, all pixel points in the image to be stabilized are converted from the stable virtual camera coordinate system back to the pixel coordinate system to obtain a stabilized image output.

2. The multi-camera synchronous electronic image stabilization method based on virtual camera imaging according to claim 1, characterized in that: There is an overlapping viewing angle area between every two cameras in the multi-camera collaborative imaging system. When the images captured by each camera in the multi-camera collaborative imaging system are obtained and stitched together, the overlapping area is also smoothed, and the pixel value corresponding to the virtual camera is calculated according to the distance from each pixel point in the overlapping area to the boundaries of the camera images on the left and right sides of the overlapping area and the pixel value of each pixel point in the overlapping area in the left and right images.

3. The multi-camera synchronous electronic image stabilization method based on virtual camera imaging according to claim 2, characterized in that: The overlapping area is smoothed by weighted averaging. The weight is assigned according to the distance between the pixel point p(x, y) in the overlapping area and the left and right camera image boundaries. After smoothing, the pixel point T in the overlapping area is virtual_value The calculation expression is: Among them, d l and d r are the distances from the pixel point p(x, y) in the overlapped area to the border of the left matching image and the border of the right matching image respectively; T left_value and T right_value are the pixel values ​​of the pixel point p(x, y) in the overlapping area in the left matching image and the right matching image respectively; T virtual_value is the pixel value of the pixel point p(x,y) in the image imaged by the virtual camera.

4. The multi-camera synchronous electronic image stabilization method based on virtual camera imaging according to claim 1, characterized in that: The step of calculating the translation offset of the virtual camera according to the perspective transformation matrix between adjacent cameras and obtaining the internal parameters of the virtual camera according to the translation offset comprises: Select a camera in the multi-camera collaborative imaging system as the target camera and obtain the intrinsic parameter K of the target camera. select ; Calculate a first perspective transformation matrix between the target camera and the leftmost camera and a second perspective transformation matrix between the target camera and the rightmost camera according to the perspective transformation matrix between adjacent cameras; Calculate the maximum coordinates and the minimum coordinates of the vertices in the images of the leftmost camera and the rightmost camera respectively according to the first perspective transformation matrix and the second perspective transformation matrix; Determine the translation offset of the virtual camera according to the maximum coordinates and the minimum coordinates of the vertices in the images of the leftmost camera and the rightmost camera, and construct a translation matrix between the camera images according to the translation offset; According to the translation matrix and the intrinsic parameter K of the target camera select Calculate the intrinsic parameter K of the virtual camera virtual .

5. The multi-camera synchronous electronic image stabilization method based on virtual camera imaging according to claim 4, characterized in that: The first perspective transformation matrix is ​​H L =H Ln …H L2 ·H L1 , the second perspective transformation matrix is ​​H R =H Rn …H R2 ·H R1 , where H Ln H is the perspective matrix of the nth camera to the left of the target camera and the n-1th camera. Rn is the perspective matrix of the nth camera and the n-1th camera to the right of the target camera, using the first perspective transformation matrix H L , the second perspective transformation matrix H R Transform the vertex coordinates of the image to obtain the minimum coordinate of the pixel coordinates of the rightmost camera image (x Rmin ,y Rmin ), the maximum coordinate is (x Rmax ,y Rmax ), the minimum coordinate of the leftmost camera image pixel coordinate is (x Lmin ,y Lmin ), the maximum coordinate is (x Lmax ,y Lmax ), determine that the width of the second image to be transformed and translated is width R =x Rmax , the second image height is height R =y Rmax +|y Rmin |, and the first image height is height L =y Lmax +|y Lmin |, the first image width is width L =|x Lmin |+w, where w is the width of the selected camera image; Set the image width of the virtual camera to x Rmax +|x Lmin |, horizontal translation offset offsetX = |x Lmin |, if height L >height R , then set the virtual camera's image height to height L , vertical offset offsetY=|y Lmin |, otherwise set the virtual camera's image height to height R , vertical translation offset offsetY=|y Rmin |; Construct the translation matrix between camera images based on the translation offset 6. The multi-camera synchronous electronic image stabilization method based on virtual camera imaging according to any one of claims 1 to 5, characterized in that: The step of constructing the perspective transformation matrix between adjacent cameras includes: After obtaining the image captured by the adjacent camera and converting it into a grayscale image, feature points in the image are extracted; Perform feature point matching on the extracted feature points to obtain matching feature point pairs; A linear equation group is constructed according to the matching feature point pairs, and the perspective transformation matrix between the current adjacent cameras is obtained after solving the linear equation group.

7. The multi-camera synchronous electronic image stabilization method based on virtual camera imaging according to any one of claims 1 to 5, characterized in that: The method also includes performing spatiotemporal synchronization calibration on the gyroscope and the target camera in the multi-camera collaborative imaging system, wherein the spatiotemporal synchronization calibration includes time synchronization and alignment of the coordinate system, wherein the time synchronization includes: Use the target camera to shoot a video of a fixed scene and simultaneously record the gyroscope data; Extract feature points between adjacent video frames and calculate the optical flow trajectory of feature points; Construct the transformation relationship between feature points in adjacent frames and estimate the change in rotation angle between frames; Compare and analyze the calculated visual rotation information with the angle obtained by integrating the gyroscope angular velocity, and find the optimal time offset Δt to align the angle change curves; Find the time delay parameter t for optimal alignment of visual motion trajectory and gyroscope integrated trajectory delay ; Using the found time delay parameter t delay Calibrate the time synchronization deviation between the gyroscope data and the target camera image.

8. The multi-camera synchronous electronic image stabilization method based on virtual camera imaging according to any one of claims 1 to 5, characterized in that: The step of converting all pixel points in the image to be stabilized from the stable virtual camera coordinate system back to the pixel coordinate system according to the image stabilization rotation matrix and the internal parameters of the virtual camera to obtain a stabilized image includes: According to the stabilization rotation matrix R and the virtual camera internal parameter K virtual Calculate the motion matrix The motion matrix P is used to perform perspective transformation on all pixels of the image to be stabilized to obtain a stabilized virtual camera image.

9. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images