Multi-camera cooperative anti-shake control system and method

Abstract

The present application belongs to the technical field of multi-camera cooperative imaging, and discloses a multi-camera cooperative anti-shake control system and method; the method comprises the following steps: generating a global hardware trigger signal by FPGA, synchronizing IMU and multi-camera acquisition, and obtaining multi-view images with space-time alignment in combination with motion parameters; then obtaining global shaking components through feature extraction, motion vector field generation, depth map analysis, etc.; fusing the global shaking components with IMU data to generate a trajectory, inputting the trajectory into LSTM to obtain a future compensation amount; based on depth layering transformation and cropping, an EIS optimized frame is obtained, a main camera is dynamically elected for super-resolution reconstruction, and finally an output of the final anti-shake video stream is obtained. Through multi-source information fusion, layered accurate compensation and dynamic optimization, the present application significantly improves the accuracy and stability of motion estimation, and the finally output video stream effectively weakens the influence of shaking, ensuring the reliability and visual continuity of the image stabilization effect in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of multi-camera collaborative imaging technology, and more specifically, to a multi-camera collaborative image stabilization control system and method. Background Technology

[0002] In the fields of AR and machine vision, multi-camera collaboration has become a core technology for improving system perception capabilities. However, the instability of image sequences caused by device jitter remains a key bottleneck restricting application experience and accuracy. For example, AR glasses construct stereoscopic vision through dual forward-looking cameras and use SLAM algorithms to locate virtual objects in real time. If the cameras jitter due to slight hand movements or head rotations, virtual objects may drift, ghost, or become misaligned with the real scene, severely damaging the immersive experience. In mobile AR applications, dynamic scenes captured by multiple cameras can experience frequent frame shifts due to jitter, causing virtual icons to jump around and reducing interaction reliability. Existing single-camera stabilization technologies struggle to cope with synchronous jitter caused by differences in the field of view of multiple cameras, while traditional multi-camera collaborative solutions lag behind in compensating for jitter in fast-moving and complex textures. Virtual registration errors often exceed the 5° viewing angle deviation threshold, failing to meet AR's requirements for sub-pixel level stability.

[0003] Chinese patent application CN120201303A discloses a method and device for multi-camera synchronous electronic image stabilization based on virtual camera imaging. The method involves: acquiring images captured by each camera in a multi-camera collaborative imaging system and stitching them together to construct a virtual camera image; the multi-camera collaborative imaging system includes multiple cameras, each acquiring images from different perspectives; calculating the translational offset of the virtual camera based on the perspective transformation matrix between adjacent cameras, and calculating the intrinsic parameters of the virtual camera based on the translational offset; the perspective transformation matrix being calculated based on the geometric positional relationship between adjacent cameras; transforming the pixel coordinate system to the coordinate system of the virtual camera to be stabilized based on the intrinsic parameters of the virtual camera; converting the compensation amount calculated based on gyroscope data to the coordinate system of the virtual camera to be stabilized to obtain a three-axis rotation angle; calculating the rotation matrix to transform the coordinate system of the virtual camera to be stabilized to the stable virtual camera coordinate system based on the three-axis rotation angle, obtaining a stabilized rotation matrix; and converting all pixels in the image to be stabilized from the stable virtual camera coordinate system back to the pixel coordinate system based on the stabilized rotation matrix and the intrinsic parameters of the virtual camera, resulting in a stabilized image output. This invention addresses the issue of synchronous electronic image stabilization for multiple cameras in multi-camera collaborative imaging systems. By constructing a virtual camera imaging model based on multi-camera perspective transformation, it effectively integrates the perspective information of multiple cameras to build a unified virtual camera viewpoint, thereby obtaining seamless and consistent images. By constructing a virtual camera to obtain a large field of view, and simultaneously achieving image stabilization for the virtual camera, it can obtain more stable video output and solve the problem of image instability caused by motion shake when the virtual camera has a large field of view.

[0004] While the above methods can meet the needs of most scenarios, research and practical application of these methods and existing technologies have revealed at least the following shortcomings:

[0005] The methods described above estimate motion based on image feature point matching. However, in complex scenes, relying solely on image feature point matching can significantly impact the accuracy of motion estimation when feature points are occluded or interfered with by similar textures. For example, in scenes with abundant greenery, similar leaf textures may lead to mismatched feature points, thus affecting the image stabilization effect.

[0006] In view of this, the present invention proposes a multi-camera collaborative image stabilization control system and method to solve the above problems. Summary of the Invention

[0007] To overcome the aforementioned deficiencies of the prior art, the present invention provides the following technical solution: a multi-camera collaborative image stabilization control system and method, comprising the following steps:

[0008] Based on FPGA, a global hardware trigger signal is generated to synchronously trigger IMU to acquire 6-DOF motion parameters and multiple cameras to acquire raw images. The motion parameters are then processed to obtain spatiotemporally aligned multi-view images.

[0009] Feature extraction is performed on multi-view images and motion parameters to obtain fused features, and a motion vector field is generated based on the fused features. The depth map is obtained by analyzing the multi-view images, and then the global jitter component is obtained by further analysis.

[0010] The global motion trajectory is generated by fusing global jitter components and real-time IMU data. The historical trajectory and motion parameters of the global motion trajectory are used as input to the LSTM network to obtain the future jitter compensation amount.

[0011] Based on the depth map and future jitter compensation, corresponding transformations are performed according to depth layers and dynamic cropping is carried out to obtain the EIS optimized frame sequence and cropping region.

[0012] Based on the camera status and EIS optimized frame sequence, the main camera is dynamically selected and multi-camera super-resolution reconstruction is performed on the cropped area to output the final stabilized video stream.

[0013] Furthermore, methods for obtaining spatiotemporally aligned multi-view images include:

[0014] The actual exposure start time of the i-th camera is calculated by adding the time when the trigger signal is emitted by the FPGA and the propagation delay of the trigger signal from the FPGA to the i-th camera; wherein, the propagation delay of the trigger signal from the FPGA to the i-th camera is determined by static timing analysis or online calibration.

[0015] At the actual exposure start time, each camera generates a 16-bit timestamp based on the FPGA clock frequency, embeds it into the original image frame header, and performs periodic calibration to obtain the original image with the calibrated timestamp.

[0016] The intrinsic and extrinsic parameter matrices of each camera are obtained using the Zhang Zhengyou calibration method, and the perspective transformation matrix between adjacent cameras is calculated based on the intrinsic and extrinsic parameter matrices.

[0017] The main camera C0 is selected as the reference coordinate system. The original image with calibrated timestamp from the i-th camera is mapped to the virtual coordinate system by combining the perspective transformation matrix to obtain the image frame.

[0018] By comparing the timestamps of image frames from each camera, the motion parameters corresponding to the timestamps of the original images with calibrated timestamps are extracted. The time difference is calculated based on the difference in timestamps and the FPGA clock frequency. Motion interpolation is then performed on the lagging frames in combination with the rotation matrix in the motion parameters based on the time difference to obtain spatiotemporally aligned multi-view images.

[0019] Furthermore, methods for periodic calibration include:

[0020] The FPGA sends synchronization messages to all cameras. The cameras record the reception time and send back a delay request. The FPGA combines the reception time and round-trip delay to calculate the clock offset of each camera and performs calibration based on the clock offset of each camera.

[0021] Furthermore, methods for obtaining fusion features include:

[0022] SIFT feature points of the r-th multi-view image are extracted using scale-space extremum detection:

[0023] The r-th multi-view image is used as input to the CNN network; SP features containing B-dimensional normalized descriptors and corresponding confidence scores are obtained.

[0024] For each point in the SIFT features, search for points in the SP features whose spatial distance is less than a distance threshold. The spatial distance is calculated using Euclidean distance to obtain the set of matching point pairs, the set of unmatched SIFT points, and the set of unmatched SP points.

[0025] The SIFT and SP features in the matching point pair set are weighted to obtain the fusion descriptor. The weighting coefficients are obtained by optimization through a natural heuristic optimization algorithm. In the SIFT features, the pixel value of the difference Gaussian image of the feature point at scale σ is used as the response value. The maximum value of the response value and the confidence score is taken as the reliability score of the fusion point.

[0026] For feature points in the SIFT unmatched point set, retain those with response values ​​not lower than the response threshold and directly include them in the fused matching point set;

[0027] For the feature points in the unmatched point set of SP, retain the points with a confidence level not lower than the confidence level and directly incorporate them into the fused matching point set;

[0028] If the number of feature points in the G*H neighborhood of the remaining points is greater than J, only retain the top K points with the highest reliability scores, where K < J, to obtain the fused features.

[0029] Furthermore, the method for obtaining the motion vector field includes:

[0030] For the fused feature points of the multi-view image in the t-th frame, search for matching points in the (t + 1)-th frame by calculating the descriptor cosine similarity, retain the qualified point pairs with a descriptor cosine similarity greater than the similarity threshold, and calculate the vectors formed by the qualified point pairs to obtain the sparse motion vectors;

[0031] Construct an X-layer image pyramid. Starting from the top layer, for each pixel in the D×D window, calculate the initial motion vector by least squares fitting the optical flow equation; constrain the optical flow vector of the feature points in the sparse motion vectors to be equal to the initial motion vector, and correct the initial motion vector according to the weights of the feature points and the optical flow vectors to obtain the initial dense motion vector field; among them, the weights of the feature points are weighted according to the Gaussian distance;

[0032] Map the initial dense vector fields of each camera to the main coordinate system through the perspective transformation matrix, and perform weighted fusion according to the reliability scores of the feature points to obtain the motion vector field.

[0033] Furthermore, the method for obtaining the depth map includes:

[0034] For the spatio-temporally aligned multi-view images, use the Census transform in the F×F window to construct the matching cost according to each pixel and the candidate disparity;

[0035] Accumulate the path costs along L path directions and select the d value corresponding to the minimum path cost as the final disparity;

[0036] Perform a left-right consistency check on the initial disparity, calculate the difference between the left-right disparity and the right-left disparity to obtain the disparity deviation, mark the pixels with a disparity deviation greater than the deviation threshold as invalid, bilinearly interpolate the known disparity of the fused feature points to fill the invalid regions, and then obtain the optimized disparity through quadratic curve fitting;

[0037] Based on the focal length and baseline distance in the camera intrinsics, convert the optimized disparity to depth, and take the average of the results of multiple cameras to obtain the depth map.

[0038] Furthermore, the method for obtaining the global jitter component includes:

[0039] Step 1, calculate the depth gradient of the depth map and construct a disparity weight matrix based on the depth gradient;

[0040] Step 2: Construct a global motion vector field based on the motion vectors. Combined with the disparity weight matrix, construct an objective function with the goal of minimizing the error between the global motion vector field and the global jitter vector. Solve the global jitter vector by iterative reweighted least squares.

[0041] Step 3: Repeat steps 1-2 for consecutive image frames to obtain single-frame jitter vectors. After filtering using the Kalman filter algorithm, the vectors are converted into time-continuous global jitter components by combining the scaling factor. The scaling factor is calculated based on the sensor size and the number of pixels to obtain the global jitter components.

[0042] Furthermore, methods for obtaining EIS optimized frame sequences include:

[0043] Based on the depth map and a preset segmentation threshold, the image is divided into foreground and background, and a layered mask marking the foreground and background regions is obtained through threshold segmentation.

[0044] The foreground layer performs a rigid transformation based on the future jitter compensation amount, i.e., rotation and translation; the background layer performs a perspective affine transformation based on the global motion trajectory, and outputs the transformed image.

[0045] A safe region is generated based on the depth map and pixel sharpness, and pixel sharpness is obtained using the Laplacian operator.

[0046] The minimum bounding rectangle of the safe region is obtained by cropping. The minimum bounding rectangle is expanded based on the predicted jitter range to obtain the cropping window. The image is cropped to the target resolution based on the cropping window, and the cropped image is output.

[0047] The cropped region is repaired with bilinear interpolation, the edges are smoothed with guided filtering, the cropping windows of consecutive frames are weighted and smoothed, and the output is an EIS-optimized frame sequence.

[0048] Furthermore, methods for obtaining the final stabilized video stream include:

[0049] Based on the status of each camera, the camera with the highest total score is selected as the master camera through weighted scoring, and a master-slave scheduling command is output.

[0050] The main camera provides the minimum bounding rectangle as the cropping area, and the secondary camera calculates the corresponding area through the perspective transformation matrix. For overlapping pixels, the pixels of the main camera and the pixels of the secondary camera are weighted to obtain the reconstructed area.

[0051] The reconstructed region is embedded into the corresponding position of the EIS optimized frame sequence, and inter-frame color correction and smoothing are performed to output the final stabilized video stream.

[0052] A multi-camera collaborative image stabilization control system, implementing the multi-camera collaborative image stabilization control method, includes:

[0053] Image alignment module: Based on FPGA, a global hardware trigger signal is generated to synchronously trigger the IMU to acquire 6-DOF motion parameters and multiple cameras to acquire raw images. The motion parameters are then processed to obtain spatiotemporally aligned multi-view images.

[0054] Global Analysis Module: Extracts features from multi-view images and motion parameters to obtain fused features, generates motion vector fields based on fused features, analyzes multi-view images to obtain depth maps, and then analyzes them to obtain global jitter components;

[0055] Jitter prediction module: It integrates global jitter components with IMU real-time data to generate a global motion trajectory, and uses the historical trajectory and motion parameters of the global motion trajectory as input to the LSTM network to obtain the future jitter compensation amount;

[0056] Optimized cropping module: Based on the depth map and future jitter compensation, it performs corresponding transformations and dynamic cropping according to depth layers to obtain EIS optimized frame sequences and cropping regions;

[0057] Image stabilization and reconstruction module: Based on the camera status and EIS optimized frame sequence, dynamically select the main camera and perform multi-camera super-resolution reconstruction on the cropped area to output the final image stabilization video stream.

[0058] The technical effects and advantages of the multi-camera collaborative image stabilization control system and method of this invention are as follows:

[0059] This invention lays a data foundation by aligning multi-view images in a spatiotemporal manner, enhances the robustness of feature matching in complex scenes by fusing features, obtains a precise motion vector field to capture pixel-level motion, combines depth maps to achieve near-far layered processing, separates global jitter components and predicts compensation amounts, generates an EIS-optimized frame sequence, and finally outputs a stabilized video stream. In complex scenes where feature points are easily occluded and similar textures interfere, it can effectively reduce miscompensation caused by defects in single feature matching or motion estimation bias. Through multi-source information fusion, layered precise compensation, and dynamic optimization, it significantly improves the accuracy and stability of motion estimation. The final output video stream effectively weakens the impact of jitter and ensures the reliability and visual coherence of the image stabilization effect in complex scenes. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the multi-camera collaborative image stabilization control method of the present invention;

[0061] Figure 2 This is a schematic diagram of the data flow in this invention;

[0062] Figure 3 This is a schematic diagram of the method for obtaining fusion features according to the present invention;

[0063] Figure 4 This is a schematic diagram of the multi-camera collaborative image stabilization control system of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Example 1:

[0066] Please see Figure 1 , Figure 2 As shown, this embodiment provides a multi-camera collaborative image stabilization control method, including the following steps:

[0067] Based on FPGA, a global hardware trigger signal is generated to synchronously trigger IMU to acquire 6-DOF motion parameters and multiple cameras to acquire raw images. The motion parameters are then processed to obtain spatiotemporally aligned multi-view images.

[0068] Methods for obtaining spatiotemporally aligned multi-view images include:

[0069] The actual exposure start time of the i-th camera is calculated by adding the time when the trigger signal is emitted by the FPGA and the propagation delay of the trigger signal from the FPGA to the i-th camera; wherein, the propagation delay of the trigger signal from the FPGA to the i-th camera is determined by static timing analysis or online calibration.

[0070] At the actual exposure start time, each camera generates a 16-bit timestamp based on the FPGA clock frequency, embeds it into the original image frame header (e.g., the 3rd to 6th bytes of the frame header), and performs periodic calibration to obtain the original image with the calibrated timestamp.

[0071] Methods for periodic calibration include:

[0072] The FPGA sends synchronization messages to all cameras. Each camera records its reception time and sends back a delay request. The FPGA then calculates the clock offset for each camera based on the reception time and round-trip delay, such as the clock offset Offset for the i-th camera. i =T tx -T rx +Delay i / 2, where T tx The timestamp for sending synchronization messages to the FPGA; T rx The timestamp for receiving synchronization messages for the camera; Delay iFor round-trip delay; calibrated according to the clock offset of each camera.

[0073] The intrinsic parameter matrix K and extrinsic parameter matrix [R|t] of each camera are obtained using Zhang Zhengyou's calibration method, where R is the extrinsic rotation matrix and t is the extrinsic translation vector. The perspective transformation matrix between adjacent cameras is calculated based on the intrinsic and extrinsic parameter matrices, such as the perspective transformation matrix between the i-th camera and the j-th camera. Among them, K i R is the intrinsic parameter matrix of the i-th camera; i Let t be the extrinsic rotation matrix of the i-th camera; i Let be the extrinsic translation vector of the i-th camera; R is the extrinsic rotation matrix of the j-th camera. j The inverse matrix; t j Let be the extrinsic translation vector of the j-th camera; Let K be the intrinsic parameter matrix of the j-th camera. j The inverse matrix;

[0074] The main camera C0 is selected as the reference coordinate system. The original image with calibrated timestamp from the i-th camera is mapped to the virtual coordinate system by combining the perspective transformation matrix to obtain the image frame.

[0075] By comparing the timestamps of image frames from each camera, the motion parameters corresponding to the timestamps of the original images with calibrated timestamps are extracted. The time difference is calculated based on the timestamp difference and the FPGA clock frequency. Motion interpolation is then performed on the lagging frames using the rotation matrix from the motion parameters based on this time difference. If the timestamp difference between camera i and camera j is greater than a preset time difference threshold (e.g., 0.5ms), the compensated lagging frame I... comp (x,y)=I lag (R imu ·(x,y) T +T imu ), where I lag For the delayed frame before compensation; R imu The selection matrix is ​​(x, y); (x, y) represents the pixel; T is the transpose of the vector; T imu Use the translation vector to obtain spatiotemporally aligned multi-view images.

[0076] The aforementioned method precisely controls the actual exposure start time of each camera and embeds a calibrated timestamp. It then performs spatial mapping using the intrinsic and extrinsic parameter matrices obtained through Zhang Zhengyou's calibration method. Based on timestamp differences and motion parameter compensation, it ensures consistency across multiple viewpoints from both temporal and spatial dimensions, providing a more reliable data foundation for subsequent motion estimation. In complex scenes, it effectively reduces the risk of mismatches caused by feature point occlusion and interference from similar textures, improving motion estimation accuracy and thus optimizing image stabilization. This avoids the impact of feature point mismatches on image stabilization performance, as seen in scenes with abundant greenery.

[0077] Feature extraction is performed on multi-view images and motion parameters to obtain fused features, and a motion vector field is generated based on the fused features. The global jitter component and local scene motion are decoupled by depth-disparity mapping to obtain the global jitter component and depth map.

[0078] Reference Figure 3 Methods for obtaining fusion features include:

[0079] SIFT feature points of the r-th multi-view image are extracted using scale-space extremum detection to obtain an A-dimensional descriptor:

[0080] Methods for obtaining A-dimensional descriptors include:

[0081] Construct a Gaussian pyramid with a scale of σ, and combine it with a Gaussian function to calculate the difference Gaussian image of the multi-view image based on σ; for example, the scale of the s-th layer in the 0-th group is σ(o,s)=σ0·2 α+s / s σ0 is the initial scale; the difference Gaussian image D(o,s,x,y) = G(x,y,σ(o,s+1)) - G(x,y,σ(o,s)), where o is the number of groups; s is the number of layers per group; G(x,y,σ) is the Gaussian function.

[0082] Taylor expansion fitting is performed on the extreme points within the A*B*C neighborhood of the difference Gaussian image, as shown in the Taylor expansion fitting result. Where D is the current extreme value; It is the first derivative; is the feature point offset; H is the Hessian matrix; and extreme point filtering is performed to obtain feature points; where A is the number of scale layers contained in the neighborhood, and B*C is the spatial dimension of the current pixel;

[0083] Methods for obtaining feature points include: obtaining the Taylor expansion fitting result, and setting the first derivative of the Taylor expansion fitting result... If the value is 0, the precise offset is obtained. Substituting the results into the Taylor expansion fit yields the extreme values. Remove extreme points with contrast values ​​below a preset threshold, such as 0.04. 0.04 is an empirical value that can be adjusted according to actual conditions.

[0084] Obtain the Hessian matrix from the Taylor expansion fitting result. Calculate the determinant of the Hessian matrix from the Hessian matrix. The trace of the Hessian matrix Tr(H) = D xx +D yy The ratio of the determinant of the Hessian matrix to the trace is calculated to remove points whose ratio is less than a preset edge threshold. The preset edge threshold is generally set based on empirical values ​​to obtain feature points.

[0085] In the Gaussian image at the scale of the feature point, calculate the gradient magnitude and gradient direction of all pixels in the neighborhood of the feature point with radius Uσ. Divide the gradient direction into 36 bins in 10° intervals, accumulate the gradient magnitude of the pixels in the neighborhood to obtain the gradient direction histogram, and take the direction corresponding to the peak value in the gradient direction histogram as the main direction.

[0086] The 16×16 region surrounding the feature point is rotated along the main direction and divided into 4×4 sub-regions. An 8-bin gradient histogram is generated for each sub-region. The 8-dimensional vectors of the 16 sub-regions are obtained and concatenated to obtain a 128-dimensional vector. After L2 normalization and truncation of high values, SIFT features containing 128-dimensional descriptors are obtained.

[0087] The r-th multi-view image is used as input to the CNN network; SP features containing B-dimensional normalized descriptors and corresponding confidence scores are obtained.

[0088] For each point in the SIFT features, search for points in the SP features whose spatial distance is less than a distance threshold. The spatial distance is calculated using Euclidean distance to obtain the set of matching point pairs, the set of unmatched SIFT points, and the set of unmatched SP points.

[0089] The SIFT and SP features in the matching point pair set are weighted to obtain the fusion descriptor. The weighting coefficients are obtained by optimization through a natural heuristic optimization algorithm. In the SIFT features, the pixel value of the difference Gaussian image of the feature point at scale σ is used as the response value. The maximum value of the response value and the confidence score is taken as the reliability score of the fusion point.

[0090] For feature points in the SIFT unmatched point set, retain those with response values ​​not lower than the response threshold and directly include them in the fused matching point set;

[0091] For feature points in the unmatched point set of SP, retain those with a confidence level not lower than the confidence level and directly include them in the fused matching point set;

[0092] If the number of feature points in the G*H neighborhood among the remaining points is greater than J, only the top K points with the highest reliability scores are retained, where K < J, to obtain the fused features.

[0093] The above method accurately captures the local details and scale-invariant information of the image through SIFT feature extraction, and combines the SP features output by the CNN network to supplement the global semantics and features at different levels. After matching and screening, weighted fusion, and neighborhood optimization, a set of fused feature points with both robustness and rich expressiveness is constructed. In complex scenes, this fused feature can effectively resist occlusion and interference from similar textures, provide a more stable and accurate feature matching basis for motion estimation, reduce the risk of false matching, and thus improve the video stabilization effect, avoid motion estimation deviation caused by the defects of single feature matching, and ensure the reliability of anti-shake processing in complex scenes.

[0094] The methods for obtaining the motion vector field include:

[0095] For the fused feature points of the multi-view image in the t-th frame, search for matching points in the (t + 1)-th frame by calculating the cosine similarity of the descriptors, retain the pairs of qualified points where the cosine similarity of the descriptors is greater than the similarity threshold, and calculate the vectors formed by the pairs of qualified points to obtain the sparse motion vectors;

[0096] Construct an X-layer image pyramid. Starting from the top layer, for each pixel in the D×D window, calculate the initial motion vector by least squares fitting of the optical flow equation. For example, if the optical flow constraint is I(x,y,t) = I(x + u,y + v,t + 1), and by Taylor expansion approximation, we get I x v + I y v + I t = 0, where I x and I y are the spatial gradients; I t is the temporal gradient; construct an overdetermined system of equations Vh = b in the D×D window, where V is the spatial gradient; h = [u,v] T is the optical flow vector; b is the temporal gradient; solve for h init = (V T V) -1 V T b; constrain the optical flow vector of the feature points in the sparse motion vectors to be equal to the initial motion vector, and correct the initial motion vector according to the weights of the feature points and the optical flow vectors to obtain the initial dense motion vector field; where the weights of the feature points are weighted according to the Gaussian distance;

[0097] Map the initial dense vector fields of each camera to the main coordinate system through the perspective transformation matrix, and perform weighted fusion according to the reliability scores of the feature points to obtain the motion vector field.

[0098] The aforementioned method first uses fused features as a foundation to obtain sparse motion vectors through descriptor matching and path cost optimization. Then, it refines the dense motion vector field by leveraging image pyramids and weighted least squares combined with depth gradients. Finally, through multi-camera mapping and fusion, it provides comprehensive and accurate pixel-level motion information for motion estimation. In complex scenes, this vector field can overcome the limitations of single-feature matching, enhance the robustness of motion description by utilizing depth and multi-camera information, effectively mitigate mismatches caused by occlusion and similar textures, accurately depict global and local motion trends, provide reliable motion basis for subsequent image processing, improve the accuracy of motion estimation, and ensure stable image stabilization performance.

[0099] Methods for obtaining depth maps include:

[0100] For spatiotemporally aligned multi-view images, a Census transform using an F×F window is employed to calculate the matching cost based on each pixel and candidate disparity, such as the matching cost for pixel (x,y). Where d is the disparity, calculated based on camera intrinsic parameters, such as... C(x,y) a,b C(x+d,y) is the Census code value at a relative offset of (a,b) within the neighborhood of pixel (x,y) in the left image; a,b This is the Census code value at a relative offset of (a,b) within the neighborhood of pixel (x+d,y) in the right image;

[0101] The path cost is accumulated along L path directions, and the d value corresponding to the minimum path cost is selected as the final disparity.

[0102] Perform a left-right consistency check on the initial disparity, such as calculating the left-right disparity and the right-left disparity. Calculate the difference between the left-right disparity and the right-left disparity to obtain the disparity deviation. Mark pixels with disparity deviation greater than the deviation threshold as invalid. Use bilinear interpolation of the known disparity of the fused feature points to fill in the invalid region. For example, for invalid pixels, find the four nearest valid feature points for interpolation. Then, obtain the optimized disparity through quadratic curve fitting.

[0103] Based on the focal length and baseline distance from the camera intrinsics, the optimized parallax is converted into depth, such as the depth of pixel (x,y). d opt (x,y) represents the disparity of pixel (x,y); f represents the focal length; B represents the baseline distance; the results from multiple cameras are averaged to obtain a depth map.

[0104] The method constructs matching costs and optimizes disparity through Census transform of multi-view images. It then converts disparity into depth using camera intrinsics, followed by optimizations such as left-right consistency checks and feature point interpolation to generate high-precision pixel-level depth information. In complex scenes, this depth map provides spatial dimensional constraints for motion estimation. By distinguishing objects at different depths, it avoids motion estimation bias caused by mismatches of feature points in similar texture regions, making the calculation of the motion vector field more closely resemble real physical motion. This enhances the accuracy of motion estimation, provides a reliable depth layering basis for image processing, and ensures the stability of image stabilization in complex scenes.

[0105] Methods for obtaining global jitter components include:

[0106] Step 1: Calculate the depth gradient of the depth map, construct the disparity weight matrix based on the depth gradient, such as by using the Sobel operator to calculate the depth gradient, calculate the magnitude based on the depth gradient, and calculate the disparity weight matrix based on the magnitude.

[0107] Step 2: Construct a global motion vector field based on the motion vectors. Combined with the disparity weight matrix, construct an objective function with the goal of minimizing the error between the global motion vector field and the global jitter vector. Solve this function using iterative reweighted least squares to separate the global jitter vector from the global motion vector field; for example, initialize the weights W. (0) (x,y)=W(x,y), solve for the initial jitter vector d (0) =(W (0)T W (0) ) -1 W (0)T v field ; Calculate the residual r (k) (x,y)=v field (x,y)-d (k) Update weights Where ∈ = 1e-6, is used to avoid a denominator of 0, and e is a mathematical constant; iterates to ||d (k+1) -d (k) ||<1e-3; Output the corresponding global jitter vector.

[0108] Step 3: Repeat steps 1-2 for consecutive image frames to obtain single-frame jitter vectors. After filtering using the Kalman filter algorithm, the vectors are converted into time-continuous global jitter components by combining the scaling factor. The scaling factor is calculated based on the sensor size and the number of pixels to obtain the global jitter components.

[0109] The aforementioned method constructs a disparity weight matrix based on depth gradients, separates the global jitter vector from the global motion vector field through weighted least squares optimization, and generates a temporally continuous jitter component sequence through iterative reweighted least squares solving and Kalman filtering smoothing. In complex scenes, this method leverages depth information to differentially weight the motion vector, highlighting the motion contribution of reliable regions and reducing the interference effects of occlusion and similar texture regions. It accurately extracts the overall camera jitter trend, providing an accurate jitter benchmark for motion compensation and avoiding image stabilization deviations caused by misjudgment of local motion. Thus, in scenes where feature points are easily disturbed, it ensures the stability of motion estimation and improves the reliability of image stabilization.

[0110] The global motion trajectory is generated by fusing global jitter components and real-time IMU data. The historical trajectory and motion parameters of the global motion trajectory are used as input to the LSTM network to obtain the future jitter compensation amount.

[0111] Using global jitter components as observations and motion parameters as control inputs, the global motion trajectory is obtained by fusing them using a Kalman filter algorithm. The historical trajectory of the global motion trajectory and motion parameters are then input into an LSTM network to obtain future jitter compensation. The future jitter compensation is converted into OIS compensation to drive the actuator. At the same time, the LSTM weights are calibrated by comparing them with the actual motion residuals through online reinforcement learning.

[0112] By fusing global jitter components with real-time IMU data, a more robust global motion trajectory is generated through Kalman filtering. An LSTM network learns from historical trajectories and motion parameters to predict future jitter compensation. Online reinforcement learning is then used to calibrate the LSTM weights based on actual motion residuals. This process effectively integrates global jitter components to provide visual jitter features, supplements high-frequency motion details with IMU data, weakens motion trajectory deviations caused by feature point interference with Kalman filtering, captures jitter trends in advance with the temporal prediction capabilities of LSTM, and dynamically corrects prediction errors. This reduces compensation lag or deviation caused by inaccurate motion estimation in complex scenes where feature points are easily occluded or interfered with by similar textures, making the OIS actuator drive more precise and significantly improving the reliability and real-time performance of image stabilization.

[0113] Based on the depth map and future jitter compensation, corresponding transformations are performed according to depth layers and dynamic cropping is carried out to obtain the EIS optimized frame sequence and cropping region.

[0114] Methods for obtaining EIS optimized frame sequences include:

[0115] Based on the depth map and a preset segmentation threshold, such as 1m, the image can be divided into near and far scenes based on common close-range scenes in AR interaction. A layered mask marking the near and far areas can be obtained through threshold segmentation. For example, in the near mask, the near area is marked as 1 and the far area is marked as 0. The far mask is obtained by calculating the complement of the near mask, that is, the near area is marked as 0 and the far area is marked as 1.

[0116] The foreground layer performs a rigid transformation based on the future jitter compensation amount, i.e., rotation and translation, as shown in the rigidly transformed pixels. Where R is the rotation matrix; (α,β,γ) is the rotation angle; (c x ,c y ) are the coordinates of the main point;

[0117] The background layer undergoes a perspective affine transformation based on the global motion trajectory, and the transformed image is output; for example, based on the global motion trajectory T... global Transformation matrix Among them, a 11 a 12 a 21 and a 22 Calculated from a global rotation matrix, allowing for moderate deformation of the distant view; t g,x and t g,y The global translation vector; the pixel after affine transformation.

[0118] Obtain the transformed data I trans (x,y)=M near (x,y)·I(x′,y′)+M far (x,y)·I(x”,y”); where I(x',y') is the gray value of pixel (x',y'); I(x”,y”) is the gray value of pixel (x”,y”);

[0119] A safe region is generated based on the depth map and pixel sharpness. Pixel sharpness is obtained using the Laplacian operator, such as the Laplacian operator value. For example, safe zone R safe = {(x,y)|Z(x,y)≥0.3m\and S(x,y)≥0.6}; Z(x,y) is the depth corresponding to the pixel (x,y);

[0120] The minimum bounding rectangle [x1, y1, x2, y2] of the safe region is obtained by cropping. The cropping window is obtained by expanding the minimum bounding rectangle based on the predicted jitter range. The image is then cropped to the target resolution based on the cropping window, and the cropped image is output. For example, the horizontal expansion Δw = 1.2 * |Δx|, and the vertical expansion Δh = 1.2 * |Δy|, where Δx is the predicted horizontal jitter displacement, and Δy is the predicted vertical jitter displacement. The scaling factor is scale = min(W).target / (x2-x1+2Δw),H target / (y2-y1+2Δh)); where W target For target horizontal resolution; H target For target vertical resolution;

[0121] The cropped region is repaired with bilinear interpolation, the edges are smoothed with guided filtering, the cropping windows of consecutive frames are weighted and smoothed, and the output is an EIS-optimized frame sequence.

[0122] The method described above uses depth maps to perform layered processing of near and far scenes. For the near scene, a rigid transformation is applied, while for the far scene, a perspective affine transformation is performed based on the global motion trajectory. Then, through safe region cropping, interpolation repair, and inter-frame smoothing, global jitter compensation is precisely applied to image regions at different depths. In complex scenes, this layered processing strategy can utilize depth information to distinguish the motion characteristics of objects at different spatial locations, avoiding misjudgments of overall motion caused by similar textures or occlusion. This makes motion compensation more consistent with real physical motion, reducing deformation or offset under a single transformation method. Therefore, even when feature point matching is susceptible to interference, targeted frame sequence optimization can still output stable and natural video frames, effectively improving image stabilization in complex scenes.

[0123] Based on the camera status and EIS optimized frame sequence, the main camera is dynamically selected and multi-camera super-resolution reconstruction is performed on the cropped area to output the final stabilized video stream.

[0124] Methods for obtaining the final stabilized video stream include:

[0125] Based on the status of each camera, such as focal length adaptation S f =exp(-|f i -0.01×D avg | / f i ); where D avg f is the average depth of the scene; i Let S be the focal length of the i-th camera; and let S be the light adaptability. L =min(1,L) i / 500); where L i The light intensity corresponding to the i-th camera; OIS error S E =exp(-E i / 3); where E i The OIS compensation error is for the i-th camera; the computing efficiency S C =1-C i / 100, where C iThe current computing power corresponds to the i-th camera; the camera with the highest total score is selected as the master camera through weighted scoring, and a master-slave scheduling instruction is output; the weighting weights are obtained by optimization based on a natural heuristic optimization algorithm;

[0126] The main camera provides a cropping region [x1, y1, x2, y2], and the secondary camera calculates the corresponding region through a perspective transformation matrix. For overlapping pixels, the pixels of the main camera and the pixels of the secondary camera are weighted to obtain the reconstructed region. The weighting weights are obtained by optimization based on a natural heuristic optimization algorithm.

[0127] The reconstructed region is embedded into the corresponding position of the EIS optimized frame sequence, and inter-frame color correction and smoothing are performed. For example, the gray world algorithm is used to unify the white balance to ensure that the inter-frame color shift is low. The inter-frame optical flow vector is calculated to see if it is greater than 1 pixel. Smoothing is performed by bilateral filtering, and the final stabilized video stream is output.

[0128] The aforementioned method utilizes dynamic election and resource allocation between master and slave cameras. The master camera focuses on the core task, while slave cameras supplement redundant fields of view. Key regions are reconstructed through perspective transformation alignment and weighted fusion. EIS-optimized frames are then embedded, and inter-frame color correction and smoothing are performed, forming a complete multi-camera collaborative image stabilization chain. In complex scenes, this method integrates the advantages of multiple cameras. The master camera ensures processing accuracy in core areas, while slave cameras supplement redundant information in occluded or similar textured areas. Combined with layered motion compensation optimized by EIS, it effectively resists local motion estimation biases caused by feature point occlusion and interference from similar textures. This ensures that the final output video stream maintains detail integrity while achieving global stability, avoiding the limitations of single-camera image stabilization in complex scenes and significantly improving video stability and visual coherence.

[0129] Example 2:

[0130] This embodiment provides a method for refining the motion vector field of depth gradient-driven motion in Embodiment 1, comprising the following steps:

[0131] After obtaining the initial dense motion vector field, the depth gradient of each pixel is calculated in conjunction with the depth map. For regions with a depth gradient greater than the high gradient threshold, the difference between its motion vector and the adjacent pixel is forced not to exceed the physical displacement threshold corresponding to the depth gradient, so as to avoid the edge features of similar texture regions from causing sticky motion deviation due to mismatch. For regions with a depth gradient less than the low gradient threshold, cross-validation of motion vectors across cameras is performed. For example, when the difference between the master and slave camera vectors is greater than 3 pixels, the data from the camera with more reliable depth is used to correct the local motion estimation error.

[0132] Example 3:

[0133] Please see Figure 4As shown, this embodiment provides a multi-camera collaborative image stabilization control system, including:

[0134] Image alignment module: Based on FPGA, a global hardware trigger signal is generated to synchronously trigger the IMU to acquire 6-DOF motion parameters and multiple cameras to acquire raw images. The motion parameters are then processed to obtain spatiotemporally aligned multi-view images.

[0135] Global Analysis Module: Extracts features from multi-view images and motion parameters to obtain fused features, generates motion vector fields based on fused features, analyzes multi-view images to obtain depth maps, and then analyzes them to obtain global jitter components;

[0136] Jitter prediction module: It integrates global jitter components with IMU real-time data to generate a global motion trajectory, and uses the historical trajectory and motion parameters of the global motion trajectory as input to the LSTM network to obtain the future jitter compensation amount;

[0137] Optimized cropping module: Based on the depth map and future jitter compensation, it performs corresponding transformations and dynamic cropping according to depth layers to obtain EIS optimized frame sequences and cropping regions;

[0138] Image stabilization and reconstruction module: Based on the camera status and EIS optimized frame sequence, dynamically select the main camera and perform multi-camera super-resolution reconstruction on the cropped area, and output the final image stabilization video stream.

[0139] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0140] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-camera collaborative image stabilization control method, characterized in that, Includes the following steps: A global hardware trigger signal is generated based on FPGA, which synchronously triggers the IMU to acquire 6-DOF motion parameters and multiple cameras to acquire raw images. The motion parameters are then processed to obtain spatiotemporally aligned multi-view images. The actual exposure start time of the i-th camera is calculated by adding the time when the trigger signal is emitted by the FPGA and the propagation delay of the trigger signal from the FPGA to the i-th camera; wherein, the propagation delay of the trigger signal from the FPGA to the i-th camera is determined by static timing analysis or online calibration. At the actual exposure start time, each camera generates a 16-bit timestamp based on the FPGA clock frequency, embeds it into the original image frame header, and performs periodic calibration to obtain the original image with the calibrated timestamp. The intrinsic and extrinsic parameter matrices of each camera are obtained using the Zhang Zhengyou calibration method, and the perspective transformation matrix between adjacent cameras is calculated based on the intrinsic and extrinsic parameter matrices. The main camera C0 is selected as the reference coordinate system. The original image with calibrated timestamp from the i-th camera is mapped to the virtual coordinate system by combining the perspective transformation matrix to obtain the image frame. By comparing the timestamps of image frames from each camera, the motion parameters corresponding to the timestamps of the original images with calibrated timestamps are extracted. The time difference is calculated based on the difference in timestamps and the FPGA clock frequency. Motion interpolation is then performed on the lagging frames in combination with the rotation matrix in the motion parameters based on the time difference to obtain spatiotemporally aligned multi-view images. Feature extraction is performed on multi-view images and motion parameters to obtain fused features, and a motion vector field is generated based on the fused features. The depth map is obtained by analyzing the multi-view images, and then the global jitter component is obtained by further analysis. The global motion trajectory is generated by fusing global jitter components and real-time IMU data. The historical trajectory and motion parameters of the global motion trajectory are used as input to the LSTM network to obtain the future jitter compensation amount. Based on the depth map and future jitter compensation, corresponding transformations are performed according to depth layers, and dynamic cropping is carried out to obtain the EIS optimized frame sequence and the cropped region: Based on the depth map and a preset segmentation threshold, the image is divided into foreground and background, and a layered mask marking the foreground and background regions is obtained through threshold segmentation. The foreground layer performs a rigid transformation based on the future jitter compensation amount, i.e., rotation and translation; the background layer performs a perspective affine transformation based on the global motion trajectory, and outputs the transformed image. A safe region is generated based on the depth map and pixel sharpness, and pixel sharpness is obtained using the Laplacian operator. The minimum bounding rectangle of the safe region is obtained by cropping. The minimum bounding rectangle is expanded based on the predicted jitter range to obtain the cropping window. The image is cropped to the target resolution based on the cropping window, and the cropped image is output. The cropped area is repaired with bilinear interpolation, the edges are smoothed with guided filtering, the cropped window of consecutive frames is weighted and smoothed, and the EIS optimized frame sequence is output. Based on the camera status and EIS optimized frame sequence, the main camera is dynamically selected and multi-camera super-resolution reconstruction is performed on the cropped area to output the final stabilized video stream.

2. The multi-camera collaborative image stabilization control method according to claim 1, characterized in that, Methods for periodic calibration include: The FPGA sends synchronization messages to all cameras. The cameras record the reception time and send back a delay request. The FPGA combines the reception time and round-trip delay to calculate the clock offset of each camera and performs calibration based on the clock offset of each camera.

3. The multi-camera collaborative image stabilization control method according to claim 1, characterized in that, Methods for obtaining fusion features include: SIFT feature points of the r-th multi-view image are extracted using scale-space extremum detection: The r-th multi-view image is used as input to the CNN network; SP features containing B-dimensional normalized descriptors and corresponding confidence scores are obtained. For each point in the SIFT features, search for points in the SP features whose spatial distance is less than a distance threshold. The spatial distance is calculated using Euclidean distance to obtain the set of matching point pairs, the set of unmatched SIFT points, and the set of unmatched SP points. The fused descriptor is obtained by weighting the SIFT and SP features in the set of matching point pairs. The weighting coefficients are optimized using a naturalistic heuristic optimization algorithm. In the SIFT features, the feature points are scaled... The pixel values ​​of the difference Gaussian image at the point are used as response values; the maximum value of the response value and the confidence score is taken as the reliability score of the fusion point. For feature points in the SIFT unmatched point set, retain those with response values ​​not lower than the response threshold and directly include them in the fused matching point set; For feature points in the unmatched point set of SP, retain those with a confidence level not lower than the confidence level and directly include them in the fused matching point set; If the number of feature points in the G H neighborhood among the remaining points is greater than J, only keep the top K points with the highest reliability scores, where K < J, to obtain the fused features.

4. The multi-camera collaborative image stabilization control method according to claim 1, characterized in that, Methods for obtaining motion vector fields include: For the fused feature points of the multi-view image in frame t, the matching points are searched in frame t+1 by calculating the descriptor cosine similarity. The qualified point pairs with the descriptor cosine similarity greater than the similarity threshold are retained, and the vector formed by the qualified point pairs is calculated to obtain the sparse motion vector. An X-layer image pyramid is constructed. Starting from the top layer, the optical flow equation is fitted to each pixel within a D×D window using least squares to calculate the initial motion vector. The optical flow vector of the feature points in the sparse motion vector is constrained to be equal to the initial motion vector. The initial motion vector is then corrected based on the weights of the feature points and the optical flow vector to obtain the initial dense motion vector field. The weights of the feature points are calculated using Gaussian distance. The initial dense vector fields of each camera are mapped to the main coordinate system through a perspective transformation matrix, and then weighted and fused according to the reliability score of feature points to obtain the motion vector field.

5. The multi-camera collaborative image stabilization control method according to claim 1, characterized in that, Methods for obtaining depth maps include: For spatiotemporally aligned multi-view images, the matching cost is calculated by constructing a matching cost based on each pixel and candidate disparity using the Census transform of an F×F window. The path cost is accumulated along L path directions, and the d value corresponding to the minimum path cost is selected as the final disparity. The initial disparity is checked for left-right consistency. The difference between the left-right disparity and the right-left disparity is calculated to obtain the disparity deviation. Pixels with disparity deviation greater than the deviation threshold are marked as invalid. The invalid region is filled with bilinear interpolation of the known disparity of the fused feature points. Finally, the optimized disparity is obtained by quadratic curve fitting. Based on the focal length and baseline distance in the camera intrinsics, the optimized parallax is converted into depth, and the results from multiple cameras are averaged to obtain a depth map.

6. The multi-camera collaborative image stabilization control method according to claim 1, characterized in that, Methods for obtaining global jitter components include: Step 1: Calculate the depth gradient of the depth map and construct the disparity weight matrix based on the depth gradient; Step 2: Construct a global motion vector field based on the motion vectors. Combined with the disparity weight matrix, construct an objective function with the goal of minimizing the error between the global motion vector field and the global jitter vector. Solve the global jitter vector by iterative reweighted least squares. Step 3: Repeat steps 1-2 for consecutive image frames to obtain single-frame jitter vectors. After filtering using the Kalman filter algorithm, the vectors are converted into time-continuous global jitter components by combining the scaling factor. The scaling factor is calculated based on the sensor size and the number of pixels to obtain the global jitter components.

7. The multi-camera collaborative image stabilization control method according to claim 1, characterized in that, Methods for obtaining the final stabilized video stream include: Based on the status of each camera, the camera with the highest total score is selected as the master camera through weighted scoring, and a master-slave scheduling command is output. The main camera provides the minimum bounding rectangle as the cropping area, and the secondary camera calculates the corresponding area through the perspective transformation matrix. For overlapping pixels, the pixels of the main camera and the pixels of the secondary camera are weighted to obtain the reconstructed area. The reconstructed region is embedded into the corresponding position of the EIS optimized frame sequence, and inter-frame color correction and smoothing are performed to output the final stabilized video stream.

8. A multi-camera collaborative image stabilization control system, implementing the multi-camera collaborative image stabilization control method according to any one of claims 1-7, characterized in that, include: Image alignment module: Based on FPGA, a global hardware trigger signal is generated to synchronously trigger the IMU to acquire 6-DOF motion parameters and multiple cameras to acquire raw images. The motion parameters are then processed to obtain spatiotemporally aligned multi-view images. Global Analysis Module: Extracts features from multi-view images and motion parameters to obtain fused features, generates motion vector fields based on fused features, analyzes multi-view images to obtain depth maps, and then analyzes them to obtain global jitter components; Jitter prediction module: It integrates global jitter components with IMU real-time data to generate a global motion trajectory, and uses the historical trajectory and motion parameters of the global motion trajectory as input to the LSTM network to obtain the future jitter compensation amount; Optimized cropping module: Based on the depth map and future jitter compensation, it performs corresponding transformations and dynamic cropping according to depth layers to obtain EIS optimized frame sequences and cropping regions; Image stabilization and reconstruction module: Based on the camera status and EIS optimized frame sequence, dynamically select the main camera and perform multi-camera super-resolution reconstruction on the cropped area to output the final image stabilization video stream.

Images