FPGA-based multi-camera anti-shake timing synchronization control system and method

Abstract

The application belongs to the technical field of signal synchronization, and discloses a multi-camera anti-shake timing synchronization control system and method based on FPGA; the method comprises the following steps: taking FPGA as a core, generating an FSYNC signal as a camera exposure reference, and obtaining a delay value through timestamp and differential measurement; collecting IMU data to obtain a global motion vector through a gradient descent algorithm; combining the two to analyze and obtain a phase offset and feed back to the FPGA, and the FPGA adjusts the FSYNC signal according to the global motion vector; the application realizes nanosecond-level timestamp recording through the homologous clock reference and hardware-level signal binding mechanism, and combines global motion vector analysis and dynamic phase offset adjustment, thereby breaking through the limitation of the prior art that relies on static time difference compensation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal synchronization, and more particularly, to a multi-camera anti-shake timing synchronization control system and method based on FPGA. BACKGROUND

[0002] In the fields of automatic driving, security monitoring, virtual reality (VR), etc., multi-camera cooperative work has become a core solution for obtaining panoramic view and three-dimensional environment perception. For example, an automatic driving vehicle needs to splice road images in real time through front-view, side-view, rear-view, etc. multi-camera to realize obstacle detection and path planning; a VR device needs to simulate human eye parallax through double cameras to build immersive experience through picture fusion. These scenes put forward strict requirements on the timing synchronization accuracy and dynamic anti-shake performance of multi-camera: if the timing is not synchronized, the time difference of multi-view images will exceed 100 ns, causing misalignment of splicing or three-dimensional reconstruction error; if anti-shake fails, picture shaking during fast motion will reduce target recognition accuracy.

[0003] A multi-camera dynamic synchronization exposure circuit and method of a photogrammetry system based on FPGA disclosed in Chinese Patent No. CN107277389B collect a photographing trigger instruction of each camera, record the different synchronization time of shutter actions of each camera according to the obtained camera photographing trigger signal, take the last arrived shutter action signal as a reference, calculate the synchronization time difference value of other cameras and the camera to which the standard shutter action signal belongs, delay the trigger signal of the shutter of other cameras according to the synchronization time difference value, and generate a synchronization photographing trigger signal of each camera. The present application processes the shutter signals of multiple cameras to make them synchronized, ensures accurate synchronization during shooting, enables simultaneous exposure of cameras, makes each photography center coincide, facilitates the splicing and post-processing of images of multiple cameras, greatly accelerates the image processing speed, and reduces the indoor working time.

[0004] Although the above method can meet most scenes, research and actual application of the above method and prior art show that the above method and prior art at least have the following defects:

[0005] The FPGA processing module of the above method records the arrival time difference of shutter trigger signals of each camera, takes the last arrived signal as a reference for fixed delay adjustment, and its control only relies on static time difference compensation, which cannot cope with dynamic changes of cameras, and the fixed delay parameter cannot be adjusted in real time, which will cause synchronization error to accumulate with time.

[0006] In view of this, the present application proposes a multi-camera anti-shake timing synchronization control system and method based on FPGA to solve the above problems. SUMMARY

[0007] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purposes, the present application provides the following technical scheme: a multi-camera anti-shake timing synchronization control method based on FPGA, comprising the following steps:

[0008] Generate FSYNC signal based on FPGA as the exposure starting reference of each camera, wherein FSYNC represents frame synchronization signal; record exposure start time and anti-shake frame output time through timestamp embedding, and obtain delay measurement value by combining difference measurement method;

[0009] Collect IMU data of each camera, wherein IMU represents inertial measurement unit, and obtain global motion vector by gradient descent algorithm optimization;

[0010] According to the delay measurement value and the global motion vector, the phase offset is obtained and fed back to the FPGA;

[0011] The FPGA adjusts the FSYNC signal according to the feedback phase offset, and obtains the adjusted FSYNC signal combined with the global motion vector;

[0012] Real-time monitoring of the adjusted FSYNC signal and the phase offset, based on the monitoring result, the phase offset is voted and optimized, the optimized offset is obtained and fed back to the FPGA for anti-shake timing synchronization control.

[0013] Further, the method for obtaining the phase offset comprises:

[0014] Control the programmable delay line of the FPGA to traverse all possible delay values with hardware minimum resolution as the step, record the corresponding anti-shake frame output time for each delay value; count the output time fluctuation corresponding to all delay values, select the continuous interval with the smallest fluctuation, and set the midpoint value of the corresponding continuous interval as the target delay reference;

[0015] Obtain the delay measurement value of the current frame, and calculate the jitter error combined with the target delay reference;

[0016] Extract the global translation component and the global rotation component from the global motion vector, calculate and normalize to obtain the motion intensity index;

[0017] Dynamically adjust the PID coefficient according to the motion intensity index; the PID coefficient includes proportional coefficient, integral coefficient and differential coefficient;

[0018] According to the current frame jitter error, the last frame jitter error and the cumulative sum of the previous k frame errors, and combining the PID control formula, the basic phase offset is calculated;

[0019] Optimize the basic phase offset based on Kalman filtering algorithm to obtain the compensation amount; limit the compensation amount to obtain the phase offset.

[0020] Further, the method for dynamically adjusting the PID coefficients according to the motion intensity index comprises:

[0021] The base proportional coefficient is calculated according to the maximum proportional coefficient, the minimum proportional coefficient and the motion intensity index;

[0022] The base integral coefficient is calculated according to the maximum integral coefficient, the minimum integral coefficient and the motion intensity index;

[0023] The base differential coefficient is calculated according to the maximum differential coefficient, the minimum differential coefficient and the motion intensity index;

[0024] The deviation of the actual delay of the current frame from the target reference is calculated to obtain a compensation residual, the residual feedback coefficient is calculated based on the compensation residual and the maximum compensation residual, and the base proportional coefficient, the base integral coefficient and the base differential coefficient are respectively modified based on the residual feedback coefficient to obtain the proportional coefficient, the integral coefficient and the differential coefficient.

[0025] Further, the method for limiting the compensation amount to obtain the phase offset comprises:

[0026] If the compensation amount is less than 0, the phase offset is 0;

[0027] If the compensation amount is not less than 0 and not greater than the preset compensation threshold, the phase offset is equal to the compensation amount;

[0028] If the compensation amount is greater than the preset compensation threshold, the phase offset is the preset compensation threshold.

[0029] Further, the method for adjusting the FSYNC signal comprises:

[0030] The ROI parameter of the current frame is obtained according to the global motion vector update;

[0031] The image is processed according to the ROI parameter, and the anti-shake frame is output;

[0032] The ROI processing delay is calculated according to the full-frame processing delay and the ROI ratio;

[0033] The interval from the exposure start time of each frame in the continuous N frames to the anti-shake frame output time is measured, and the average value is calculated to obtain the full-frame processing delay;

[0034] The ROI processing delay is calculated according to the full-frame processing delay, the proportional coefficient and the ROI ratio;

[0035] The actual delay of the current frame is calculated based on the random jitter and the ROI processing delay;

[0036] The phase offset amount of compensation is calculated based on the actual delay of the current frame and the target delay reference;

[0037] The compensated phase offset is converted into a control word based on the resolution, so that the FSYNC signal controls the triggering of the next frame exposure according to the compensated phase offset, thus obtaining the adjusted FSYNC signal.

[0038] Furthermore, methods for obtaining the ROI parameters of the current frame include:

[0039] The ROI ratio is calculated based on the maximum region ratio, minimum region ratio, limit attenuation coefficient, and motion intensity.

[0040] Based on the image resolution W*H of the full frame image, the original ROI center (W / 2, H / 2), the motion tracking coefficient, and the global motion vector, the offset center coordinates are calculated.

[0041] The ROI parameters are obtained by concatenating the ROI scale with the offset center coordinates.

[0042] Furthermore, methods for obtaining optimized offsets include:

[0043] The phase offset A is obtained by the preset main compensation unit based on the method of obtaining phase offset, and the phase offset B is obtained by the preset slave compensation unit based on the method of obtaining phase offset;

[0044] Determine whether A and B are valid calculation results;

[0045] The voting threshold is calculated based on the preset minimum voting threshold, the benchmark fluctuation coefficient, the global motion intensity preparation, and the maximum value of the historical fluctuation of the master and slave units.

[0046] Based on the validity of A and B and their corresponding differences, the values ​​are compared with the voting threshold, and the optimized offset is output based on the comparison results.

[0047] Furthermore, methods for determining whether A and B are valid calculation results include:

[0048] Determine if the delay value is within the preset physical range; if so, the delay value is considered valid; otherwise, it is considered invalid.

[0049] Determine whether each component of the global motion vector satisfies the mechanical limit constraints of the camera. If it does, the global motion vector is considered valid; otherwise, it is considered invalid.

[0050] If either the delay value or the global motion vector is determined to be invalid, then sub-flag 1 is abnormal; otherwise, sub-flag 1 is normal.

[0051] Determine whether the output of the PID controller is within the hardware delay line range. If it is, the output of the PID controller is considered valid; otherwise, it is considered invalid.

[0052] Determine whether the output of the Kalman filter algorithm exceeds the preset output threshold. If it does, the output of the PID controller is deemed invalid; otherwise, it is deemed valid.

[0053] If either the output of the PID controller or the output of the Kalman filter algorithm is determined to be invalid, then flag 2 is abnormal; otherwise, flag 2 is normal.

[0054] Determine whether the difference between the completion time and the start time is greater than the preset frame period. If so, determine that sub-flag 3 is abnormal; otherwise, mark sub-flag 3 as normal.

[0055] Count the number of abnormal occurrences of sub-flag 1, sub-flag 2 and sub-flag 3 in the last R frames. If the number of abnormal occurrences of sub-flag 4 is greater than the abnormal threshold, then sub-flag 4 is determined to be abnormal; otherwise, sub-flag 4 is marked as normal.

[0056] The status of each of the four sub-flags of A or B is comprehensively judged. When all four sub-flags are normal, the status is judged to be normal; otherwise, the status is judged to be abnormal.

[0057] Furthermore, methods for obtaining global motion vectors include:

[0058] The motion vectors and IMU data of each camera are collected. The motion vectors include the translation and rotation components of the camera relative to the previous frame; the IMU data includes acceleration and angular velocity.

[0059] Data alignment and filtering are performed on motion vectors and IMU data to obtain standard motion vectors and standard IMU data;

[0060] Select the b-th camera as the reference camera and use the corresponding standard motion vector as the global reference vector;

[0061] The standard motion vectors of each camera are transformed from their own coordinate system to the global coordinate system to obtain the global motion vector; the own coordinate system is established with the optical center corresponding to the camera as the origin;

[0062] Set a global transformation matrix, and define the objective function by combining the weight of the i-th camera, the global reference vector, and the transformation vector. The global transformation matrix minimizes the deviation between the global motion vectors of all cameras and the reference vector after transformation by the matrix. The optimal global transformation matrix is ​​obtained by iteratively solving the gradient descent method. The optimal global transformation matrix is ​​then applied to the reference vector to obtain the global motion vector.

[0063] The FPGA-based multi-camera image stabilization timing synchronization control system, implementing the FPGA-based multi-camera image stabilization timing synchronization control method, includes:

[0064] The reference measurement module generates an FSYNC signal based on FPGA as the exposure start reference for each camera; it records the exposure start time and the stabilization frame output time by embedding timestamps, and calculates the delay measurement value by combining the differential measurement method.

[0065] Motion analysis module: Collects IMU data from each camera, optimizes it using gradient descent algorithm, and calculates the global motion vector;

[0066] Offset analysis module: Analyzes the delay measurement value and global motion vector to obtain the phase offset and feeds it back to the FPGA;

[0067] Signal adjustment module: The FPGA adjusts the FSYNC signal based on the feedback phase shift and the global motion vector to obtain the adjusted FSYNC signal;

[0068] Offset optimization module: Real-time monitoring of the adjusted FSYNC signal and phase offset, voting optimization of the phase offset based on the monitoring results, obtaining the optimized offset and feeding it back to the FPGA for anti-jitter timing synchronization control.

[0069] The technical effects and advantages of the multi-camera image stabilization timing synchronization control system and method based on FPGA of this invention are as follows:

[0070] This invention uses an FPGA to generate an FSYNC signal as an exposure reference, combined with a timestamp differential measurement method to capture and process dynamic changes in delay in real time, eliminating clock domain conversion errors. Then, by acquiring IMU data and motion vectors, a global motion vector is obtained through gradient descent optimization, providing a unified motion reference for multi-camera collaboration. Next, the PID coefficients are dynamically adjusted based on motion intensity indicators, and combined with Kalman filtering and amplitude limiting, to achieve accurate phase shift calculation. Finally, a master-slave dual-path verification and dynamic voting optimization mechanism ensure the reliability of the phase shift. This closed-loop design of "measurement-analysis-compensation-verification" enables the system to respond in real time to camera motion changes and handle dynamic factors such as load fluctuations, reducing synchronization errors, compressing single-frame calculation time, and meeting the real-time requirements of cameras. Simultaneously, a multi-level anomaly detection mechanism significantly improves system robustness, achieving an upgrade from passive alignment to active adaptation, providing a high-precision and highly reliable solution for multi-camera anti-shake timing synchronization. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating the FPGA-based multi-camera image stabilization timing synchronization control method of the present invention.

[0072] Figure 2 This is a schematic diagram of the data flow in this invention;

[0073] Figure 3This is a schematic diagram of the data flow in Embodiment 2 of the present invention;

[0074] Figure 4 This is a schematic diagram of the FPGA-based multi-camera image stabilization timing synchronization control system of the present invention. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0076] Please see Figure 1 , Figure 2 As shown, this embodiment provides a multi-camera image stabilization timing synchronization control method based on FPGA, including the following steps:

[0077] The frame synchronization signal FSYNC is generated based on FPGA and used as the exposure start reference for each camera. The exposure start time and the anti-shake frame output time are recorded by embedding timestamps, and the delay measurement value is calculated by combining the differential measurement method.

[0078] Methods for obtaining delay measurements include:

[0079] The exposure start time and the stabilization frame output time are recorded by a timestamp register. The 32-bit counter of the timestamp is driven by the on-chip PLL of the FPGA, and its counting reference is the same as the FSYNC signal of the trigger layer. The exposure start time is strictly aligned with the FSYNC signal, and the stabilization frame output time is bound to the DDR write signal of the stabilization module to capture the actual output moment.

[0080] The rising edge of the FSYNC signal is marked as the exposure start time, and the DOR write completion signal is marked as the stabilization frame output time. The delay measurement value is calculated based on the exposure start time, the stabilization frame output time, and the start time and end time of the local reference path synchronously recorded in the FPGA.

[0081] The aforementioned method, through a shared clock reference and hardware-level signal binding mechanism, strictly aligns the exposure start time with the FSYNC signal and binds the stabilization frame output time with the DDR write signal in real time. Combined with a 32-bit synchronization counter, it achieves nanosecond-level timestamp recording, fundamentally overcoming the limitations of existing technologies that rely on static time difference compensation. By using the rising edge of FSYNC as the exposure start reference and the DDR write completion signal as the output termination marker, this method constructs a dynamic delay measurement system covering the entire "exposure-processing-output" process. This enables the system to capture delay fluctuations caused by changes in camera status in real time and obtain the true processing delay through differential calculation. The design of the timestamp counter and the trigger layer FSYNC sharing the same source eliminates clock domain conversion errors, ensuring that the delay measurement value accurately reflects the dynamic characteristics of the current processing link. This provides precise input for subsequent dynamic phase offset adjustments, effectively suppressing the time accumulation effect of synchronization errors and achieving real-time response and closed-loop control to dynamic changes in the camera.

[0082] The inertial measurement unit (IMU) data from each camera is collected, and the global motion vector is calculated by optimizing the gradient descent algorithm.

[0083] Methods for obtaining global motion vectors include:

[0084] The motion vectors and IMU data of each camera are collected. The motion vectors include the translation and rotation components of the camera relative to the previous frame; the IMU data includes acceleration and angular velocity.

[0085] Data alignment and filtering are performed on motion vectors and IMU data to obtain standard motion vectors and standard IMU data;

[0086] Select the b-th camera as the reference camera and use the corresponding standard motion vector as the global reference vector;

[0087] The standard motion vectors of each camera are transformed from their own coordinate system to the global coordinate system to obtain the global motion vector; the own coordinate system is established with the optical center corresponding to the camera as the origin;

[0088] Set a global transformation matrix, and define the objective function by combining the weight of the i-th camera, the global reference vector, and the transformation vector. The global transformation matrix minimizes the deviation between the global motion vectors of all cameras and the reference vector after transformation by the matrix. The optimal global transformation matrix is ​​obtained by iteratively solving the gradient descent method. The optimal global transformation matrix is ​​then applied to the reference vector to obtain the global motion vector.

[0089] This method, which collects IMU data and motion vectors from each camera and performs alignment filtering, selects a reference camera, transforms the coordinate system to the global coordinate system, and optimizes the global motion vector using a gradient descent algorithm, overcomes the limitations of existing technologies that rely solely on static time difference compensation. By capturing the dynamic motion state of each camera in real time, this method integrates scattered local motion information into a unified global motion reference, accurately reflecting the consistency and differences in motion among multiple cameras in dynamic scenes. Based on this global motion vector, the system no longer relies on fixed delay parameters but dynamically adjusts the synchronization strategy according to the real-time motion characteristics of each camera. This fundamentally avoids the accumulation of synchronization errors caused by dynamic changes in cameras, achieving real-time perception and collaborative control of the dynamic motion state of multiple cameras, and providing an accurate motion benchmark for subsequent dynamic delay compensation.

[0090] Based on the analysis of the delay measurement and global motion vector, the phase offset is obtained and fed back to the FPGA;

[0091] Methods for obtaining phase shift include:

[0092] The programmable delay line of the FPGA is controlled to traverse all possible delay values ​​with the minimum hardware resolution as the step size. For each delay value, the corresponding anti-shake frame output time is recorded. The output time fluctuations corresponding to all delay values ​​are counted, and the continuous interval with the smallest fluctuation is selected. The midpoint value of the corresponding continuous interval is set as the target delay reference.

[0093] Obtain the delay measurement value of the current frame and calculate the jitter error by combining it with the target delay benchmark; if the jitter error is greater than 0, the next frame needs to be triggered in advance; if the jitter error is less than 0, the next frame needs to be triggered in a delayed manner.

[0094] Global translation and global rotation components are extracted from the global motion vector, and the motion intensity index is obtained by calculating and normalizing them.

[0095] The PID coefficients are dynamically adjusted based on the exercise intensity index; the PID coefficients include the proportional coefficient, integral coefficient, and derivative coefficient.

[0096] Methods for dynamically adjusting PID coefficients based on exercise intensity indicators include:

[0097] The basic proportional coefficient is calculated by combining the maximum and minimum proportional coefficient values ​​with the exercise intensity index.

[0098] The basic integral coefficient is calculated by combining the maximum and minimum integral coefficients with the exercise intensity index;

[0099] The basic differential coefficients are obtained by combining the maximum and minimum values ​​of the differential coefficients with the exercise intensity index;

[0100] The deviation between the actual delay of the current frame and the target reference is calculated to obtain the compensation residual. The residual feedback coefficient is calculated based on the compensation residual and the maximum value of the compensation residual. The basic proportional coefficient, basic integral coefficient and basic differential coefficient are corrected based on the residual feedback coefficient to obtain the proportional coefficient, integral coefficient and differential coefficient.

[0101] The method of dynamically adjusting PID coefficients based on motion intensity incorporating both motion intensity and compensation effect into the coefficient adjustment logic effectively overcomes the limitation of existing technologies where fixed parameters cannot adapt to dynamic changes. This method first uses the motion intensity index as the core, combining the maximum / minimum values ​​of each coefficient to calculate the basic proportional, integral, and derivative coefficients. This allows the basic trend of the coefficients to directly respond to the intensity of camera movement, fundamentally eliminating the inadequacy of static parameters for dynamic scenes. Then, by calculating residual feedback coefficients through compensation residuals, the basic coefficients are corrected in real time. The better the compensation effect (smaller the compensation residual), the smaller the correction magnitude; the worse the compensation effect (larger the compensation residual), the larger the correction magnitude, forming a closed loop of "motion state → basic coefficients → real-time effect correction." This dynamic adjustment mechanism allows the PID coefficients to match the dynamic changes of the camera in real time, ensuring that the response characteristics of the proportional, integral, and derivative coefficients are always consistent with the current scene requirements. This avoids the response lag or overshoot problems of fixed parameters in dynamic scenes, effectively suppressing the accumulation of synchronization errors and achieving accurate and real-time adaptation to the dynamic changes of the camera.

[0102] The basic phase offset is calculated based on the sum of the jitter error of the current frame, the jitter error of the previous frame, and the cumulative error of the previous k frames, combined with the PID control formula.

[0103] The basic phase offset is optimized based on the Kalman filter algorithm to obtain the compensation amount; the compensation amount is then subjected to amplitude limiting to obtain the phase offset.

[0104] The aforementioned method fundamentally overcomes the limitations of existing technologies that rely on static time difference compensation through dynamic benchmark setting and real-time compensation mechanisms. First, it determines the target delay benchmark through full-range delay scanning, providing a stable reference for dynamic adjustment. Then, based on real-time comparison of the current frame delay measurement with the target benchmark, it accurately calculates jitter error to capture dynamic changes in the camera, overcoming the static limitations of fixed delay parameters. Simultaneously, it extracts motion intensity indicators through global motion vectors and dynamically adjusts PID coefficients, enabling the proportional, integral, and derivative coefficients to adaptively change with the intensity of motion. Combined with real-time error correction from PID control and jitter prediction from Kalman filtering, it achieves dynamic optimization of the basic phase offset. Finally, the phase offset obtained after amplitude limiting is fed back to the FPGA adjustment trigger signal in real time. This process forms a closed loop of "measurement-analysis-compensation," ensuring that the compensation amount for each frame cycle accurately matches the current camera state, effectively suppressing the time accumulation effect of synchronization errors, and achieving real-time response and active correction to dynamic changes in the camera.

[0105] Methods for obtaining phase shift by limiting the compensation amount include:

[0106] If the compensation amount is less than 0, then the phase offset is 0;

[0107] If the compensation amount is not less than 0 and not greater than the preset compensation threshold, then the phase offset is equal to the compensation amount;

[0108] If the compensation amount is greater than the preset compensation threshold, then the phase offset is the preset compensation threshold.

[0109] The method of limiting the compensation amount to obtain the phase shift provides a reliable physical boundary for the dynamic adjustment mechanism by constraining the compensation amount within the effective range achievable by the hardware, effectively overcoming the limitations of existing technologies with fixed delay parameters in dynamic scenes. In this method, when the compensation amount is less than 0, the phase shift is forced to 0, avoiding synchronization disorder caused by ineffective negative delay adjustment; when the compensation amount is between 0 and a preset threshold, the real-time calculated value is directly used, preserving the flexibility to dynamically respond to camera changes; when the compensation amount exceeds the threshold, the preset maximum value is taken to prevent compensation failure caused by hardware over-limitation. This processing ensures that the phase shift can follow the dynamic changes of the camera in real time, and avoids the accumulation of errors caused by over-compensation or ineffective compensation through hardware constraints, making the adjustment of each frame period accurately effective within the physically feasible range, thereby suppressing the divergence of synchronization errors over time and achieving a balance between dynamic compensation and hardware capabilities.

[0110] The FPGA adjusts the FSYNC signal based on the feedback phase shift and the global motion vector to obtain the adjusted FSYNC signal.

[0111] Methods for obtaining the adjusted FSYNC signal include:

[0112] The ROI parameters of the current frame are obtained by updating the global motion vectors.

[0113] The ROI ratio is calculated based on the maximum region ratio, minimum region ratio, limit attenuation coefficient, and motion intensity.

[0114] Based on the image resolution W*H of the full frame image, the original ROI center (W / 2, H / 2), the motion tracking coefficient, and the global motion vector, the offset center coordinates are calculated.

[0115] The ROI scale is concatenated with the offset center coordinates to obtain the ROI parameters;

[0116] The ROI scale is concatenated with the offset center coordinates to obtain the ROI parameters;

[0117] The ROI processing latency is calculated based on the full-frame processing latency and the ROI ratio.

[0118] Measure the interval between the exposure start time and the stabilization frame output time of each frame in N consecutive frames, calculate the average value, and obtain the full frame processing delay;

[0119] The ROI processing latency is calculated based on the full-frame processing latency, the scaling factor, and the ROI ratio; the scaling factor is obtained by least squares fitting.

[0120] The actual delay of the current frame is calculated based on random jitter and ROI processing delay; the random jitter is obtained by statistical modeling to obtain the mean, and then optimized based on the Kalman filter algorithm;

[0121] The compensated phase offset is calculated based on the actual delay of the current frame and the target delay reference.

[0122] The compensated phase offset is converted into a control word based on the resolution, so that the FSYNC signal controls the triggering of the next frame exposure according to the compensated phase offset, thus obtaining the adjusted FSYNC signal.

[0123] This method, which combines FPGA with global motion vector adjustment of the FSYNC signal, completely overcomes the limitations of existing static time difference compensation technologies by dynamically adapting to camera status and implementing real-time closed-loop control. The method first dynamically updates the ROI parameters based on global motion vectors, calculates the ROI ratio based on motion intensity, and adjusts the ROI center coordinates according to the motion direction, ensuring that the ROI parameters match the dynamic changes of the camera in real time. Then, it calculates the ROI processing delay by measuring the full-frame processing delay and using a ratio coefficient fitted with least squares, ensuring that the delay calculation closely matches the actual hardware characteristics. Simultaneously, it introduces Kalman filtering optimization with random jitter to accurately capture the actual delay of the current frame. Based on this, it calculates the phase offset based on the deviation between the actual delay and the target reference, and converts it into a control word to adjust the FSYNC signal, forming a complete closed loop of "motion state → ROI adaptation → delay measurement → phase compensation → signal adjustment". This process allows the adjustment of the FSYNC signal to no longer rely on fixed delay parameters, but can respond in real time to dynamic factors such as camera motion changes and load fluctuations. The adjustment in each frame cycle is based on the current actual state, effectively blocking the accumulation path of synchronization errors and achieving precise adaptation and active control of dynamic scenes.

[0124] The adjusted FSYNC signal and phase offset are monitored in real time. Based on the monitoring results, the phase offset is optimized by voting. The optimized offset is then fed back to the FPGA for anti-jitter timing synchronization control.

[0125] Methods for obtaining optimized offsets include:

[0126] The phase offset A is obtained by the preset main compensation unit based on the method of obtaining phase offset, and the phase offset B is obtained by the preset slave compensation unit based on the method of obtaining phase offset;

[0127] Determine whether A and B are valid calculation results;

[0128] Methods for determining whether A and B are valid calculation results include:

[0129] Determine if the delay value is within the preset physical range; if so, the delay value is considered valid; otherwise, it is considered invalid.

[0130] Determine whether each component of the global motion vector satisfies the mechanical limit constraints of the camera. If it does, the global motion vector is considered valid; otherwise, it is considered invalid.

[0131] If either the delay value or the global motion vector is determined to be invalid, then sub-flag 1 is abnormal; otherwise, sub-flag 1 is normal.

[0132] Determine whether the output of the PID controller is within the hardware delay line range. If it is, the output of the PID controller is considered valid; otherwise, it is considered invalid.

[0133] Determine whether the output of the Kalman filter algorithm exceeds the preset output threshold. If it does, the output of the PID controller is deemed invalid; otherwise, it is deemed valid.

[0134] If either the output of the PID controller or the output of the Kalman filter algorithm is determined to be invalid, then flag 2 is abnormal; otherwise, flag 2 is normal.

[0135] Determine whether the difference between the completion time and the start time is greater than the preset frame period. If so, determine that sub-flag 3 is abnormal; otherwise, mark sub-flag 3 as normal.

[0136] Count the number of abnormal occurrences of sub-flag 1, sub-flag 2 and sub-flag 3 in the last R frames. If the number of abnormal occurrences of sub-flag 4 is greater than the abnormal threshold, then sub-flag 4 is determined to be abnormal; otherwise, sub-flag 4 is marked as normal.

[0137] The status of each of the four sub-flags of A or B is comprehensively judged. When all four sub-flags are normal, the status is judged to be normal; otherwise, the status is judged to be abnormal.

[0138] The method for determining whether A and B are valid calculation results provides a reliable validity verification mechanism for dynamic compensation by conducting end-to-end anomaly detection on input data, calculation process, time constraints, and historical trends. This effectively compensates for the lack of data validity judgment in existing static compensation technologies. The method first checks physical rationality, verifying whether the delay value is within a preset range and whether the global motion vector meets mechanical limits, ensuring the adaptability of input data to the dynamic state of the camera. Then, it checks whether the PID output is within the hardware range and whether the Kalman filter result exceeds limits, ensuring the compliance of the calculation process. Simultaneously, it uses time constraints to ensure that the calculation is completed within the frame period, meeting real-time requirements. Finally, it combines historical anomaly statistics from the last R frames to identify persistent faults to avoid misjudgments due to transient interference. This multi-level detection forms a three-dimensional verification of "input-process-time-history," ensuring that A and B are only judged as valid when there are no anomalies throughout the entire link. It eliminates erroneous data from the source into the compensation process, avoiding the accumulation of synchronization errors caused by invalid parameters. This provides reliable basic data for the dynamic adjustment mechanism, enabling phase offset optimization to truly respond to the dynamic changes of the camera, rather than being misled by outliers.

[0139] The voting threshold is calculated based on the preset minimum voting threshold, the benchmark fluctuation coefficient, the global motion intensity preparation, and the maximum value of the historical fluctuation of the master and slave units.

[0140] Based on the validity of A and B and their corresponding differences, compare them with the voting threshold, and output the optimized offset based on the comparison result;

[0141] Methods for outputting optimized offsets based on comparison results include:

[0142] When both A and B are valid, and the difference between A and B is not higher than the voting threshold, the weighted average of the two is taken as the optimization offset; the weights can be obtained based on a natural heuristic optimization algorithm.

[0143] When both A and B are valid, and the difference between A and B is higher than the voting threshold, the more reliable value is selected based on the average phase offset of the last M frames, combined with the deviation between the current master and slave units and historical trends.

[0144] When only one unit in A and B is valid, the output of the valid unit is directly used, and a reliability correction is added. This can be obtained by optimization based on a natural heuristic optimization algorithm and used as the optimization offset.

[0145] When both A and B are invalid, the average phase offset of the last M frames is directly output as the optimized offset, and a system alarm is triggered, while the master and slave units are forcibly restarted.

[0146] A method for obtaining the final phase offset by real-time monitoring and adjustment of the FSYNC signal and phase offset, and by voting optimization, effectively overcomes the limitations of existing static compensation technologies in handling dynamic changes through a master-slave dual-path verification and dynamic decision-making mechanism. In this method, the master and slave compensation units calculate the phase offsets A and B respectively, combining validity judgment to ensure the reliability of the input data. Then, the voting threshold is dynamically adjusted based on global motion intensity and historical fluctuations, allowing the judgment criteria to adapt to the camera's motion state in real time. For different scenarios, optimization strategies are implemented: a weighted average is used when both are valid and the deviation is small; when the deviation is too large, a more reliable value is selected based on historical trends; when only one is valid, the reliability correction amount is used to approximate the historical average; and when both are invalid, the historical average is used, forming a multi-level fault-tolerance mechanism. This design captures the dynamic changes in phase offset in real time and eliminates outliers and integrates valid information through voting optimization, ensuring that the final phase offset accurately matches the current camera state. This avoids the problem of error accumulation with dynamic changes under fixed delay parameters, achieving an upgrade from "passive alignment" to "active adaptation," and providing a more robust dynamic compensation basis for anti-shake timing synchronization control. Example

[0147] Please see Figure 3 As shown, this embodiment provides a cross-camera dynamic deviation calibration method applied to Embodiment 1, including the following steps:

[0148] Using the FSYNC signal of the b-th camera as a reference, the actual exposure time difference of other cameras is measured, and the individual deviation compensation value is calculated. During actual adjustment, the individual deviation compensation value is superimposed on the optimized offset of each camera as the final offset and fed back to the FPGA for anti-shake timing synchronization control.

[0149] The above method dynamically captures the actual exposure differences caused by individual hardware differences of each camera, transforms static hardware deviations into individual compensation values ​​that can be adjusted in real time, and eliminates fixed deviations caused by hardware differences by superimposing exclusive individual compensation values. It avoids the synchronization error caused by the accumulation of individual differences over time under a single fixed delay parameter, so that the final offset of each camera can both adapt to dynamic changes and correct hardware characteristic differences. It achieves synergy between dynamic response and individual calibration, and effectively improves the accuracy and stability of multi-camera image stabilization timing synchronization control. Example

[0150] Please see Figure 4 As shown, this embodiment provides a multi-camera image stabilization timing synchronization control system based on FPGA, including:

[0151] The reference measurement module generates an FSYNC signal based on FPGA as the exposure start reference for each camera; it records the exposure start time and the stabilization frame output time by embedding timestamps, and calculates the delay measurement value by combining the differential measurement method.

[0152] Motion analysis module: Collects IMU data from each camera, optimizes it using gradient descent algorithm, and calculates the global motion vector;

[0153] Offset analysis module: Analyzes the delay measurement value and global motion vector to obtain the phase offset and feeds it back to the FPGA;

[0154] Signal adjustment module: The FPGA adjusts the FSYNC signal based on the feedback phase shift and the global motion vector to obtain the adjusted FSYNC signal;

[0155] Offset optimization module: Real-time monitoring of the adjusted FSYNC signal and phase offset, voting optimization of the phase offset based on the monitoring results, obtaining the optimized offset and feeding it back to the FPGA for anti-jitter timing synchronization control.

[0156] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0157] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-camera image stabilization timing synchronization control method based on FPGA, characterized in that, Includes the following steps: The FSYNC signal is generated based on FPGA and used as the exposure start reference for each camera. The exposure start time and the anti-shake frame output time are recorded by embedding timestamps, and the delay measurement value is calculated by combining the differential measurement method. Collect IMU data from each camera, optimize using gradient descent algorithm, and calculate the global motion vector; Based on the analysis of the delay measurement and global motion vector, the phase offset is obtained and fed back to the FPGA; Methods for obtaining phase shift include: The programmable delay line of the FPGA is controlled to traverse all delay values ​​with the minimum hardware resolution as the step size. The corresponding anti-shake frame output time is recorded for each delay value. The output time fluctuation corresponding to all delay values ​​is counted, and the continuous interval with the smallest fluctuation is selected. The midpoint value of the corresponding continuous interval is set as the target delay reference. Obtain the delay measurement value of the current frame and calculate the jitter error by combining it with the target delay benchmark; Global translation and global rotation components are extracted from the global motion vector, and the motion intensity index is obtained by calculating and normalizing them. The PID coefficients are dynamically adjusted based on the exercise intensity index; the PID coefficients include the proportional coefficient, integral coefficient, and derivative coefficient. The basic phase offset is calculated based on the sum of the jitter error of the current frame, the jitter error of the previous frame, and the cumulative error of the previous k frames, combined with the PID control formula. The basic phase offset is optimized based on the Kalman filter algorithm to obtain the compensation amount; the compensation amount is then subjected to amplitude limiting to obtain the phase offset. The FPGA adjusts the FSYNC signal based on the feedback phase shift and the global motion vector to obtain the adjusted FSYNC signal. The adjusted FSYNC signal and phase offset are monitored in real time. Based on the monitoring results, the phase offset is optimized by voting. The optimized offset is then fed back to the FPGA for anti-jitter timing synchronization control.

2. The FPGA-based multi-camera image stabilization timing synchronization control method according to claim 1, characterized in that, Methods for dynamically adjusting PID coefficients based on exercise intensity indicators include: The basic proportional coefficient is calculated by combining the maximum and minimum proportional coefficient values ​​with the exercise intensity index. The basic integral coefficient is calculated by combining the maximum and minimum integral coefficients with the exercise intensity index; The basic differential coefficients are obtained by combining the maximum and minimum values ​​of the differential coefficients with the exercise intensity index; The deviation between the actual delay of the current frame and the target reference is calculated to obtain the compensation residual. The residual feedback coefficient is calculated based on the compensation residual and the maximum value of the compensation residual. The basic proportional coefficient, basic integral coefficient and basic differential coefficient are corrected based on the residual feedback coefficient to obtain the proportional coefficient, integral coefficient and differential coefficient.

3. The FPGA-based multi-camera image stabilization timing synchronization control method according to claim 1, characterized in that, Methods for obtaining phase shift by limiting the compensation amount include: If the compensation amount is less than 0, then the phase offset is 0; If the compensation amount is not less than 0 and not greater than the preset compensation threshold, then the phase offset is equal to the compensation amount; If the compensation amount is greater than the preset compensation threshold, then the phase offset is the preset compensation threshold.

4. The FPGA-based multi-camera image stabilization timing synchronization control method according to claim 1, characterized in that, Methods for obtaining the adjusted FSYNC signal include: The ROI parameters of the current frame are obtained by updating the global motion vectors. The image is processed based on the ROI parameters, and the stabilized frame is output. The ROI processing latency is calculated based on the full-frame processing latency and the ROI ratio. Measure the interval between the exposure start time and the stabilization frame output time of each frame in N consecutive frames, calculate the average value, and obtain the full frame processing delay; The ROI processing latency is calculated based on the full-frame processing latency, the scaling factor, and the ROI ratio. The actual latency of the current frame is calculated based on random jitter and ROI processing latency. The compensated phase offset is calculated based on the actual delay of the current frame and the target delay reference. The compensated phase offset is converted into a control word based on the resolution, so that the FSYNC signal controls the triggering of the next frame exposure according to the compensated phase offset, thus obtaining the adjusted FSYNC signal.

5. The FPGA-based multi-camera image stabilization timing synchronization control method according to claim 4, characterized in that, Methods for obtaining the ROI parameters of the current frame include: The ROI ratio is calculated based on the maximum region ratio, minimum region ratio, limit attenuation coefficient, and motion intensity. Based on the image resolution W*H of the full frame image, the original ROI center (W / 2, H / 2), the motion tracking coefficient, and the global motion vector, the offset center coordinates are calculated. The ROI parameters are obtained by concatenating the ROI scale with the offset center coordinates.

6. The FPGA-based multi-camera image stabilization timing synchronization control method according to claim 1, characterized in that, Methods for obtaining optimized offsets include: The phase offset A is obtained by the preset main compensation unit based on the method of obtaining phase offset, and the phase offset B is obtained by the preset slave compensation unit based on the method of obtaining phase offset; Determine whether A and B are valid calculation results; The voting threshold is calculated based on the preset minimum voting threshold, the benchmark fluctuation coefficient, the global motion intensity preparation, and the maximum value of the historical fluctuation of the master and slave units. Based on the validity of A and B and their corresponding differences, the values ​​are compared with the voting threshold, and the optimized offset is output based on the comparison results.

7. The FPGA-based multi-camera image stabilization timing synchronization control method according to claim 6, characterized in that, Methods for determining whether A and B are valid calculation results include: Determine if the delay value is within the preset physical range; if so, the delay value is considered valid; otherwise, it is considered invalid. Determine whether each component of the global motion vector satisfies the mechanical limit constraints of the camera. If it does, the global motion vector is considered valid; otherwise, it is considered invalid. If either the delay value or the global motion vector is determined to be invalid, then sub-flag 1 is abnormal; otherwise, sub-flag 1 is normal. Determine whether the output of the PID controller is within the hardware delay line range. If it is, the output of the PID controller is considered valid; otherwise, it is considered invalid. Determine whether the output of the Kalman filter algorithm exceeds the preset output threshold. If it does, the output of the PID controller is deemed invalid; otherwise, it is deemed valid. If either the output of the PID controller or the output of the Kalman filter algorithm is determined to be invalid, then flag 2 is abnormal; otherwise, flag 2 is normal. Determine whether the difference between the completion time and the start time is greater than the preset frame period. If so, determine that sub-flag 3 is abnormal; otherwise, mark sub-flag 3 as normal. Count the number of abnormal occurrences of sub-flag 1, sub-flag 2 and sub-flag 3 in the last R frames. If the number of abnormal occurrences of sub-flag 4 is greater than the abnormal threshold, then sub-flag 4 is determined to be abnormal; otherwise, sub-flag 4 is marked as normal. The status of each of the four sub-flags of A or B is comprehensively judged. When all four sub-flags are normal, the status is judged to be normal; otherwise, the status is judged to be abnormal.

8. The FPGA-based multi-camera image stabilization timing synchronization control method according to claim 1, characterized in that, Methods for obtaining global motion vectors include: The motion vectors and IMU data of each camera are collected. The motion vectors include the translation and rotation components of the camera relative to the previous frame; the IMU data includes acceleration and angular velocity. Data alignment and filtering are performed on motion vectors and IMU data to obtain standard motion vectors and standard IMU data; Select the b-th camera as the reference camera and use the corresponding standard motion vector as the global reference vector; The standard motion vectors of each camera are transformed from their own coordinate system to the global coordinate system to obtain the global motion vector; the own coordinate system is established with the optical center corresponding to the camera as the origin; Set a global transformation matrix, and define the objective function by combining the weight of the i-th camera, the global reference vector, and the transformation vector. The global transformation matrix minimizes the deviation between the global motion vectors of all cameras and the reference vector after transformation by the matrix. The optimal global transformation matrix is ​​obtained by iteratively solving the gradient descent method. The optimal global transformation matrix is ​​then applied to the reference vector to obtain the global motion vector.

9. A multi-camera image stabilization timing synchronization control system based on FPGA, implementing the multi-camera image stabilization timing synchronization control method based on FPGA as described in any one of claims 1-8, characterized in that, include: Reference measurement module: Generates FSYNC signal based on FPGA, which serves as the exposure start reference for each camera; The exposure start time and the stabilized frame output time are recorded by embedding a timestamp, and the delay measurement value is calculated by combining the differential measurement method. Motion analysis module: Collects IMU data from each camera, optimizes it using gradient descent algorithm, and calculates the global motion vector; Offset analysis module: Analyzes the delay measurement value and global motion vector to obtain the phase offset and feeds it back to the FPGA; Methods for obtaining phase shift include: The programmable delay line of the FPGA is controlled to traverse all delay values ​​with the minimum hardware resolution as the step size. The corresponding anti-shake frame output time is recorded for each delay value. The output time fluctuation corresponding to all delay values ​​is counted, and the continuous interval with the smallest fluctuation is selected. The midpoint value of the corresponding continuous interval is set as the target delay reference. Obtain the delay measurement value of the current frame and calculate the jitter error by combining it with the target delay benchmark; Global translation and global rotation components are extracted from the global motion vector, and the motion intensity index is obtained by calculating and normalizing them. The PID coefficients are dynamically adjusted based on the exercise intensity index; the PID coefficients include the proportional coefficient, integral coefficient, and derivative coefficient. The basic phase offset is calculated based on the sum of the jitter error of the current frame, the jitter error of the previous frame, and the cumulative error of the previous k frames, combined with the PID control formula. The basic phase offset is optimized based on the Kalman filter algorithm to obtain the compensation amount; the compensation amount is then subjected to amplitude limiting to obtain the phase offset. Signal adjustment module: The FPGA adjusts the FSYNC signal based on the feedback phase shift and the global motion vector to obtain the adjusted FSYNC signal; Offset optimization module: Real-time monitoring of the adjusted FSYNC signal and phase offset, voting optimization of the phase offset based on the monitoring results, obtaining the optimized offset and feeding it back to the FPGA for anti-jitter timing synchronization control.

Images