Multi-camera anti-shake time sequence synchronous control system and method based on FPGA (Field Programmable Gate Array)

Abstract

The invention belongs to the technical field of signal synchronization, and discloses a multi-camera anti-shake time sequence synchronization control system and method based on an FPGA. The method comprises the following steps: taking an FPGA as a core, generating an FSYNC signal as a camera exposure reference, and obtaining a delay value through a timestamp and differential measurement; acquiring I MU data and obtaining a global motion vector through a gradient descent algorithm; analyzing by combining the two to obtain phase deviation and feeding back the phase deviation to the FPGA, and adjusting the FSYNC signal by combining the global motion vector by the FPGA according to the phase deviation; according to the method, nanosecond timestamp recording is realized through a homologous clock reference and hardware-level signal binding mechanism, and the limitation of dependence on static time difference compensation in the prior art is broken through in combination with global motion vector analysis and dynamic phase offset adjustment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of signal synchronization, and more particularly, to a multi-camera anti-shake timing synchronization control system and method based on FPGA. BACKGROUND

[0002] In the fields of automatic driving, security monitoring, virtual reality (VR), etc., multi-camera collaborative work has become a core solution for obtaining panoramic view and three-dimensional environment perception. For example, an automatic driving vehicle needs to splice road images in real time through front-view, side-view, rear-view, etc. multi-camera to realize obstacle detection and path planning; a VR device needs to simulate human eye parallax through double cameras to build immersive experience through picture fusion. These scenes put forward strict requirements on the timing synchronization accuracy and dynamic anti-shake performance of multi-camera: if the timing is not synchronized, the time difference of multi-view images will exceed 100ns, causing misalignment of splicing or three-dimensional reconstruction error; if anti-shake fails, picture shaking during fast motion will reduce target recognition accuracy.

[0003] A multi-camera dynamic synchronization exposure circuit and method of a photogrammetry system based on FPGA disclosed in Chinese Patent No. CN107277389B collect a photographing trigger instruction of each camera, record the different synchronization time of shutter actions of each camera according to the obtained camera photographing trigger signal, take the last arrived shutter action signal as a reference, calculate the synchronization time difference value of other cameras and the camera to which the standard shutter action signal belongs, delay the trigger signal of the shutter of other cameras according to the synchronization time difference value, and generate a synchronization photographing trigger signal of each camera. The present application processes the shutter signals of multiple cameras to make them synchronized, ensures accurate synchronization during shooting, enables simultaneous exposure of cameras, makes each photography center coincide, facilitates the splicing and post-processing of images of multiple cameras, greatly accelerates the image processing speed, and reduces the indoor working time.

[0004] Although the above method can meet most scenes, research and actual application of the above method and prior art show that the above method and prior art at least have the following defects:

[0005] The FPGA processing module of the above method records the arrival time difference of shutter trigger signals of each camera, and performs fixed delay adjustment taking the last arrived signal as a reference. The control thereof only depends on static time difference compensation and cannot cope with dynamic changes of cameras. Moreover, the fixed delay parameter cannot be adjusted in real time, which will cause synchronization error to accumulate with time.

[0006] In view of this, the present application proposes a multi-camera anti-shake timing synchronization control system and method based on FPGA to solve the above problems. SUMMARY

[0007] In order to overcome the above-mentioned defects of the prior art, in order to achieve the above-mentioned purposes, the present application provides the following technical scheme: a multi-camera anti-shake timing synchronization control method based on FPGA, comprising the following steps:

[0008] Generate FSYNC signal based on FPGA as the exposure starting reference of each camera; record the exposure start time and anti-shake frame output time through timestamp embedding, and calculate the delay measurement value by combining the differential measurement method;

[0009] Collect the IMU data of each camera, and calculate the global motion vector by optimizing through the gradient descent algorithm;

[0010] According to the delay measurement value and the global motion vector, the phase offset is obtained and fed back to the FPGA;

[0011] The FPGA adjusts the FSYNC signal according to the feedback phase offset, and obtains the adjusted FSYNC signal combined with the global motion vector;

[0012] Real-time monitoring of the adjusted FSYNC signal and the phase offset, based on the monitoring result, the phase offset is voted and optimized, the optimized offset is obtained and fed back to the FPGA for anti-shake timing synchronization control.

[0013] Further, the method for obtaining the phase offset comprises:

[0014] Control the programmable delay line of the FPGA to traverse all possible delay values with the hardware minimum resolution as the step, record the corresponding anti-shake frame output time for each delay value; count the output time fluctuation corresponding to all delay values, select the continuous interval with the smallest fluctuation, and set the midpoint value of the corresponding continuous interval as the target delay reference;

[0015] Obtain the delay measurement value of the current frame, and calculate the jitter error combined with the target delay reference;

[0016] Extract the global translation component and the global rotation component from the global motion vector, calculate and normalize to obtain the motion intensity index;

[0017] Dynamically adjust the PID coefficient according to the motion intensity index; the PID coefficient includes the proportional coefficient, the integral coefficient and the differential coefficient;

[0018] According to the current frame jitter error, the last frame jitter error and the cumulative sum of the previous k frame errors, and combining the PID control formula, the basic phase offset is calculated;

[0019] Optimize the basic phase offset based on the Kalman filtering algorithm to obtain the compensation amount; limit the compensation amount to obtain the phase offset.

[0020] Further, the method for dynamically adjusting the PID coefficients according to the motion intensity index comprises:

[0021] The base proportional coefficient is calculated according to the maximum proportional coefficient, the minimum proportional coefficient and the motion intensity index;

[0022] The base integral coefficient is calculated according to the maximum integral coefficient, the minimum integral coefficient and the motion intensity index;

[0023] The base differential coefficient is calculated according to the maximum differential coefficient, the minimum differential coefficient and the motion intensity index;

[0024] The deviation of the actual delay of the current frame from the target reference is calculated to obtain a compensation residual, the residual feedback coefficient is calculated based on the compensation residual and the maximum compensation residual, and the base proportional coefficient, the base integral coefficient and the base differential coefficient are respectively modified based on the residual feedback coefficient to obtain the proportional coefficient, the integral coefficient and the differential coefficient.

[0025] Further, the method for limiting the compensation amount to obtain the phase offset comprises:

[0026] If the compensation amount is less than 0, the phase offset is 0;

[0027] If the compensation amount is not less than 0 and not greater than the preset compensation threshold, the phase offset is equal to the compensation amount;

[0028] If the compensation amount is greater than the preset compensation threshold, the phase offset is the preset compensation threshold.

[0029] Further, the method for adjusting the FSYNC signal comprises:

[0030] The ROI parameter of the current frame is obtained according to the global motion vector update;

[0031] The image is processed according to the ROI parameter, and the anti-shake frame is output;

[0032] The ROI processing delay is calculated according to the full-frame processing delay and the ROI ratio;

[0033] The interval from the exposure start time of each frame in the continuous N frames to the anti-shake frame output time is measured, and the average value is calculated to obtain the full-frame processing delay;

[0034] The ROI processing delay is calculated according to the full-frame processing delay, the proportional coefficient and the ROI ratio;

[0035] The actual delay of the current frame is calculated based on the random jitter and the ROI processing delay;

[0036] The phase offset amount of compensation is calculated based on the actual delay of the current frame and the target delay reference;

[0037] The compensated phase offset is converted into a control word according to a resolution, so that the FSYNC signal controls triggering of exposure of a next frame according to the compensated phase offset, and an adjusted FSYNC signal is obtained.

[0038] Further, the method for obtaining the ROI parameter of the current frame comprises:

[0039] According to the maximum region ratio, the minimum region ratio, the limit attenuation coefficient and the motion intensity, an ROI ratio is calculated and obtained.

[0040] According to the image resolution W*H of the full frame image, the original ROI center (W / 2, H / 2), the motion following coefficient and the global motion vector, an offset center coordinate is calculated and obtained.

[0041] The ROI ratio and the offset center coordinate are spliced to obtain the ROI parameter.

[0042] Further, the method for obtaining the optimized offset comprises:

[0043] A phase offset A is obtained based on the preset master compensation unit based on the method for obtaining the phase offset, and a phase offset B is obtained based on the preset slave compensation unit based on the method for obtaining the phase offset.

[0044] It is judged whether A and B are valid calculation results.

[0045] A voting threshold is calculated based on a preset minimum voting threshold, a reference fluctuation coefficient, a global motion intensity preparation and a maximum value of historical fluctuations of the master-slave unit.

[0046] According to the effectiveness of A and B and the corresponding difference value, the voting threshold is compared, and the optimized offset is output according to the comparison result.

[0047] Further, the method for judging whether A and B are valid calculation results comprises:

[0048] It is judged whether the delay value is within a preset physical range, and if yes, the delay value is determined to be valid, otherwise, it is determined to be invalid.

[0049] It is judged whether each component of the global motion vector satisfies the mechanical limit constraint of the camera, and if yes, the global motion vector is determined to be valid, otherwise, it is determined to be invalid.

[0050] If either the delay value or the global motion vector is determined to be invalid, a sub-flag 1 is marked as abnormal, otherwise, the sub-flag 1 is marked as normal.

[0051] It is judged whether the output of the PID controller is within the range of the hardware delay line, and if yes, the output of the PID controller is determined to be valid, otherwise, it is determined to be invalid.

[0052] determining whether the output of the Kalman filtering algorithm exceeds a preset output threshold, if yes, determining that the output of the PID controller is invalid, otherwise determining that the output of the PID controller is valid;

[0053] if either the output of the PID controller or the output of the Kalman filtering algorithm is determined to be invalid, marking sub-flag 2 as abnormal, otherwise marking sub-flag 2 as normal;

[0054] determining whether the difference between the completion time and the start time is greater than a preset frame period, if yes, determining that sub-flag 3 is abnormal, otherwise marking sub-flag 3 as normal;

[0055] counting the number of abnormalities of sub-flag 1, sub-flag 2 and sub-flag 3 in the last R frames, if the number of sub-flag abnormalities is greater than an abnormality threshold, determining that sub-flag 4 is abnormal, otherwise marking sub-flag 4 as normal;

[0056] comprehensively determining the state of each of the four sub-flags of A or B, when the four sub-flags are all normal, determining that the state is normal, otherwise determining that the state is abnormal.

[0057] Further, the method for obtaining the global motion vector comprises:

[0058] collecting motion vectors and IMU data of each camera, the motion vector including a translation component and a rotation component of the camera relative to the previous frame; the IMU data including acceleration and angular velocity;

[0059] performing data alignment and filtering processing on the motion vector and the IMU data to obtain standard motion vectors and standard IMU data;

[0060] selecting the bth camera as a reference camera, and taking the corresponding standard motion vector as a global reference vector;

[0061] converting the standard motion vectors of each camera to a global coordinate system based on its own coordinate system to obtain global motion vectors; wherein the own coordinate system is established with the optical center of the camera as the origin;

[0062] setting a global transformation matrix, defining an objective function in combination with the weight of the ith camera, the global reference vector and the conversion vector; the global transformation matrix makes the global motion vectors of all cameras have the minimum deviation from the reference vector after being transformed by the matrix; the optimal global transformation matrix is obtained by iterative solution through gradient descent method, and the global motion vector is obtained by applying the optimal global transformation matrix to the reference vector.

[0063] The multi-camera anti-shake time sequence synchronization control system based on FPGA implements the multi-camera anti-shake time sequence synchronization control method based on FPGA, comprising:

[0064] Benchmarking module: generate FSYNC signal based on FPGA as the exposure starting reference of each camera; record the exposure start time and anti-shake frame output time through timestamp embedding, and obtain the delay measurement value by combining the difference measurement method;

[0065] Motion analysis module: collect IMU data of each camera, and obtain the global motion vector by gradient descent algorithm optimization;

[0066] Offset analysis module: analyze the delay measurement value and the global motion vector to obtain the phase offset and feedback to the FPGA;

[0067] Signal adjustment module: FPGA adjusts FSYNC signal according to the feedback phase offset and combines the global motion vector to obtain the adjusted FSYNC signal;

[0068] Offset optimization module: real-time monitor the adjusted FSYNC signal and the phase offset, and vote optimization on the phase offset based on the monitoring result to obtain the optimized offset and feedback to the FPGA for anti-shake timing synchronization control.

[0069] The technical effects and advantages of the FPGA-based multi-camera anti-shake timing synchronization control system and method of the application are as follows:

[0070] The application generates FSYNC signal as exposure reference through FPGA, combines timestamp difference measurement method to capture and process the dynamic change of delay in real time, eliminates clock domain conversion error; then collects IMU data and motion vector, and obtains global motion vector through gradient descent algorithm optimization, provides unified motion reference for multi-camera cooperation; then dynamically adjusts PID coefficient based on motion intensity index, and combines Kalman filter and amplitude limiting processing to realize accurate calculation of phase offset; finally, through master-slave dual path verification and dynamic voting optimization mechanism, the reliability of the phase offset is ensured; the closed-loop design of "measurement-analysis-compensation-verification" makes the system can respond to camera motion changes, process dynamic factors such as load fluctuation in real time, reduce synchronization error, compress single frame calculation time, meet the real-time requirement of camera, at the same time, through multi-level abnormality detection mechanism, the system robustness is greatly improved, realizes the upgrade from passive alignment to active adaptation, provides high-precision and high-reliability solution for multi-camera anti-shake timing synchronization. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 The flowchart of the FPGA-based multi-camera anti-shake timing synchronization control method of the application is shown in the figure;

[0072] Figure 2 The data flow diagram of the application is shown in the figure;

[0073] Figure 3A data flow direction schematic diagram of embodiment 2 of the present application;

[0074] Figure 4 A structure schematic diagram of the FPGA-based multi-camera anti-shake timing synchronization control system of the present application. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0076] Embodiment 1

[0077] Please refer to Figure 1 、 Figure 2 The present embodiment provides a FPGA-based multi-camera anti-shake timing synchronization control method, which comprises the following steps:

[0078] The FSYNC signal is generated based on FPGA, serving as the exposure starting reference of each camera. The exposure start time and the anti-shake frame output time are recorded by time stamp embedding, and the delay measurement value is obtained by combining the differential measurement method.

[0079] The method for obtaining the delay measurement value comprises:

[0080] The exposure start time and the anti-shake frame output time are recorded by the time stamp register. The 32-bit counter of the time stamp is driven by the FPGA internal PLL, and the counting reference is homologous to the FSYNC signal of the trigger layer. The exposure start time is strictly aligned with the FSYNC signal, and the anti-shake frame output time is bound with the DDR write signal of the anti-shake module, which is used to capture the real output time.

[0081] The rising edge of the FSYNC signal is marked as the exposure start time, and the DOR write completion signal is marked as the anti-shake frame output time. The delay measurement value is obtained according to the exposure start time and the anti-shake frame output time, and the start time of the local reference path and the end time of the local reference path recorded by the FPGA time stamp synchronization.

[0082] The method strictly aligns the exposure start time with the FSYNC signal, binds the anti-jitter frame output time with the DDR write signal in real time, and realizes nanosecond-level timestamp recording by combining a 32-bit synchronization counter, thereby fundamentally solving the limitations of the prior art relying on static time difference compensation. The method uses the rising edge of the FSYNC as the exposure start reference and the DDR write completion signal as the output termination marker to build a dynamic delay measurement system covering the whole process of "exposure-processing-output", so that the system can capture the delay fluctuation caused by the change of the camera state in real time, and obtain the real processing delay through differential calculation. The design of the timestamp counter and the trigger layer FSYNC eliminates the clock domain conversion error, ensures that the delay measurement value can accurately reflect the dynamic characteristics of the current processing link, provides accurate input for subsequent dynamic phase offset adjustment, thereby effectively suppressing the time accumulation effect of synchronization error and realizing real-time response and closed-loop control of the dynamic change of the camera.

[0083] The IMU data of each camera is collected, and a global motion vector is calculated by gradient descent algorithm optimization;

[0084] The method for obtaining the global motion vector comprises:

[0085] The motion vector and the IMU data of each camera are collected, the motion vector includes the translation component and the rotation component of the camera relative to the previous frame, and the IMU data includes the acceleration and the angular velocity;

[0086] The motion vector and the IMU data are subjected to data alignment and filtering processing to obtain a standard motion vector and a standard IMU data;

[0087] The bth camera is selected as a reference camera, and the corresponding standard motion vector is selected as a global reference vector;

[0088] The standard motion vectors of the cameras are converted to a global coordinate system based on their own coordinate systems to obtain global motion vectors, wherein the own coordinate system is established with the optical center of the camera as the origin;

[0089] A global transformation matrix is set, and a target function is defined in combination with the weight of the ith camera, the global reference vector and the conversion vector; the global transformation matrix makes the global motion vectors of all cameras have the minimum deviation from the reference vector after being transformed by the matrix; the optimal global transformation matrix is obtained by iterative solution through the gradient descent method, and the global motion vector is obtained by applying the optimal global transformation matrix to the reference vector.

[0090] The IMU data and motion vectors of each camera are collected and filtered, and the global motion vector is obtained by selecting a reference camera, converting the coordinate system to a global coordinate system, and optimizing the gradient descent algorithm. The method breaks through the limitation of the prior art which only relies on static time difference compensation. The method can accurately reflect the motion consistency and difference of multiple cameras in a dynamic scene by capturing the dynamic motion state of each camera in real time and integrating the dispersed local motion information into a unified global motion reference. Based on the global motion vector, the system can dynamically adjust the synchronization strategy according to the real-time motion characteristics of each camera, thereby avoiding the accumulation of synchronization errors caused by dynamic changes of the camera and realizing real-time perception and cooperative control of the dynamic motion state of the multiple cameras, which provides an accurate motion reference for subsequent dynamic delay compensation.

[0091] According to the delay measurement value and the global motion vector, the phase offset is obtained and fed back to the FPGA;

[0092] The method for obtaining the phase offset comprises:

[0093] The programmable delay line of the FPGA is controlled to traverse all possible delay values with a step of the hardware minimum resolution, and the output time corresponding to each delay value is recorded; the output time fluctuations corresponding to all delay values are counted, and the continuous interval with the smallest fluctuation is selected, and the midpoint value of the continuous interval is set as the target delay reference;

[0094] The delay measurement value of the current frame is obtained, and the jitter error is calculated based on the target delay reference; if the jitter error is greater than 0, the next frame is triggered in advance; if the jitter error is less than 0, the next frame is triggered later;

[0095] The global translation component and the global rotation component are extracted from the global motion vector, and the motion intensity index is calculated and normalized;

[0096] The PID coefficient is dynamically adjusted according to the motion intensity index; the PID coefficient includes a proportional coefficient, an integral coefficient and a differential coefficient;

[0097] The method for dynamically adjusting the PID coefficient according to the motion intensity index comprises:

[0098] The basic proportional coefficient is calculated based on the maximum proportional coefficient, the minimum proportional coefficient and the motion intensity index;

[0099] The basic integral coefficient is calculated based on the maximum integral coefficient, the minimum integral coefficient and the motion intensity index;

[0100] The basic differential coefficient is calculated based on the maximum differential coefficient, the minimum differential coefficient and the motion intensity index;

[0101] The deviation of the actual delay of the current frame from the target reference is calculated to obtain a compensation residual, a residual feedback coefficient is calculated based on the compensation residual and a maximum value of the compensation residual, and the base proportional coefficient, the base integral coefficient and the base differential coefficient are respectively modified based on the residual feedback coefficient to obtain the proportional coefficient, the integral coefficient and the differential coefficient.

[0102] The method for dynamically adjusting the PID coefficient according to the motion intensity index effectively breaks through the limitation that the fixed parameters in the prior art cannot adapt to dynamic changes by taking both the motion intensity and the compensation effect into the coefficient adjustment logic. The method first takes the motion intensity index as the core, calculates the base proportional, integral and differential coefficients in combination with the maximum / minimum values of each coefficient, so that the base trend of the coefficients can directly respond to the degree of camera motion, and thus the adaptation of static parameters to dynamic scenes is avoided from the source. Then, the residual feedback coefficient is calculated through the compensation residual to modify the base coefficient in real time. The better the compensation effect (the smaller the compensation residual), the smaller the modification range, and the worse the compensation effect (the larger the compensation residual), the larger the modification range, forming a closed loop of “motion state→base coefficient→real-time effect modification”. This dynamic adjustment mechanism enables the PID coefficient to match the dynamic changes of the camera in real time, ensures that the response characteristics of the proportional coefficient, the integral coefficient and the differential coefficient are always consistent with the current scene requirements, and avoids the response lag or overshoot problem of fixed parameters in dynamic scenes, thereby effectively suppressing the time accumulation of synchronization error and achieving precise and real-time adaptation to the dynamic changes of the camera.

[0103] The base phase offset is calculated based on the current frame jitter error, the last frame jitter error and the cumulative sum of the previous k frame errors in combination with the PID control formula;

[0104] The base phase offset is optimized based on the Kalman filtering algorithm to obtain a compensation amount, and the compensation amount is subjected to amplitude limiting to obtain the phase offset.

[0105] The method fundamentally solves the limitation of the prior art relying on static time difference compensation by dynamic reference setting and real-time compensation mechanism. The method first determines a target delay reference through full-range delay scanning to provide a stable reference for dynamic adjustment, and then based on real-time comparison of the current frame delay measurement value and the target reference, accurately calculates the jitter error to capture the dynamic changes of the camera, breaking through the static limitation of fixed delay parameters. At the same time, by extracting the motion intensity index from the global motion vector and dynamically adjusting the PID coefficient, the proportional coefficient, integral coefficient and differential coefficient can adaptively change with the degree of motion, combined with real-time error correction of PID control and jitter prediction of Kalman filter, the dynamic optimization of the basic phase offset is realized, and finally the phase offset obtained through amplitude limiting processing can be fed back to the FPGA to adjust the trigger signal. This process forms a closed loop of "measurement-analysis-compensation", ensuring that the compensation amount of each frame period can accurately match the current camera state, effectively suppressing the time accumulation effect of synchronization error, and realizing real-time response and active correction to the dynamic changes of the camera.

[0106] The method for amplitude limiting the compensation amount to obtain the phase offset comprises:

[0107] If the compensation amount is less than 0, the phase offset is 0;

[0108] If the compensation amount is not less than 0 and not greater than a preset compensation threshold, the phase offset is equal to the compensation amount;

[0109] If the compensation amount is greater than the preset compensation threshold, the phase offset is the preset compensation threshold.

[0110] The method for amplitude limiting the compensation amount to obtain the phase offset provides a reliable physical boundary for the dynamic adjustment mechanism by constraining the compensation amount within the effective range that can be realized by hardware, effectively compensating for the limitations of fixed delay parameters in dynamic scenes in the prior art. In the above method, when the compensation amount is less than 0, the phase offset is forced to be 0, avoiding synchronization disorder caused by invalid negative delay adjustment; when the compensation amount is between 0 and the preset threshold, the real-time calculated value is directly used, retaining the flexibility of dynamically responding to camera changes; when the compensation amount exceeds the threshold, the preset maximum value is taken, preventing compensation failure caused by hardware over-limiting. This processing not only ensures that the phase offset can follow the dynamic changes of the camera in real time, but also avoids error accumulation caused by excessive compensation or invalid compensation by hardware constraints, so that the adjustment of each frame period is accurately effective within the physically feasible range, thereby suppressing the divergence of synchronization error over time and achieving a balance between dynamic compensation and hardware capability.

[0111] The FPGA adjusts the FSYNC signal based on the feedback phase offset, and obtains an adjusted FSYNC signal;

[0112] The method for obtaining the adjusted FSYNC signal comprises:

[0113] updating the ROI parameter of the current frame according to the global motion vector;

[0114] calculating the ROI ratio according to the maximum area ratio, the minimum area ratio, the limit attenuation coefficient and the motion intensity;

[0115] calculating the offset center coordinates according to the image resolution W*H of the full frame image, the original ROI center (W / 2, H / 2), the motion following coefficient and the global motion vector;

[0116] splicing the ROI ratio and the offset center coordinates to obtain the ROI parameter;

[0117] splicing the ROI ratio and the offset center coordinates to obtain the ROI parameter;

[0118] calculating the ROI processing delay according to the full frame processing delay and the ROI ratio;

[0119] measuring the interval from the exposure start time to the anti-shake frame output time of each frame in the continuous N frames, calculating the average value to obtain the full frame processing delay;

[0120] calculating the ROI processing delay according to the full frame processing delay, the ratio coefficient and the ROI ratio; the ratio coefficient is obtained by least square fitting;

[0121] calculating the actual delay of the current frame based on the random jitter and the ROI processing delay; the random jitter is obtained by statistical modeling to obtain the mean value, and then optimized based on the Kalman filtering algorithm;

[0122] calculating the compensated phase offset based on the actual delay of the current frame and the target delay reference;

[0123] converting the compensated phase offset into a control word according to the resolution, so that the FSYNC signal controls triggering the exposure of the next frame according to the compensated phase offset, to obtain the adjusted FSYNC signal.

[0124] The method for adjusting the FSYNC signal by the FPGA based on the global motion vector completely breaks through the limitation of the static time difference compensation of the prior art. The method first dynamically updates the ROI parameter based on the global motion vector, calculates the ROI ratio according to the motion intensity, adjusts the ROI center coordinate in combination with the motion direction, so that the ROI parameter can match the dynamic change of the camera in real time; then calculates the ROI processing delay by measuring the full-frame processing delay in combination with the proportional coefficient fitted by the least square method, ensures that the delay calculation is in line with the actual hardware characteristics, and introduces the Kalman filter optimization with random jitter to accurately capture the actual delay of the current frame. On this basis, the phase offset is calculated based on the deviation of the actual delay and the target reference, and is converted into a control word to adjust the FSYNC signal, forming a complete closed loop of "motion state -> ROI adaptation -> delay measurement -> phase compensation -> signal adjustment". This process makes the adjustment of the FSYNC signal no longer dependent on fixed delay parameters, but can respond to dynamic factors such as camera motion changes and processing load fluctuations in real time. The adjustment of each frame period is based on the current actual state, effectively blocking the accumulation path of the synchronization error, and achieving accurate adaptation and active control of the dynamic scene.

[0125] Real-time monitoring of the adjusted FSYNC signal and the phase offset, voting optimization of the phase offset based on the monitoring results, obtaining the optimized offset and feeding back to the FPGA for anti-shake timing synchronization control.

[0126] The method for obtaining the optimized offset comprises:

[0127] The preset master compensation unit obtains a phase offset A based on the method for obtaining the phase offset, and the preset slave compensation unit obtains a phase offset B based on the method for obtaining the phase offset;

[0128] Judging whether A and B are valid calculation results;

[0129] The method for judging whether A and B are valid calculation results comprises:

[0130] Judging whether the delay value is within a preset physical range, if yes, determining that the delay value is valid, otherwise determining that it is invalid;

[0131] Judging whether each component of the global motion vector meets the mechanical limit constraint of the camera, if yes, determining that the global motion vector is valid, otherwise determining that it is invalid;

[0132] If either the delay value or the global motion vector is determined to be invalid, the sub-flag 1 is marked as abnormal, otherwise the sub-flag 1 is marked as normal;

[0133] Judging whether the output of the PID controller is within the hardware delay line range, if yes, determining that the output of the PID controller is valid, otherwise determining that it is invalid;

[0134] determining whether the output of the Kalman filtering algorithm exceeds a preset output threshold, if yes, determining that the output of the PID controller is invalid, otherwise determining that it is valid;

[0135] If either the output of the PID controller or the output of the Kalman filtering algorithm is determined to be invalid, mark sub-flag 2 as abnormal, otherwise mark sub-flag 2 as normal;

[0136] determining whether the difference between the completion time and the start time is greater than a preset frame period, if yes, determining that sub-flag 3 is abnormal, otherwise marking sub-flag 3 as normal;

[0137] counting the number of abnormalities of sub-flag 1, sub-flag 2 and sub-flag 3 in the last R frames, if the number of sub-flag abnormalities is greater than an abnormality threshold, determining that sub-flag 4 is abnormal, otherwise marking sub-flag 4 as normal;

[0138] comprehensive determination is made according to the state of each of the four sub-flags of A or B, when all four sub-flags are normal, the state is determined to be normal, otherwise the state is determined to be abnormal.

[0139] The method for determining whether A and B are valid calculation results provides a reliable effectiveness verification mechanism for dynamic compensation through full-link abnormality detection of input data, calculation process, time constraints and historical trends, effectively making up for the defects of the lack of data effectiveness judgment in the static compensation of the prior art. The method first checks whether the delay value is within a preset range and whether the global motion vector conforms to the mechanical limit from the perspective of physical reasonableness, ensuring the adaptability of the input data to the dynamic state of the camera; then it checks whether the PID output is within the hardware range and whether the Kalman filtering result is out of range, ensuring the compliance of the calculation process; at the same time, it ensures that the calculation is completed within the frame period through time constraint judgment, meeting the real-time requirement; finally, it identifies continuous faults by combining the historical abnormality statistics of the last R frames to avoid misjudgment caused by transient interference. This multi-level detection forms a three-dimensional verification of "input-process-time-history", ensuring that A and B are only determined to be valid when there is no abnormality in the whole link, eliminating incorrect data from the source into the compensation process, avoiding the accumulation of synchronization errors caused by invalid parameters, providing reliable basic data for the dynamic adjustment mechanism, and enabling the optimization of phase shift to truly respond to the dynamic changes of the camera, rather than being misled by abnormal values.

[0140] The voting threshold is calculated based on a preset minimum voting threshold, a reference fluctuation coefficient, a global motion intensity preparation, and a maximum value of the historical fluctuations of the master and slave units;

[0141] According to the effectiveness of A and B and the corresponding difference, the voting threshold is compared, and the optimized offset is output according to the comparison result;

[0142] The method for outputting the optimized offset according to the comparison result includes:

[0143] When A and B are both valid, and the difference between A and B is not higher than the voting threshold, the weighted average of the two is taken as the optimized offset; the weighting weight can be obtained based on natural heuristic optimization algorithm optimization;

[0144] When A and B are both valid, and the difference between A and B is higher than the voting threshold, the average value of the phase offset of the near M frames is taken as the reference, and the more reliable value is selected according to the deviation of the current master-slave unit and the historical trend;

[0145] When only one of A and B is valid, the output of the valid unit is directly used, and a reliability correction amount is superimposed, which can be obtained based on natural heuristic optimization algorithm optimization, as the optimized offset;

[0146] When A and B are both invalid, the average value of the phase offset of the near M frames is directly output as the optimized offset, and the system alarm is triggered at the same time, and the master-slave unit is forced to restart.

[0147] The method of real-time monitoring the adjusted FSYNC signal and the phase offset and performing voting optimization to obtain the final phase offset breaks through the limitation of the static compensation of the prior art that cannot cope with dynamic changes through the master-slave double-path verification and dynamic decision mechanism. In the method, the master-slave compensation units calculate the phase offset A and B respectively, and the reliability of the input data is ensured in combination with the validity judgment, and the voting threshold is dynamically adjusted based on the global motion intensity and the history fluctuation, so that the judgment standard can adapt to the camera motion state in real time. For the optimization strategy of different scenes, when both A and B are valid and the deviation is small, the weighted average is used, when the deviation is too large, the more reliable value is selected in combination with the historical trend, when only one of A and B is valid, the historical average value is approached through the reliability correction amount, and when both A and B are invalid, the historical average value is used to form a multi-level fault tolerance mechanism. This design not only captures the dynamic changes of the phase offset through real-time monitoring, but also eliminates abnormal values and integrates valid information through voting optimization, so that the final phase offset can accurately match the current camera state, avoiding the problem that the error accumulates with dynamic changes under fixed delay parameters, and realizing the upgrade from "passive alignment" to "active adaptation", which provides a dynamic compensation basis with stronger robustness for anti-shake timing synchronization control.

[0148] Embodiment 2:

[0149] Please refer to Figure 3 As shown in FIG. 1, the embodiment provides a cross-camera dynamic deviation calibration method applied to embodiment 1, which includes the following steps:

[0150] Taking the FSYNC signal of the bth camera as a reference, the actual exposure time difference of other cameras is measured, and individual deviation compensation values are calculated. In actual adjustment, the individual deviation compensation value is superimposed as the final offset for the optimized offset of each camera and fed back to the FPGA for anti-shake timing synchronization control.

[0151] The method dynamically captures actual exposure differences of each camera caused by individual differences of hardware, converts static hardware deviation into individual compensation values which can be adjusted in real time, eliminates fixed deviation caused by hardware differences by superimposing exclusive individual compensation values, avoids synchronization error caused by individual differences accumulated over time under a single fixed delay parameter, and makes the final offset of each camera adapt to dynamic changes and correct hardware characteristic differences, thereby realizing the cooperation of dynamic response and individual calibration and effectively improving the precision and stability of multi-camera anti-shake timing synchronization control.

[0152] Embodiment 3:

[0153] Referring to Figure 4 The embodiment provides a multi-camera anti-shake timing synchronization control system based on FPGA, which comprises:

[0154] The reference measurement module generates an FSYNC signal based on FPGA as a reference for starting exposure of each camera, records exposure start time and anti-shake frame output time through timestamp embedding, and calculates a delay measurement value by combining a differential measurement method;

[0155] The motion analysis module collects IMU data of each camera, optimizes the data through a gradient descent algorithm, and calculates a global motion vector;

[0156] The offset analysis module analyzes the delay measurement value and the global motion vector, obtains a phase offset, and feeds back the phase offset to the FPGA;

[0157] The signal adjustment module adjusts the FSYNC signal based on the feedback phase offset and the global motion vector, and obtains an adjusted FSYNC signal;

[0158] The offset optimization module monitors the adjusted FSYNC signal and the phase offset in real time, optimizes the phase offset based on the monitoring result, obtains an optimized offset, and feeds back the optimized offset to the FPGA for anti-shake timing synchronization control.

[0159] The above merely describes specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0160] Finally, the above merely describes preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multi-camera anti-shake timing synchronization control method based on FPGA, characterized in that, The method comprises the following steps: generating an FSYNC signal based on an FPGA as a starting reference for exposure of each camera; recording exposure start time and anti-shake frame output time through timestamp embedding, and calculating a delay measurement value by combining a differential measurement method; collecting IMU data of each camera, and calculating a global motion vector through gradient descent algorithm optimization; analyzing the delay measurement value and the global motion vector to obtain a phase offset and feed back to the FPGA; adjusting the FSYNC signal based on the feedback phase offset and combining the global motion vector to obtain an adjusted FSYNC signal; real-time monitoring of the adjusted FSYNC signal and the phase offset, voting optimization of the phase offset based on the monitoring result, obtaining an optimized offset and feeding back to the FPGA for anti-shake timing synchronization control.

2. The FPGA-based multi-camera anti-shake timing synchronization control method according to claim 1, characterized in that, The method for obtaining the phase offset comprises: controlling the programmable delay line of the FPGA to traverse all delay values with a hardware minimum resolution as a step, recording the corresponding anti-shake frame output time for each delay value; counting the output time fluctuations corresponding to all delay values, selecting the continuous interval with the smallest fluctuation, and setting the midpoint value of the corresponding continuous interval as a target delay reference; obtaining the delay measurement value of the current frame, combining the target delay reference to calculate a jitter error; extracting a global translation component and a global rotation component from the global motion vector, calculating and normalizing to obtain a motion intensity index; dynamically adjusting the PID coefficient according to the motion intensity index; the PID coefficient includes a proportional coefficient, an integral coefficient and a differential coefficient; calculating a basic phase offset based on the current frame jitter error, the last frame jitter error and the cumulative sum of the previous k frame errors, and combining the PID control formula; optimizing the basic phase offset based on the Kalman filtering algorithm to obtain a compensation amount; limiting the compensation amount to obtain the phase offset.

3. The FPGA-based multi-camera anti-shake timing synchronization control method according to claim 2, characterized in that, The method for dynamically adjusting the PID coefficient according to the motion intensity index comprises: calculating a basic proportional coefficient based on the maximum proportional coefficient, the minimum proportional coefficient and the motion intensity index; calculating a basic integral coefficient based on the maximum integral coefficient, the minimum integral coefficient and the motion intensity index; calculating a basic differential coefficient based on the maximum differential coefficient, the minimum differential coefficient and the motion intensity index; calculating the deviation of the actual delay of the current frame from the target reference to obtain a compensation residual, calculating a residual feedback coefficient based on the compensation residual and the maximum compensation residual, and modifying the basic proportional coefficient, the basic integral coefficient and the basic differential coefficient based on the residual feedback coefficient to obtain the proportional coefficient, the integral coefficient and the differential coefficient.

4. The FPGA-based multi-camera anti-shake timing synchronization control method according to claim 2, characterized in that, The method for limiting the compensation amount to obtain the phase offset comprises: if the compensation amount is less than 0, the phase offset is 0; if the compensation amount is not less than 0 and not greater than a preset compensation threshold, the phase offset is equal to the compensation amount; if the compensation amount is greater than the preset compensation threshold, the phase offset is the preset compensation threshold.

5. The FPGA-based multi-camera anti-shake timing synchronization control method according to claim 1, characterized in that, The method for adjusting the FSYNC signal comprises: updating the ROI parameter of the current frame based on the global motion vector; processing the image based on the ROI parameter to output an anti-shake frame; calculating an ROI processing delay based on the full-frame processing delay and the ROI ratio; The interval from the exposure start time to the anti-shake frame output time of each frame in the continuous N frames is measured, an average value is calculated, and a full-frame processing delay is obtained; The ROI processing delay is calculated according to the full-frame processing delay, the scale factor and the ROI scale; The actual delay of the current frame is calculated based on the random jitter and the ROI processing delay; The compensated phase offset is calculated based on the actual delay of the current frame and the target delay reference; The compensated phase offset is converted into a control word according to the resolution, and the FSYNC signal controls the triggering of the next frame exposure according to the compensated phase offset, to obtain an adjusted FSYNC signal.

6. The FPGA-based multi-camera anti-shake timing synchronization control method according to claim 5, characterized in that, The method for obtaining the ROI parameter of the current frame comprises: The ROI scale is calculated according to the maximum region scale, the minimum region scale, the limit attenuation coefficient and the motion intensity; The offset center coordinates are calculated according to the image resolution W*H of the full-frame image, the original ROI center (W / 2, H / 2), the motion following coefficient and the global motion vector; The ROI parameter is obtained by splicing the ROI scale and the offset center coordinates.

7. The FPGA-based multi-camera anti-shake timing synchronization control method according to claim 1, characterized in that, The method for obtaining the optimized offset comprises: The phase offset A is obtained by the preset master compensation unit based on the method for obtaining the phase offset, and the phase offset B is obtained by the preset slave compensation unit based on the method for obtaining the phase offset; It is judged whether A and B are valid calculation results; The voting threshold is calculated based on the preset minimum voting threshold, the reference fluctuation coefficient, the global motion intensity preparation and the maximum value of the master-slave unit historical fluctuation; The optimized offset is output according to the comparison between the validity of A and B and the corresponding difference value and the voting threshold.

8. The FPGA-based multi-camera anti-shake timing synchronization control method according to claim 7, characterized in that, The method for judging whether A and B are valid calculation results comprises: It is judged whether the delay value is within the preset physical range, if yes, the delay value is determined to be valid, otherwise, it is determined to be invalid; It is judged whether each component of the global motion vector meets the mechanical limit constraint of the camera, if yes, the global motion vector is determined to be valid, otherwise, it is determined to be invalid; If either the delay value or the global motion vector is determined to be invalid, the sub-flag 1 is marked as abnormal, otherwise, the sub-flag 1 is marked as normal; It is judged whether the output of the PID controller is within the hardware delay line range, if yes, the output of the PID controller is determined to be valid, otherwise, it is determined to be invalid; It is judged whether the output of the Kalman filter algorithm exceeds the preset output threshold, if yes, the output of the PID controller is determined to be invalid, otherwise, it is determined to be valid; If either the output of the PID controller or the output of the Kalman filter algorithm is determined to be invalid, the sub-flag 2 is marked as abnormal, otherwise, the sub-flag 2 is marked as normal; It is judged whether the difference between the completion time and the start time is greater than the preset frame period, if yes, the sub-flag 3 is determined to be abnormal, otherwise, the sub-flag 3 is marked as normal; The abnormal times of the sub-flag 1, the sub-flag 2 and the sub-flag 3 of the recent R frames are counted, if the sub-flag abnormal times are greater than the abnormal threshold, the sub-flag 4 is determined to be abnormal, otherwise, the sub-flag 4 is marked as normal; The states of the four sub-flags of A or B are comprehensively judged respectively, when the four sub-flags are all normal, the state is determined to be normal, otherwise, the state is determined to be abnormal.

9. The FPGA-based multi-camera anti-shake timing synchronization control method according to claim 1, characterized in that, The method for obtaining the global motion vector comprises: Collect motion vectors and IMU data of each camera, the motion vector contains the translation component and rotation component of the camera relative to the last frame; the IMU data contains acceleration and angular velocity; Align and filter the motion vector and IMU data to obtain standard motion vector and standard IMU data; Select the bth camera as the reference camera, and the corresponding standard motion vector as the global reference vector; Convert the standard motion vector of each camera to the global coordinate system based on its own coordinate system to obtain the global motion vector; wherein the own coordinate system is established with the optical center of the camera as the origin; Set the global transformation matrix, and define the objective function combining the weight of the ith camera, the global reference vector and the conversion vector; the global transformation matrix makes the deviation of the global motion vector of all cameras after the transformation of the matrix from the reference vector minimum; the optimal global transformation matrix is obtained by iterative solution through gradient descent method, and the optimal global transformation matrix is applied to the reference vector to obtain the global motion vector.

10. A FPGA-based multi-camera anti-shake timing synchronization control system, implementing the FPGA-based multi-camera anti-shake timing synchronization control method of any one of claims 1-9, characterized in that, It includes: Reference measurement module: generate FSYNC signal based on FPGA as the exposure starting reference of each camera; Record the exposure start time and anti-shake frame output time by timestamp embedding, and obtain the delay measurement value by combining the differential measurement method; Motion analysis module: collect IMU data of each camera, and calculate the global motion vector by gradient descent algorithm optimization; Offset analysis module: analyze the delay measurement value and the global motion vector to obtain the phase offset and feedback to the FPGA; Signal adjustment module: FPGA adjusts FSYNC signal according to the feedback phase offset and combines the global motion vector to obtain the adjusted FSYNC signal; Offset optimization module: real-time monitor the adjusted FSYNC signal and the phase offset, vote optimization on the phase offset based on the monitoring result, obtain the optimized offset and feedback to the FPGA for anti-shake timing synchronization control.

Images