Multi-scene adaptive shooting control system of motion camera

Abstract

The invention relates to the technical field of image acquisition and processing, and discloses a multi-scene adaptive shooting control system for a motion camera, and the system comprises an electromechanical coupling vibration feature extraction module which is used for collecting an angular velocity signal of an inertial measurement unit; the inertial view field prediction module is used for predicting the posture of the camera at the exposure moment of the next frame based on the kinematic state space model; the target luminous flux calculation module is used for calculating an ideal photoelectric charge cumulant required by meeting a preset gray scale response according to the brightness estimation value; the discontinuous integral control module is used for generating an integral gating function according to the instantaneous phase; and the dynamic duty ratio gain compensation module is used for calculating effective integration time generated by discontinuous integration. According to the invention, through discontinuous integration under electromechanical coupling control and feedforward exposure based on inertial prediction, stable imaging without distortion and accurate exposure in a composite scene of high-frequency vibration and illumination dramatic change is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image acquisition and processing, in particular to a multi-scene adaptive shooting control system for a motion camera. BACKGROUND

[0002] As a compact shooting device, motion cameras are widely used in various high-dynamic and high-vibration scenes, such as being fixed on unmanned aerial vehicles, racing cars or extreme sports equipment. In such applications, the camera inevitably bears sustained high-frequency mechanical vibration from the engine, propeller or road bumps of the vehicle. For CMOS image sensors that generally use a rolling shutter, this vibration will cause a slight difference between the exposure time of different rows of pixels in the image frame and the camera pose. When the vibration frequency is coupled with the row scanning period of the rolling shutter, image distortion, commonly known as the Jello Effect, will occur, and high-frequency jitter will also cause motion blur, reducing the clarity of the image.

[0003] To suppress the influence of such vibration, the existing technology usually uses electronic image stabilization (EIS) or optical image stabilization (OIS) technology. Electronic image stabilization is a post-processing method that crops the effective imaging area and uses gyroscope data for image compensation, but this will cause loss of frame and reduction of resolution, and cannot fundamentally eliminate the Jello Effect that has occurred in the acquisition stage. Optical image stabilization moves the lens group or sensor through mechanical structure to offset the jitter, and the compensation bandwidth and stroke are usually designed for low-frequency disturbances such as handheld shaking, and for high-frequency, high-amplitude structured vibration generated by devices such as unmanned aerial vehicles, the suppression effect is limited, and it is difficult to completely eliminate the internal distortion caused by the coupling of the rolling shutter and residual vibration.

[0004] In addition, when the motion camera is at high speed through a scene with a dramatic change in lighting conditions (for example, from a forest to an open area), it also faces the problem of automatic exposure (AE) response lag. Traditional automatic exposure systems are mostly feedback control, that is, the exposure parameters (such as shutter time and gain) of the next frame are calculated according to the brightness statistical information of the image frame that has been acquired. This "image first, light later" mode will cause the exposure decision to be based on outdated lighting information when the camera pose or scene lighting changes rapidly, resulting in a short period of overexposure or underexposure in the shooting picture, affecting the continuity of the video stream. SUMMARY

[0005] In view of the deficiencies of the prior art, the motion camera multi-scene adaptive shooting control system is provided, which solves the composite technical problems that the prior art cannot eliminate image distortion from the physical acquisition level in a high-frequency mechanical vibration environment, and the exposure response lag caused by the feedback exposure control mechanism in a high-speed motion scene, so that the imaging clarity and brightness accuracy cannot be ensured at the same time.

[0006] To achieve the above object, the application is implemented by the following technical solutions: The first aspect of the application provides a motion camera multi-scene adaptive shooting control system, which comprises: An electromechanical coupling vibration feature extraction module is used to collect the angular velocity signal of an inertial measurement unit, and extract the mechanical structural vibration feature and instantaneous phase of the carrier through time-frequency analysis; An inertial field of view prediction module is used to predict the attitude of the camera at the next frame exposure time based on a kinematic state space model, and determine the brightness estimation value of the predicted field of view region in the preset ambient light illumination manifold based on the predicted attitude; A target luminous flux calculation module is used to calculate the ideal photocharge accumulation amount required to meet the preset gray response according to the brightness estimation value, and determine the corresponding nominal shutter time and nominal analog gain; A non-continuous integration control module is used to generate an integration gating function according to the instantaneous phase, control the image sensor to perform non-continuous integration in a single frame acquisition period, and the integration gating function only opens the charge accumulation in the vibration relative stationary interval; A dynamic duty cycle gain compensation module is used to calculate the effective integration time generated by non-continuous integration, determine the time loss compensation coefficient according to the ratio of the nominal shutter time to the effective integration time, and adjust the analog gain of the image sensor to compensate for the loss of luminous flux.

[0007] Preferably, the electromechanical coupling vibration feature extraction module specifically comprises a time-frequency transform analysis unit, a main vibration frequency locking unit and a phase information separation unit. The time-frequency transform analysis unit is used to perform short-time Fourier transform on the angular velocity signal to construct a time-frequency distribution function. The main vibration frequency locking unit is used to calculate the power spectral density and search for the energy extreme point in the high-frequency interval to lock the dominant mechanical vibration frequency. The phase information separation unit is used to extract the instantaneous vibration signal of the dominant mechanical vibration frequency component and calculate its instantaneous phase, which is used as the reference benchmark of the non-continuous integration control module.

[0008] In one embodiment, the inertial field-of-view prediction module comprises a state space construction unit, a Kalman filter prediction unit, and an optical axis orientation calculation unit. The state space construction unit is configured to construct a system state vector comprising camera pose quaternion and angular velocity. The Kalman filter prediction unit is configured to iteratively calculate based on the system state vector and state transition matrix, and output a predicted pose quaternion of the next frame. The optical axis orientation calculation unit is configured to calculate the orientation of the camera optical axis vector in the world coordinate system, so as to index the ambient light manifold.

[0009] Further, the Kalman filter prediction unit is configured to perform the following operations: The prior state estimation subunit is configured to call the optimal posterior state estimation value at the previous time and left-multiply the state transition matrix to obtain a prior state prediction vector comprising uncorrected camera pose quaternion components and angular velocity components. The prior covariance calculation subunit is configured to calculate the prior error covariance matrix. The state vector output subunit is configured to intercept the quaternion part in the prior state prediction vector and perform normalization processing, and output the predicted pose quaternion.

[0010] Preferably, the ambient light manifold is a historical brightness statistical model constructed based on a spherical coordinate system. The inertial field-of-view prediction module further comprises a manifold projection query unit and a brightness integral estimation unit, which are respectively configured to determine the spherical range covered by the predicted field-of-view region and perform integral operation on the historical brightness statistical value in the range, so as to obtain the brightness estimation value.

[0011] In one embodiment, the target light flux calculation module comprises an ideal charge quantization unit and a reference parameter mapping unit. The ideal charge quantization unit is configured to calculate the ideal photocharge accumulation amount based on the brightness estimation value, the target image gray value, and the photoelectric conversion efficiency. The reference parameter mapping unit is configured to match the combination of the nominal shutter time and the nominal analog gain according to the reciprocity law principle.

[0012] Further, the reference parameter mapping unit is configured to perform the following operations: The linear region constraint subunit is configured to set the maximum linear analog gain threshold of the image sensor. The reciprocity law matching subunit is configured to search for the parameter combination under the threshold constraint. The parameter priority determination subunit is configured to, when there are multiple combinations that satisfy the condition, preferentially select the combination with the nominal shutter time less than the single-frame time window.

[0013] Preferably, the non-continuous integral control module specifically comprises a stability criterion calculation unit and a gate signal generation unit. The stability criterion calculation unit is configured to calculate a vibration angular velocity module value of the instantaneous vibration signal as a vibration stability criterion function. The gate signal generation unit is configured to compare the criterion function with a preset vibration threshold value, generate an integral gate function in an open state when the criterion function is less than the threshold value, and generate an integral gate function in a closed state otherwise.

[0014] Further, the stability criterion calculation unit specifically performs the following operations: The instantaneous vibration signal is decomposed into three-axis orthogonal vibration components by a quadrature component decomposition subunit; the arithmetic square root of the sum of squares of the three-axis components is calculated by a Euclidean norm operation subunit to obtain the vibration angular velocity module value; and the module value is assigned as the vibration stability criterion function by a criterion function assignment subunit.

[0015] Further, the non-continuous integral control module further comprises a charge well driving unit configured to apply the integral gate function to the pixel control circuit of the image sensor. When the integral gate function is in the open state, the pixel charge well is in a photoelectron capture mode; and when the integral gate function is in the closed state, the pixel charge well is in a charge holding mode and does not perform a reset operation.

[0016] Preferably, the dynamic duty cycle gain compensation module specifically comprises an effective time integration unit, a compensation coefficient solving unit, and a final gain adjustment unit. The effective time integration unit is configured to time-integrate the integral gate function to obtain an effective integration time. The compensation coefficient solving unit is configured to calculate the ratio of the nominal shutter time to the effective integration time to obtain a time loss compensation coefficient. The final gain adjustment unit is configured to multiply the nominal analog gain by the coefficient to generate a final compensation gain, and drive the analog front-end amplification circuit.

[0017] The second aspect of the present application provides a motion camera multi-scene adaptive shooting control method, which comprises the following steps: S1, collecting the angular velocity signal of the inertial measurement unit, extracting the mechanical structural vibration characteristics of the carrier through time-frequency analysis, and separating out the instantaneous phase corresponding to the dominant mechanical vibration frequency; S2, predicting the attitude of the camera at the next frame exposure time based on the kinematic state space model using Kalman filtering, projecting the predicted attitude to the preset environmental light manifold graph, and thereby determining the brightness estimation value of the predicted field of view region; S3, calculating the ideal photocharge accumulation amount required to meet the preset gray scale response according to the brightness estimation value, and determining the corresponding nominal shutter time and nominal analog gain based on the reciprocity law principle; S4, generating an integral gating function according to the instantaneous phase, driving the image sensor to open the charge accumulation only in the relatively stationary interval in a single frame acquisition period, and performing discontinuous integration; S5, calculating the effective integration time generated by the discontinuous integration in real time, calculating the ratio of the nominal shutter time and the effective integration time to obtain a time loss compensation coefficient, adjusting the analog gain of the image sensor by using the time loss compensation coefficient to compensate for the loss of light flux, and outputting a final image.

[0018] The present application provides a motion camera multi-scene adaptive shooting control system. The present application has the following beneficial effects: 1. The present application avoids imaging distortion directly in the photoelectric signal acquisition stage by combining mechanical and electrical coupling vibration feature extraction with discontinuous integration control. The system uses an inertial measurement unit to lock the phase of mechanical vibration, controls the image sensor to open the charge well for integration only in the relatively stationary interval with low vibration speed, and suspends integration in the violent vibration interval. This physical layer timing control method can eliminate the jelly effect and motion blur caused by the coupling of sensor line-by-line scanning and high-frequency mechanical vibration from the source, avoiding the problems of cutting the frame or introducing calculation artifacts in traditional digital anti-shake algorithms.

[0019] 2. The present application uses inertial field of view prediction and environmental light flow graph to solve the automatic exposure lag problem in high-speed motion scenes. By constructing a kinematic state space model and running Kalman filtering, the system can predict the camera pose and the corresponding field of view area at the next exposure time in advance, and directly index the light distribution model constructed historically to obtain the brightness estimate value. This feedforward exposure parameter planning mechanism allows the camera to determine the target light flux before the field of view changes dramatically, ensuring the exposure response speed and accuracy in rapidly alternating light and dark environments.

[0020] 3. The present application establishes a light-mechanical-electrical closed-loop control mechanism based on dynamic duty cycle gain compensation, ensuring the imaging brightness consistency in the discontinuous integration mode. For the effective exposure time lost due to avoiding vibration, the system calculates the duty cycle loss in real time and automatically adjusts the analog gain to compensate for the light flux gap according to the principle of energy conservation. This mechanism deeply couples the mechanical vibration state and the photoelectric conversion parameters, which can maintain the gray response of the output image consistent with the preset target while ensuring the image without distortion, achieving the balance between anti-vibration interference and precise exposure. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The system architecture of the present application is shown in the figure; Figure 2 The method flowchart of the present application is shown in the figure.

[0022] Wherein, 10, electromechanical coupling vibration feature extraction module; 20, inertial field of view prediction module; 30, target light flux calculation module; 40, non-continuous integral control module; 50, dynamic duty cycle gain compensation module. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the specification of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0024] Reference Figures 1-2 , Figure 1 It is an architecture diagram of a motion camera multi-scene adaptive shooting control system according to an embodiment of the present application. The present application provides a motion camera multi-scene adaptive shooting control system, which is configured to cooperatively control the image acquisition timing and analog gain in a dynamic environment.

[0025] The motion camera multi-scene adaptive shooting control system includes a high-frequency inertial measurement unit, a central processor, and a CMOS image sensor. The high-frequency inertial measurement unit is connected with the central processor through a high-speed data bus, for real-time output of motion data of a vehicle. The central processor is connected with the high-frequency inertial measurement unit and the CMOS image sensor respectively, for receiving the motion data and sending an exposure control signal and a gain configuration instruction to the CMOS image sensor.

[0026] The central processor internally runs a plurality of functional logic modules, specifically including: an electromechanical coupling vibration feature extraction module 10, an inertial field of view prediction module 20, a target light flux calculation module 30, a non-continuous integral control module 40, and a dynamic duty cycle gain compensation module 50. These modules work cooperatively to form an optoelectromechanical closed-loop control circuit.

[0027] The electromechanical coupling vibration feature extraction module 10 is used to receive an angular velocity signal from the high-frequency inertial measurement unit. The electromechanical coupling vibration feature extraction module 10 separates the high-frequency mechanical structural vibration features of the vehicle and locks the instantaneous phase corresponding to the dominant mechanical vibration frequency by performing time-frequency analysis processing on the angular velocity signal. The instantaneous phase data is output to the non-continuous integral control module 40 as a reference benchmark for timing control.

[0028] The inertial field-of-view prediction module 20 is configured to receive motion state data from the high-frequency inertial measurement unit. The inertial field-of-view prediction module 20 predicts the three-dimensional pose of the camera at the next frame exposure time using a kinematic state space model, and indexes the corresponding spatial region in the preset ambient light manifold based on the predicted pose, thereby outputting the brightness estimation value of the predicted field-of-view region.

[0029] The target light flux calculation module 30 is configured to receive the brightness estimation value output by the inertial field-of-view prediction module 20. The target light flux calculation module 30 calculates the ideal photocharge accumulation amount required for correct exposure according to the preset gray scale response target and photoelectric conversion efficiency. Based on the ideal photocharge accumulation amount and the reciprocity law principle, the target light flux calculation module 30 further determines the corresponding nominal shutter time and nominal analog gain.

[0030] The non-continuous integration control module 40 is configured to receive the instantaneous phase output by the electromechanically coupled vibration feature extraction module 10. The non-continuous integration control module 40 generates an integration gating function according to the instantaneous phase, which is a sequence of pulse signals and only presents a high level when the vibration is in the relative stationary interval. The non-continuous integration control module 40 directly applies the integration gating function to the pixel control circuit of the CMOS image sensor to control the pixel charge well to perform non-continuous photocharge accumulation within a single frame acquisition period.

[0031] The dynamic duty cycle gain compensation module 50 is configured to monitor the integration gating function generated by the non-continuous integration control module 40. The dynamic duty cycle gain compensation module 50 counts the effective duration of the gating signal within a single frame period to obtain the effective integration time.

[0032] The dynamic duty cycle gain compensation module 50 further calculates the ratio of the nominal shutter time to the effective integration time to obtain a time loss compensation coefficient. The dynamic duty cycle gain compensation module 50 modifies the nominal analog gain determined by the target light flux calculation module 30 using the time loss compensation coefficient to generate a final compensation gain, and writes the final compensation gain to the analog front-end amplification circuit of the CMOS image sensor to complete the readout of image data.

[0033] Through the connection and interaction of the above modules, the motion camera multi-scene adaptive shooting control system realizes the timing avoidance of mechanical vibration at the physical level, and realizes the dynamic compensation of light flux loss at the electrical level.

[0034] Reference Figures 1-2 The electromechanically coupled vibration feature extraction module 10 specifically includes a time-frequency transform analysis unit, a main vibration frequency locking unit, and a phase information separation unit. These units are connected in sequence and are configured to process the original inertial data obtained from the high-frequency inertial measurement unit.

[0035] The time-frequency transform analysis unit is configured to receive a digital angular velocity signal outputted by the high-frequency inertial measurement unit. Let The angular velocity vector collected at time t is , which contains both the rigid body motion component of the vehicle and the elastic vibration component of the mechanical structure. In order to locate the vibration features in both time and frequency domains, the time-frequency transform analysis unit performs a short-time Fourier transform on the principal axis component of the angular velocity signal.

[0036] Specifically, the time-frequency transform analysis unit uses a sliding window function to intercept a signal segment, and calculates the frequency spectrum of the segment. Define as the complex time-frequency distribution function of the angular velocity signal at time and frequency , and its calculation formula is as follows: ; In the formula, is the complex time-frequency distribution function; indicates the base of the natural logarithm; is the frequency; indicates the integral infinitesimal; indicates the original input angular velocity signal; indicates a sliding window function with a center position at time ; is the integral variable; is the imaginary unit. The sliding window function is configured as a finite-length real window, such as a Hanning window or a Hamming window, to suppress spectral leakage.

[0037] The main vibration frequency locking unit is configured to receive the above-mentioned complex time-frequency distribution function , and convert it into a power spectral density distribution. The main vibration frequency locking unit first calculates the square of the modulus to obtain the power spectral density . Subsequently, the main vibration frequency locking unit performs a peak search algorithm within a preset high-frequency vibration interval . The high-frequency vibration interval is set to cover the common mechanical resonance frequency range, so as to exclude low-frequency motion disturbances such as operator hand jitter or vehicle turning.

[0038] The main vibration frequency locking unit determines the dominant mechanical vibration frequency at the current time by the following optimization objective function: ; In the formula, is the dominant mechanical vibration frequency; indicates the power spectral density function; indicates the “argument of the maximum value”; is the high-frequency vibration interval; denotes the argument of the value range, i.e. the frequency belongs to the closed interval . This limits the search space of the operator.

[0039] The formula indicates that at time , the frequency point with the largest power spectral density is selected as the current dominant mechanical vibration frequency. If there are multiple peaks with similar energy, the component with the highest frequency value is selected because the high-frequency component contributes most significantly to the jelly effect.

[0040] The phase information separation unit is configured to separate a single-frequency narrowband vibration component from the original wideband signal based on the determined dominant mechanical vibration frequency . This unit uses an adaptive bandpass filter or directly extracts the complex value of the corresponding frequency point from the time-frequency distribution function to reconstruct the instantaneous vibration signal .

[0041] The phase information separation unit further calculates the phase angle of the instantaneous vibration signal. This phase angle reflects the specific position of the mechanical vibration waveform at the current time, which is at the peak, trough, or zero-crossing point. The analytical expression for phase extraction is as follows: ; wherein is the phase angle; denotes the inverse tangent function; denotes the value of the complex time-frequency distribution function at the specific time and the dominant mechanical vibration frequency ; and represent the imaginary part and the real part of the complex number, respectively. The extracted phase information and the instantaneous vibration signal are transmitted in real time to the subsequent non-continuous integral control module 40 for accurate definition of the vibration relative to the stationary time window.

[0042] Referring to Figures 1-2 , the inertial field of view prediction module 20 specifically includes a state space construction unit, a Kalman filter prediction unit, and an optical axis pointing calculation unit. The core function of this inertial field of view prediction module 20 is to utilize the low-delay characteristics of inertial data to calculate the physical space region covered by the next frame of image and its lighting conditions in advance before the image sensor is exposed.

[0043] The state space construction unit is configured to establish a mathematical model describing the motion characteristics of the camera. The state space construction unit encapsulates the three-dimensional pose and angular velocity of the camera into a system state vector. The system state vector is defined as follows: where is the state vector; is a superscript, representing the transpose operation of a matrix or vector; represents the pose quaternion of the camera at time t, used to represent the rotational state of the camera; represents the three-axis angular velocity at time t. This state vector encompasses all the kinematic information required to predict the future pointing of the camera.

[0044] The Kalman filter prediction unit is configured to perform a time update step based on the constructed state space model. Since the sampling frequency of the camera is much higher than the image frame rate, this unit utilizes the discretized kinematic equations for recursion.

[0045] The Kalman filter prediction unit performs prior state estimation. This process utilizes the posterior state estimation value at the previous time step and the state transition matrix to calculate the prior state prediction vector at the current time step. The state prediction equation is as follows: where is the state transition matrix, describing how the angular velocity drives the rotational evolution of the quaternion; is the control input matrix; is the external control input (such as the torque command of an active gimbal, which is zero when there is no active control).

[0046] The Kalman filter prediction unit performs the update of the prior error covariance to quantify the uncertainty of the prediction. The covariance prediction equation is as follows: where represents the prior error covariance matrix; is a superscript, representing the transpose operation of a matrix or vector; is the state transition matrix; represents the posterior error covariance matrix at the previous time step; represents the process noise covariance matrix, used to simulate the uncertainty and small disturbances of the system model.

[0047] ​​​​By the above calculation, the Kalman filter prediction unit outputs a predicted attitude quaternion . To ensure the validity of the rotation representation, the unit normalizes the quaternion before output.

[0048] The optical axis pointing calculation unit is configured to map the predicted attitude into a specific ambient lighting space. The unit maintains a pre-defined ambient lighting manifold , which is a luminance statistical model constructed based on a spherical coordinate system, where and represent the azimuth and elevation angles of the spherical coordinate system, respectively.

[0049] The optical axis pointing calculation unit first calculates the camera optical axis vector in the world coordinate system, based on the predicted attitude quaternion . The optical axis pointing calculation unit determines the predicted field of view region in the spherical coordinate system, based on the field of view angle parameters of the camera.

[0050] Finally, the optical axis pointing calculation unit calculates the predicted average luminance estimate by integrating the luminance distribution of the ambient lighting manifold within the predicted field of view region. The formula for the luminance estimate is as follows: ; where is the average luminance estimate; is the predicted attitude quaternion; represents the ambient lighting manifold; represents an area integral (double integral) over the two-dimensional region ; represents the azimuth angle calculated from the predicted attitude ; represents the elevation angle calculated from the predicted attitude ; represents the integral infinitesimal of the azimuth angle; represents the integral infinitesimal of the elevation angle.

[0051] This formula indicates that the system does not perform photometry based on the current image, but rather queries the luminance value in the historical lighting model based on the predicted future attitude, thereby eliminating the lag in lighting changes caused by rapid motion.

[0052] Referring to Figures 1-2 , the target luminous flux calculation module 30 specifically includes an ideal charge quantization unit and a reference parameter mapping unit. The target luminous flux calculation module 30 undertakes the key task of converting ambient luminance information into specific camera execution parameters.

[0053] Ideal charge quantization units are configured based on predicted brightness estimates. Determine the energy target required for imaging. To ensure the grayscale of the output image is within the optimal dynamic range, the ideal charge quantization unit is preset with a target image grayscale value. (For example, the value corresponding to neutral gray). This unit incorporates the photoelectric conversion efficiency parameter of the image sensor. (Unit: electrons / lumen·second), calculate the ideal photocharge accumulation required to satisfy this grayscale response. .

[0054] ; In the formula, This represents the ideal cumulative photocharge. The grayscale value of the target image; This is the inverse function value of the gamma correction coefficient; used to restore nonlinear gray values ​​to linear optical energy domain values. Let be the combined system response constant, which includes lens transmittance, quantum efficiency, and system gain factor. This formula establishes the absolute amount of charge that should accumulate in the pixel potential well, without considering time constraints.

[0055] The reference parameter mapping unit is configured to decompose the abstract charge quantity into specific exposure time and gain parameters. According to the reciprocity law, the accumulated photocharge is a function of the exposure time and the analog gain (and incident light intensity). Let the nominal shutter time be... The nominal analog gain is Then it satisfies the following constraints: ; In the formula, This represents the ideal cumulative photocharge. This is an estimate of the average brightness. To set the nominal shutter speed; This is the nominal analog gain.

[0056] To solve for specific parameter combinations, the benchmark parameter mapping unit introduces linear region constraint logic. This benchmark parameter mapping unit first sets the maximum allowable linear analog gain threshold for the image sensor. and minimum photoelectric response time The reference parameter mapping unit searches for the optimal solution in the parameter space, prioritizing low gain to ensure the signal-to-noise ratio.

[0057] Specifically, the reference parameter mapping unit first attempts to adjust the base gain. The required shutter speed is calculated below: ; In the formula, ideal photo charge accumulation amount; average brightness estimation value; denotes the base analog gain; denotes the calculated shutter time.

[0058] If the calculated is less than the upper limit of the single-frame time window , the nominal shutter time is directly set as and .

[0059] If the calculated exceeds the upper limit of the single-frame time window, it indicates that the ambient light is insufficient. At this time, the unit fixes the nominal shutter time as the maximum allowed value (i.e. ), and reversely calculates the required gain compensation multiple until the maximum linear analog gain threshold is reached. Through this priority determination strategy, the system can maximize the signal-to-noise ratio performance of the image while ensuring the accuracy of the exposure amount. The finally determined parameters will be sent as reference instructions to the subsequent control link.

[0060] Referring to Figures 1-2 , the non-continuous integral control module 40 specifically includes a stability criterion calculation unit and a gate signal generation unit. The function of the non-continuous integral control module 40 is to convert macro mechanical vibration information into micro pixel-level exposure control timing.

[0061] The stability criterion calculation unit is configured to receive the instantaneous vibration signal output from the electromechanical coupling vibration feature extraction module 10. This signal is a three-dimensional vector representing the instantaneous angular velocity of the camera caused by mechanical vibration. In order to quantify the intensity of the vibration at the current time, the stability criterion calculation unit calculates the Euclidean norm of the instantaneous vibration signal, i.e. the vibration angular velocity module, and takes it as the vibration stability criterion function .

[0062] The calculation formula of the vibration stability criterion function is as follows: ; In the formula, is the vibration stability criterion function; is the instantaneous vibration signal; , , are the orthogonal components of the instantaneous vibration signal along the three axes of the camera body coordinate system, respectively; The value of Angular motion velocity induced by vibration. When the vibration displacement is at the peak or trough, its angular velocity is minimum, i.e. tends to zero, which is the vibration relative stationary interval.

[0063] The gate signal generation unit is configured to convert the continuous vibration stability criterion function into a discrete binary integral gating function . The gate signal generation unit is internally provided with a preset vibration threshold , which defines the maximum acceptable vibration angular velocity.

[0064] The gate signal generation unit generates the integral gating function through the following comparison logic: ; wherein, is the integral gating function with a value of 0 or 1; is the vibration stability criterion function; is the vibration threshold; represents the conditional judgment, which means “if”. Corresponding to the “on” state of the integral gating function, the image sensor is instructed to perform charge accumulation; corresponding to the “off” state of the integral gating function, the image sensor is instructed to suspend charge accumulation. The generated integral gating function is a high-frequency pulse sequence that is switched on and off multiple times within a single frame acquisition period.

[0065] The integral gating function is directly applied to the pixel-level control circuit of the CMOS image sensor. In a specific embodiment, the signal controls the transfer gate within the pixel cell.

[0066] When (on state), the transfer gate is turned on, and the photoelectrons generated by the photodiode under illumination are transferred and accumulated into the floating diffusion region or a dedicated storage node.

[0067] When , the transfer gate is turned off, and the charge channel between the photodiode and the storage node is cut off. At this time, the charge accumulated in the storage node is maintained, neither increased nor reset or read out, until the integral gating function next time becomes the on state. This process is repeated within a single frame time, thereby realizing the physical layer exposure time splitting.

[0068] Referring to Figures 1-2, the dynamic duty cycle gain compensation module 50 specifically comprises an effective time integration unit, a compensation coefficient calculation unit and a final gain adjustment unit. The dynamic duty cycle gain compensation module 50 functions to maintain the target light flux constant by adjusting the analog gain in the case of a decrease in effective exposure time caused by non-continuous integration.

[0069] The effective time integration unit is configured to receive the integration gate function output by the non-continuous integration control module 40. The unit time-integrates the binary gate signal in a complete single-frame acquisition period to accurately calculate the total length of time during which the pixel charge well is actually in an open state, i.e., the effective integration time .

[0070] ; wherein and represent the start and end times of the single-frame acquisition period, respectively; is the integration gate function with a value of 0 or 1. The calculation result quantifies the time actually used for photoelectric charge accumulation due to avoidance of vibration; represents an integration infinitesimal, indicating that the integration is performed with respect to the time variable .

[0071] The compensation coefficient calculation unit is configured to receive the nominal shutter time determined by the target light flux calculation module 30 and the effective integration time calculated by the effective time integration unit. The compensation coefficient calculation unit calculates the ratio of the two to determine the light flux compensation multiple required to compensate for the time loss, i.e., the time loss compensation coefficient .

[0072] The formula for calculating the time loss compensation coefficient is as follows: ; wherein is the time loss compensation coefficient; is the shutter time required in an ideal case without vibration; is the shutter time actually available in a vibrating environment. When (i.e., no stable integration window is found throughout the frame period), is set to a preset maximum value to avoid division by zero errors and provide a gain in the limit case. The coefficient indicates the multiple by which the analog gain needs to be amplified in order to achieve the same total light flux as the nominal shutter time.

[0073] The final gain adjustment unit is configured to receive the time loss compensation coefficient and the nominal analog gain determined by the target luminous flux calculation module 30 . The final gain adjustment unit multiplies the two to calculate the theoretical final compensation gain .

[0074] To ensure that the image signal does not enter the nonlinear region or saturation region, the final gain adjustment unit also compares the calculated theoretical gain with the maximum linear analog gain threshold of the image sensor , and takes the smaller value as the final compensation gain . The final gain determination formula is as follows: ; In the formula, is the final compensation gain; is the is the nominal analog gain; is the time loss compensation coefficient; is the maximum linear analog gain threshold; represents the minimum value function. The final compensation gain is written into the gain control register of the CMOS image sensor, used to drive its analog front-end amplification circuit, and the signal is amplified by a corresponding amplitude when the image data is read out, so as to complete the accurate compensation of the luminous flux loss caused by the non-continuous integration.

[0075] Referring to Figures 1-2 , Figure 2 is a flowchart of a motion camera multi-scene adaptive shooting control method according to an embodiment of the present application. The motion camera multi-scene adaptive shooting control method provided by the present application is applied to the motion camera multi-scene adaptive shooting control system in the foregoing embodiment, and specifically includes the following steps: S100, vibration feature and phase extraction: the system first continuously collects a digital angular velocity signal from the high-frequency inertial measurement unit 200. The signal is sent to the electromechanical coupling vibration feature extraction module 10. The time-frequency transform analysis unit in the electromechanical coupling vibration feature extraction module 10 performs a short-time Fourier transform on the signal to construct a two-dimensional time-frequency distribution function describing the change of signal frequency components over time. Then, the main vibration frequency locking unit calculates the power spectral density of the time-frequency distribution and searches for the maximum value point of energy in a preset high-frequency interval (for example, 50Hz to 300Hz). The frequency corresponding to the maximum value point is determined as the dominant mechanical vibration frequency. Finally, the phase information separation unit separates the corresponding narrowband instantaneous vibration signal from the original signal according to the locked dominant mechanical vibration frequency, and calculates the instantaneous phase of the signal. The instantaneous phase data and the instantaneous vibration signal are used as the control basis for the subsequent steps.

[0076] S200, Field of view prediction and brightness estimation: The system performs field of view prediction simultaneously. The inertial field of view prediction module 20 receives all motion data (attitude and angular velocity) from the high-frequency inertial measurement unit. The state space construction unit within the inertial field of view prediction module 20 integrates the camera attitude quaternion and angular velocity into a system state vector. The Kalman filter prediction unit performs a time update step based on the state vector and state transition matrix, iteratively calculates the predicted attitude quaternion of the camera at the next frame image exposure time. The optical axis pointing calculation unit receives the predicted attitude and calculates the pointing vector of the camera optical axis in the world coordinate system accordingly. The pointing vector is used to index the pre-set environmental illumination manifold graph, and combined with the field of view angle parameter of the camera to determine the spherical region covered by the predicted field of view. By integrating the brightness statistical value in this region, a brightness estimation value representing the average brightness of the region is finally obtained.

[0077] S300, Exposure parameter planning: The target light flux calculation module 30 receives the brightness estimation value output by step S200. The ideal charge quantization unit calculates the ideal photoelectric charge accumulation amount that the pixel needs to accumulate to achieve the average gray value target of the image according to the pre-set target image average gray value and the response efficiency of the photoelectric conversion system. Then, the reference parameter mapping unit matches the exposure parameter combination required to generate the ideal charge amount according to the reciprocity law principle. The reference parameter mapping unit preferentially selects a lower analog gain to obtain a higher signal-to-noise ratio, and calculates the required shutter time under this premise. If the calculated shutter time exceeds the maximum allowed time of a frame, the shutter time is fixed as the maximum value, and the analog gain is increased accordingly until the gain reaches the upper limit of the pre-set linear region. Finally, the target light flux calculation module 30 outputs a set of nominal shutter time and nominal analog gain.

[0078] S400, Non-continuous integration control: The non-continuous integration control module 40 receives the instantaneous vibration signal output by step S100. The stability criterion calculation unit calculates the Euclidean norm of the three-dimensional vibration signal in real time, i.e. the vibration angular velocity modulus, and takes the modulus value as the vibration stability criterion function. The gate signal generation unit compares the value of the criterion function with a pre-set vibration threshold value. When the criterion function value is less than the threshold value, it indicates that the camera is near the peak or trough of the vibration waveform at this moment, and is relatively stationary, at which time the integration gate function in the open state is generated; otherwise, the integration gate function in the closed state is generated. The gate function is applied to the pixel control circuit of the CMOS image sensor 400 in the form of a high-frequency pulse sequence, driving the charge well to perform segmented charge accumulation within a single frame period.

[0079] S500, gain compensation and image output: at the end of the single frame acquisition period, the dynamic duty cycle gain compensation module 50 performs compensation calculation. The effective time integration unit first performs time integration on the integral gating function generated in step S400 to obtain the effective integration time during which the charge accumulation is actually performed. The compensation coefficient solving unit calculates the ratio of the nominal shutter time determined in step S300 to the effective integration time to obtain the time loss compensation coefficient. The final gain adjustment unit multiplies the nominal analog gain by the compensation coefficient to obtain the final compensation gain, and performs amplitude limiting processing to ensure that the gain value does not exceed the maximum linear gain of the sensor. The final compensation gain is configured to the analog front-end amplification circuit of the image sensor. During the data readout process, the pixel signal is amplified by the gain, thereby completing the light flux compensation for the exposure time loss, and finally outputting a frame of image with accurate exposure and no motion distortion.

Claims

1. A multi-scene adaptive shooting control system for an action camera, characterized in that, include: The electromechanical coupling vibration feature extraction module is used to collect the angular velocity signal of the inertial measurement unit and extract the mechanical structure vibration features and instantaneous phase of the vehicle through time-frequency analysis; The inertial field of view prediction module is used to predict the camera's attitude at the next frame exposure time based on the kinematic state space model, and to determine the brightness estimate of the predicted field of view region based on the predicted attitude in the preset ambient lighting manifold map. The target luminous flux calculation module is used to calculate the ideal photocharge accumulation required to meet the preset grayscale response based on the brightness estimate, and to determine the corresponding nominal shutter time and nominal analog gain. The discontinuous integration control module is used to generate an integral gating function based on the instantaneous phase, and control the image sensor to perform discontinuous integration within a single frame acquisition cycle. The integral gating function only enables charge accumulation in the relatively static vibration range. The dynamic duty cycle gain compensation module is used to calculate the effective integration time generated by discontinuous integration, determine the time loss compensation coefficient based on the ratio of the nominal shutter time to the effective integration time, and adjust the analog gain of the image sensor to compensate for light flux loss.

2. The action camera multi-scene adaptive shooting control system according to claim 1, characterized in that, The electromechanical coupling vibration feature extraction module includes: The time-frequency transformation analysis unit is used to perform a short-time Fourier transform on the angular velocity signal acquired by the inertial measurement unit to construct the time-frequency distribution function of the angular velocity signal; The main vibration frequency locking unit is used to calculate the power spectral density of the time-frequency distribution function, search for energy extrema in a preset high-frequency range, and determine the frequency corresponding to the energy extrema as the dominant mechanical vibration frequency. The phase information separation unit is used to extract the instantaneous vibration signal corresponding to the dominant mechanical vibration frequency component from the angular velocity signal, and calculate the instantaneous phase of the instantaneous vibration signal as a reference for the discontinuous integral control module.

3. The action camera multi-scene adaptive shooting control system according to claim 1, characterized in that, The inertial field-of-view prediction module includes: State space construction unit, used to construct a system state vector containing camera pose quaternions and angular velocities; The Kalman filter prediction unit is used to iteratively calculate and output the predicted pose quaternion for the next exposure time using the Kalman filter algorithm based on the system state vector and state transition matrix. The optical axis pointing calculation unit is used to calculate the pointing of the camera optical axis vector in the world coordinate system based on the predicted attitude quaternion, and is used to index the ambient lighting manifold.

4. The action camera multi-scene adaptive shooting control system according to claim 3, characterized in that, The Kalman filter prediction unit specifically includes: The prior state estimation subunit is used to retrieve the optimal posterior state estimate from the previous time step, multiply it by the state transition matrix to obtain the prior state prediction vector at the current time step. The prior state prediction vector contains uncorrected camera attitude quaternion components and angular velocity components. The prior covariance calculation subunit is used to calculate the product of the state transition matrix and the posterior error covariance matrix of the previous time step, and then multiply the product result by the transpose of the state transition matrix on the right, and finally add the preset process noise covariance matrix to obtain the prior error covariance matrix of the current time step. The state vector output subunit is used to extract the corresponding quaternion part from the prior state prediction vector, normalize it, and output the processed result as the predicted pose quaternion for the next frame exposure time to the optical axis pointing calculation unit.

5. The action camera multi-scene adaptive shooting control system according to claim 1, characterized in that, The target luminous flux calculation module includes: An ideal charge quantization unit is used to calculate the ideal photocharge accumulation based on the brightness estimate and the preset target image grayscale value, combined with the response efficiency of the photoelectric conversion system. The reference parameter mapping unit is used to match the combination of nominal shutter time and nominal analog gain required to generate the ideal photocharge accumulation amount according to the ideal photocharge accumulation amount and the reciprocity law principle.

6. The action camera multi-scene adaptive shooting control system according to claim 5, characterized in that, The reference parameter mapping unit specifically includes: The linear region constraint subunit is used to set the maximum linear analog gain threshold and the minimum photoelectric response time of the image sensor. A reciprocity matching subunit is used to search for a numerical combination of the nominal shutter time and the nominal analog gain under the constraint of the maximum linear analog gain threshold, such that the product of the nominal shutter time and the nominal analog gain is proportional to the ideal photocharge accumulation. The parameter priority determination subunit is used to prioritize the combination whose nominal shutter time is less than the preset single frame time window when there are multiple combinations of values ​​that satisfy the ideal photocharge accumulation, and output the selected combination to the dynamic duty cycle gain compensation module.

7. The action camera multi-scene adaptive shooting control system according to claim 1, characterized in that, The discontinuous integral control module includes: The stability criterion calculation unit is used to calculate the vibration angular velocity magnitude of the instantaneous vibration signal as the vibration stability criterion function; The gating signal generation unit is used to compare the vibration stability criterion function with a preset vibration threshold. When the vibration stability criterion function is less than the vibration threshold, an integral gating function in the open state is generated; when the vibration stability criterion function is greater than or equal to the vibration threshold, an integral gating function in the closed state is generated.

8. A multi-scene adaptive shooting control system for an action camera according to claim 7, characterized in that, The stability criterion calculation unit specifically includes: The orthogonal component decomposition subunit is used to receive the instantaneous vibration signal output by the phase information separation unit and decompose the instantaneous vibration signal into three-axis orthogonal vibration components along the inertial measurement unit's body coordinate system. The Euclidean norm operation subunit is used to calculate the square values ​​of the three-axis orthogonal vibration components respectively, sum the three square values, and perform an arithmetic square root operation on the summation result to obtain the vibration angular velocity magnitude. The criterion function assignment subunit is used to directly assign the vibration angular velocity magnitude value output by the Euclidean norm operation subunit to the vibration stability criterion function, and transmit the numerical sequence of the function to the gate signal generation unit in real time.

9. A multi-scene adaptive shooting control system for an action camera according to claim 1, characterized in that, The dynamic duty cycle gain compensation module includes: The effective time integration unit is used to perform time integration on the integral gating function within a single frame acquisition period to calculate the effective integration time. The compensation coefficient calculation unit is used to calculate the ratio of the nominal shutter time to the effective integration time to obtain the time loss compensation coefficient; The final gain adjustment unit is used to multiply the nominal analog gain by the time loss compensation coefficient to generate the final compensation gain and drive the analog front-end amplification circuit of the image sensor.

10. A multi-scene adaptive shooting control method for an action camera, applied to the multi-scene adaptive shooting control system for an action camera as described in any one of claims 1-9, characterized in that, Includes the following steps: S1. Acquire the angular velocity signal of the inertial measurement unit, extract the mechanical structure vibration characteristics of the vehicle through time-frequency analysis, and separate the instantaneous phase corresponding to the dominant mechanical vibration frequency; S2. Based on the kinematic state-space model, Kalman filtering is used to predict the camera's pose at the next frame exposure time. The predicted pose is then projected onto a preset ambient lighting manifold to determine the brightness estimate of the predicted field of view. S3. Calculate the ideal photocharge accumulation required to meet the preset grayscale response based on the brightness estimate, and determine the corresponding nominal shutter time and nominal analog gain based on the reciprocity law principle; S4. Generate an integral gating function based on the instantaneous phase, and drive the image sensor to activate charge accumulation only in the relatively static vibration range within a single frame acquisition cycle, and perform discontinuous integration; S5. Calculate the effective integration time generated by the discontinuous integration in real time, calculate the ratio of the nominal shutter time to the effective integration time to obtain the time loss compensation coefficient, and use the time loss compensation coefficient to adjust the analog gain of the image sensor to compensate for the light flux loss, and output the final image.

Images