Lossless video compression method for action cameras

Abstract

The application discloses a lossless video compression method for a motion camera, comprising the following steps: acquiring a sequence of original video frames and inertial measurement unit data and aligning a time reference, and extracting a three-axis jerk signal; evaluating a coding pipeline delay to determine a look-ahead prediction window, and using the signal in the window to perform a sudden change precursor prediction to generate a coding strategy label; using the inertial measurement unit data to calculate a row-by-row rotation parameter and construct an inverse spatial mapping model, performing inverse geometric sampling on a pre-reconstructed video reference frame to generate a rolling shutter perception prediction frame; calculating a prediction residual according to the coding strategy label combined with the prediction frame, performing reversible transformation and entropy coding on the prediction residual, and encapsulating and outputting a lossless compressed video code stream together with auxiliary reconstruction information. The application is beneficial to reducing defects caused by traditional prediction lag, reducing geometric projection deviation, and improving lossless compression efficiency and reconstruction fidelity in a high-speed scene.

Description

Technical Field

[0001] This invention relates to video image processing and source coding technology, and in particular to a lossless high-fidelity video compression method for high-speed scenes with action cameras. Background Technology

[0002] When action cameras record extreme sports or high-speed dynamic scenes, the equipment is often accompanied by violent spatial tumbling and high-frequency mechanical vibrations. In these high-speed motion scenarios, the captured high-resolution video sequences exhibit strong temporal dynamics and spatial topological complexity. To preserve the most original visual information within limited storage space and transmission bandwidth, lossless high-fidelity video compression technology becomes a crucial foundational support. Its main technical goal is to minimize temporal and spatial redundancy between video sequence frames without sacrificing any of the original image's physical acquisition accuracy.

[0003] Recent technologies for video stream compression generally rely on pixel-domain algorithms for inter-frame motion estimation and compensation. In this architecture, the video encoder typically passively determines whether to switch encoding modes by evaluating the residual energy after completing inter-frame prediction for the current frame. Furthermore, for widely used rolling shutter image sensors, existing conventional compression schemes either forcibly perform spatial interpolation resampling of the original video frames before encoding to achieve distortion correction, or completely ignore the temporal differences caused by line-by-line sequential exposure in the prediction loop, still using a globally uniform translational motion model for block-level matching. However, these existing technologies exhibit significant limitations when dealing with complex high-speed motion. On one hand, the reactive response mechanism leads to a systematic lag in the encoder's adaptation to drastic dynamic changes, easily causing a surge in local residuals and wasting encoding computational resources under severe conditions. On the other hand, facing image geometric distortion induced by high-speed rotation, existing compensation strategies not only struggle to maintain the original pixel accuracy necessary for lossless compression, but also experience a systematic amplification of spatial prediction bias with increasing motion intensity, leading to the failure of the inter-frame prediction reference. Therefore, there is an urgent need to study a video compression method that is highly robust in high-speed dynamic scenes and can help suppress spatiotemporal geometric distortion errors. Summary of the Invention

[0004] The purpose of this invention is to provide a lossless video compression method for action cameras, in order to solve the aforementioned problems existing in the prior art.

[0005] To address the aforementioned problems in the existing technology, the present invention provides the following technical solution: acquiring the original video frame sequence of a motion camera and the corresponding inertial measurement unit data, and aligning the time reference of the original video frame sequence and the inertial measurement unit data; Triaxial accelerometer signals are extracted based on the aligned inertial measurement unit data; Evaluate the coding pipeline latency of the current frame to determine the look-ahead prediction window; The triaxial acceleration signal within the prospective prediction window is used to perform abrupt change precursor prediction and generate encoding strategy labels. The row-by-row rotation parameters are calculated using the inertial measurement unit data, and an inverse spatial mapping model is constructed based on the row-by-row rotation parameters; The inverse spatial mapping model is used to perform inverse geometric sampling on the pre-reconstructed video reference frame to generate a rolling shutter sensing prediction frame; Based on the prediction path defined by the encoding strategy label, and combined with the rolling shutter sensing prediction frame, the prediction residual of the current frame in the original video frame sequence is calculated; The predicted residual is subjected to reversible transformation and entropy encoding, and the encoded predicted residual is encapsulated with auxiliary reconstruction information containing the line-by-line rotation parameters to output a lossless compressed video stream.

[0006] According to one aspect of this application, calculating line-by-line rotation parameters using the inertial measurement unit data includes: Determine the time interpolation ratio between the start exposure time of each frame in the original video frame sequence and the sampling time of two adjacent inertial measurement units; Based on the time interpolation ratio, linear interpolation is performed on the inertial measurement unit data at the sampling times of the two adjacent inertial measurement units; Obtain the subsampling period precision instantaneous angular velocity corresponding to the initial exposure time, and use the subsampling period precision instantaneous angular velocity to calculate the line-by-line rotation parameters.

[0007] According to one aspect of this application, after generating the rolling shutter sensing prediction frame, the method further includes performing local compensation backoff correction: The rolling shutter sensing prediction frame is divided according to the preset line band width, and the mean square value of the prediction residual of each line band region relative to the current frame is evaluated respectively. When the mean square value of the predicted residual in the target line band region is detected to be significantly higher than the mean square value of the residual in the whole frame, the target line band region is determined to be an abnormal region. For the abnormal region, the process reverts to the block-level motion search process, regenerates the predicted pixel value of the target line band region using the searched block-level translation motion vector, and refreshes the corresponding position in the rolling shutter sensing prediction frame.

[0008] In this process, the inverse spatial mapping and the subpixel interpolation operation are both performed using fixed-point integer arithmetic at the encoding and decoding ends. The line-by-line rotation parameters are encapsulated in the bitstream as auxiliary reconstruction information and transmitted to the decoding end, so that the decoding end can generate a prediction frame that is exactly consistent with the bits at the encoding end using the same fixed-point integer arithmetic and the same line-by-line rotation parameters.

[0009] The beneficial effects of this invention are as follows: ① A look-ahead prediction mechanism based on inertial measurement units is introduced. By evaluating the coding pipeline delay in real time and dynamically setting the look-ahead window, combined with an exponentially weighted sliding statistical algorithm to generate an adaptive mutation threshold, this scheme can detect motion precursors and proactively route predicted paths before drastic changes occur in actual image pixels. This mechanism shifts the decision-making time from delayed post-event remediation to pre-event prevention, suppressing the surge in local residuals caused by inter-frame prediction failures. It initially solves the problem of systemic lag and wasted computational resources caused by the passive response of traditional video coding when dealing with high-speed motion.

[0010] ② Unlike the traditional block-level translation model, this application reconstructs a row-by-row inverse spatial mapping model. By introducing a second-order correction for row number self-consistency and a quaternion incremental chain multiplication integral for large-angle rotation, continuous geometric deformation is approximately restored, and complex spatial projection errors are suppressed to the sub-pixel level, alleviating problems such as the amplification of spatial prediction deviation caused by rolling shutter geometric distortion induced by high-speed rotation.

[0011] ③ Furthermore, this scheme applies a fixed-point subpixel interpolation specification. By utilizing fixed normalized shift and rounding parameters, floating-point calculation drift is reduced, ensuring that the entire signal chain from prediction to transformation is closed within the restricted integer domain. Ultimately, this scheme improves the video data compression efficiency of action cameras under extreme conditions while maintaining the original pixel fidelity. Attached Figure Description

[0012] Figure 1 This is a flowchart of the lossless video compression method for action cameras according to the present invention.

[0013] Figure 2 This is a flowchart for evaluating the coding pipeline delay of the current frame to determine the look-ahead prediction window.

[0014] Figure 3 This is a flowchart illustrating the generation of encoding strategy tags for this invention.

[0015] Figure 4 This is a flowchart of the inverse spatial mapping model constructed based on the row-by-row rotation parameters of the present invention. Detailed Implementation

[0016] like Figures 1 to 4 The specific processing procedure of the present invention is described as shown.

[0017] Example 1: A lossless video compression method for motion cameras based on hardware latency adaptation and geometric inverse rendering is provided, mainly including the following steps: Step 101: Obtain the original video frame sequence of the action camera and the corresponding inertial measurement unit data, and align the time reference of the original video frame sequence and the inertial measurement unit data; Specifically, the original video frame sequence can be a frame-by-frame scan image read from the output interface of the motion camera image sensor, with each frame carrying a specific hardware timestamp.

[0018] In this embodiment, the color space of the original video frame sequence is YUV4:2:0 format, and the original data bit depth B of the luminance component is 8 bits. Those skilled in the art will understand that this method is also applicable to other bit depth configurations such as 10 bits or 12 bits.

[0019] The inertial measurement unit data includes raw sequences of triaxial acceleration and triaxial angular velocity recorded at a high sampling rate.

[0020] To reduce sensor physical bias, after acquiring the data from the inertial measurement unit, calibration compensation is typically performed using the factory-calibrated zero bias vector and scale factor matrix, and a low-pass filter is applied to suppress high-frequency quantization noise and mechanical vibration interference.

[0021] Since image sensors and inertial measurement units typically operate in different hardware clock domains, it is necessary to use a synchronization pulse signal as an anchor point to map the timestamps of the original video frame sequence and the inertial sampling time reference to a unified clock domain.

[0022] By aligning the time reference, the positioning deviation of the initial exposure time of the video frame on the inertial time axis can be constrained within an effective range, providing a time alignment basis for subsequent high-precision physical geometric compensation.

[0023] In some alternative implementations, for devices with fixed delay deviations, frame-by-frame compensation can be performed using pre-calibrated clock offsets to establish a unified synchronization time mapping table and record the precise index position of the start exposure time of each frame.

[0024] Step 102: Extract triaxial accelerometer signals based on the aligned inertial measurement unit data; In this embodiment, the triaxial acceleration signal refers to the first derivative of acceleration with respect to time, a physical quantity that can sensitively characterize the severity of sudden changes in the motion state of the motion camera. Specifically, a central difference operation is performed on the triaxial acceleration time series based on the sampling time interval.

[0025] As an example, the calculation operation can be represented as: ; in, For the first The triaxial accelerometer vector at each sampling time. The three-axis acceleration vector for the next adjacent time step. The three-axis acceleration vector is the value at the previous adjacent time step. This is the sampling time interval for the inertial measurement unit.

[0026] Furthermore, after extracting the triaxial jerk vector, the square root of the sum of the squares of its components yields the corresponding jerk scalar amplitude. By extracting this signal, complex spatial motion is transformed into a one-dimensional energy scalar sequence, providing a reliable physical criterion for subsequently determining whether drastic changes are about to occur in the video footage.

[0027] Step 103: Evaluate the coding pipeline delay of the current frame to determine the look-ahead prediction window; In this step, the encoding pipeline latency refers to the time span from the end of the exposure of the last row of pixels in the current frame until the video encoder actually initiates the encoding decision for that frame. Constrained by the image signal processing pipeline and data transmission bus, this latency exhibits dynamic variations depending on the hardware device or operating mode.

[0028] By assessing this delay in real time, a time interval extending into the future is determined on the physical timeline—the look-ahead window. The length and position of this window determine how much inertial data, not yet consumed by video frames, the encoder can read in advance.

[0029] Step 104: Use the triaxial acceleration signal in the look-ahead prediction window to perform abrupt change precursor prediction and generate coding strategy labels; In practice, the triaxial acceleration signal corresponding to the look-ahead prediction window is extracted, and the short-time energy aggregation value of the signal within the window is calculated.

[0030] The aggregated value is compared with a preset or adaptively updated mutation detection threshold. When the energy aggregated value exceeds the threshold, it is determined that there is a precursor to a motion mutation in the upcoming frame time interval.

[0031] Based on the determination result, a specific coding strategy label is assigned to the current frame, such as identifying it as normal mode or transient enhancement mode.

[0032] Traditional video encoders typically switch encoding modes passively only after the residual is too large after prediction is completed. However, this step utilizes the objective physical condition that the inertial sampling rate is much higher than the video frame rate to actively switch strategies before the sudden change is actually reflected in the image pixels, thus suppressing the loss of encoding efficiency and sudden increase in bit rate caused by inter-frame prediction failure.

[0033] Step 105: Calculate the line-by-line rotation parameters using the inertial measurement unit data, and construct an inverse spatial mapping model based on the line-by-line rotation parameters; Because action cameras generally use rolling shutter sensors, their line-by-line sequential exposure method results in differences in the actual acquisition time of pixels in different rows within the same frame. At high speeds, this time difference translates into geometric distortion within the frame.

[0034] Using the aligned time reference, the exposure time corresponding to each horizontal row is found, and the angular velocity data near that time is integrated over time to obtain the cumulative rotation angle of the camera around the pitch axis, yaw axis and roll axis, thereby obtaining the row-by-row rotation parameters.

[0035] By combining the camera's intrinsic parameter matrix, the row-by-row rotation parameters are transformed into an inverse spatial mapping model describing the spatial projection relationship. This model defines the mapping rule for any pixel coordinate in the current frame coordinate system to be traced back to the physical space of the previous moment.

[0036] Step 106: Use the inverse spatial mapping model to perform inverse geometric sampling on the pre-reconstructed video reference frame (the encoded frame is cached after local reconstruction) to generate a rolling shutter sensing prediction frame. In this step, the core logic of reverse geometric sampling is to find the source from the target. For the current frame to be encoded... Each integer pixel coordinate point in the row, for example, coordinate point The value is input into the inverse spatial mapping model to calculate the reference sampling position of the pixel in the pre-reconstructed video reference frame. Since the calculated reference sampling position usually contains a decimal part, subpixel interpolation needs to be performed at this position in the video reference frame to obtain the predicted pixel value. After traversing and concatenating all predicted pixel values ​​line by line, the complete rolling shutter sensing prediction frame can be obtained.

[0037] By using reverse geometric sampling instead of forward mapping, it is possible to ensure that every integer coordinate position in the current frame is queried exactly once, overcoming the defects of pixel holes or multiple pixel overlapping mapping in the prediction frame.

[0038] This step is mainly applicable to inter-frame coding scenarios where a usable reference frame exists. For scenarios where no usable reference frame exists, such as the first frame of a video sequence or a randomly accessed frame, intra-frame coding mode is used to complete the encoding independently. This is a standard procedure in the field of video coding and will not be elaborated further.

[0039] Step 107: Based on the prediction path defined by the encoding strategy label, calculate the prediction residual of the current frame in the original video frame sequence in combination with the rolling shutter sensing prediction frame; Read the generated encoding strategy labels and execute branch routing.

[0040] When the encoding strategy label indicates the normal mode, the rolling shutter sensing prediction frame is used as the reference for the current frame. The original pixel value of the current frame in the original video frame sequence is subtracted from the predicted pixel value one by one to calculate the pixel-level prediction residual.

[0041] When the label indicates transient enhancement mode, it means that the image is in a period of drastic change and transition. Other reference mechanisms such as block-level spatial prediction can be introduced to perform energy comparison and select the optimal residual generation method.

[0042] This process enables the acquisition of minimized residual energy under different motion intensities.

[0043] Step 108: Perform reversible transformation and entropy encoding on the prediction residual, and encapsulate the encoded prediction residual with auxiliary reconstruction information containing the line-by-line rotation parameters to output a lossless compressed video stream.

[0044] To meet the fundamental requirements of lossless high-fidelity compression, the spatial domain prediction residual is divided into multiple coding blocks, and an invertible transformation operation based on integer arithmetic is applied to each coding block to map the residual signal to the transform domain coefficient representation.

[0045] Since it is a lossless mode, the transformed coefficients are not rounded or quantized, but are directly fed into the context adaptive binary arithmetic encoder for probability model estimation and bitstream compression.

[0046] The compressed residual bitstream is used as the main payload, and auxiliary reconstruction information such as line-by-line rotation parameters are incorporated according to the agreed syntax structure to encapsulate and form a lossless compressed video stream.

[0047] After receiving the bitstream, the decoding end can independently reconstruct the same inverse spatial mapping model based on the embedded line-by-line rotation parameters, thereby achieving pixel-level absolute lossless restoration.

[0048] Assuming an action camera is mounted on a cyclist's helmet, and the camera experiences a rapid yaw rotation while going downhill through a curve, the inertial measurement unit (IMU) has already captured the energy surge of the triaxial jerk signal within the look-ahead window corresponding to the encoding pipeline delay before the current frame has been fully exposed.

[0049] Based on this, it was determined that a sudden change in the image would occur in the future frame, and the encoding strategy label was proactively switched to transient enhancement mode. Simultaneously, using angular velocity data within the same time period, the rotation matrix of each row of the current frame was calculated line by line, and an inverse spatial mapping model was constructed. This model traces each integer pixel coordinate of the current frame back to its sub-pixel position in the previous video reference frame, and after fixed-point interpolation, generates a rolling shutter-aware prediction frame.

[0050] Because the predicted frame precisely matches the line-by-line geometric offset introduced by the rolling shutter at the physical level, the prediction residual is significantly reduced compared to traditional block-level translation compensation. In transient enhancement mode, the spatial prediction residual and the inter-frame prediction residual with lower energy are further compared at the coding block level to ensure that the residual in the abrupt transition region is also effectively suppressed. The low-energy prediction residual is then transformed and entropy-coded to generate a compact lossless compressed bitstream.

[0051] According to one aspect of this application, after the encoding output of the current frame is completed, an inverse reversible transformation is performed on the encoded prediction residual, and the residual after transformation is added pixel by pixel to the rolling shutter sensing prediction frame to obtain the local reconstructed frame of the current frame.

[0052] The locally reconstructed frame is cached in the reference frame buffer and used as the video reference frame when performing inverse geometric sampling on subsequent frames. For the first frame of the video sequence, since there is no available reference frame, this frame is treated as an intra-coded frame, and only the spatial prediction mode is used to generate the prediction residual. After encoding, its local reconstruction result is cached as the initial reference frame for subsequent frames.

[0053] Example 2: Based on the above examples, the layered time synchronization mechanism between video frames and inertial measurement units is described in detail, especially how to achieve offline calibration and online interpolation mechanism for coarse and fine layered time alignment in cross-clock domain acquisition.

[0054] Step 201, aligning the time base of the original video frame sequence with that of the inertial measurement unit data, is performed using a pre-calibrated clock offset; this clock offset is pre-acquired through the following offline calibration steps: In action camera systems, the timestamps of video frames and the sampling times of the inertial measurement unit are typically generated by separate hardware timers. Due to limitations in hardware startup sequence and clock drift, there is a fixed time discrepancy between the two.

[0055] Conventional coarse synchronization can only limit the deviation to no more than one inertial sampling period. Under high-speed rotation conditions with an angular velocity of 200 degrees per second, a coarse synchronization deviation of 1 millisecond can lead to a rotation angle error of about 0.2 degrees, which in turn produces a projection displacement of 3 to 4 pixels at the edge of the image with a focal length of 1000 pixels. This displacement is close to the amount of geometric distortion deviation that rolling shutter compensation attempts to eliminate.

[0056] To provide a time reference capable of supporting sub-pixel level compensation accuracy, a pre-calibrated clock offset with microsecond-level accuracy needs to be obtained in advance. This embodiment provides an offline calibration method based on joint visual-inertial optimization for system initialization.

[0057] Step 202: Using the candidate clock offset value as the optimization variable, calculate the inter-frame optical flow displacement and the corresponding inertial measurement unit integral displacement respectively; Specifically, during the calibration process at the encoder startup stage, a sequence of visual frames with a clear motion pattern is selected, preferably a video clip with uniform rotation characteristics.

[0058] For this segment, feature points between adjacent visual frames are extracted, and the optical flow displacement between visual frames is calculated.

[0059] Meanwhile, given a series of clock offset candidate values, the integration time axis is realigned according to each candidate value, and the angular velocity data within this period is integrated over time to obtain the corresponding inertial measurement unit integrated displacement.

[0060] Step 203: The pre-calibrated clock offset is obtained by minimizing the residual between the inter-frame optical flow displacement and the integral displacement of the inertial measurement unit.

[0061] The squared spatial distance between the inter-frame optical flow displacement and the integral displacement of the inertial measurement unit is calculated for each pair of visual frames, and then accumulated over multiple frames to obtain the target cost function.

[0062] Specifically, the optimization calculation relationship is expressed as follows: ; In the formula, The sum of squared residuals accumulated over multiple frames. For integer indices of the frame sequence, This indicates a summation operation on the squared residuals of all valid frames. For the first Visual inter-frame optical flow displacement from one frame to an adjacent frame. For a given clock offset candidate value The integral displacement of the inertial measurement unit is obtained by time integration.

[0063] By iteratively updating the candidate clock offset value τ, we find the one that maximizes the sum of squared cumulative residuals. The optimal solution that reaches the minimum value is the pre-calibrated clock offset. This offline calibration step can achieve alignment accuracy on the order of tens of microseconds, providing a reliable microsecond-level time anchor point for subsequent rotation matrix calculations.

[0064] Furthermore, as an online alternative or supplement to the aforementioned offline visual-inertial joint optimization scheme, in scenarios where no additional offline calibration steps are required, subsampling period linear interpolation can be used to achieve precise synchronization.

[0065] Step 204: Calculate the line-by-line rotation parameters using inertial measurement unit data, including: When calculating line-by-line rotation parameters, the accuracy of the integration starting point directly determines the accuracy of the rotation calculation. When the initial exposure time of a video frame lies between the sampling times of two discrete inertial measurement units, the instantaneous physical quantities at that absolute physical moment cannot be directly read from the hardware buffer. This process is similar to the one described above and will not be repeated here.

[0066] Step 205: Determine the time interpolation ratio between the start exposure time of each frame in the original video frame sequence and the sampling time of two adjacent inertial measurement units; Obtain the start exposure time of the current frame after coarse synchronization mapping, and locate the two adjacent sampling nodes before and after this time on the continuous inertial time axis. Calculate the time offset of this start exposure time relative to the previous sampling node, and divide this time offset by the fixed sampling time interval of the inertial measurement unit to obtain the normalized time interpolation ratio.

[0067] Specifically, the calculation relationship is expressed as follows: ; In the formula, This is the normalized time interpolation ratio. For the first The start exposure time of the frame, The first one located before that exposure time Sampling time of each inertial measurement unit The sampling time interval is a fixed inertial measurement unit.

[0068] Step 206: Based on the time interpolation ratio, perform linear interpolation on the inertial measurement unit data at the sampling times of the two adjacent inertial measurement units to obtain the sub-sampling period precision instantaneous angular velocity corresponding to the initial exposure time; Get the The instantaneous angular velocity vector at the sampling moment and the first sampling moment The instantaneous angular velocity vectors at each sampling time are used. A weighted fusion operation is performed on these two discrete angular velocity vectors using the calculated normalized time interpolation ratio. The interpolation relationship is expressed as: ; in, The instantaneous angular velocity vector generated by interpolation at the target time. This is the normalized time interpolation ratio. For the first The instantaneous angular velocity vector recorded at each sampling moment. For the next adjacent The instantaneous angular velocity vector recorded at each sampling moment.

[0069] Step 207: Obtain the subsampling period precision instantaneous angular velocity corresponding to the initial exposure time, and use the subsampling period precision instantaneous angular velocity to calculate the line-by-line rotation parameters.

[0070] Through the weighted fusion calculation described above, an instantaneous angular velocity with sub-sampling period precision exceeding the physical sampling hardware frequency was obtained in terms of temporal resolution. This instantaneous angular velocity was then used for subsequent line-by-line exposure interval integration. This online interpolation method is applied directly during video encoder runtime, refining the effective temporal resolution of 1 millisecond to a continuous floating-point precision matching the bandwidth of the angular velocity signal. In this way, without increasing offline calibration computation resources at the encoding end, the inter-frame prediction geometric projection deviation caused by hardware clock asynchrony is reduced.

[0071] Example 3: This example mainly describes the process of dynamically adjusting the look-ahead window size based on pipeline delay, and the process of constructing an adaptive mutation threshold and hysteresis smoothing mechanism using exponentially weighted statistics. The main steps are as follows: Step 301: Monitor the time span between the end of exposure of the original video frame sequence and the start of encoding of the current frame to determine the encoding pipeline delay; Specifically, the latency generated by the image signal processing pipeline differs between ultra-low latency video transmission systems and conventional camera recording systems. To obtain an accurate time span, the average pipeline latency is estimated during the encoder initialization phase by statistically analyzing the states of multiple consecutive frames of data. The specific calculation formula is as follows: ; in, To estimate the obtained encoding pipeline delay, The number of consecutive video frames used in the statistical calibration. This indicates that the sum of the deviations is calculated for a specified number of frame sequences. For the first The latest sample index of the inertial measurement unit buffer corresponding to the frame coding start time. For the first The sampling index of the inertial measurement unit corresponding to the end of the exposure of the last row of pixels in the frame. This is the fixed sampling time interval for the inertial measurement unit. This estimation allows for an accurate assessment of the actual latency capability under the current hardware operating conditions.

[0072] Step 302: Based on the ratio of the encoding pipeline delay to the video frame period, classify the look-ahead hierarchical mode; In this embodiment, the acquired encoding pipeline delay is compared with the set video frame period. Let the video frame rate be f. video The video frame period is then 1 / f video By judging t pipe The system adaptively routes to different decision branches within its time period interval, thereby ensuring that the system has deterministic look-ahead behavior under any delay constraint.

[0073] Step 303: When the look-ahead grading mode is the full look-ahead mode, set the look-ahead prediction window to cover the entire subsequent frame period; When a video frame period is detected with a coding pipeline delay greater than or equal to twice the video frame period, the look-ahead hierarchical mode is set to full look-ahead mode. In this mode, the prediction time span is taken as two full frame periods, enabling the system to perceive motion abrupt changes in the next two time intervals in advance when encoding the current frame.

[0074] Step 304: When the look-ahead grading mode is a partial look-ahead mode or a downgraded mode, the look-ahead prediction window is truncated according to the actual length of the coding pipeline delay, and the trigger weight of the mutation precursor prediction on the coding strategy label is adjusted accordingly.

[0075] When the coding pipeline delay is between 1 and 2 video frame periods, a partial look-ahead mode is entered. In this mode, the time span of the look-ahead prediction window is truncated to the actual coding pipeline delay length, covering only part or all of the time interval of the next frame. When the coding pipeline delay is strictly less than 1 video frame period, a degraded mode is entered. In degraded mode, the look-ahead prediction window may contain only a few inertial sampling points, resulting in decreased statistical reliability. Correspondingly, the trigger weight of the precursor prediction result on the generation of coding strategy labels is reduced, and the strategy switching is instead based more on the comparison result of the instantaneous energy of the current frame itself and the adaptive mutation threshold, thereby preventing misjudgments when there are insufficient samples.

[0076] Adjusting the trigger weight can be understood as correspondingly adjusting the triggering conditions of the mutation precursor prediction on the encoding strategy label, or correspondingly reducing the decision proportion of the mutation precursor prediction in the forward prediction window in the generation of the encoding strategy label.

[0077] In some scenarios, the final trigger determination of the strategy label can be set as a logical AND operation between the instantaneous energy determination result of the current frame and the look-ahead energy determination result. In full look-ahead mode, the transient enhancement mode is triggered only when the look-ahead energy exceeds the adaptive mutation threshold. In partial look-ahead mode, both the instantaneous energy of the current frame and the look-ahead energy are required to exceed the corresponding thresholds for triggering. In degraded mode, the look-ahead determination result is ignored, and the determination is based solely on the instantaneous energy of the current frame. Those skilled in the art will understand that the above weight adjustment method is not limited to logical gating implementation, and other conventional methods such as weighted summation can also be used.

[0078] Step 305: Extract the amplitude of the triaxial accelerometer signal in the current frame and perform sliding aggregation to obtain a short-time energy sequence; Using the initial exposure time of the current frame to be encoded as a reference, a fixed-length sliding window is opened along the historical timeline, and the scalar amplitude values ​​within this window are aggregated. The calculation relationship for this short-time energy aggregation is as follows: ; in, The short-time energy sequence value generated for the current frame. This represents the number of inertial measurement unit (IMU) sampling points contained within one video frame period. This indicates that the values ​​of each historical sampling point within the sliding window are summed. The scalar amplitude of the triaxial accelerometer signal. This is the index in the inertial sampling sequence that maps the start exposure time of the current frame. This is a sliding variable within the window.

[0079] Step 306: Use a pre-configured forgetting factor to perform smoothing iteration on the short-time energy sequence, and extract the exponential moving mean and moving standard deviation; To accommodate the significant differences in acceleration baselines across different motion scenarios, an exponentially weighted moving average algorithm is used to track the motion baseline in real time. The calculation relationship for smooth iteration is as follows: ; ; in, The updated exponential moving average of the current frame. For the pre-configured forgetting factor, The short-time energy of the current frame. This is the exponential moving average saved from the previous frame. The sliding variance of the current frame. This is the sliding variance of the previous frame. The sliding standard deviation can be extracted by taking the square root of the sliding variance.

[0080] In practical configurations, the forgetting factor The specific value can be adaptively adjusted according to the dynamic switching frequency of the motion scene. When the motion mode switching is more frequent, the factor can be increased appropriately to speed up the baseline tracking rate, and vice versa to enhance the smoothing effect. Those skilled in the art can determine the appropriate value without creative effort.

[0081] Step 307: Amplify the sliding standard deviation with a pre-configured sensitivity control coefficient and superimpose it onto the exponential moving mean to generate an adaptive mutation threshold; This process utilizes the confidence interval construction method in statistics to calculate the following relationship: ; in, The adaptive mutation threshold for the current frame. It is the exponential moving average. For the pre-configured sensitivity control coefficients, This represents the sliding standard deviation.

[0082] To further illustrate the update process of the adaptive mutation threshold, it is assumed that the normalized accelerometer energy acquired by the system enters a stable transition period. A pre-configured forgetting factor is set. Sensitivity control coefficient Assuming the exponential moving average of the previous frame... Sliding variance The corresponding standard deviation is If a slight perturbation occurs in the current frame, the extracted short-time energy... Substitute into the smoothing iteration process to calculate: the exponential moving average of the current frame. Current frame sliding variance Extract the current moving standard deviation. The resulting adaptive mutation threshold The aforementioned exponentially weighted iterative mechanism enables the adaptive mutation threshold to be dynamically adjusted in accordance with the changing trend of the motion baseline, thereby suppressing the technical defect of misjudging normal fluctuations as mutations.

[0083] Step 308: Aggregate the energy of the three-axis acceleration signals within the look-ahead prediction window as look-ahead energy; After determining the length of the forward prediction window, the energy aggregation value of the signal within that future window interval is calculated. The calculation relationship is as follows: ; in, Forward-looking energy, To predict the number of inertial measurement unit sampling points included within the look-ahead window, This indicates that the summation operation is performed on each sampling point within the future time window. The scalar amplitude of the triaxial accelerometer signal. Map the starting exposure time of the current frame to the index in the inertial sequence. For sliding variables that extend into the future.

[0084] Step 309: When the look-ahead energy exceeds the adaptive mutation threshold, trigger the generation of the encoding strategy label indicating the transient enhancement mode. When the look-ahead energy does not exceed the adaptive mutation threshold, generate the encoding strategy label indicating the normal mode.

[0085] The calculated look-ahead energy is compared with the adaptive mutation threshold of the current frame. To avoid false triggering caused by single-point energy peaks when the threshold is exceeded, the system further introduces a continuity confirmation condition, requiring that the proportion of sampling points exceeding the adaptive mutation threshold within the look-ahead prediction window must reach a preset ratio. Once the confirmation condition is met, the precursor determination takes effect, and the current frame to be encoded is marked as transient enhancement mode.

[0086] Step 310: After the encoding strategy tag indicating the transient enhancement mode is triggered in the current frame, the transient enhancement mode is forced to be maintained for a preset number of subsequent video frames; Because motion mutations in real physical space typically have a certain duration, and energy signals are prone to oscillations when crossing threshold boundaries, a hysteresis smoothing mechanism is introduced to eliminate coding strategy jitter caused by signal oscillations. Once a frame enters transient enhancement mode, a preset number of consecutive video frames are forced to remain in the same mode. The preset number typically matches the typical duration of motion mutations, for example, set to 2 to 4 frames, thus ensuring that the rising edge of the mutation event is fully covered.

[0087] Step 311: Continuously monitor the energy state of the triaxial accelerometer signal corresponding to subsequent video frames until the falling edge condition that no sudden change precursor is generated in multiple consecutive video frames is met. Then, allow the encoding strategy label to return to the normal mode to eliminate the jitter phenomenon of the strategy label oscillating frame by frame.

[0088] During the lock-in period, the signal's descent trend is evaluated frame by frame. A falling edge condition is determined only when the short-time energies of multiple consecutive frames in the lock-in period are all strictly below the corresponding adaptive mutation threshold, and the look-ahead energies do not trigger mutation precursors. The mode lock is then released, allowing the coding strategy labels of subsequent frames to fall back to the normal inter-frame prediction mode. Through this control logic, the algorithm improves the stability of encoder state transitions and avoids the additional computational overhead caused by frequent switching of the prediction architecture at mutation edges.

[0089] Example 4: The following describes the specific process of constructing an inverse spatial mapping model to perform inverse geometric sampling for input frames in different preprocessing states.

[0090] Step 401: Determine whether the original video frame sequence has undergone lens distortion correction processing; Specifically, when outputting video streams, action camera systems may provide image data with different geometric characteristics depending on the configuration of the image signal processor. Consumer-grade action cameras typically perform lens distortion correction internally, and the output image pixel coordinates satisfy a linear mapping relationship with an ideal pinhole camera model. However, in raw data acquisition scenarios, the acquired data may be uncorrected raw frames. These frames retain the radial distortion caused by wide-angle lenses, with image edges potentially shifting by tens of pixels. Therefore, before constructing the inverse spatial mapping model, it is necessary to determine whether the currently processed raw video frame sequence has already undergone lens distortion correction to select a matching coordinate projection mapping method.

[0091] Step 402: When a frame is confirmed to be calibrated, the pre-loaded camera intrinsic matrix and its inverse matrix are used, combined with the transpose operation of the row-by-row rotation parameters, to construct the row-by-row inverse homography matrix as the inverse spatial mapping model. In this embodiment, the linear pinhole model is precisely valid when the frame is confirmed to be calibrated. The pre-loaded camera intrinsic matrix and its inverse matrix are obtained, along with the row-by-row rotation parameters in orthogonal rotation matrix form corresponding to the current row. To achieve the reverse mapping mode, the target pixel coordinates of the current frame need to be reversed back to the source coordinate system of the video reference frame. Specifically, the calculation process involves multiplying the camera intrinsic matrix, the transpose of the row-by-row rotation parameters, and the inverse of the camera intrinsic matrix sequentially to obtain the third-order inverse homography matrix used for reverse mapping.

[0092] The calculation relationship is expressed as follows: ; in, For the first Frame number The row-by-row inverse homography matrix corresponding to each row, For the pre-loaded camera intrinsic parameter matrix, For the first Frame number The transpose matrix of the row-by-row rotation parameters. This is the inverse of the pre-loaded camera intrinsic parameter matrix.

[0093] Furthermore, when performing reverse geometric sampling, for the current frame... Given the integer column pixel coordinates of a row, they are converted into a homogeneous coordinate vector and left-multiplied by the aforementioned row-wise inverse homography matrix. Homogeneous coordinate normalization is then performed to obtain the sub-pixel reference sampling position in the video reference frame. This reverse mapping mechanism ensures that each integer pixel position in the current frame is queried exactly once, thus avoiding pixel overlap and holes that may occur with forward mapping.

[0094] Step 403: When an uncorrected frame is confirmed, the inverse spatial mapping model containing a nonlinear coordinate mapping link is constructed by using the pre-configured distortion model and anti-distortion model, the pre-loaded camera intrinsic parameter matrix, and the row-by-row rotation parameters.

[0095] In some alternative implementations, when an uncorrected frame is identified, a single linear matrix multiplication cannot accurately describe the spatial mapping relationship with severe distortion. Distortion and anti-distortion models describing the lens's physical characteristics are pre-obtained. For the current frame... Given the integer column pixel coordinates of a row, a non-linear coordinate mapping link is constructed to perform a series of transformations in sequence.

[0096] The projection result of left-multiplying the homogeneous coordinate vector of the distorted pixels in the current frame by the inverse of the camera intrinsic parameter matrix is ​​calculated, and this result is input into the anti-distortion model to obtain the distortion-free normalized coordinate vector. This first-stage mapping relationship is expressed as: ; in, To obtain the distortion-free normalized coordinate vector of the current frame, This is the nonlinear function corresponding to the anti-distortion model. The inverse of the camera intrinsic parameter matrix. This is the homogeneous coordinate vector of the target distorted pixel in the current frame.

[0097] Multiply the transpose matrix of the row-by-row rotation parameters by the distortion-free normalized coordinate vector, apply an inverse spatial rotation, and obtain the distortion-free normalized coordinate vector corresponding to the reference frame.

[0098] The second-stage mapping relationship is represented as follows: ; in, For the reference frame side distortion-free normalized coordinate vector, This is the transpose matrix of the row-by-row rotation parameters. It is a distortion-free normalized coordinate vector.

[0099] Based on this, the distortion-free normalized coordinate vector of the reference frame is input into the pre-configured distortion model, and the output is multiplied by the camera intrinsic parameter matrix and remapped back to the pixel coordinate system containing distortion features, thereby obtaining the actual sub-pixel reference sampling position in the video reference frame.

[0100] The mapping relationship in the third stage is expressed as follows: ; in, To obtain the homogeneous coordinate vector of subpixel distortion of the video reference frame, For the camera intrinsic parameter matrix, This is the nonlinear function corresponding to the distortion model. The reference frame is a distortion-free normalized coordinate vector.

[0101] The inverse spatial mapping model, which is composed of nonlinear functions and linear matrix multiplication, can calculate the inverse geometric projection position of the uncorrected frame under three-dimensional rotational motion pixel by pixel, thus meeting the high-precision motion compensation requirements in scenarios without distortion correction.

[0102] Example 5: How to correct the time starting point deviation caused by vertical pixel offset during the reverse mapping process.

[0103] Step 501, in the process of performing reverse geometric sampling on the pre-reconstructed video reference frame using the inverse spatial mapping model to generate the rolling shutter-aware prediction frame, further includes performing a second-order correction for line number self-consistency: Specifically, when performing a reverse mapping on a given line of the current frame, the sampled line number in the calculated video reference frame is usually not equal to the given line number of the current frame. There is a time deviation between the exposure time of the actual sampled line in the video reference frame and the integration starting point assumed when constructing the initial inverse spatial mapping model.

[0104] For example, the current frame number The line is reverse-mapped to the reference frame and falls on the first line. Okay. Due to the pitch rotation angle between frames. The vertical displacement of a pixel is approximately: ; Due to the actual sampling row number ≠ Hypothetical line number There is a time deviation in the exposure time. : ; Within this time deviation, the camera has an angular velocity It rotates by an additional angle, namely the second-order correction. : ; Project the rotation angle correction back into the pixel domain and multiply it again. : .

[0105] In the formula, The pixel row number of the current frame image; Reverse map the r-th row pixel of the current frame to the corresponding pixel row number in the reference frame; Accumulated rotation angle between frames in the pitch direction; The vertical focal length is the camera intrinsic parameter matrix; The time deviation between the actual exposure time of a pixel and the assumed exposure time; The line scan period for a single row of pixels in the image sensor; The camera's rotational angular velocity; This is a second-order correction for the rotation angle; This represents the pixel projection deviation in the vertical direction.

[0106] This time deviation has a linear product relationship with the line scan period, which leads to the introduction of a second-order correction for the rotation angle. This second-order correction for the rotation angle is positively correlated with the product of the camera angular velocity, the intrinsic longitudinal focal length, the inter-frame cumulative rotation angle, and the line scan period.

[0107] To assess the impact of this deviation, the projected displacement of this second-order correction in the pixel domain is calculated. This displacement is positively correlated with the product of the camera angular velocity, the square of the intrinsic longitudinal focal length, the cumulative inter-frame rotation angle, and the line scan period. The calculation relationship for this displacement can be expressed as: ; in, This represents the pixel projection deviation in the vertical direction. This is the absolute value of the camera's angular velocity, specifically the absolute value of the camera's instantaneous angular velocity on the dominant rotation axis that causes pixel displacement in the vertical direction. The vertical focal length is the intrinsic parameter matrix of the camera. The absolute value of the cumulative rotation angle between frames in the pitch direction. This refers to the row scan cycle parameter.

[0108] A pre-defined pixel threshold is used as the trigger mechanism. When the absolute value of the initial pixel offset in the vertical direction is greater than 2, the system starts an iterative correction process; otherwise, it maintains a zero-order approximation to save computational overhead.

[0109] Step 502: For a given line number of the current frame, obtain the initial reference sampling line number corresponding to it in the video reference frame through the initial mapping of the inverse spatial mapping model; Perform the 0th round of initial estimation. Using the given row number of the current frame as the end point of the integration interval, and the exposure time of the same row number in the video reference frame as the initial integration start point, calculate the rotation matrix within the first integration interval.

[0110] The rotation matrix is ​​used to construct an inverse spatial mapping model for the initial state, and an inverse mapping operation is performed on the homogeneous coordinate vector formed by the given column coordinates and the given row number in the current frame. The vertical coordinate component in the mapping result is extracted and used as the initial reference sampling row number.

[0111] Step 503: Based on the deviation between the initial reference sampling row number and the given row number, redefine the time integration interval of the inertial measurement unit data to calculate the corrected row-by-row rotation parameters; Entering the first round of correction. Since the initial reference sampling line number obtained in the previous stage can more accurately reflect the actual physical sampling exposure time of the video reference frame, the time integration starting point of the inertial measurement unit data is replaced with the exposure time corresponding to the initial reference sampling line number.

[0112] The corrected time integration interval is defined as starting from the exposure time of the initial reference sampling line number and ending at the exposure time of the given line number in the current frame. Based on this corrected time integration interval, the angular velocity sequence is re-integrated to obtain the corrected line-by-line rotation parameters.

[0113] Since the initial reference sample row numbers obtained by reverse mapping different column coordinates within the same given row number may differ, this means that the corrected row-by-row rotation parameters are related not only to the row number but also to the column number.

[0114] To control computational costs, only three column coordinate points—the first, middle, and last column of the given row—are extracted for the aforementioned redefinition and integration calculations. For the remaining column coordinate points within the given row, their corresponding correction parameters are obtained by performing linear interpolation on the correction rotation parameters of these three column coordinate points.

[0115] Step 504: Update the inverse spatial mapping model using the corrected line-by-line rotation parameters, and perform inverse geometric sampling again. Obtain the corrected reference sampling coordinates to generate the rolling shutter sensing prediction frame.

[0116] The inverse spatial mapping model is reconstructed using the obtained corrected line-by-line rotation parameters, and a second inverse geometric sampling is performed for the same coordinate points in the current frame. Since the correction amount of the starting point of the time integration is a second-order small quantity, after one round of iteration, the output secondary sampling coordinate position can converge to sub-pixel accuracy.

[0117] In some alternative implementations, in order to maintain the efficient computational structure of sharing a single inverse homography matrix row by row, a piecewise linear approximation method can be used instead of the above-mentioned pixel-by-pixel iterative correction.

[0118] Specifically, the current frame is divided into 16 to 32 segments in the vertical direction.

[0119] The complete correction process of initial mapping, integration interval redefinition, and secondary mapping is performed only once in the center row of each row segment to obtain the corrected row-by-row rotation parameters for that row segment's center row. For the remaining rows within each row segment, linear interpolation in the vertical direction is directly performed using the corrected row-by-row rotation parameters of the center rows of adjacent row segments to obtain the corresponding interpolation rotation matrix. This alternative approach can control the residual coordinate projection deviation at the sub-pixel level while maintaining the consistency and simplicity of the computational structure.

[0120] Example 6: This example describes the specific implementation process of extracting the accurate rotation matrix using quaternion incremental chain multiplication to suppress the integration error caused by the noncommutativity of three-dimensional rotation under extreme conditions such as high-speed spin.

[0121] Specifically, the line-by-line rotation parameters are calculated using inertial measurement unit data, including: When an action camera is in normal operating conditions, the cumulative rotation angle within a single frame interval is on the order of a very small magnitude. The first-order linear approximation of the Rodriguez formula is usually used to obtain the rotation matrix.

[0122] However, in extreme applications such as high-speed free-fall spin, the rotation in three-dimensional space belongs to a Lie group structure, which is mathematically noncommutative. Directly performing trapezoidal integration and linearly summing the angular velocity time series over a single-span integration interval leads to the inability to ignore second-order cross terms, resulting in unacceptable projection errors of several or even tens of pixels. Therefore, it is necessary to adaptively switch the mathematical solution path of the integration algorithm at the underlying level according to the drastic change in the actual rotational attitude.

[0123] Step 601: Evaluate the cumulative rotation within the current row exposure range; Here, the integration interval of the current line (i.e., the exposure interval of the current line) refers to the time span from the exposure time of the corresponding line in the video reference frame to the exposure time of the current line in the current frame.

[0124] Specifically, the frame currently marked as transient enhancement mode or its time-axis neighboring frames are obtained, and the inter-frame cumulative rotation angle modulus within the integration interval of the current line to be processed is calculated.

[0125] The cumulative rotation angular modulus, as a one-dimensional physical feature, directly reflects the degree of drastic change in the spatial attitude of the camera within a single frame exposure time. Therefore, a rotation threshold is set. For example, the specific value of this rotation threshold can be set to 0.1 radians.

[0126] The obtained inter-frame cumulative rotation angle magnitude is compared with the preset rotation threshold, and this is used as the sole criterion for switching the integral solution model.

[0127] Step 602: When the cumulative rotation exceeds a preset rotation threshold, the line exposure interval is divided into multiple sampling sub-intervals, and the incremental rotation quaternion corresponding to each sampling sub-interval is extracted based on the inertial measurement unit data. When the evaluation determines that the cumulative rotation angle modulus exceeds the preset rotation threshold, the single integral interval that originally spanned the corresponding exposure times of the two lines of the video reference frame and the current frame is uniformly divided into a preset number of sampling sub-intervals.

[0128] As an example of specific parameter constraints, in a system with a video configuration frame rate of 60 frames per second and an inertial measurement unit sampling frequency of 1000 Hz, the single integration interval is divided into 16 sampling sub-intervals, and the time span of each sampling sub-interval is exactly one inertial measurement unit sampling time interval.

[0129] For each segmented sampling sub-interval, the instantaneous angular velocities at the start and end points of the sub-interval are first extracted. The trapezoidal integral method is then used to calculate the incremental rotation angle vector within that specific sampling sub-interval. The calculation relationship for this incremental rotation angle vector is expressed as: ; in, For the first The incremental rotation angle vector of each sampling sub-interval, The instantaneous angular velocity recorded at the beginning of this sub-interval. The instantaneous angular velocity recorded at the end of this sub-interval. This represents the sampling time interval for this sampling sub-interval.

[0130] At this computational granularity, because the span of a single subinterval is extremely short, the magnitude of its local rotation angle is much smaller than the limit that produces non-commutative errors. Therefore, the linear cumulative approximation holds completely within the subinterval. This incremental rotation angle vector is mapped and transformed into the corresponding incremental rotation quaternion. This mapping transformation relationship is expressed as: ; in, To extract the generated incremental rotation quaternions, The incremental rotation angle vector The calculated modulus, It is a cosine mapping function. It is a sine mapping function. This is the transpose operation for vector coordinates.

[0131] To ensure the robustness of the software engineering system under extreme conditions and to avoid division-by-zero exceptions when the vector magnitude approaches zero during floating-point division operations, the magnitude value is checked for boundary conditions before being substituted into the above trigonometric function mapping.

[0132] When the modulus is detected to be strictly less than a preset small truncation threshold, such as 10... -8 When the power is raised to the power of 1, the incremental rotation quaternion is obtained directly using an approximate equation based on Taylor expansion.

[0133] The approximate equation's computational relationship can be expressed as follows: ; in, To extract the generated incremental rotation quaternions, This is the incremental rotation angle vector corresponding to the sampling sub-interval. This is the transpose operation for vector coordinates.

[0134] After completing the calculation of any of the above branches, the vector magnitude normalization process is forcibly performed on the output incremental rotated quaternion.

[0135] Step 603: Perform quaternion chain multiplication operations on the incremental rotated quaternions of each of the sampling sub-intervals in sequence to obtain the cumulative quaternion; Initialize a constant vector (1,0,0,0) with four components in memory. T The quaternion serves as the basic unit and is used as the starting point for the cumulative chain of quaternions.

[0136] Based on the chronological order of the physical timestamp sequence, the incremental rotation quaternion generated in each sampling sub-interval is extracted one by one, and Hamiltonian quaternion multiplication is performed on it and the current cached cumulative state result.

[0137] The iterative chain multiplication calculation relationship is expressed as follows: ; in, Update the accumulated quaternion obtained in the current step. This is the previous accumulated quaternion cached before the multiplication step is executed. For the corresponding number Incremental rotation quaternions extracted from each time sampling sub-interval This is the standard Hamiltonian quaternion multiplication operator.

[0138] Furthermore, in order to suppress quaternion eigenmode drift caused by the computer performing multiple floating-point multiplications and rounding in highly intensive operation loops, the system has a reset mechanism.

[0139] After each chain multiplication iteration operation with a preset number of steps, such as 10 steps, the magnitude of the sum of squares of the current cumulative quaternion is extracted, and each dimension component is divided by the magnitude to force a periodic magnitude normalization correction operation, thereby ensuring that the cumulative quaternion is always strictly on the unit hypersphere.

[0140] Step 604: Extract matrix elements from the accumulated quaternion to generate an accurate rotation matrix as the row-by-row rotation parameters. When the accumulated rotation degree does not exceed the preset rotation threshold, the row-by-row rotation parameters are generated using a first-order linear approximation based on the accumulated rotation angle vector within the row exposure interval.

[0141] After traversing and completing the chain multiplication iteration task corresponding to all sub-intervals of the current entire row exposure time interval, the final closed cumulative quaternion is obtained. This cumulative quaternion is known to contain a first real scalar and three suffix imaginary vector components. Based on the orthogonal matrix analytic mapping rule from Lie algebra to rotation groups, the quadratic terms and cross-product terms of these four components are expanded and combined to calculate nine independent values, which are then filled into the nine element coordinate positions of the third-order matrix.

[0142] The analytical extraction of computational relationships is expanded using matrix coordinates. For example, the computational relationship for the elements in the first row and first column of the matrix diagonal is represented as follows: ; in, For the first row and first column diagonal elements of the generated exact rotation matrix, This is the second imaginary part of the final accumulated quaternion. This is the third imaginary component of the final accumulated quaternion.

[0143] Similarly, the calculation relationship for extracting the non-diagonal elements of the first row and second column of the matrix is ​​expressed as follows: ; in, For the first row and second column off-diagonal elements of the generated exact rotation matrix, For the real part of the final accumulated quaternion, This is the first imaginary part of the final accumulated quaternion. This is the second imaginary component. This is the third imaginary component.

[0144] By employing parallel analytical mapping rules, all nine target matrix elements are calculated and combined to generate the third-order precise rotation matrix corresponding to the current row. This precise rotation matrix, generated based on quaternion closed-loop integrals, can rigorously reflect the actual physical attitude evolution of the camera under violent tumbling. It is directly overwritten as the row-by-row rotation parameters for that row and seamlessly fed into the downstream inverse spatial mapping model construction stage.

[0145] When completing inter-frame compensation and finally encapsulating and outputting auxiliary data to the lossless compressed video stream, this parameter system can be written by the video encoder as an equivalent rotation angle vector of three floating-point components, or it can be directly transmitted in a compact payload format using the final cumulative quaternion of four floating-point components. This provides the decoder with a symmetrical deterministic reconstruction basis that suppresses cumulative truncation errors.

[0146] Example 7: When converting a physical floating-point model to an engineering integer encoder, the main processing steps for issues such as fixed-point closure specification, compensation back-off mechanism for local anomaly line bands, and differentiated prediction residuals are as follows: Step 701: Use a pre-set integer domain filter to perform filtering operations on the reference pixel data at the non-integer coordinates to obtain intermediate precision features; In the reverse mapping operation, the reference sampling position of the obtained video reference frame is usually a non-integer coordinate. To obtain the specific pixel value at this sub-pixel position, a fixed-point sub-pixel interpolation process consistent with the target coding standard is adopted.

[0147] Generally, a pre-defined integer domain filter is used to obtain intermediate precision features from the data at the non-integer coordinates. Specifically, a pre-defined integer domain filter is extracted, such as an 8-tap half-pixel filter for the luminance component. The system applies this integer domain filter to the integer row pixel values ​​around the reference sampling position, performing convolution operations in the horizontal and vertical directions.

[0148] Since the sum of the coefficients of the integer domain filter is greater than 1, the bit width of the feature data output after convolution filtering will be expanded. The expanded result is used as the intermediate precision feature.

[0149] Step 702: Apply fixed normalized shift and rounding bias parameters to process the intermediate precision features, and output an integer prediction value limited by the bit depth of the original data, so as to generate the rolling shutter sensing prediction frame in a strictly closed integer domain.

[0150] To meet the fundamental requirements of lossless compression, the irreversible rounding error introduced by floating-point interpolation must be eliminated. After obtaining the intermediate precision characteristics, a deterministic arithmetic right shift operation is performed.

[0151] The fixed-point constraint calculation relationship is expressed as follows: ; in, The output is the integer predicted value. These are intermediate-precision features obtained through filter convolution calculations. For fixed rounding bias parameters, For arithmetic right shift operator, It is a fixed normalized shift parameter.

[0152] After this shift operation, a further pruning operation is performed on the result, forcibly limiting its numerical range to 0 to 2. B -1 Within the closed interval, where B is the bit depth of the original data.

[0153] Through the above fixed-point interpolation process, the encoder and decoder use the exact same integer operation sequence to generate the same prediction benchmark, so that the entire link from prediction, residual calculation to reversible transformation can be strictly closed in the integer domain.

[0154] Step 703: Divide the rolling shutter sensing prediction frame according to the preset line band width, evaluate the mean square value of the prediction residual of each line band region relative to the current frame, and calculate the mean square value of the prediction residual of all line band regions to obtain the mean square value of the residual of the whole frame. After generating the rolling shutter sensing prediction frame, its prediction quality is evaluated region by region.

[0155] Specifically, the image is horizontally divided according to a preset line band width, which can be configured as the height of the encoding block.

[0156] Extract the predicted pixels within each line band region, and subtract them one by one from the corresponding original pixels in the original video frame sequence to calculate the mean square value of the prediction residual for that line band region.

[0157] Simultaneously, the mean squared value of the global prediction residuals covering all pixels in the entire frame is calculated. The arithmetic mean of the mean squared values ​​of the prediction residuals for all line band regions is then calculated to obtain the mean squared value of the residuals for the entire frame.

[0158] Step 704: When the predicted residual mean square value of the target line band region is detected to be significantly greater than the mean square value of the full frame residuals (i.e., greater than a preset multiple of the mean square value of the full frame residuals), the target line band region is determined to be an abnormal region. The mean square value of the predicted residuals for each line band region is compared with the mean square value of the global predicted residuals. When the value of a target line band region exceeds a preset multiple of the global value, it is identified as an abnormal region.

[0159] For example, the preset multiplier can be set to 2. This abnormal state is usually caused by independently moving foreground objects or sudden changes in scene depth, making it impossible for a single global rotation model to fully describe the true motion and displacement of the local area.

[0160] Step 705: For the abnormal region, revert to the block-level motion search process, regenerate the predicted pixel value of the target line band region using the searched block-level translation motion vector, and refresh the corresponding position in the rolling shutter sensing prediction frame.

[0161] For target line bands identified as abnormal regions, the rotation compensation based on the inverse spatial mapping model is stopped, and the conventional translation motion search module built into the video encoder is invoked.

[0162] Using the abnormal region in the current frame as the matching target, perform a block-by-block translation search within the reference frame to extract the block-level translation motion vector that minimizes the block matching cost.

[0163] The pixel is moved using the block-level translation motion vector, and the predicted pixel value of the local line band region is regenerated. This value is then used to replace the abnormal predicted pixels that were originally caused by the failure of the rotation model.

[0164] This local compensation backoff correction mechanism ensures high-precision prediction of pure rotating background regions while providing a robust compensation fallback solution for foreground regions containing complex translational parallax.

[0165] Step 706: When the coding strategy label indicates transient enhancement mode, calculate the block-level spatial prediction residual of the current coding block using the neighboring pixels that have completed local reconstruction, and calculate the block-level inter-frame prediction residual of the current coding block based on the rolling shutter sensing prediction frame. After local correction, the encoding strategy label generated for the current frame is read. When the encoding strategy label indicates transient enhancement mode, it means that the current frame is in a transition period of sudden motion change, and the reliability of relying solely on the time dimension reference frame is at risk of decline.

[0166] In this state, a dual prediction evaluation process is initiated at the coding block level.

[0167] For the current block to be encoded, on the one hand, the block-level spatial prediction residual is calculated according to the spatial angle direction prediction mode with reference to the reconstructed adjacent pixels; on the other hand, the block-level inter-frame prediction residual is calculated based on the rolling shutter sensing prediction frame corrected by the above steps.

[0168] Step 707: Compare the energy of the block-level spatial prediction residual and the block-level inter-frame prediction residual, and select the one with smaller energy as the prediction residual of the corresponding coding block.

[0169] The absolute values ​​of the block-level spatial prediction residuals and the sum of the absolute values ​​of the block-level inter-frame prediction residuals are calculated separately and used as a measure of alignment energy.

[0170] The smaller of the two values ​​is selected, and its corresponding residual matrix is ​​used as the prediction residual of the final output of the current coding block.

[0171] This adaptive routing selection based on the energy minimization criterion enables intelligent allocation of spatial and temporal predicted paths within abrupt frames, further reducing the data load to be encoded.

[0172] After the residual generation is completed, an invertible integer transformation is applied to the predicted residual to map the spatial domain residual to the transform domain, and then input into a context-adaptive binary arithmetic encoder for entropy compression.

[0173] The final compressed residual is then combined with the corresponding auxiliary reconstruction parameters. It should be noted that, regarding the encapsulation format of the auxiliary reconstruction parameters, either the line-by-line rotation parameters can be extracted from the rotation matrix in reverse and written into a rotation angle vector with three floating-point components, or the cumulative quaternion containing four components obtained during the large-angle integration stage can be directly encapsulated into the frame header structure, thus outputting the final lossless compressed video stream along with the compressed residual.

[0174] It should be noted that in a certain loop-outside preprocessing method, the original rolling shutter frame is first resampled into an equivalent global shutter frame by an independent rolling shutter correction module, and then the corrected frame is sent as input to the standard video encoder. Let the original frame be... The corrected frame is ,in Indicates rolling shutter correction transformation. This represents resampling interpolation error. The encoder... During encoding, block-level motion compensation is used from the reference frame. Generate prediction frames The coding residual is In lossless mode, the encoder needs to be fully preserved. All information. Because the encoder does not know... The specific parameters of the transformation, It does not contain geometric compensation information for inter-frame rotation. The data still includes inter-frame geometric differences caused by camera rotation and resampling errors. Both constitute the main source of residual energy.

[0175] In the in-loop integration method of this application, the encoder directly processes the original rolling shutter frame. Encoding is performed. Using the line-by-line rotation parameters provided by the inertial measurement unit, the reference frame is mapped inversely within the encoder's prediction loop. Generate rolling shutter sensing prediction frames The mapping calculation uses integer-field fixed-point interpolation to ensure precise bit consistency between the encoder and decoder. The encoded residual is... .because It already includes information on inter-frame geometric transformations caused by camera rotation. The redundant portion is no longer included. The residual energy mainly comes from elements that cannot be explained by the rotation model, such as scene depth changes, foreground motion, and measurement noise. In a typical static scene with camera rotation, the inter-frame differences caused by camera rotation account for the main part of the original inter-frame residual energy. The in-loop integration method directly eliminates this part of the energy through geometric compensation, which significantly reduces the residual information entropy and thus improves the lossless compression efficiency.

[0176] In summary, at the implementation level of inter-frame prediction, existing solutions use inertial measurement unit (IMU) data for frame-level coding parameter selection, while the predicted frames within the encoder are still generated by standard block-level motion compensation. In contrast, this application directly uses the line-by-line rotation parameters measured by the IMU to construct a line-by-line inverse spatial mapping model, generating predicted frames that consider the rolling shutter effect within the encoder's prediction loop. This means that the predicted frame itself already contains the geometric transformation information caused by camera rotation, and the inter-frame residual no longer contains this redundancy, thus directly reducing the information entropy of the residual in lossless coding scenarios.

[0177] Regarding the coupling of rolling shutter processing and encoding, existing schemes treat rolling shutter correction as a preprocessing step outside the encoding loop, where the resampling interpolation error of the correction frame is irreversibly mixed into the encoded signal. In contrast, this application directly embeds the line-by-line geometric mapping into the prediction loop, using integer-domain fixed-point interpolation to ensure that the encoder and decoder generate bit-accurate prediction frames. The residual is losslessly encoded in the integer domain, without introducing additional distortion. This implementation requires addressing the encoder-decoder consistency constraints that are not addressed by preprocessing methods, including the integer-domain closure of fixed-point subpixel interpolation, the transmission of line-by-line rotation parameters as auxiliary reconstruction information along with the bitstream, and second-order correction for line number self-consistency.

[0178] Regarding the decision-making mechanism of the encoding strategy, existing schemes passively adjust encoding parameters based on the current motion speed or amplitude detected by sensors, such as increasing quantization parameters when rapid motion is detected. This application, however, utilizes a look-ahead window formed by combining jerk signals with the inherent delay of the encoding pipeline to proactively route the encoding strategy from inter-frame geometric prediction to spatial prediction before abrupt changes in motion state. This avoids the encoding efficiency loss caused by a surge in residuals after the geometric model fails due to motion abrupt changes, requiring a passive switch. There is a functional coupling relationship between the jerk signal and the progressive rotation geometric model: zero jerk indicates a constant rotational angular velocity, in which case the progressive rotation model can accurately compensate for inter-frame geometric transformations, making it suitable for inter-frame geometric prediction mode; significantly non-zero jerk indicates an abrupt change in rotation state, at which point the rigid body assumption of the progressive rotation model may fail, and a proactive switch to spatial prediction mode should be initiated. This forward-looking strategy routing mechanism based on physical signals is not addressed in existing sensor-assisted encoding schemes and rolling shutter correction schemes.

Claims

1. A lossless video compression method for action cameras, characterized in that, include: Acquire the raw video frame sequence from the action camera and the corresponding inertial measurement unit data, and align their time bases; Triaxial accelerometer signals are extracted based on the aligned inertial measurement unit data; Evaluate the coding pipeline latency of the current frame to determine the look-ahead prediction window; The triaxial acceleration signal within the prospective prediction window is used to perform abrupt change precursor prediction and generate encoding strategy labels. The row-by-row rotation parameters are calculated using the inertial measurement unit data, and an inverse spatial mapping model is constructed based on the row-by-row rotation parameters; The inverse spatial mapping model is used to perform inverse geometric sampling on the pre-reconstructed video reference frame to generate a rolling shutter sensing prediction frame; Based on the prediction path defined by the encoding strategy label, and combined with the rolling shutter sensing prediction frame, the prediction residual of the current frame in the original video frame sequence is calculated; The predicted residual is subjected to reversible transformation and entropy encoding, and the encoded predicted residual is encapsulated with auxiliary reconstruction information containing the line-by-line rotation parameters to output a lossless compressed video stream.

2. The method according to claim 1, characterized in that, Evaluate the coding pipeline latency of the current frame to determine the look-ahead window, including: Monitor the time span between the end of exposure of the original video frame sequence and the start of encoding of the current frame to determine the encoding pipeline delay; Based on the ratio of the encoding pipeline delay to the video frame period, a look-ahead hierarchical mode is defined. When the look-ahead grading mode is the full look-ahead mode, the look-ahead prediction window is set to cover the entire subsequent frame period; When the look-ahead grading mode is a partial look-ahead mode or a downgraded mode, the look-ahead prediction window is truncated according to the actual length of the coding pipeline delay, and the trigger weight of the mutation precursor prediction on the coding strategy label is adjusted accordingly.

3. The method according to claim 2, characterized in that, Using the triaxial accelerometer signal within the look-ahead prediction window, abrupt change precursor prediction is performed, generating encoding strategy labels, including: The amplitude of the triaxial acceleration signal within the current frame is extracted and subjected to sliding aggregation to obtain a short-time energy sequence; The short-time energy sequence is smoothed iteratively using a pre-configured forgetting factor to extract the exponential moving mean and moving standard deviation; The sliding standard deviation is amplified by a pre-configured sensitivity control coefficient and superimposed on the exponential moving mean to generate an adaptive mutation threshold; The energy of the triaxial acceleration signals within the look-ahead prediction window is aggregated as the look-ahead energy; When the look-ahead energy exceeds the adaptive mutation threshold, the generation of the encoding strategy label indicating the transient enhancement mode is triggered.

4. The method according to claim 1, characterized in that, Constructing an inverse spatial mapping model based on the row-by-row rotation parameters includes: Determine whether the original video frame sequence has undergone lens distortion correction processing; When a frame is confirmed to be calibrated, the pre-loaded camera intrinsic matrix and its inverse matrix are used, combined with the transpose operation of the row-by-row rotation parameters, to construct the row-by-row inverse homography matrix as the inverse space mapping model. When a frame is identified as uncorrected, the inverse spatial mapping model, which includes a nonlinear coordinate mapping link, is constructed by using the pre-configured distortion model and anti-distortion model, the pre-loaded camera intrinsic parameter matrix, and the row-by-row rotation parameters.

5. The method according to claim 1, characterized in that, In the process of generating a rolling shutter-aware prediction frame by performing reverse geometric sampling on a pre-reconstructed video reference frame using the aforementioned inverse spatial mapping model, the process also includes performing a second-order correction for line number self-consistency: For a given line number of the current frame, the initial reference sampling line number corresponding to it in the video reference frame is obtained through the initial mapping of the inverse spatial mapping model; Based on the deviation between the initial reference sampling row number and the given row number, the time integration interval of the inertial measurement unit data is redefined in order to calculate the corrected row-by-row rotation parameters; The inverse spatial mapping model is updated using the corrected row-by-row rotation parameters, and inverse geometric sampling is performed again.

6. The method according to claim 1, characterized in that, Calculating line-by-line rotation parameters using the inertial measurement unit data includes: Assess the cumulative rotation within the current line's exposure range; When the cumulative rotation exceeds a preset rotation threshold, the line exposure interval is divided into multiple sampling sub-intervals, and the incremental rotation quaternion corresponding to each sampling sub-interval is extracted based on the inertial measurement unit data. Perform quaternion chain multiplication operations sequentially on the incremental rotated quaternions of each of the aforementioned sampling sub-intervals to obtain the cumulative quaternion; Matrix elements are extracted from the accumulated quaternions to generate an exact rotation matrix as the row-by-row rotation parameters.

7. The method according to claim 1, characterized in that, Generating a rolling shutter-aware prediction frame also includes fixed-point subpixel interpolation of the non-integer coordinates generated by inverse geometric sampling: The reference pixel data at the non-integer coordinates is filtered using a pre-defined integer domain filter to obtain intermediate precision features; The intermediate precision features are processed by applying fixed normalized shift and rounding bias parameters, and the output is an integer prediction value limited by the bit depth of the original data, so as to generate the rolling shutter sensing prediction frame in a strictly closed integer domain.

8. The method according to claim 3, characterized in that, Based on the prediction path defined by the encoding strategy label, and combined with the rolling shutter-aware prediction frame, the prediction residual of the current frame in the original video frame sequence is calculated, including: When the coding strategy label indicates transient enhancement mode, the block-level spatial prediction residual of the current coding block and the block-level inter-frame prediction residual based on the rolling shutter sensing prediction frame are calculated respectively. The energy of the block-level spatial prediction residual and the block-level inter-frame prediction residual are compared, and the one with smaller energy is selected as the prediction residual of the corresponding coding block.

9. The method according to claim 1, characterized in that, Aligning the original video frame sequence with the time reference of the inertial measurement unit data is performed using a pre-calibrated clock offset; this clock offset is pre-acquired through the following offline calibration steps: Using the clock offset candidate value as the optimization variable, the inter-frame optical flow displacement and the corresponding inertial measurement unit integral displacement are calculated respectively. The pre-calibrated clock offset is obtained by minimizing the residual between the inter-frame optical flow displacement and the integral displacement of the inertial measurement unit.

10. The method according to claim 1, characterized in that, After generating the encoding strategy labels, the application of hysteresis smoothing criteria is also included: After the encoding strategy tag indicating transient enhancement mode is triggered in the current frame, the transient enhancement mode is forced to be maintained for a preset number of subsequent video frames; The energy state of the triaxial accelerometer signal corresponding to subsequent video frames is continuously monitored until the falling edge condition that no sudden change precursor is generated in multiple consecutive video frames is met, at which point the encoding strategy label is allowed to return to the normal mode.

Images