Binocular camera low latency synchronization method

Abstract

The application belongs to the technical field of image processing, and particularly relates to a binocular camera low-delay synchronization method, which comprises the following steps: establishing data queues for a main-view acquisition stream and an auxiliary-view acquisition stream respectively, and acquiring acquisition frames with acquisition time identifiers from the data queues; taking the acquisition time identifier of an acquisition frame in the main-view acquisition stream as a unified target time; and based on at least two acquisition frames in the auxiliary-view acquisition stream within the neighborhood of the unified target time, generating a re-projected auxiliary-view frame at the unified target time by executing a cross-view stream time re-projection algorithm; taking the acquisition frame in the main-view acquisition stream as a reference acquisition frame, and outputting the reference acquisition frame and the re-projected auxiliary-view frame together to form a pair of time frame pairs. The application not only improves the synchronization accuracy and stability under low-delay conditions, but also reduces hardware dependence and storage overhead, and has the advantages of strong real-time performance, high adaptability, simple implementation and good scalability.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a low-latency synchronization method for binocular cameras. Background Technology

[0002] In recent years, with the widespread application of multi-camera imaging, stereo vision, augmented reality, and visual navigation in unmanned systems, synchronous imaging and temporal calibration of multiple cameras have become crucial aspects of vision system design. Traditional multi-camera synchronization schemes typically rely on hardware triggering, achieving inter-frame time alignment by sharing the same clock signal or an external synchronization line. While this method offers high reliability in fixed platforms or industrial equipment, in lightweight mobile terminals, intelligent driving sensor clusters, and consumer imaging devices, hardware synchronization lines often cannot strictly guarantee low latency, low drift, and flexible deployment, resulting in significant time-axis deviations in multi-view images. Due to microsecond to millisecond-level inconsistencies in exposure, buffering, and transmission paths of different cameras, even frames nominally "acquired simultaneously" will still experience physical time drift in the actual scene, causing misalignment and blurring in the calculated depth map, disparity map, or fused image.

[0003] To reduce hardware dependence, researchers have begun exploring software-based temporal correction and inter-frame reconstruction methods. Existing techniques often employ temporal interpolation and optical flow extrapolation to synchronize data from multiple cameras. Specifically, at the time point of the main view camera frame, a reprojected image is generated based on the optical flow estimation between adjacent frames of the secondary view camera. Common implementations include dense optical flow-based frame interpolation, motion-compensated video reconstruction, and deep learning-based temporal alignment networks. These methods achieve good results in static scenes, but still have several limitations in complex scenes. First, traditional optical flow extrapolation methods are highly sensitive to motion estimation errors. In cases of rapid camera movement or partial object occlusion, the inconsistency between the forward and reverse optical flow fields can cause extrapolated pixel position drift. Most existing methods only use unidirectional optical flow or symmetrical optical flow averaging to generate intermediate frames, making it difficult to effectively determine occluded areas and truly reprojectable pixels, resulting in obvious ghosting or fragmented structures in the synthesized frames. Summary of the Invention

[0004] Therefore, the main objective of this invention is to provide a low-latency synchronization method for binocular cameras. This method establishes an independent zero-copy / lock-free circular queue between the primary and secondary view acquisition streams, enabling efficient transmission and lock-free access of acquisition frame data, thereby reducing system latency and avoiding multi-threaded contention. The method uses the acquisition time identifier of a specific acquisition frame in the primary view acquisition stream as a unified target time. Multiple frames within the temporal neighborhood are selected in the secondary view acquisition stream, and a cross-view stream temporal reprojection algorithm is executed to generate a reprojected secondary view frame at the corresponding time, thus achieving high-precision alignment of the two cameras in the temporal dimension. Cross-view stream temporal reprojection utilizes bidirectional consistency checks of the forward and backward optical flow fields to automatically eliminate occlusion and unstable regions, and performs time extrapolation and interpolation sampling based on reliable pixels to generate a geometrically and temporally consistent secondary view image. After outputting the time-frame pairs, this scheme also employs an adaptive fine-tuning control mechanism for the acquisition time difference to ensure stable synchronization during long-term operation. Compared with existing hardware synchronization and unidirectional interpolation methods, this invention not only improves synchronization accuracy and stability under low latency conditions, but also reduces hardware dependence and storage overhead. It has the advantages of strong real-time performance, high adaptability, simple implementation and good scalability, and can be widely used in the fields of stereo vision, augmented reality, autonomous driving and multi-sensor fusion.

[0005] The technical solution adopted in this invention is as follows:

[0006] A low-latency synchronization method for binocular cameras includes the following steps:

[0007] Establish data queues for the main view acquisition stream and the secondary view acquisition stream respectively, and obtain the acquisition frames with acquisition time identifiers from the data queues;

[0008] The acquisition time identifier of a certain acquisition frame in the main view acquisition stream is used as the unified target time; and based on at least two acquisition frames in the secondary view acquisition stream that are in the neighborhood of the unified target time, a reprojected secondary view frame at the unified target time is generated by executing the cross-view stream time reprojection algorithm.

[0009] The capture frames in the main view capture stream are used as reference capture frames, and together with the reprojected auxiliary view frames, they form a pair of time frames and are output.

[0010] Furthermore, the steps of establishing data queues for the main view acquisition stream and the auxiliary view acquisition stream specifically include: establishing independent zero-copy / lock-free circular queues for the main view acquisition stream and the auxiliary view acquisition stream respectively. The zero-copy / lock-free circular queue consists of a continuously pre-allocated storage area, a single write sequence identifier, a single read sequence identifier, and a read-only view handle; when the driver layer generates an acquisition frame, it directly maps the acquisition frame to a read-only view handle and writes it into the corresponding zero-copy / lock-free circular queue; the reader completes lock-free dequeueing by advancing the read sequence identifier.

[0011] Furthermore, the step of using the acquisition time identifier of a certain acquisition frame in the main view acquisition stream as the unified target time specifically includes: selecting the acquisition frame corresponding to the latest and most stable acquisition time identifier in the data queue of the main view acquisition stream as the reference acquisition frame for the unified target time.

[0012] Furthermore, the steps of executing the cross-viewstream temporal reprojection algorithm include: constructing image pyramids for the reference acquisition frame and the preceding and following acquisition frames selected before and after the unified target time in the secondary view acquisition stream; obtaining the forward initial displacement field and the backward initial displacement field through block matching search at the smallest layer of the image pyramid, and propagating them layer by layer to the original resolution layer to obtain the forward optical flow field and the backward optical flow field.

[0013] Furthermore, after the block matching search, a subpixel refinement step is also included: calculating the gray-level difference within the block for multiple sampling points around the matching position, fitting a quadratic surface using the gray-level difference, and obtaining the subpixel displacement increment by finding the minimum point of the quadratic surface to generate forward and backward optical flow fields with subpixel precision.

[0014] Furthermore, the layer-by-layer propagation includes: upsampling the displacement field of the lower layer to the higher layer as the initial value, and then correcting each block in the higher layer. The correction includes checking the grayscale difference. If the grayscale difference exceeds a preset threshold, displacement correction is attempted in the neighborhood and sub-pixel thinning is performed again.

[0015] Furthermore, after obtaining the forward and backward optical flow fields, a bidirectional consistency cross-checking step is also included: using the reference acquisition frame as a benchmark, the forward and backward optical flow fields are used together to perform two projections, and the position difference formed by the two projections is compared. Pixels with position differences exceeding a preset threshold are marked as occluded pixels.

[0016] Furthermore, the steps for generating reprojection auxiliary frames include: for each reprojectable pixel that is not marked as an occluded pixel, based on the relative relationship between the unified target time and the acquisition time identifiers of the preceding and following neighbor acquisition frames, time extrapolation is performed along the directions of the forward optical flow field and the backward optical flow field, respectively, and candidate reprojection results are obtained through interpolation sampling.

[0017] Furthermore, after obtaining the candidate reprojection results, the process also includes: arbitrating projection conflicts when multiple source locations are mapped to the same target pixel; and performing scanline sealing or neighborhood filling on the hole regions caused by occlusion or extrapolation failure.

[0018] Furthermore, it also includes a fine-tuning control step: recording the difference in the acquisition time markers of the time frame pairs; if the difference continues to deviate from the preset center, then fine-tuning the read sequence markers of the data queue of the auxiliary view acquisition stream by advancing or lagging by a unit step.

[0019] By employing the above technical solutions, this invention achieves the following beneficial effects: The low-latency synchronization method for binocular cameras proposed in this invention can achieve precise alignment of the primary and secondary view acquisition streams at a unified target time through software-level frame-time domain reconstruction and time fine-tuning mechanisms without relying on dedicated hardware synchronization signals, thereby significantly improving the temporal consistency and spatial registration accuracy of the binocular vision system. This method establishes zero-copy / lock-free circular queues for both the primary and secondary view acquisition streams, enabling secure transmission of image frames between the acquisition and processing stages with minimal memory copying costs. This effectively reduces waiting delays caused by multi-threaded contention and ensures the system maintains stable data throughput even under high frame rate environments. By selecting the acquisition time identifier of the latest and most stable acquisition frame in the primary view acquisition stream as the unified target time, this invention establishes a unified reference benchmark on the timeline, allowing the secondary view acquisition stream to perform cross-view stream temporal reprojection near this time to construct a reprojected secondary view frame that strictly corresponds to the primary view. Cross-view temporal reprojection employs a bidirectional optical flow field combined with consistency cross-checking. This automatically identifies non-reprojectable pixel regions when handling complex motion and local occlusion, thereby reducing artifacts and ghosting caused by optical flow errors. Compared to traditional unidirectional optical flow extrapolation schemes, this method performs temporal extrapolation and interpolation sampling only on reliable pixels after bidirectional displacement consistency detection, thus exhibiting higher robustness and fidelity in scenarios involving rapid motion, nonlinear motion, and sudden changes in illumination. Through continuous recording of temporal deviations and a sequence identification fine-tuning mechanism, the system can adaptively correct the acquisition time difference between the primary and secondary view acquisition streams, keeping temporal drift at an extremely low level during long-term operation and ensuring consistency and continuity between multi-frame outputs. This invention not only reduces the response latency of the binocular system under high load conditions but also avoids alignment errors caused by inconsistent hardware clocks, achieving real-time binocular synchronous output in highly dynamic scenes. In summary, this method achieves a balance between system architecture, algorithm robustness, and real-time performance, and can be widely applied in fields such as augmented reality, stereo reconstruction, autonomous driving, and high-precision ranging. It has significant technical advantages such as simple structure, low implementation cost, high synchronization accuracy, and strong portability. Attached Figure Description

[0020] Figure 1 A flowchart illustrating the low-latency synchronization method for binocular cameras provided in an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram illustrating the principle of cross-viewstream time alignment provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram illustrating the selection of time extrapolation path and number of advancement steps provided in an embodiment of the present invention;

[0023] Figure 4This is a schematic diagram of minimum block matching search provided in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of end-to-end delay comparison curves at different resolutions provided in an embodiment of the present invention. Detailed Implementation

[0025] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0026] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0027] Example 1: Reference Figure 1 The low-latency synchronization method for binocular cameras specifically includes the following processes:

[0028] First, data queues and acquisition frames are established. Data queues are created for both the main view acquisition stream and the secondary view acquisition stream. The capacity of the data queue can be set to 32 acquisition frames, and the total capacity of the continuously pre-allocated storage area can be set to 256 megabytes. Each acquisition frame occupies a fixed size space in the queue, and the address is aligned to a 64-byte boundary. After the driver layer completes writing a frame, it writes the acquisition time marker to the frame's description information and sets it to a readable state. The reader only retrieves one frame from the readable frames at a time, avoiding reading frames in progress. This arrangement ensures the consistency between the acquisition time marker and the pixel data, preventing partially written frames from entering subsequent processes. The reader executes this process for both the main view acquisition stream and the secondary view acquisition stream, acquiring acquisition frames with acquisition time markers.

[0029] Subsequently, a unified target time is determined and a reference acquisition frame is selected. The reader retrieves the latest frame from the data queue of the main view acquisition stream and checks whether the frame is readable. If it is readable, the acquisition time corresponding to that frame is marked as the unified target time, and the frame is also used as the reference acquisition frame. The main view acquisition stream is used as the unified time reference because it serves as a reference for downstream positioning and alignment in most applications. Prioritizing alignment to the main view acquisition stream reduces the burden of pose extrapolation in subsequent downstream applications and achieves shorter end-to-end latency with the same bandwidth.

[0030] Next, for the secondary view acquisition stream, at least two acquisition frames are selected within the same target time neighborhood. Specifically, in the secondary view acquisition stream's data queue, the frame with an acquisition time marker less than the same target time that is the most recent is selected as the preceding acquisition frame, and the frame with an acquisition time marker greater than the same target time that is the most recent is selected as the following acquisition frame. If the acquisition frequency is 60 frames per second and the trigger deviation between cameras is between 3 and 8 milliseconds, typically one frame can be selected around the same target time. If only frames exist in one side of the neighborhood, in an optional implementation, two adjacent frames on the same side can be used to replace the preceding and following acquisition frames, but subsequent processing is still based on these two frames.

[0031] Next, a cross-stream temporal reprojection algorithm is executed to generate a reprojected secondary view frame at a unified target time. The specific implementation of this algorithm includes the following sequential processing.

[0032] First, an image pyramid is constructed. Four-layer image pyramids are built for the reference frame, the preceding neighbor frame, and the following neighbor frame. The width and height of each layer are halved compared to the layer above. During construction, a 3x3 neighborhood mean smoothing is performed first, followed by size reduction. The reason for setting the pyramid to four layers is that this number achieves a sufficient balance between field of view coverage and computational speed on common 1920x1080 resolution acquisition frames. Empirical testing shows that this step can be completed in less than 6 milliseconds per frame on a general-purpose processor.

[0033] Second, block matching search is performed on the smallest layer of the image pyramid to obtain the forward and backward initial displacement fields. The steps are as follows: a regular grid is established on the smallest layer of the reference acquisition frame with a step size of 14 pixels, and each grid block is set to 11 x 11. For each grid block, a search is performed around the same grid center in the corresponding layer of the preceding acquisition frame with a radius of 6 pixels. The search evaluation is based on the sum of the absolute values ​​of the grayscale differences within the block, and the smallest sum is taken as the corresponding position of the grid block, forming the forward initial displacement field. The same search is performed on the following adjacent acquisition frames to form the backward initial displacement field. The reason for using this block size and search radius is to balance the matching stability of areas with weak texture with the computational cost of the search, and the arrangement of a step size smaller than the block side length ensures coverage without creating holes.

[0034] Third, sub-pixel refinement is performed to improve matching accuracy. Nine equally spaced sampling points are collected around the optimal integer pixel position of each grid block, with three sampling points each in the horizontal and vertical directions. The absolute values ​​of the gray-level differences within the block are summed for each of the nine sampling points, and the sub-pixel offset at the minimum position is estimated using a quadratic surface fitting method. This offset is superimposed on the integer pixel position to obtain the corresponding position with sub-pixel accuracy, thereby generating the forward and backward optical flow fields with sub-pixel accuracy. The reason for using nine-point sampling and quadratic surface fitting is that this method has a single-peak approximation characteristic on local continuous textures, which can obtain stable sub-pixel positioning without increasing the complexity of the model. The measured average positioning error in a uniform texture area is less than 0.3 pixels.

[0035] Fourth, perform layer-by-layer propagation and correction until the original resolution layer is reached. The displacement field of the lower layer is upsampled to the higher layer at twice the scale as the initial displacement. For each grid block, align it to the preceding or following frame of the corresponding layer according to the current displacement, and calculate the sum of the absolute values ​​of the grayscale differences within the block. If this value is greater than 25, correction is considered necessary. The correction steps include trying finite discrete displacements of ±1 pixel in the horizontal, vertical, and two diagonal directions, and replacing the current displacement with the result that has the smaller grayscale difference. After replacement, nine-point sampling and quadratic surface fitting are performed again to obtain the sub-pixel increment. After completing all grid blocks in this layer, upsampling is performed again to a higher layer until returning to the original resolution layer. Setting 25 as the threshold is because in an 8-bit grayscale image, this value can effectively distinguish between successfully aligned and unaligned blocks, while keeping the number of corrections less than 3, ensuring that the overall latency is controlled to no more than 12 milliseconds per frame.

[0036] Fifth, perform bidirectional consistency cross-checking and mark occluded pixels. Using the reference acquisition frame as a baseline, the forward optical flow field projects the center position of each grid block of the preceding acquisition frame onto the coordinates of the reference acquisition frame. Then, the backward optical flow field projects the corresponding position of the reference acquisition frame back onto the coordinates of the following acquisition frame. The position difference formed by the forward and backward projections is compared block by block. If the difference exceeds 1 pixel, the pixel area covered by that block is marked as an occluded pixel. The reason for using 1 pixel as the difference limit is that after the aforementioned sub-pixel refinement, the position difference of the non-occluded area is usually concentrated within 0.6 pixels. Setting the limit to 1 pixel can effectively isolate occlusion and matching degradation.

[0037] Sixth, temporal extrapolation is performed based on optical flow to generate candidate reprojection results. For each reprojectable pixel not marked as an occluded pixel, two extrapolation paths are generated based on the unified target time and the order of the preceding and following acquisition times. When the unified target time is closer to the preceding acquisition time, one step is advanced along the forward optical flow direction and bilinear interpolation sampling is performed in the preceding acquisition frame, while two steps are advanced along the reverse direction of the backward optical flow direction and bilinear interpolation sampling is performed in the following acquisition frame, resulting in two candidate reprojection results. When the unified target time is closer to the following acquisition time, one step is advanced along the reverse direction of the backward optical flow direction and sampling is performed in the following acquisition frame, while two steps are advanced along the forward optical flow direction and sampling is performed in the preceding acquisition frame, resulting in two candidate reprojection results. The reason for this arrangement is that by asymmetrically selecting the number of advancement steps, the side that is closer is projected with a shorter path in space, thereby reducing the accumulation of interpolation errors, while the other side supplements possible occlusion reveal information through a longer path. Setting the step size to 1 or 2 can cover the common intervals between uniform target time and two neighboring acquisition times under camera deviation conditions of 60 frames per second and 3 to 8 milliseconds. If the frame rate is higher, such as 120 frames per second, a combination of step size 1 and 3 can be used to compensate for insufficient pixel displacement at shorter time intervals.

[0038] Seventh, perform projection conflict arbitration and hole handling. When multiple source locations are mapped to the same target pixel, calculate the brightness difference between each source location and the reference acquisition frame at the same location, and select the one with the smaller brightness difference as the value of the target pixel. If the brightness differences of two source locations are the same, select the one with the smaller row index and column index, which ensures determinism and reproducibility. Perform scanline sealing on hole regions caused by occlusion or extrapolation failure. Scanline sealing involves scanning from left to right along each row. When a hole is encountered, the nearest valid pixel on the left is copied to the right until the edge of the reference acquisition frame or a non-hole pixel is encountered. Then, scan again from right to left along each row to fill any remaining holes. The reason for using this bidirectional scanning is to maintain structural continuity without introducing external information, while reducing striping in edge regions. For hole regions with an area greater than 49 pixels, neighborhood filling can be added. An optional approach is to take the nearest non-empty pixel in the four neighborhoods of the hole and perform multi-value copying.

[0039] Eighth, perform edge-preserving refinement to improve structural clarity. Using a reference acquisition frame as a guide, perform guided filtering on the generated candidate reprojection results. The window size can be set to 7x7, reducing smoothing intensity near significant edges of the reference acquisition frame while maintaining consistent smoothness in non-edge areas. Guided filtering is used because it allows for edge alignment of the reprojected secondary view frame based on the structure of the reference acquisition frame, suppressing blurring caused by interpolation.

[0040] Ninth, perform dual-candidate pixel-by-pixel selection to obtain the reprojected auxiliary view frame. For each pixel, compare the brightness difference between the two candidate reprojection results and the reference acquisition frame at the same position, and take the one with the smaller brightness difference as the final pixel value. If the brightness differences are the same, prioritize the candidate result from the side closer to the unified target time. The reason for this selection is that the closer side is more consistent in time, which can reduce motion blur in dynamic scenes. The frame generated at this point is the reprojected auxiliary view frame at the unified target time.

[0041] Finally, the reference acquisition frame and the reprojected secondary view frame are combined to form a time frame pair and output. During output, the frame pair order is kept consistent with the main view acquisition stream to ensure the assumption of time consistency in downstream processing holds true. Within a complete processing cycle, the total latency from selecting a unified target time to generating the time frame pair can be controlled to no more than 20 milliseconds per frame on common general-purpose processors. This value is derived from the cumulative maximum processing time of the aforementioned steps. In actual deployment, when the resolution is 1280 x 720, this total latency can be reduced to no more than 12 milliseconds per frame.

[0042] In one alternative implementation, the data queue employs a zero-copy, lock-free ring structure. A combination of contiguous pre-allocated storage areas, single write sequence identifiers, single read sequence identifiers, and read-only view handles avoids unnecessary data copying and mutual exclusion waiting. This structure reduces memory copying by at least once per frame under the same hardware conditions, while allowing the reader to complete lock-free dequeueing simply by advancing the read sequence identifier. This approach significantly reduces data dwell time in the processing flow, improves the availability of frames near a unified target time, and further reduces end-to-end latency. When the acquisition frame resolution is higher, such as 3840x2160, the number of pyramid layers can be increased to five, the grid step size to 20 pixels, the block size to 15x15, and the search radius to 8 pixels to maintain matching stability under large field-of-view motion. If the scene is predominantly close-up with minimal motion, the number of layers can be reduced to three, and the step size to 10 pixels to improve speed. The system can be configured once during startup based on the statistical range of the average trigger deviation between the main view acquisition stream and the secondary view acquisition stream. For example, when the deviation is concentrated between 2 and 5 milliseconds, a combination of step sizes of 1 and 1 is used; when the deviation is concentrated between 6 and 12 milliseconds, a combination of step sizes of 1 and 2 is used. For smaller hole regions, such as those no larger than 25 pixels, only nearest-neighbor replication can be used to obtain sharper boundaries. For larger hole regions, such as those larger than 100 pixels, a checkerboard diffusion method with alternating rows and columns can be used to fill them. The reason for this classification is that small holes have a less significant impact on visual continuity, while large holes need to avoid stripe aggregation caused by unidirectional replication.

[0043] Example 2: Separate data queues are established for the primary and secondary view acquisition streams, using zero-copy / lock-free circular queues. A contiguous pre-allocated storage area is allocated its capacity once during initialization; this capacity can be set to 256 megabytes. It is divided into 32 acquisition frame slots using a fixed frame slot method, with the starting address of each acquisition frame slot aligned to a 64-byte boundary. Both the single-write sequence identifier and the single-read sequence identifier start from 0 and always advance in increments of 1. The read-only view handle describes the visible range of a frame in the circular space using both index and length information, along with an acquisition time identifier.

[0044] When the driver layer generates a capture frame, it executes a single sequential write process. First, pixel data is written to the target capture frame slot. After writing, the capture time identifier is written to the description information of that slot, and the valid bit of the read-only view handle is set to valid. Then, the single write sequence identifier is advanced. This order ensures that when the reader observes that the read-only view handle is valid, the pixel data and capture time identifier have already been written, thus preventing the reading of incomplete frames. The reader starts checking whether the read-only view handle is valid from the slot pointed to by the single read sequence identifier. If valid, it returns the read-only view handle and advances the single read sequence identifier; if invalid, it keeps the single read sequence identifier unchanged and retryes in the next loop. Since writing and reading each have only one advance direction, and the advance step size is always 1, there is no contention for the same identifier at any time, and the entire process maintains consistency without the need for locking.

[0045] When the single write sequence identifier and the single read sequence identifier wrap around numerically, the slot is located by dividing by the total number of slots, without introducing an additional mapping table. The read-only view handle does not need to be explicitly zeroed after the read operation is completed; the next write will overwrite the slot and reset the valid bit, thus completing natural reclamation. The reason for using contiguous pre-allocated storage areas is to ensure a compact physical layout, reduce the probability of cache misses and page switching, and reduce unpredictable jitter at high frame rates. The reason for using read-only view handles is that they can directly transmit the location description of pixel data in the storage area; downstream processing does not need to copy data or hold write permissions, thus avoiding the waiting caused by write-read interleaving.

[0046] In one alternative implementation, the number of slots can be increased to 64, and the capacity of the contiguously pre-allocated storage area can be set to 512 megabytes to accommodate acquisition frames with a resolution of 3840 x 2160. Before writing, it is checked whether the next slot is being held by the reader. If the holding time exceeds one frame interval, the earlier slot is overwritten first, keeping the frames in the queue as close to the current time as possible. All of the above alternative implementations maintain zero-copy and lock freedom, and do not change the unidirectional advancement characteristics of the single write sequence identifier and the single read sequence identifier.

[0047] Example 3: In the main view acquisition stream's data queue, the latest and most stable acquisition frame corresponding to the acquisition time identifier is selected as the reference acquisition frame for the unified target time. Specifically, in each loop, the instantaneous value of the single write sequence identifier is read first and recorded as the current highest write position. Then, starting from the slot corresponding to the current highest write position, the read-only view handle of that slot is checked for validity. Once the read-only view handle is set to valid, it indicates that the driver layer has completed the pixel data writing and acquisition time identifier writing, and the frame can be considered stable. If the slot is valid, it is directly regarded as the latest and most stable acquisition frame, and the acquisition time identifier of this acquisition frame is used as the unified target time, making this acquisition frame the reference acquisition frame. If the slot is invalid, the previous slot is checked sequentially until a valid read-only view handle is found or the number of checks reaches 3 slots. The reason for limiting the number of checks to 3 is to avoid long backtracking causing the unified target time to fall into an early historical moment, thereby maintaining low latency characteristics.

[0048] Using the latest and most stable acquisition frame as the unified target time has two direct benefits. First, the reference acquisition frame comes from the main view acquisition stream, so subsequent downstream pose estimation and alignment based on the main view acquisition stream do not require time extrapolation, reducing accumulated errors. Second, the stability is determined by the write order and the valid bits of the read-only view handle, so the reader does not need to use historical statistics or maintain complex states, thus ensuring the determinism of the selection result.

[0049] In one alternative implementation, a short wait window is set, with a duration of 1 to 2 milliseconds. If the read-only view handle of the slot corresponding to the highest write position becomes valid within the window, that frame is immediately selected as the reference acquisition frame. This wait window can reduce the number of backtracking steps when some sensors have slight write delays, making the target time closer to the actual acquisition time. Another alternative implementation involves temporarily downgrading to using the second newest valid frame if no valid new frame is found after three consecutive loops. This downgrading only lasts for one loop, after which the latest valid frame is used.

[0050] Example 4: Before executing the cross-viewstream temporal reprojection algorithm, image pyramids are constructed for the reference acquisition frame, the previous neighbor acquisition frame, and the next neighbor acquisition frame. Each frame's image pyramid consists of four layers. Starting from the original resolution layer, the width and height are halved in the next layer. Before each layer reduction, a 3x3 neighborhood mean smoothing is performed, followed by size reduction. With four layers set at an acquisition frame resolution of 1920x1080, construction can be completed within 6 milliseconds per frame, while ensuring that the global trend is sufficient to cover a large field of view motion at the smallest layer.

[0051] The forward and backward initial displacement fields are obtained through block matching search at the smallest layer of the image pyramid. The operation involves placing a regular grid with a 14-pixel step size at the smallest layer of the reference acquisition frame, with the grid block size set to 11x11. For each grid block, discrete positions are traversed within a search region centered on the grid in the corresponding layer of the preceding acquisition frame. The search region radius is set to 6 pixels. The evaluation method is to sum the absolute values ​​of the grayscale differences within the block, and the position with the smallest evaluation value is taken as the corresponding position of that grid block in the preceding acquisition frame, thus forming the forward initial displacement field. The backward initial displacement field is obtained in the corresponding layer of the following acquisition frame using the same method. Choosing an 11x11 block size provides sufficient contrast in sparse texture areas, and the 14-pixel step size, being smaller than the block side length, achieves a balance between coverage and redundancy. The 6-pixel search radius at the smallest layer can cover common hand-held jitter and medium-speed motion.

[0052] The initial forward and backward displacement fields of the smallest layer are propagated layer by layer to the original resolution layer to obtain the forward and backward optical flow fields. Layer-by-layer propagation includes three actions: First, the displacement field of the lower layer is upsampled to the higher layer at a ratio of 2. Second, each grid block in the higher layer is corrected by aligning the block with the current displacement in the corresponding frame and calculating the sum of the absolute values ​​of the grayscale differences within the block. If this value is greater than 25, the current displacement needs correction. Correction is performed in a finite-trial manner, attempting a forward and reverse displacement of 1 pixel each in the horizontal, vertical, and two diagonal directions, for a total of eight attempts in four directions. The attempt with the smaller grayscale difference replaces the current displacement. Third, a quadratic surface fitting with nine-point sampling is performed at the replaced integer pixel position to obtain the sub-pixel increment, which is then superimposed to obtain the sub-pixel result of this layer. After completing all grid blocks in this layer, upsampling continues to the higher layer until the original resolution layer. Setting the threshold to 25 directly reduces the number of meaningless corrections, ensuring that a single layer process averages no more than two attempts, thereby controlling the propagation phase time to no more than 12 milliseconds per frame. Through this coarse-to-fine propagation, displacement information is given a general direction over a large area by the smallest layer, and at the high-resolution layer, it is stably converged into the forward and backward optical flow fields through limited neighborhood attempts and nine-point refinement.

[0053] First, a large-scale displacement is compressed using an image pyramid. Then, block matching search finds reliable integer pixel correspondences at discrete locations. Subsequently, layer-by-layer propagation and finite discrete correction are used to refine the image at a high-resolution layer. This sequence avoids the high cost of performing a large-scale search directly at the original resolution layer, ensuring that each frame only depends on the reference frame, the previous neighbor frame, and the next neighbor frame within the same round. The arrangement of grid step size smaller than the block size allows for overlap between adjacent blocks. If a local match fails subsequently, it can be compensated by the neighboring valid blocks during propagation. Ultimately, a dense and usable forward and backward optical flow field is formed at the original resolution layer.

[0054] In one alternative implementation, at a resolution of 3840 x 2160, the number of image pyramid layers is increased to five, the grid step size is set to 20 pixels, the block size is set to 15 x 15, and the search radius is set to 8 pixels to accommodate a larger displacement range. When dealing with near-field scenes with small motion amplitudes, the number of image pyramid layers can be reduced to three, and the grid step size can be set to 10 pixels to improve processing speed. Another alternative implementation introduces an early stopping strategy during the layer-by-layer propagation phase. When the sum of the absolute values ​​of the grayscale differences within a block is less than 12, upsampling directly proceeds to the next layer to avoid unnecessary attempts in already well-aligned areas. An additional extended search can be added at the minimum layer, with an extension radius of 9 pixels, but this is only enabled once when candidate frames are missing in the previous frame loop, ensuring that effective forward and backward initial displacement fields are still obtained even with a small number of lost frames.

[0055] Example 5: After completing the block matching search at the smallest layer of the image pyramid and obtaining the forward and backward initial displacement fields, sub-pixel thinning is performed to generate sub-pixel precision forward and backward optical flow fields. The implementation process includes the following sequential actions.

[0056] First, a nine-point sampling layout is established around the optimal integer pixel position of each grid block. The nine sampling points are located at three positions (left, center, and right) in both the row and column directions, with a sampling interval of 0.5 pixels. These nine points together form an equally spaced 3x3 grid around the integer pixel position. The 0.5-pixel interval is used because this interval maintains the unimodality of the error surface within the nine-point neighborhood under common imaging noise levels, thus facilitating the stable determination of the minimum position.

[0057] Subsequently, the absolute values ​​of the grayscale differences within each of the nine sampling points are summed to obtain nine scalar difference values. During the calculation, bilinear interpolation is used to align the sampling between the reference acquisition frame and the preceding or following acquisition frame to ensure stable grayscale comparison even at non-integer positions. The nine differences are used to fit a local error quadratic surface. Specifically, a fixed nine-point coefficient table is used to linearly combine the nine differences to obtain the coefficient set of the local error quadratic surface. Then, the offsets in the row and column directions are directly calculated through a fixed vertex solution step. The advantage of using a fixed coefficient table and fixed solution steps is that all calculations can be solidified into an addition and multiplication sequence, which facilitates instruction pipelining and parallel execution, while avoiding the introduction of uncertain iterations at runtime.

[0058] Next, the obtained sub-pixel offsets in the row and column directions are superimposed on the integer pixel positions to form sub-pixel candidate positions. To prevent excessive offsets in areas with extremely weak texture, the offset in each direction is limited to no more than 0.75 pixels. Then, the absolute sum of the intra-block grayscale differences is recalculated at each sub-pixel candidate position as a verification value. If the verification value is less than the smallest difference among the nine points, the sub-pixel candidate position is accepted; if the verification value is not less than the smallest difference among the nine points, the position corresponding to the smallest difference among the nine points is returned to. This acceptance and rejection rule avoids misjudgments due to noise or interpolation, ensuring that sub-pixel correction only occurs when it genuinely improves the match.

[0059] The above steps were applied to the preceding and following acquisition frames respectively to obtain the forward and backward optical flow fields with sub-pixel accuracy. For a typical setting of 1920x1080 resolution, 14-pixel grid spacing, and 11x11 block size, the average computation time for nine-point fitting and one verification is no more than 6 milliseconds per frame on a general-purpose processor, and the average positioning error in areas with uniform texture can be reduced to below 0.3 pixels. The reason for using nine-point fitting in conjunction with verification is that the actual error terrain can be approximated by a smooth parabolic surface.

[0060] Example 6: After obtaining the initial values ​​of the forward and backward optical flow fields with sub-pixel precision, the process propagates layer by layer along the image pyramid from low to high, and each grid block is corrected at each higher layer. After the correction is completed, sub-pixel thinning is performed again until the original resolution layer is reached. The implementation process includes three closely linked sets of actions.

[0061] The first group involves upsampling. The displacement results of the lower layer are upsampled by a scale of 2, directly mapping them to the initial displacement of the mesh block at the same location in the higher layer. This approach allows large-scale motion trends to be transferred to higher-resolution layers at a lower cost, avoiding large-scale searches at high-resolution layers.

[0062] The second group is for correction. For each grid block in a higher layer, the preceding or following adjacent acquisition frames are aligned and sampled according to the current displacement, and the sum of the absolute values ​​of the gray-level differences within the block is calculated as the current error. If the current error does not exceed 25, the displacement remains unchanged and the process moves to the next grid block. If the current error is greater than 25, finite discrete displacements of 1 pixel each are attempted in the horizontal, vertical, and two diagonal directions, for a total of 8 attempts. In each attempt, the sum of the absolute values ​​of the gray-level differences within the block is recalculated, and the attempt with the smallest error is selected as the new integer pixel position. Setting 25 as the threshold is considered because, under the condition of an 8-bit grayscale image, this value can form a clear boundary with the stable matching interval when the block size is 11 by 11, thereby avoiding unnecessary attempts on already well-aligned blocks.

[0063] The third group involves further sub-pixel refinement. For each grid block after integer pixel correction, the nine-point sampling, fixed coefficient table fitting, vertex solving, and verification process described in section five is repeated to obtain the sub-pixel result for that layer. If the error after verification does not decrease compared to the integer pixel position, the integer pixel position is kept unchanged to ensure monotonic improvement. After completing all grid blocks in this layer, the process proceeds to a higher layer and repeats the above three groups of actions until the original resolution layer is reached, thus obtaining the final forward and backward optical flow fields. To control runtime, the number of correction and refinement loops can be limited to no more than two times per grid block. Empirical testing shows that the entire propagation and correction time at a 1920x1080 resolution does not exceed 12 milliseconds per frame.

[0064] This sequential arrangement of upsampling followed by finite discrete correction and then subpixel refinement has two advantages. First, large-scale motions that have already converged in lower layers are directly carried over to higher-resolution layers, avoiding repeated searches in higher-resolution layers. Second, discrete 8-directional attempts can overcome small steps of poor local matching, and then nine-point refinement locks the position to the neighborhood of continuous minimum values, thus balancing global convergence and local accuracy.

[0065] Example 7: After obtaining the forward and backward optical flow fields at the original resolution layer, a bidirectional consistency cross-check is performed and occluded pixels are marked. The implementation process includes three steps: pairwise projection towards the pixel or mesh center, difference comparison, and occlusion expansion.

[0066] First, using a reference acquisition frame as a baseline, the position of the preceding acquisition frame is projected onto the coordinates of the reference acquisition frame using a forward optical flow field, resulting in the forward projected position. Then, the corresponding position of the reference acquisition frame is back-projected onto the coordinates of the following acquisition frame using a backward optical flow field, resulting in the backward projected position. Both projections employ bilinear interpolation to ensure continuity at sub-pixel positions.

[0067] Subsequently, the forward and backward projected positions are compared point by point. The comparison method involves calculating the absolute difference in the row direction and the absolute difference in the column direction, and taking the larger of the two as the position difference metric for that point. If the position difference metric is greater than 1 pixel, the point is marked as an occluded pixel; if the position difference metric is no greater than 1 pixel, the point is marked as a reprojectable pixel. The reason for choosing the larger metric is that it can more sensitively reflect alignment failures in either direction, and can more promptly eliminate unreliable correspondences in scenes with fast movement or edge transitions.

[0068] Finally, the newly marked occluded pixels undergo a limited expansion to cover potentially unstable regions immediately adjacent to the edges. The expansion method involves extending the neighborhood by 1 pixel in both the row and column directions, temporarily promoting unmarked pixels within this neighborhood to occluded status. Projection positions that exceed boundaries or fall into invalid regions are directly marked as occluded pixels. After expansion, the final set of occluded pixels and the set of reprojectable pixels are obtained. This set is used for subsequent temporal extrapolation and candidate reprojection generation; extrapolation and interpolation are performed only on reprojectable pixels to avoid error propagation caused by occlusion.

[0069] This bidirectional consistency cross-checking arrangement ensures that the forward and backward optical flow fields provide a consistent geometric interpretation at the same location. Inconsistency indicates occlusion or matching degradation at that location, and pre-marking avoids introducing erroneous pixels during temporal extrapolation. Setting the threshold to 1 pixel allows for tolerance to the real scene even after nine-point subpixel refinement, while clearly separating inconsistent areas for subsequent scanline sealing or neighborhood filling.

[0070] In one alternative implementation, the position difference metric is adjusted from taking the larger value to the sum of the absolute values ​​of the row difference and the column difference to improve sensitivity to slant errors; or the extended neighborhood is increased from 1 pixel to 2 pixels for situations with fast movement speeds or complex scene occlusion boundaries. Another alternative implementation involves adding a single-iteration refinement before marking occluded pixels. The refinement method involves attempting to backtrack 0.5 pixels unidirectionally along a row or column within a range where the position difference metric is close to 1 pixel. If the difference between the two projections decreases significantly after backtracking, the pixel is not marked as an occluded pixel, thereby reducing over-marking near the boundary.

[0071] Example 8: After completing the bidirectional consistency cross-validation and obtaining reprojectable pixels, two candidate reprojection results are generated for each reprojectable pixel. The process first locates the extrapolation starting point. For the forward path, the position of the preceding neighboring acquisition frame corresponding to the reprojectable pixel, established during the aforementioned forward projection process, is used as the starting point. At this position, the displacement value of the forward optical flow field is read using bilinear interpolation to ensure stable values ​​for non-integer positions. For the backward path, the position of the following neighboring acquisition frame corresponding to the reprojectable pixel, established during the aforementioned two projection processes, is used as the starting point. At this position, the displacement value of the backward optical flow field is read using bilinear interpolation. The direct advantage of selecting the starting point and reading method in this way is that it maintains the same coordinate system and the same interpolation rule as the bidirectional consistency cross-validation, reducing the error propagation caused by coordinate transformation.

[0072] The number of advance steps is then determined based on the relationship between the unified target time and the preceding and following acquisition time markers. When the unified target time is closer to the preceding acquisition time, one step is advanced on the forward path and two steps are advanced on the backward path. When the unified target time is closer to the following acquisition time, one step is advanced on the backward path and two steps are advanced on the forward path. Each advance is equivalent to moving one pixel along the direction of the corresponding optical flow field, and after each step, the pixel value is sampled in the corresponding acquisition frame using bilinear interpolation. The advance uses gradual accumulation rather than a one-time jump because gradual advancement can continuously follow the displacement direction at locations where texture and motion change gradually, reducing sampling deviation when crossing corners. The combination of 1 and 2 steps covers common time intervals at 60 frames per second and a trigger deviation between the two cameras within 3 to 8 milliseconds, enabling time extrapolation to be completed in fewer steps and avoiding the introduction of complex inferences.

[0073] If the starting or intermediate position falls outside the boundary of the acquisition frame during path advancement, it is replaced with the nearest valid pixel position using boundary clamping, and advancement continues. If it falls into an area previously marked as an occluded pixel, path advancement is stopped and marked as extrapolation failure. After advancement, two endpoint positions are recorded, and bilinear interpolation is used to sample the corresponding preceding and following acquisition frames to obtain two candidate reprojection results. The reason for retaining two candidate reprojection results is that the two time directions provide independent evidence for the same position, which can provide a more robust decision source in subsequent selection and projection conflict arbitration.

[0074] Example 9: After obtaining the candidate reprojection results, generating the reprojected secondary view frame requires handling two types of problems. The first type is the arbitration of projection conflicts when multiple source locations are mapped to the same target pixel, and the second type is the filling of hole regions.

[0075] Projection conflict arbitration primarily uses consistency with the reference acquisition frame as the criterion. For multiple source locations mapped to the same target pixel, the absolute value of the brightness difference between the pixel value at each source location in its original acquisition frame and the brightness difference at the same location in the reference acquisition frame is calculated. The source location with the smaller difference is selected as the definitive source of the target pixel. If the differences between two source locations are the same, the one with the smaller row index is selected; if the row indices are also the same, the one with the smaller column index is selected. This hierarchical decision-making ensures that each arbitration results in a definite and reproducible outcome, while maintaining the best structural consistency with the reference acquisition frame, thereby reducing motion blur. For color images, the above comparison is performed on all three channels during arbitration, and the sum of the absolute values ​​of the channel differences is used as a substitute measure of brightness difference to ensure consistency across the three channels.

[0076] Hole regions include unassigned areas caused by occluded pixels and unassigned areas caused by extrapolation failure. The process begins with scanline sealing. Scanning from left to right along each row, if a hole is encountered, the nearest valid pixel on the left is continuously copied until the edge of the reference acquisition frame or a non-hole pixel is encountered. Then, scanning again from right to left along each row, copying the nearest valid pixel on the right to the left until stopping. These two scans maintain structural continuity without introducing external images and achieve rapid closure in narrow gaps. When the hole width or height exceeds 7 pixels and the scanline sealing still leaves unfilled areas, neighborhood filling is performed. Neighborhood filling is performed in 3x3 window units. Within the window, non-empty pixels are counted, and the pixel value with the highest number is selected as the filling value for the center pixel of the hole. If the number is the same, the pixel with the brightness closest to the reference acquisition frame at the same position is selected. For hole regions exceeding 100 pixels in area, 4-connected region marking can be performed first, followed by neighborhood filling once from top left to bottom right and then from bottom right to top left within each connected region to avoid stripes caused by unidirectional copying. The direct effect of the above sequence arrangement is to prioritize the restoration of linear structures, then maintain texture consistency in local majority areas, and ultimately balance coherence and detail.

[0077] In one alternative implementation, edge alignment fine-tuning between candidate reprojection results is performed before projection conflict arbitration. Specifically, a 3x3 window is taken at each corresponding position of the candidate reprojection results. If the reference acquisition frame has a significant edge at that position, the candidate reprojection result aligned with the reference acquisition frame in the gradient direction is preferentially selected. Another alternative implementation involves adding a column-oriented scan after scan line sealing is completed. Column-oriented copying helps repair residual vertical slits formed during row-oriented scans. A 5x5 median filter can also be performed after neighborhood filling, affecting only pixels that were previously holes, to suppress isolated outliers caused by copying.

[0078] Example 10: A fine-tuning control step is performed after the output of the time synchronization frame pair. This step first records the difference in the acquisition time markers of the time synchronization frame pair. The recorded content is the time difference between the acquisition time marker of the reference acquisition frame and the median value of the acquisition time markers of the preceding and following acquisition frames used to generate the reprojection auxiliary view frame in the auxiliary view acquisition stream. The median value is used because the reprojection auxiliary view frame is formed by extrapolating from the preceding and following acquisition frames, and the median value can more accurately reflect the time center of this time synchronization. When this difference continues to deviate from the preset center, the read sequence marker of the data queue of the auxiliary view acquisition stream is fine-tuned by advancing or lagging by a unit step.

[0079] The specific procedure is as follows: immediately after each output, determine the sign and absolute time of the difference. If the difference is positive and the absolute time is not less than 1 millisecond, increment the positive bias count by 1 and reset the negative bias count to zero. If the difference is negative and the absolute time is not less than 1 millisecond, increment the negative bias count by 1 and reset the positive bias count to zero. If the absolute time of the difference is less than 1 millisecond, reset both counts to zero and consider the output to be aligned. When the positive bias count reaches 3, advance the read sequence identifier by one unit step, advancing it by one acquisition frame position, and reset both counts to zero. A count of 3 on the current side indicates that three consecutive outputs have deviated in the same direction. When the negative bias count reaches 3, lag the read sequence identifier by one unit step, reversing it by one acquisition frame position, and reset both counts to zero. To avoid oscillations caused by frequent adjustments, a maximum of one unit step advance or lag adjustment is performed in any 10 outputs. The advance or backtracking of the sequence identifier is completed within the zero-copy / lock-free circular queue. The advance and backtracking directions are always consistent with a step of 1, which does not require locking and thus does not violate the lock-free dequeueing characteristic.

[0080] After each unit step, the selection of the next unified target time, along with the selection of the preceding and following acquisition frames, will be closer to the time center of the reference acquisition frame. The direct effect is to reduce the total number of steps in the two time extrapolation paths, thereby reducing interpolation errors and end-to-end latency. If a difference reversal occurs after the unit step and the absolute time is still greater than or equal to 1 millisecond, further steps will be paused for at least 5 outputs to wait for a new stable trend to form, preventing oscillations near the boundary.

[0081] Example 11: Assume the acquisition frame resolution is 1920 x 1080. The frame rate is 60. The main view acquisition stream and the secondary view acquisition stream each establish a zero-copy / lock-free circular data queue with a capacity of 32 bytes. A contiguous pre-allocated storage area of ​​256 megabytes is provided, and the acquisition frame slot addresses are aligned to 64-byte boundaries. After each pixel data write, the driver layer writes an acquisition time marker and sets the read-only view handle to valid, then advances the single-write sequence marker. The reader only advances the read sequence marker when the read-only view handle is valid, thus ensuring the consistency between the acquisition frame and the acquisition time marker.

[0082] In a single processing loop, the latest and most stable frame from the main view acquisition stream is selected as the reference frame, and its corresponding acquisition time identifier is used as the unified target time. The example gives a unified target time of 1000.000 milliseconds. The secondary view acquisition stream can read the previous neighboring frame with an acquisition time identifier of 994.000 milliseconds and the next neighboring frame with an acquisition time identifier of 1010.667 milliseconds within the neighborhood of this time. A four-layer image pyramid is constructed for the reference frame, the previous neighboring frame, and the next neighboring frame. Before each layer reduction, a 3x3 neighborhood mean smoothing is performed, and then the width and height are each halved from the previous layer. This arrangement facilitates stabilizing large-scale displacements first, followed by small-scale corrections at higher resolution layers, reducing search costs and mismatches. Block matching search is performed at the smallest layer. The grid step size is 14 pixels, the block size is 11x11, and the search radius is 6 pixels. The calculation details for a single grid block and the explanation of all parameters are given below.

[0083] Let the grayscale function of the reference acquisition frame be... Here and This represents the integer pixel coordinates in the coordinate system of the reference acquisition frame. Let the grayscale function of the previous adjacent acquisition frame be... Let the pixel coordinates of the block center be... Let the intra-block offset index be... and ,in Represents an integer offset in the line direction relative to the block center. This represents an integer offset in the column direction relative to the block center, with values ​​covering a square with a side length of 11. Let the corresponding search displacement be... ,in Indicates the integer pixel displacement in the row direction. Represents the integer pixel displacement along the column. Let the block pixel set be... ,in The cost of the sum of absolute differences within a block is defined as follows: .in This indicates that when the search displacement is The cost of block matching at that time. Minimum. Provide the best correspondence at the integer pixel level. The instance is taken from the center of a grid block at the smallest layer. The search yields the minimum cost when... If several samples in the neighborhood are , Therefore, the current optimal value is .

[0084] Let the row-oriented one-dimensional cost function be... Here Represents displacement relative to integers The row-to-subpixel offset, The cost is expressed in pixels and can be decimal. Let the column-oriented one-dimensional cost function be... Here Represents displacement relative to integers The column-direction subpixel offset is estimated using a three-point parabolic interpolation vertex solution. Row-direction estimation uses... The value of the cost, the column estimate uses The cost value. The sub-pixel displacement increments in the row and column directions are respectively ; in and These represent the optimal sub-pixel increments in the row and column directions, respectively. Substituting the values ​​above, we obtain... .

[0085] The integer displacement and the subpixel increment are combined to obtain the subpixel displacement vector in the previous neighbor direction. ; here This represents the subpixel displacement vector from the reference acquisition frame to the previous adjacent acquisition frame. Its first component is the row displacement, and the second component is the column displacement. The same method is used to obtain the displacement vector in the direction of the subsequent adjacent acquisition frame. ;in Represents the optimal integer pixel displacement in the next adjacent direction. and This represents the sub-pixel increment in the row and column directions of the next neighbor.

[0086] The displacement field of the minimum layer is upsampled to a higher layer by a ratio of 2 as the initial value. In the higher layer, the sum of the absolute differences within each grid block is calculated. If this value is greater than 25, finite discrete displacements of ±1 pixel are attempted in the horizontal, vertical, and two diagonal directions, for a total of 8 attempts. The one with the smaller displacement value is replaced, and the sub-pixel thinning from Section 5 is performed again. This process is repeated layer by layer until the original resolution layer is reached, thus forming the forward and backward optical flow fields.

[0087] Using the reference acquisition frame as a baseline, define the coordinate difference between the orthogonal and back projections. Let... and These represent the row and column differences between the two projections in the reference acquisition frame coordinate system, respectively. Let the consistency metric be... ;when Pixels are classified as reprojectable when Pixels are identified as occluded. Instance pixels are calculated. ,then It belongs to reprojectable pixels.

[0088] Deterministic calculations for two time extrapolation paths are given. Assume arbitrary displacement vectors. Let its Euclidean norm be... Let the unit direction vector be... ;in express Length, Indicates and Unit vectors in the same direction. The unit vector in the preceding direction is... Let the path position sequence be... Here Indicates the advancement of the first The continuous pixel coordinates after the step, Let be a real number representing the row coordinate. Let be the real number of the column coordinate. Let the extrapolation start point be... .

[0089] Further progress On the previous adjacent acquisition frame Perform bilinear interpolation. Let the floor function be... Let represent the largest integer not exceeding the independent variable. Let the decimal part be . Here Indicates the decimal offset in the horizontal direction. This represents a decimal offset in the vertical direction. Let the value of the four neighboring pixels be... ; Bilinear interpolation is . Substitution Example pixels with four neighboring regions ,get .

[0090] This value is recorded as a candidate reprojection result from a single advance from the previous neighboring path. , here This represents the pixel value of candidate reprojection result one. A two-step advancement is used in the subsequent neighbor direction. The unit vector is... .

[0091] Let the starting point be .first step Step Two Similarly, bilinear interpolation is performed. Let... Let the four neighboring sample pixels be ,get This value is recorded as the candidate reprojection result of two advances from the subsequent neighboring path. Here This represents the pixel value of candidate reprojection result two. If there are multiple source locations for the same target pixel, the one with higher consistency with the reference acquisition frame is used. Let the corresponding pixel in the reference acquisition frame be... Define the brightness difference between the two candidates and the reference. Substituting the values, we get... .because Therefore, choose When the two are equal, the candidate reprojection result from the side closer to the unified target time is selected. For occlusion holes in the entire frame, scanline sealing is first performed along the row direction, and then multi-valued neighborhood filling with a 3x3 window is performed on the remaining holes to maintain structural continuity and detail coherence.

[0092] The pixel-by-pixel selection results are used to construct a reprojected secondary view frame, which is then combined with the reference acquisition frame to form a time-synchronized frame pair for output. After output, a time deviation evaluation and unit step fine-tuning are performed. Let the reference acquisition frame time be... Let the median value of the previous and next neighbor acquisition times used for this reprojection be... Let the time deviation be... Here Indicates the acquisition time identifier of the reference acquisition frame. This represents the midpoint between the acquisition time markers of the preceding and following acquisition frames. This indicates the time deviation of the reference acquisition frame relative to the median value. If it occurs three times consecutively... and A unit step delay is applied to the read sequence identifier of the data queue of the secondary view acquisition stream in milliseconds. This ensures that the preceding and following acquisition frames selected in the next round are more symmetrical in time, thereby reducing the total extrapolation steps and lowering end-to-end latency. To avoid oscillation, a maximum of one unit step is applied in any 10 outputs.

[0093] The construction time for the four-layer image pyramid should not exceed 6 milliseconds per frame. Layer-by-layer propagation and finite discrete correction should not exceed 12 milliseconds per frame. The combined time for bidirectional consistency cross-validation, two time extrapolations, and candidate selection should not exceed 4 milliseconds per frame. Scanline sealing and neighborhood filling should not exceed 2 milliseconds per frame when the occlusion area does not exceed 5%. The typical end-to-end latency should not exceed 20 milliseconds per frame.

[0094] In one alternative implementation, at a frame rate of 120, the closer side advances by 1 step and the farther side advances by 3 steps. In low-light conditions, the block size is increased from 11x11 to 13x13, while the finite discrete correction trigger threshold is adjusted from 25 to 30 to enhance robustness. For fast-moving scenes, the minimum layer search radius can be temporarily increased from 6 to 9, enabled only once when the minimum cost of two consecutive grid blocks exceeds 200, and then reverted to 6.

[0095] refer to Figure 2 , Figure 2 This paper systematically demonstrates the core principles and implementation mechanism of cross-viewstream temporal alignment in this invention. The diagram corresponds to a technical solution that uses the primary view acquisition stream as a unified time reference and performs temporal reprojection based on neighboring frames in the secondary view acquisition stream. This forms the fundamental framework for achieving low-latency synchronization of binocular cameras. Figure 2As shown, a horizontal timeline is drawn at the top of the graph, with arrows pointing to the right and labeled "Time," indicating the forward flow of time. Multiple time scales are evenly distributed along the timeline, labeled t0, t1, t2, t3, t4, t5, and t6 from left to right. These scales represent discrete time sampling points. The interval between adjacent scales corresponds to the camera's acquisition cycle; for example, at an acquisition frequency of 60 frames per second, the interval between adjacent scales is approximately 16.67 milliseconds. The timeline provides a unified reference coordinate system for the time positions of frames in subsequent acquisition streams. Below the timeline is a timeline diagram of the main view acquisition stream. This timeline is represented by a horizontal solid line, with "Main View Stream" labeled at the left end in a larger and bolder font, indicating its dominant role as a time reference. On the main view stream timeline, multiple rectangles are drawn according to the time scale positions, each rectangle representing an acquisition frame. Labeled F0, F1, F2, F3, and F4 from left to right, these labels correspond to the sequence numbers of the acquisition frames. The center of the rectangle is aligned with the scale on the timeline, indicating the acquisition time of each frame. For example, the center of frame F0 is aligned with time t0, frame F1 with time t1, and so on. Among the multiple acquisition frames in the main view stream, frame F2 is specially marked with a bold border, and the font size of the label inside the box is correspondingly increased. An indicator line is drawn downwards from frame F2, with an arrow at the end pointing to the text labeled "Reference Acquisition Frame." This label further emphasizes the special role of frame F2 in the entire alignment process. The time scale corresponding to frame F2 is t2, and a vertical dashed line is drawn upwards and downwards from this time position, running through the entire graph. The text above the dashed line is labeled "Unified Target Time." This dashed line serves as a time reference benchmark in the entire graph, and all time alignment operations are targeted at the unified target time represented by this dashed line. Below the main view stream timeline is the timeline of the secondary view acquisition stream. This timeline is also represented by a horizontal solid line, with the text "Secondary View Stream" labeled at the left end of the line. On the secondary view stream timeline, multiple rectangles representing acquisition frames are also drawn, labeled A0, A1, A2, A3, and A4 from left to right. It's important to note that the temporal positions of the frames in the secondary view stream are not perfectly aligned with their corresponding positions in the primary view stream. For example, the center of frame A0 is located between t0 and t1, while the center of frame A1 is located between t1 and t2. This temporal offset reflects the triggering deviation between the stereo cameras. This triggering deviation is typically in the range of 3 to 8 milliseconds. Among the multiple acquisition frames in the secondary view stream, frames A1 and A2 are specially marked with bold borders. An indicator line is drawn downwards from frame A1, ending with the label "Previous Acquisition Frame." Another indicator line is drawn downwards from frame A2, ending with the label "Next Acquisition Frame." Observing the positions of these two frames on the timeline reveals that frame A1 was acquired before the unified target time t2, while frame A2 was acquired after the unified target time t2; the two frames are located in the preceding and following neighborhoods of the unified target time, respectively. This surrounding positional relationship is a prerequisite for performing temporal reprojection. Figure 1 The lower middle section of the frame connects the preceding and following acquired frames A1 and A2 with a new rectangle using dashed lines. This new rectangle is drawn with a dashed border, labeled "A'" inside, and "Reprojection Auxiliary View Frame" on the right. The dashed border indicates that this frame is not directly acquired but is a virtual frame generated by an algorithm. Dashed lines extend from the lower right corner of frame A1 and the lower left corner of frame A2, converging in a V-shape to frame A', visually indicating that frame A' is generated by fusing frames A1 and A2 using a temporal reprojection algorithm. The reprojection auxiliary view frame A' is precisely aligned with the unified target time t2 on the timeline and is at the same moment as the reference acquired frame F2, thus achieving cross-viewstream time synchronization. Figure 1 The quantitative relationship of time deviation is also marked below the auxiliary video stream timeline. Vertical line segments are drawn downwards from the center position of frame A1 and the unified target time t2, respectively. The bottom of the two line segments are connected by a horizontal line segment, and the bottom of the connecting line segment is marked "Δt1", representing the time difference between the preceding adjacent acquisition frame and the unified target time. Similarly, vertical line segments are drawn downwards from the unified target time t2 and the center position of frame A2, and the bottom of the connecting line segment is marked "Δt2", representing the time difference between the following adjacent acquisition frame and the unified target time. These two time difference markings clearly quantify the relative positional relationship between the preceding and following neighboring frames and the unified target time, providing a basis for the selection of subsequent time extrapolation steps.

[0096] Figure 3 This paper details the time extrapolation strategy and adaptive step selection mechanism employed by the present invention in generating reprojected secondary view frames. A horizontal timeline is drawn at the top of the figure, with arrows pointing to the right indicating the forward progression of time. Three key moments are marked on the timeline, from left to right: the previous neighbor moment, the unified target time, and the next neighbor moment. These three moments are clearly marked on the timeline with vertical lines and dots, with the unified target time emphasized by a bold line and a larger dot, indicating its central role as the time alignment reference. A vertical dashed line is drawn downwards from the unified target time point, running through the entire figure and dividing it into left and right time regions, visually reflecting the demarcation effect of the unified target time. Figure 3A large rectangle is drawn in the central region, labeled "Case 1: Unified target time is closer to the previous neighbor time." This box illustrates the processing strategy when the distance between the unified target time and the previous neighbor time is less than the distance to the subsequent neighbor time. The box is divided into two parallel parts, corresponding to the extrapolation process of the forward path and the backward path, respectively. The forward path part is labeled with the text "Forward Path:", and an extrapolation diagram is drawn to its right. A solid dot represents the starting position of the previous neighbor time, from which a thick arrow of moderate length points to a hollow dot on the right. Above the arrow is labeled "Advance 1 step", indicating an equivalent displacement of 1 pixel step along the forward optical flow direction. The hollow dot at the end of the arrow is represented by a thick border, and the right side is labeled "Candidate Result 1". This design reflects the core idea of ​​the algorithm: when the target time is closer to a certain side, a shorter advancement step is used for that side to reduce interpolation accumulation error and maintain higher spatial accuracy. The backward path section, labeled "Backward Path:", illustrates the extrapolation process from another temporal direction. Two consecutive arrows emanate from the solid dot on the right (representing the starting point of the next adjacent time step), pointing to the left and ultimately reaching the hollow dot. The arrows are labeled "Advanced 2 Steps," indicating the need to traverse a larger time interval. The hollow dot at the end is labeled "Candidate Result 2." While the larger number of advancement steps in this path may lead to greater interpolation errors, it provides independent evidence from another temporal direction, which is valuable for revealing occluded regions and handling rapid motion. Figure 3The lower region of the image shows another large rectangle labeled "Case 2: Unified Target Time Closer to the Next Neighbor Time". This box illustrates the processing strategy in the opposite case, specifically the choice of the number of advance steps when the unified target time is closer to the next neighbor time. In this case, the forward path needs to advance 2 steps to reach the vicinity of the target time from the previous neighbor time, while the backward path only needs to advance 1 step. This interchange of advance steps clearly demonstrates the adaptive nature of the algorithm: the choice of advance steps is not fixed but dynamically adjusted according to the relative position of the unified target time. By comparing the two cases, it can be found that regardless of which side the unified target time is biased towards, the algorithm always maintains a common principle: advance 1 step on the closer side and 2 steps on the farther side. This asymmetric advancement strategy has several advantages. First, the short path extrapolation on the closer side can make full use of time proximity, reducing interpolation errors caused by motion and illumination changes. Second, although the long path extrapolation on the farther side may have slightly lower accuracy, it can provide additional constraint information, playing an important role, especially in handling complex scenes such as occlusion and revelation. Third, the two candidate results generated by the two paths are compared in the subsequent projection conflict arbitration and pixel-by-pixel selection steps. The optimal result is selected based on consistency with the reference acquisition frame, further improving the reprojection quality. The combination of advance steps of 1 and 2 has been fully validated under the conditions of an acquisition frequency of 60 frames per second and a camera trigger deviation of 3 to 8 milliseconds. Under these conditions, the inter-frame time interval is approximately 16.7 milliseconds, and the camera deviation accounts for approximately 20% to 50%. The combination of advance steps of 1 and 2 can effectively cover the common interval range of unified target time and the acquisition time of the preceding and following neighborhoods. For higher frame rates, such as 120 frames per second, a combination of advance step sizes of 1 and 3 can be used to adapt to shorter inter-frame intervals and smaller pixel displacements.

[0097] refer to Figure 4Minimum layer block matching search is a crucial step performed at the third layer (minimum layer) of the image pyramid, aiming to quickly obtain the forward and backward initial displacement fields. The specific implementation process is as follows: First, a regular grid is established on the minimum layer (240×135 pixels resolution) of the reference acquisition frame. The grid is arranged with sampling points evenly placed across the entire image plane at a step size of 14 pixels. Each sampling point serves as the center of the grid, expanding outwards to form a rectangular grid block of size 11×11 pixels. The choice of 11×11 pixels as the block size is based on the consideration that this size can contain sufficient grayscale variation information in areas with weak texture to achieve stable matching, while avoiding the computational burden and failure of rigidity assumptions caused by excessively large block sizes. The reason for choosing 14 pixels as the grid step size is that this step size is less than the side length of the block (11 pixels), thus creating spatial coverage between adjacent grid blocks, ensuring that no unsampled void areas appear in the image plane, and providing dense initial constraints for subsequent layer-by-layer propagation. Subsequently, for each grid block, a discrete position search is performed in the corresponding layer of the preceding acquisition frame. The search area is determined as follows: Using the center position of the grid block in the reference acquisition frame as a reference, a search range with a radius of 6 pixels is defined around the same coordinate position in the preceding acquisition frame. The discrete positions covered by this search range form a 13×13 pixel candidate region. For each integer pixel position within this candidate region, an 11×11 pixel block of the same size as the reference block is extracted from the preceding acquisition frame. The absolute value of the grayscale difference between corresponding pixels within the two blocks is calculated, and the sum of the 169 pixel differences within the entire block is used to obtain the matching evaluation value for that position. After traversing all candidate positions within the search range, the position with the smallest evaluation value is selected as the corresponding position of the grid block in the preceding acquisition frame. The coordinate difference between this corresponding position and the center position of the reference block constitutes the displacement vector of the grid block in the forward initial displacement field.

[0098] like Figure 5As shown, the horizontal axis represents image resolution, with five typical resolution specifications marked from left to right: 640×480, 1280×720, 1920×1080, 2560×1440, and 3840×2160. These five resolutions cover the entire range from standard definition to 4K ultra-high definition, comprehensively reflecting the algorithm's performance characteristics under different data scales. The vertical axis represents processing latency in milliseconds, measured from 0 to 40 milliseconds, with a uniformly distributed scale, setting a scale line every 5 milliseconds. Processing latency is defined as the total time from selecting a unified target time to completing the generation of time-paired frames, encompassing all computational stages such as image pyramid construction, block matching search, layer-by-layer propagation, subpixel thinning, bidirectional consistency check, time extrapolation, projection conflict arbitration, and hole handling. Two curves are plotted in the figure for comparison. The curve of the method of this invention is represented by a solid line, and data points are marked with solid dots, with the specific latency value marked next to each data point. As the curve shows, the processing latency increases with resolution, but the rate of increase is relatively gradual. Specifically, the latency is 6 milliseconds at 640×480 resolution, 12 milliseconds at 1280×720 resolution, 20 milliseconds at 1920×1080 resolution, 30 milliseconds at 2560×1440 resolution, and 44 milliseconds at 3840×2160 resolution. This data indicates that the method of this invention can achieve an end-to-end latency of 20 milliseconds at commonly used full HD resolutions, meeting the requirements of real-time vision applications. The curve for traditional interpolation methods is represented by a long dashed line, with data points marked by hollow circles. The rate of increase of this curve is significantly faster than that of the method of this invention, reflecting that the computational complexity of traditional methods increases more significantly with resolution. The specific data are as follows: 8 milliseconds latency at 640×480 resolution, 18 milliseconds latency at 1280×720 resolution, 31 milliseconds latency at 1920×1080 resolution, 47 milliseconds latency at 2560×1440 resolution, and 72 milliseconds latency at 3840×2160 resolution. Comparing the two curves, the advantage of the method of this invention is 11 milliseconds at Full HD resolution, and the advantage expands to 28 milliseconds at 4K resolution, fully verifying the performance advantage of this invention under high-resolution conditions.

[0099] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative. Those skilled in the art can omit, substitute, and modify the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function and achieve substantially the same result according to substantially the same method falls within the scope of the present invention. Therefore, the scope of the present invention is defined only by the appended claims.

Claims

1. A low-latency synchronization method for binocular cameras, characterized in that, Includes the following steps: Establish data queues for the main view acquisition stream and the secondary view acquisition stream respectively, and obtain the acquisition frames with acquisition time identifiers from the data queues; The acquisition time identifier of a certain acquisition frame in the main view acquisition stream is used as the unified target time; and based on at least two acquisition frames in the secondary view acquisition stream that are in the neighborhood of the unified target time, a reprojected secondary view frame at the unified target time is generated by executing the cross-view stream time reprojection algorithm. The acquisition frames in the main view acquisition stream are used as reference acquisition frames, and together with the reprojected auxiliary view frames, they form a pair of time frame pairs and are output. The steps of executing the cross-viewstream temporal reprojection algorithm include: constructing image pyramids for the reference acquisition frame and the preceding and following acquisition frames selected before and after the unified target time in the secondary view acquisition stream; obtaining the forward initial displacement field and the backward initial displacement field through block matching search at the smallest layer of the image pyramid, and propagating them layer by layer to the original resolution layer to obtain the forward optical flow field and the backward optical flow field. After obtaining the forward and backward optical flow fields, a bidirectional consistency cross-checking step is also included: using the reference acquisition frame as a benchmark, the forward and backward optical flow fields are used together to perform two projections, and the position difference formed by the two projections is compared. Pixels with position differences exceeding a preset threshold are marked as occluded pixels. The steps for generating reprojection auxiliary frames include: for each reprojectable pixel that is not marked as an occluded pixel, based on the relative relationship between the unified target time and the acquisition time identifiers of the preceding and following neighbor acquisition frames, time extrapolation is performed along the directions of the forward optical flow field and the backward optical flow field, respectively, and candidate reprojection results are obtained through interpolation sampling.

2. The method according to claim 1, characterized in that, The specific steps for establishing data queues for the main view acquisition stream and the auxiliary view acquisition stream include: establishing independent zero-copy / lock-free circular queues for the main view acquisition stream and the auxiliary view acquisition stream respectively. The zero-copy / lock-free circular queue consists of a continuously pre-allocated storage area, a single write sequence identifier, a single read sequence identifier, and a read-only view handle; when the driver layer generates an acquisition frame, it directly maps the acquisition frame to a read-only view handle and writes it into the corresponding zero-copy / lock-free circular queue; the reader completes lock-free dequeueing by advancing the read sequence identifier.

3. The method according to claim 1, characterized in that, The specific steps of using the acquisition time identifier of a certain acquisition frame in the main view acquisition stream as the unified target time include: selecting the acquisition frame corresponding to the latest and most stable acquisition time identifier in the data queue of the main view acquisition stream as the reference acquisition frame for the unified target time.

4. The method according to claim 1, characterized in that, Following the block matching search, a subpixel refinement step is also included: calculating the intra-block grayscale difference for multiple sampling points around the matching position, fitting a quadratic surface using the grayscale difference, and obtaining the subpixel displacement increment by finding the minimum point of the quadratic surface to generate the forward and backward optical flow fields with subpixel precision.

5. The method according to claim 4, characterized in that, Layer-by-layer propagation includes: upsampling the displacement field of the lower layer to the higher layer as the initial value, and then correcting each block in the higher layer. The correction includes checking the grayscale difference. If the grayscale difference exceeds the preset threshold, displacement correction is attempted in the neighborhood and sub-pixel thinning is performed again.

6. The method according to claim 5, characterized in that, After obtaining the candidate reprojection results, the process also includes: arbitrating projection conflicts when multiple source locations are mapped to the same target pixel; and performing scanline sealing or neighborhood filling for hole regions caused by occlusion or extrapolation failure.

7. The method according to claim 1, characterized in that, It also includes a fine-tuning control step: recording the difference in the acquisition time markers of the time frame pairs; if the difference continues to deviate from the preset center, then fine-tuning the read sequence markers of the data queue of the auxiliary view acquisition stream by advancing or lagging by a unit step.

Images