Methods, computing devices, computer-readable storage media, and computer program products for video stream synchronization control

Abstract

The application relates to a method, a computing device, a computer readable storage medium and a program product for video stream synchronization control. The method comprises: in response to a trigger signal, synchronously sending an exposure signal to multiple cameras, so that the multiple cameras perform synchronous exposure and collect multiple video data based on the exposure signal; respectively buffering the multiple video data to corresponding buffer modules in multiple buffer modules; and based on a frame start signal of each video data and a buffer threshold of the corresponding buffer module, synchronously outputting at least part of the multiple video data from the multiple buffer modules. The application can perform high-precision and high-stability video synchronous collection under effective hardware resources.

Description

Technical Field

[0001] The embodiments of this application generally relate to the field of data flow control, and more specifically to a method, computing device, computer-readable storage medium, and computer program product for video stream synchronization control. Background Technology

[0002] With the development of video data processing fields such as vehicle surround view, security monitoring, and industrial vision, gigabit multimedia serial links (GMSL) are used to control multiple video streams, thereby enabling the acquisition and frame synchronization of multiple video streams, so as to facilitate subsequent image stitching, fusion, and intelligent analysis.

[0003] Traditional multi-channel solutions typically employ either hardware synchronization or buffer-based approaches. Hardware synchronization uses a deserializer to synchronize exposures across multiple cameras; however, limitations in the accuracy and stability of the hardware synchronization signal make strict frame-level synchronization difficult. Buffer-based solutions buffer multiple video streams for synchronized output. While this approach can handle some timing discrepancies, it places high demands on system hardware resources, leading to poor synchronization reliability and system stability when resources are limited or when the video stream experiences abnormal fluctuations.

[0004] In summary, the shortcomings of traditional multi-channel video acquisition solutions are: difficulty in achieving strict frame synchronization output, and poor synchronization reliability and stability under limited hardware resources. Summary of the Invention

[0005] This application provides a method, computing device, computer-readable storage medium, and computer program product for video stream synchronization control. This application enables high-precision and highly stable video synchronization acquisition with effective hardware resources.

[0006] According to a first aspect of this application, a method for video stream synchronization control is provided, the method comprising: in response to a trigger signal, synchronously sending an exposure signal to multiple cameras, such that the multiple cameras perform synchronous exposure and acquire multiple video data based on the exposure signal; caching the multiple video data into corresponding cache modules in a plurality of cache modules; and synchronously outputting at least a portion of the video data from the plurality of cache modules based on a frame start signal of each video data and a cache threshold of the corresponding cache module.

[0007] In some embodiments, the plurality of cache modules are configured one-to-one with the multiple cameras to cache each video data stream acquired; and the cache threshold is used to indicate the maximum playback duration corresponding to the video data cached by the current cache module.

[0008] In some embodiments, synchronously outputting at least a portion of the video data from the plurality of cache modules based on the frame start signal of each video data stream and the corresponding cache threshold of the cache module includes: determining whether the frame start signal of each video data stream has arrived; in response to determining that the frame start signals of the plurality of video data streams have all arrived, determining whether the video data cached in the plurality of cache modules is all less than the corresponding cache threshold; in response to determining that the video data cached in the plurality of cache modules is all less than the corresponding cache threshold, synchronously outputting the plurality of video data from the plurality of cache modules; and in response to determining that the video data cached in the cache module where the frame start signal has arrived is equal to the corresponding cache threshold, forcibly outputting the video data in the cache module where the cached video data is equal to the corresponding cache threshold.

[0009] In some embodiments, the synchronous output of at least a portion of the video data from the plurality of cache modules based on the frame start signal of each video data stream and the corresponding cache threshold of the cache module further includes: in response to determining that the frame start signal of at least one video data stream has not arrived, determining whether the video data cached in the cache module where the frame start signal has arrived is less than the corresponding cache threshold; in response to determining that the video data cached in the cache module where the frame start signal has arrived is less than the corresponding cache threshold, video data can continue to be cached in that cache module.

[0010] In some embodiments, the synchronous output of at least a portion of the video data from the plurality of cache modules based on the frame start signal of each video data stream and the cache threshold of the corresponding cache module further includes: in response to determining that the video data cached in at least one cache module is equal to the corresponding cache threshold, forcibly outputting the video data in the cache module whose cached video data is equal to the corresponding cache threshold.

[0011] In some embodiments, the method further includes: in response to determining that the video data cached by at least a portion of the plurality of cache modules is equal to a corresponding cache threshold, outputting a synchronization anomaly identification signal, the synchronization anomaly identification signal being used to indicate that there is a synchronization anomaly among the output video data.

[0012] In some embodiments, the cache threshold is greater than the synchronization time difference after the multiple video data sources are synchronized.

[0013] In some embodiments, the buffer threshold is set based on the number of clock cycles and the transmission rate of the multi-channel video data stream.

[0014] In some embodiments, the synchronous output of at least a portion of the video data from the plurality of cache modules based on the frame start signal of each video data stream and the cache threshold of the corresponding cache module further includes: in response to determining that the video data cached by at least a portion of the cache modules is equal to the corresponding cache threshold, sending a video transmission pause signal to the corresponding multi-channel camera so that the camera of the corresponding channel stops sending the acquired video data back to the cache module.

[0015] In some embodiments, synchronously sending exposure signals to multiple cameras so that the multiple cameras can synchronously expose and acquire multiple video data based on the exposure signals includes: based on the multiple video data, the deserializer adjusts the timing of sending the exposure signals to synchronize the acquired multiple video data sources.

[0016] In some embodiments, the trigger signal is a pulse signal with a duty cycle.

[0017] In some embodiments, the method further includes: synchronously transmitting the trigger signal to one or more downstream deserializers connected thereto, so that the downstream deserializers send the exposure signal to other multi-channel cameras connected thereto based on the trigger signal to synchronously acquire downstream multi-channel video data.

[0018] According to a second aspect of this application, a computing device is also provided. The computing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the computing device to perform the method of the first aspect of this application.

[0019] According to a third aspect of this application, a computer-readable storage medium is also provided. This computer-readable storage medium stores a computer program, which, when executed by a machine, performs the method of the first aspect of this application.

[0020] According to a fourth aspect of this application, a computer program product is also provided, comprising a computer program that, when executed by a machine, performs the method of the first aspect of this application.

[0021] By employing the scheme of this application, multiple cameras are synchronously exposed and multiple video data are acquired through trigger signals, achieving synchronization of video acquisition sources and fundamentally reducing the initial deviation between multiple video data streams. Secondly, when receiving video data, a corresponding buffer module is set for each stream, and the frame start signal and buffer threshold are used in conjunction to ensure frame-synchronized output of at least some video data. In some embodiments, anomaly protection exceeding the threshold is further implemented. Specifically, under normal conditions, the arrival of the frame start signal ensures precise alignment and output of multiple video streams; when an abnormal delay in a single video stream causes its buffer data volume to reach a preset threshold, a protection mechanism is triggered to force the output of that stream's data and mark it as abnormal, thereby ensuring the continuous operation of the entire system. In this way, this application effectively solves the problem of video data asynchrony caused by transmission and processing delays in multiple video streams, enabling high-precision and highly stable synchronous video acquisition with limited hardware resources.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0023] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0024] Figure 1 A schematic structural diagram of a video stream synchronization control system provided by an embodiment of this application is shown.

[0025] Figure 2 A schematic flowchart of a method for video stream synchronization control provided by an embodiment of this application is shown.

[0026] Figure 3 A schematic structural diagram of a video stream synchronization control system provided in some embodiments of this application is shown.

[0027] Figure 4 A schematic flowchart illustrating a method for synchronously outputting at least a portion of video data from multiple video streams from multiple cache modules, as provided in some embodiments of this application, is shown.

[0028] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0029] Preferred embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0030] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0031] As mentioned above, in traditional multi-channel GMSL video acquisition schemes, such as those based on hardware synchronization, the GMSL deserializer generates or forwards a synchronization trigger signal and broadcasts it to all cameras via the GMSL link so that all cameras can expose simultaneously. However, during hardware source synchronization, the video stream experiences jitter and attenuation after transmission and hardware interface processing, leading to data pixel misalignment, signal offset, and consequently, frame start signal misalignment. This results in incomplete synchronization, making it difficult for the transmitted video stream to meet the fine-grained requirements of image stitching, fusion, and frame alignment. The synchronization deviation generated by the deserializer is tens of microseconds. If source synchronization is not performed and buffering is used directly, the usage of RAM or DDR memory becomes extremely high, resulting in excessive resource waste.

[0032] For example, in a cache module-based solution, GMSL buffers multiple video streams and outputs them together after the start signals of each video frame arrive, thus achieving inter-frame synchronization between different data streams. However, due to the limited hardware resources of GMSL, in other words, the depth of the cache module in GMSL is limited. Inter-frame synchronization requires setting the depth of the cache module according to the transmission rate and the amount of data per frame. Therefore, limited hardware resources constrain the range of conditions for multi-channel video synchronization. In addition, when using a cache module for inter-frame synchronization, it is impossible to perform pre-processing for source synchronization of multiple video streams, resulting in excessive initial time deviations. This can easily lead to cache data accumulation and rapid overflow, resulting in video data loss, errors, and synchronization failure. In summary, the range of frame synchronization conditions in a cache module-based solution is limited by the cache depth, making it difficult to achieve a balance between synchronization accuracy and cache safety. Therefore, the shortcomings of traditional multi-channel video acquisition solutions are: difficulty in achieving strict frame synchronization output, and poor synchronization reliability and stability under limited hardware resources.

[0033] To at least partially address one or more of the aforementioned problems and other potential issues, an example embodiment of this application proposes a method for video stream synchronization control, the method comprising: in response to a trigger signal, synchronously sending an exposure signal to multiple cameras, such that the multiple cameras perform synchronous exposure and acquire multiple video data based on the exposure signal; caching the multiple video data into corresponding cache modules in a plurality of cache modules; and synchronously outputting at least a portion of the video data from the plurality of cache modules based on a frame start signal of each video data and a cache threshold of the corresponding cache module.

[0034] Therefore, this application constructs a dynamic synchronization mechanism at the hardware logic level using the frame start signal and buffer threshold. Since the buffer threshold can be flexibly defined through configuration information, and the buffer state is pre-verified before data output, it can meet the real-time synchronization processing requirements of multiple video streams, possesses high synchronization flexibility, and has good system fault tolerance. Furthermore, this application generates corresponding dynamic output conditions for each video stream through a combination of the frame start signal and the buffer threshold, enabling video data buffering upon detection of the frame start signal. This effectively solves the pixel misalignment caused by hardware synchronization deviation and the overflow problem caused by insufficient buffer depth in traditional solutions, achieving a balance between synchronization accuracy and buffer safety.

[0035] Figure 1 A schematic diagram of a computing device 100 for implementing a method for video stream synchronization control according to an embodiment of this application is shown. Figure 1 As shown, the computing device 100 may have one or more processors and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor. The processors include dedicated processors such as graphics processing units (GPUs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), general-purpose computing on graphics processing units (GPGPUs), and general-purpose processing units such as CPUs.

[0036] The computing device 100 includes at least a synchronous exposure module 102, a video buffer module 104, and a video output module 106.

[0037] The synchronous exposure module 102 is configured, for example, to synchronously send an exposure signal to multiple cameras in response to a trigger signal, so that the multiple cameras can synchronously expose and acquire multiple video data based on the exposure signal.

[0038] The video caching module 104 is configured, for example, to cache the multiple video data into corresponding cache modules in multiple caching modules.

[0039] The video output module 106 is configured, for example, to synchronously output at least a portion of the video data from the plurality of buffer modules based on the frame start signal of each video data stream and the buffer threshold of the corresponding buffer module.

[0040] The following will combine Figure 2 and Figure 3 This application describes a method 200 for video stream synchronization control, based on embodiments of the present application. It should be understood that method 200 can, for example, be implemented in... Figure 1 The described computing device 100 performs the action. Method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of this application is not limited in this respect.

[0041] At step 202, the computing device 100 responds to the trigger signal by synchronously sending an exposure signal to the multiple cameras, so that the multiple cameras can synchronously expose and acquire multiple video data based on the exposure signal.

[0042] Regarding the trigger signal, it may, for example, instruct the deserializer's deserialization chip (e.g., MAX96712) to write a synchronous exposure configuration so that the deserializer, in response to the trigger signal, synchronously sends exposure signals (e.g., an Fsync signal) to multiple cameras. In some embodiments, the trigger signal may be, for example, a pulse signal with a duty cycle.

[0043] Regarding the exposure signal, it instructs, for example, multiple cameras to acquire multiple video data in response to the exposure signal.

[0044] Regarding multi-channel cameras, for example, it refers to the integration of multiple cameras in the same system. Each camera performs video acquisition in response to an exposure signal sent by a deserializer at the central processing unit, and sends the acquired video data to the deserializer via an independent data path. The deserializer is responsible for receiving and processing the independent signals from each camera, thereby ensuring parallel transmission of multiple video streams.

[0045] In some embodiments, a method for synchronously sending exposure signals to multiple cameras so that the multiple cameras can synchronously expose and acquire multiple video data based on the exposure signals includes, for example, the deserializer adjusting the timing of sending the exposure signals based on the multiple video data to synchronize the acquired multiple video data sources. For example, after receiving the synchronously sent exposure signals, the frame start signals sent back by the multiple cameras after transmission and hardware interface processing may not be completely synchronized. Therefore, the deserializer adjusts the timing of sending the exposure signals of each camera based on the returned multiple video data to control the video stream return deviation within a small range, thereby achieving video data source synchronization. Furthermore, the range of conditions for subsequent inter-frame synchronization is also reduced, the amount of video data to be buffered is reduced, and thus the requirement for buffer depth is lowered. It should be understood that in this document, "video data" refers to the original frame information after acquisition by the multiple cameras and before transmission; "video stream" refers to the dynamic sequence of data after transmission and hardware interface processing. The synchronization problem discussed in this document occurs during the video stream formation stage.

[0046] In step 204, the computing device 100 caches the multiple video data into corresponding cache modules in multiple cache modules.

[0047] Regarding the caching module, it may, for example, indicate a first-in-first-out (FIFO) caching module for real-time caching of corresponding video data. In some embodiments, multiple acquired video data streams are connected one-to-one to independent caching modules. In some embodiments, the caching module is also configured, for example, to detect the frame start signal of each video data stream in real time. It should be understood that the depth of each caching module can be the same or different.

[0048] In some embodiments, multiple caching modules are configured one-to-one with the multiple cameras to cache the video data collected from each channel.

[0049] At step 206, the computing device 100 synchronously outputs at least a portion of the video data from the plurality of cache modules based on the frame start signal of each video data stream and the cache threshold of the corresponding cache module.

[0050] In some embodiments, the cache threshold is used to indicate the maximum playback duration corresponding to the video data cached by the current cache module, that is, the maximum playable duration corresponding to the amount of cached data at a preset bitrate. The cache threshold, for example, is greater than the synchronization time difference after synchronizing the multiple video data sources. The synchronization time difference, for example, indicates the inherent synchronization time difference between each video data stream, that is, the difference between the first arriving video stream and the last arriving video stream, to ensure that when the last video stream arrives, the cache will not be prematurely cleared due to timeout, thereby avoiding the discarding of some later-arriving video data or its inability to align with other streams.

[0051] In some embodiments, the buffer threshold is set, for example, based on the number of clock cycles and the transmission rate of the multiple video data streams. In some embodiments, the method for calculating the number of clock cycles includes, for example, calculating the ratio of the maximum arrival deviation of the multiple video streams to the transmission rate of the multiple video data streams.

[0052] For example, after source synchronization processing, the maximum time deviation between the frame start signals of multiple video data streams. 10 microseconds ( The serial data stream corresponding to multiple video data streams operates at a clock frequency of 100MHz, with a single clock cycle. 10 nanoseconds ( As shown in formula (1), to eliminate this deviation, the minimum number of clock cycles required for the cache depth is at least 1000 clock cycles.

[0053] Formula 1

[0054] Furthermore, the cache threshold, besides being greater than the synchronization time difference after synchronizing the multiple video data sources, must also be less than the maximum depth of the cache module. For example, the maximum depth of the cache module is a power of 2, such as 2048. More preferably, the cache threshold can be 60% to 80% of the maximum depth of the cache module to balance synchronization alignment and cache overflow protection. The clock period corresponding to the cache threshold should be greater than... To ensure that the cache threshold can cover the maximum deviation after source synchronization, the cache threshold can be set, for example, to 1500 clock cycles, which can completely cover... The system minimizes synchronization deviations, ensuring the buffer doesn't overflow before all frame start signals arrive. If one video stream experiences a delay due to transmission anomalies, it is forcibly output when the buffered data reaches 1500 clock cycles, preventing data accumulation and transmission errors. Simultaneously, the deviations of each video stream are kept within the threshold time range, guaranteeing overall synchronization. Furthermore, the 1500 clock cycles threshold provides some depth space redundancy compared to the maximum depth of 2048 clock cycles, allowing for appropriate buffer fault tolerance when downstream data becomes congested and sends a stop-transmission notification to upstream. This approach facilitates an optimal balance between buffer protection and synchronization performance, preventing insufficient buffer space and data loss caused by excessively large depth settings.

[0055] In some embodiments, the computing device 100 synchronously transmits the trigger signal to one or more downstream deserializers connected thereto, so that the downstream deserializers send the exposure signal to other connected multi-channel cameras based on the trigger signal to synchronously acquire downstream multi-channel video data. Regarding the synchronous transmission of the trigger signal to one or more downstream deserializers connected thereto, it may, for example, indicate that the trigger signal is passed from the deserializer to the downstream deserializer, and in response to the trigger signal, the downstream deserializer sends an exposure signal to its connected downstream multi-channel cameras, such that the downstream cameras begin acquiring video data simultaneously.

[0056] In the above scheme, multiple cameras are synchronously exposed and multiple video data are acquired by triggering a signal to achieve source synchronization. The multiple video data are then cached into corresponding cache modules in multiple cache modules. Based on the frame start signal of each video data and the corresponding cache threshold of the cache module, at least a portion of the video data from the multiple cache modules is synchronously output, thereby achieving frame synchronization. Therefore, this application, through a two-stage synchronization strategy of "source synchronization" and "frame synchronization," first controls the initial deviation of the returned multi-channel video data within a small range, and then achieves complete output synchronization through the alignment of the frame start signals, solving the problem of difficulty in achieving high-precision and high-stability video synchronization acquisition in multi-channel video acquisition schemes. In addition, a threshold protection mechanism forces the system to output when abnormally delayed video data occurs, thereby ensuring the normal operation of the entire system and solving the problem of overall stuttering caused by single-channel anomalies in video data return.

[0057] Figure 3 Schematic diagrams of a video stream synchronization control system 300 in some embodiments of this application are shown. For example... Figure 3As shown, system 300 includes, for example, a deserializer 302 and a lower-level deserializer 304. Deserializer 302 is connected to a multi-channel camera, for example, and the multi-channel camera 306 includes, for example, a first camera 3062, a second camera 3064, a third camera 3066, and a fourth camera 3068. This application does not limit the specific number of multi-channel cameras.

[0058] Regarding the deserializer 302, it is configured, for example, to synchronously send exposure signals (for example, as shown in reference 310) to multiple cameras in response to a transmitted trigger signal (e.g., as shown in reference 312) so that the first camera 3062, the second camera 3064, the third camera 3066 and the fourth camera 3068 in the multiple cameras perform synchronous exposure based on the exposure signals and acquire multiple video data.

[0059] Regarding multi-channel cameras, for example, a one-to-one corresponding buffer module is set up to cache the video data of the corresponding channel in real time. For example, the first camera 3062 is equipped with a first buffer module 3082, the second camera 3064 is equipped with a second buffer module 3084, and so on.

[0060] In some embodiments, the aforementioned caching module is further configured, for example, to: detect and extract the start-of-frame signal, detect the amount of cached data, and output the cached video data in response to the synchronization control logic of the deserializer 302. For example, when the deserializer 302 detects that all caching modules have detected the start-of-frame signal, it controls all caching modules (e.g., the first caching module 3082, the second caching module 3084, the third caching module 3086, and the fourth caching module 3088) to synchronously output one frame of video data.

[0061] Regarding the lower-level deserializer 304, it is configured, for example, to be electrically connected to the deserializer 302 in order to receive a trigger signal sent by the deserializer 302, and in response to the trigger signal to send an exposure signal to other multi-channel cameras (not shown) connected thereto to synchronously acquire lower-level multi-channel video data.

[0062] In the above scheme, the global exposure trigger signal is synchronously sent to all cameras through the collaborative work of the deserializer, multiple cameras, and buffer module, realizing video synchronization at the acquisition source. In addition, during the receiving process, the deserialized video data of each channel is buffered and aligned based on the buffer module, thereby eliminating the slight delay caused by transmission and hardware interface, and realizing the synchronous output of strictly aligned multi-channel video frames.

[0063] The following will combine Figure 3 and Figure 4An exemplary flowchart describes a method 400 (i.e., step 206 in method 200) for synchronously outputting at least a portion of video data from multiple buffer modules according to embodiments of this application. In some embodiments, the dual logic of method 400 includes, for example, synchronous output based on the arrival of all frame start signals, and protective forced output based on a single buffer reaching a threshold. It should be understood that method 400 may, for example, be... Figure 1 The described computing device 100 performs the operation. Method 400 may also include additional actions not shown and / or the actions shown may be omitted; the scope of this application is not limited in this respect.

[0064] At step 402, the computing device 100 determines whether the frame start signal of each video data in the multiple video data has arrived.

[0065] For example, such as Figure 3 As shown, the computing device (not shown) continuously monitors whether the frame start signal of each video data stream has arrived in the first buffer module 3082, the second buffer module 3084, the third buffer module 3086, and the fourth buffer module 3088.

[0066] If it is determined in step 402 that the frame start signal of each video data in the multiple video data has arrived (step 402 determines "yes"), then in step 404, the computing device 100 determines whether the video data cached in the multiple cache modules is less than the corresponding cache threshold.

[0067] For example, in response to determining that the frame start signals of multiple video data streams (e.g., the first cache module 3082, the second cache module 3084, the third cache module 3086, and the fourth cache module 3088) have all arrived, the computing device further determines whether the video data cached in the first cache module 3082, the second cache module 3084, the third cache module 3086, and the fourth cache module 3088 are all less than the corresponding cache threshold.

[0068] If, in step 402, it is determined that the frame start signal for at least one video data stream has not arrived (step 402 determines "No"), then in step 406, the computing device 100 determines whether the video data cached in the cache modules where the frame start signal has arrived is less than the corresponding cache threshold. Simultaneously, for video data whose frame start signal has not yet arrived, the system can continue to wait for that video data stream (not shown in the figure). For example, if the frame start signal in the fourth cache module 3088 has not arrived, while caching the video data acquired by the first camera 3062, the second camera 3064, and the third camera 3066 into their respective first cache modules 3082, second cache module 3084, and third cache module 3086, the system continues to wait for the frame start signal for the video data acquired by the fourth camera 3068.

[0069] If, in step 406, it is determined that the video data cached in the cache module where the frame start signal has arrived is equal to the corresponding cache threshold (step 406 determines "No"), then in step 410, the cached video data is forcibly output to be equal to the video data in the cache module where the cached data is equal to the corresponding cache threshold.

[0070] On the other hand, if the video data cached in the cache module where the frame start signal has been determined to have arrived in step 406 is less than the corresponding cache threshold (step 406 determines "yes"), then video data can continue to be cached in that cache module (not shown in the figure).

[0071] If it is determined in step 404 that the video data cached in the plurality of cache modules is all less than the corresponding cache threshold (step 404 determines "yes"), then in step 408, the computing device 100 synchronously outputs the multi-channel video data from the plurality of cache modules.

[0072] In some embodiments, in response to the arrival of the frame start signal for each video data stream in the first cache module 3082, the second cache module 3084, the third cache module 3086, and the fourth cache module 3088, and the fact that the cached video data streams are all less than the corresponding cache threshold, the computing device synchronously outputs multiple video data streams.

[0073] On the other hand, if it is determined in step 404 that the video data cached in at least one cache module is equal to the corresponding cache threshold (step 404 determines "no"), then in step 410, the computing device 100 forcibly outputs the video data in the cache module whose cached video data is equal to the corresponding cache threshold.

[0074] In some embodiments, if it is determined that the frame start signal of each of the multiple video data streams has arrived, and if it is detected that the video data cached by at least one of the buffer modules has reached a preset threshold, then the cached video data is forcibly output, and the cached video data of the remaining streams is output synchronously. This effectively avoids extending the overall system waiting time due to slow caching of some data streams, thus preventing the accumulation of stuttering and latency—that is, the phenomenon where small delays in each frame accumulate frame by frame, ultimately leading to significant output lag—and thus improving the overall real-time performance of the output multiple video streams.

[0075] In other embodiments, in response to determining that at least one of the multiple video data streams has not received a frame start signal, but the video data cached by the buffer module of one of the streams has reached a preset threshold, the computing device forces the output of the cached video data, and outputs the remaining streams only after the remaining streams meet their own output conditions (e.g., the frame start signals of the remaining streams have all arrived and are less than the preset threshold).

[0076] For example, such as Figure 3 As shown, in response to the arrival of frame start signals for multiple video data streams (e.g., the first buffer module 3082, the second buffer module 3084, the third buffer module 3086, and the fourth buffer module 3088), it is further determined whether the video data buffered by at least some of the buffer modules 3082, 3084, 3086, and 3088 is equal to the corresponding buffer threshold. If so, the video data buffered in the corresponding buffer module is output. Through this method, video data from different channels are output independently when their respective buffers reach their thresholds, without mandatory time alignment constraints. This achieves real-time video data output, improves video data throughput, and avoids the problem of the entire system being blocked due to waiting for slow or faulty channels.

[0077] In some embodiments, the method 400 may optionally include step 412, wherein when step 404 determines that the video data cached in at least one cache module is equal to the corresponding cache threshold (step 404 determines "No"), the computing device 100 further outputs a synchronization anomaly flag signal. Note that step 412 is shown as after step 410 in the figure for illustrative purposes, but those skilled in the art will understand that step 412 may also be executed before step 410, or in parallel with step 410.

[0078] In some embodiments, the computing device 100, in response to determining that the video data cached by at least a portion of the plurality of cache modules is equal to a corresponding cache threshold, outputs a synchronization anomaly identification signal, the synchronization anomaly identification signal indicating that a synchronization anomaly exists between the output video data. Here, "the cached video data is equal to the corresponding cache threshold" can mean that the duration of the cached video data is equal to the corresponding cache threshold (in clock cycles, such as 1500 clock cycles as described above).

[0079] For example, if the duration of the video data cached by the first cache module 3082, the second cache module 3084, and the third cache module 3086 equals the corresponding cache threshold, while the fourth cache module 3088 has not reached the corresponding cache threshold, the threshold protection mechanism is triggered, and the video data cached by the first cache module 3082, the second cache module 3084, and the third cache module 3086 is output, along with a synchronization anomaly flag signal. In some embodiments, this synchronization anomaly flag can be used for reporting, enabling real-time monitoring and alarming of the video synchronization status. This method ensures that the synchronization effect of other pathways is maintained even in the event of an anomaly, and also improves the monitorability of the anomaly output.

[0080] Furthermore, in some embodiments, the method for synchronously outputting at least a portion of the video data from the plurality of cache modules based on the frame start signal of each video data stream and the corresponding cache threshold of the cache module further includes, for example, sending a video transmission pause signal to the corresponding multi-channel camera in response to determining that the video data cached by at least a portion of the cache modules is equal to the corresponding cache threshold, so as to stop the camera of the corresponding channel from transmitting the acquired video data back to the cache module.

[0081] For example, assuming the duration of the video data cached by the first cache module 3082, the second cache module 3084, and the third cache module 3086 is equal to the corresponding cache threshold, the deserializer 302 sends a video transmission pause signal to the corresponding first camera 3062, second camera 3064, and third camera 3066 to pause the transmission of acquired video data. In this way, the deserializer 302 proactively notifies the corresponding camera to pause video data transmission, effectively preventing data loss due to overflow of the corresponding cache module. This ensures the integrity and synchronization of multi-channel video transmission even with limited cache hardware resources.

[0082] In the above scheme, by detecting and judging the frame start signals of multiple video streams, and controlling the synchronous output of data when all signals arrive normally, complete frame-level alignment is achieved. This ensures that the output video data meets the core synchronization accuracy requirements of applications such as image stitching and fusion. Furthermore, by setting a buffer threshold for the buffer module, when an anomaly in a single video stream causes its buffer data volume to reach a preset threshold, protection logic is triggered to force the output of that stream's data, effectively preventing data loss or system crashes caused by buffer overflow. This ensures that even when individual streams are abnormal, the system can still maintain operation and output usable data (while simultaneously marking the anomaly), significantly improving the system's robustness and data security. Without increasing additional hardware costs, the scheme balances the contradiction between high synchronization accuracy and hardware resource limitations.

[0083] It should be understood that the method involved in this application can be implemented using various programmable devices such as FPGA, MCU, and DSP, and the number of video data channels can be flexibly expanded according to needs, making it suitable for various scenarios requiring frame synchronization, such as vehicle surround view, security monitoring, and multi-camera systems. Furthermore, the frame start signal detection and synchronization control logic upon which this solution relies has good compatibility, can be adapted to various hardware platforms such as FPGA, MCU, and DSP, and the number of video data channels can be flexibly expanded according to actual needs, resulting in a highly scalable system architecture.

[0084] The various processes and procedures described above, such as methods 200 and 400, can be executed at a computing device. This computing device may include, for example, at least one processor (at least one graphics processor and at least one central processing unit); and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor. In some embodiments, methods 200 and 400 may be implemented as a computer software program or program product tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing device via read-only memory (ROM) and / or a communication unit. When the computer program is loaded into random-access memory (RAM) and executed by the GPU and CPU, one or more actions of methods 200 and 400 described above can be performed.

[0085] This application may be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this application. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0086] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions can be provided to the central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the central processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0088] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0089] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps loaded in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0090] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors.

Claims

1. A method for video stream synchronization control, characterized in that, The method includes: In response to a trigger signal, an exposure signal is synchronously sent to multiple cameras so that the multiple cameras can perform synchronous exposure and acquire multiple video data based on the exposure signal; The multi-channel video data is cached into corresponding cache modules in multiple cache modules; and Based on the frame start signal of each video data stream and the corresponding cache threshold of the cache module, at least a portion of the video data from the multiple cache modules is synchronously output.

2. The method according to claim 1, wherein the plurality of cache modules are configured in one-to-one correspondence with the multi-channel camera to cache each channel of video data acquired; and the cache threshold is used to indicate the maximum playback duration corresponding to the video data cached by the current cache module.

3. The method according to claim 2, wherein, Based on the frame start signal of each video data stream and the corresponding buffer threshold of the buffer module, synchronously outputting at least a portion of the video data from the multiple buffer modules includes: Determine whether the frame start signal of each video data stream in the multi-channel video data has arrived; In response to determining that the frame start signals of the multiple video data have all arrived, it is determined whether the video data cached in the multiple cache modules is all less than the corresponding cache threshold; In response to determining that the video data cached in the plurality of cache modules is all less than the corresponding cache threshold, the multi-channel video data is synchronously output from the plurality of cache modules; and In response to the determination that the video data cached in the buffer module has arrived equals the corresponding buffer threshold, the system forcibly outputs the video data cached in the buffer module equal to the corresponding buffer threshold.

4. The method according to claim 3, wherein, Based on the frame start signal of each video data stream and the corresponding buffer threshold of the buffer module, synchronously outputting at least a portion of the video data from the multiple buffer modules further includes: In response to determining that the frame start signal of at least one video data stream has not arrived, determine whether the video data cached in the buffer module where the frame start signal has arrived is less than the corresponding buffer threshold; If the video data cached in the cache module that determines that the start of frame signal has arrived is less than the corresponding cache threshold, then video data can continue to be cached in that cache module.

5. The method according to claim 3, wherein, Based on the frame start signal of each video data stream and the corresponding buffer threshold of the buffer module, synchronously outputting at least a portion of the video data from the multiple buffer modules further includes: In response to determining that the video data cached in at least one cache module is equal to the corresponding cache threshold, the cached video data in the cache module is forcibly output to be equal to the video data in the cache module corresponding to the cache threshold.

6. The method according to claim 5, further comprising: In response to determining that at least some of the cache modules cache video data equal to the corresponding cache threshold, a synchronization anomaly identification signal is output, the synchronization anomaly identification signal being used to indicate that there is a synchronization anomaly among the output video data.

7. The method according to claim 2, wherein the buffer threshold is greater than the synchronization time difference after the multiple video data sources are synchronized.

8. The method according to claim 2, wherein the buffer threshold is set based on the number of clock cycles and the transmission rate of the multi-channel video data.

9. The method according to claim 2, wherein the cache threshold is less than the maximum depth value of the cache module.

10. The method according to claim 3, wherein, Based on the frame start signal of each video data stream and the corresponding buffer threshold of the buffer module, synchronously outputting at least a portion of the video data from the multiple buffer modules further includes: In response to determining that the video data cached by at least some of the plurality of cache modules is equal to the corresponding cache threshold, a video transmission pause signal is sent to the corresponding multi-channel camera so that the camera on the corresponding channel stops sending the acquired video data back to the cache module.

11. The method according to claim 1, wherein synchronously sending exposure signals to multiple cameras so that the multiple cameras perform synchronous exposure based on the exposure signals and acquire multiple video data streams comprises: Based on the multi-channel video data, the deserializer adjusts the timing of sending the exposure signal to synchronize the acquired multi-channel video data sources.

12. The method according to claim 1, wherein the trigger signal is a pulse signal with a duty cycle.

13. The method according to claim 1, further comprising: The trigger signal is synchronously transmitted to one or more downstream deserializers connected to it, so that the downstream deserializers can send the exposure signal to other connected multi-channel cameras based on the trigger signal to synchronously acquire downstream multi-channel video data.

14. A computing device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method according to any one of claims 1-13.

16. A computer program product, characterized in that, Includes a computer program, which, when executed by a machine, performs the method according to any one of claims 1-13.

Images