Multi-camera image frame synchronization method and system, vehicle and storage medium

Abstract

The invention discloses a multi-camera image frame synchronization method and system, a vehicle and a storage medium, and belongs to the technical field of automatic driving, and the method comprises the steps: obtaining image frames of a plurality of cameras, storing the image frames in a same image frame queue, sequentially extracting target image frames from the image frame queue, and reading the timestamps of the target image frames; determining a first difference value between the timestamp of the target image frame and a reference timestamp of a preset preparation area; based on the relation between the first difference value and a preset first error threshold value and a preset second error threshold value, the synchronization state of the target image frame is judged, and the second error threshold value is larger than the first error threshold value; and executing a corresponding synchronization action based on the synchronization state. According to the scheme, storage redundancy and synchronization algorithm complexity can be effectively reduced, and accurate alignment of multi-camera image frames on the time scale and real-time performance of synchronization can be guaranteed.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a method, system, vehicle, and storage medium for synchronizing multi-camera image frames. Background Technology

[0002] With the rapid development of advanced driver assistance systems (ADAS), the collaborative operation of multiple cameras, including front-view, rear-view, surround-view, and side-view cameras, has become standard in vehicle perception systems. The image data collected by these cameras supports key driving functions such as automatic parking, automatic emergency braking, blind spot monitoring, and adaptive cruise control. However, when a vehicle is traveling at high speed, images captured by cameras at different locations at different times can reflect significant differences in the physical world. For example, at a vehicle speed of 120 km / h, a 10-millisecond error corresponds to a displacement of approximately 33 centimeters. Clearly, even a tiny time error can lead to misjudgments by ADAS systems, seriously threatening driving safety.

[0003] Currently, most software synchronization solutions use multiple queues to store data from different cameras, which is complex and has spatial redundancy. This can easily lead to the loss of valid data or block the entire synchronization process due to the delay of a single image frame, resulting in a large delay in the synchronization of image frames from multiple cameras. Summary of the Invention

[0004] This invention provides a method, system, vehicle, and storage medium for synchronizing image frames from multiple cameras, aiming to solve the problems of poor image frame synchronization and real-time performance caused by hardware synchronization errors, link transmission errors, or system failures.

[0005] Firstly, a method for synchronizing image frames from multiple cameras is provided, including: Image frames from multiple cameras are acquired and stored in the same image frame queue. Target image frames are then extracted sequentially from the image frame queue, and the timestamps of the target image frames are read. Determine a first difference between the timestamp of the target image frame and the reference timestamp of a preset preparation area, wherein the preparation area is used to cache a set of image frames being synchronized; Based on the relationship between the first difference and a preset first error threshold and a preset second error threshold, the synchronization state of the target image frame is determined; the first error threshold is used to determine image frame synchronization, the second error threshold is used to distinguish different image frames, and the second error threshold is greater than the first error threshold. Based on the synchronization state, perform the corresponding synchronization action.

[0006] In some embodiments, determining the synchronization state of the target image frame based on the relationship between the first difference and preset first error thresholds and preset second error thresholds includes: If the absolute value of the first difference is less than or equal to the first error threshold, it is determined that the target image frame is synchronized with the image frame in the preparation area; If the first difference is positive and greater than the second error threshold, or if the first difference is negative and its absolute value is greater than a preset multiple of the second error threshold, it is determined that the target image frame is out of sync with the image frames in the preparation area. If the first difference is positive and greater than the first error threshold and less than or equal to the second error threshold, or if the first difference is negative and its absolute value is greater than the first error threshold and less than or equal to a preset multiple of the second error threshold, the synchronization state of the target image frame is determined to be uncertain.

[0007] In some embodiments, performing the corresponding synchronization action based on the synchronization state includes: If it is determined that the target image frame is synchronized with the image frame in the preparation area, the target image frame is placed into the preparation area; If it is determined that the target image frame is out of sync with the image frames in the preparation area, and the timestamp of the target image frame is greater than the reference timestamp of the preparation area, the operation of migrating the set of image frames in the preparation area to a preset ready area is performed, and the target image frame is placed into the preparation area. The ready area is used to cache the set of image frames that have been synchronized.

[0008] In some embodiments, performing the corresponding synchronization action based on the synchronization state further includes: If it is determined that the target image frame is out of sync with the image frames in the preparation area, the timestamp of the target image frame is less than the reference timestamp of the preparation area, and the absolute value of the first difference is greater than a preset multiple of the second error threshold, the reference timestamp of the preparation area is determined to be abnormal, and the operation of clearing the preparation area is performed.

[0009] In some embodiments, performing the corresponding synchronization action based on the synchronization state further includes: If the synchronization state of the target image frame is uncertain, calculate a second difference between the timestamp of the target image frame and the reference timestamp of the ready area; If the absolute value of the second difference is less than or equal to the first error threshold, the target image frame is placed into the ready area; If the absolute value of the second difference is greater than the first error threshold, the target image frame is discarded.

[0010] In some embodiments, the step of migrating the set of image frames in the preparation area to the ready area includes: Clear the ready area and move all image frames in the preparation area to the ready area; Assign the reference timestamp value of the preparation area to the reference timestamp of the ready area; And, clear the preparation area.

[0011] In some embodiments, it also includes: In the initial state, the reference timestamp of the preparation area is set to an invalid value; If the preparation area is empty, the timestamp of the first image frame placed in the preparation area is set as the reference timestamp of the preparation area.

[0012] Secondly, a multi-camera image frame synchronization system is also provided, including: The data acquisition module is used to acquire image frames from multiple cameras and store them in the same image frame queue, and to extract target image frames sequentially from the image frame queue and read the timestamps of the target image frames. The timestamp difference determination module is used to determine the first difference between the timestamp of the target image frame and the reference timestamp of the preset preparation area, wherein the preparation area is used to cache the set of image frames being synchronized. The synchronization state determination module is used to determine the synchronization state of the target image frame based on the relationship between the first difference and a preset first error threshold and a preset second error threshold; the first error threshold is used to determine the synchronization of the image frame, the second error threshold is used to distinguish different image frames, and the second error threshold is greater than the first error threshold. The synchronization execution module is used to perform corresponding synchronization actions based on the synchronization state.

[0013] Thirdly, a vehicle is also provided, including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the steps of any of the methods described above.

[0014] Fourthly, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program being loaded by a processor to perform the steps of any of the methods described above.

[0015] Beneficial effects: This application avoids the space waste and management complexity caused by multiple independent queues by uniformly storing image frames from multiple cameras into the same queue. Then, it calculates the first difference between the timestamp of the target image frame and the reference timestamp of the preparation area, and determines the synchronization status by combining it with two preset error thresholds. Finally, it performs corresponding actions such as storing in the preparation area, migrating to the ready area, clearing the preparation area, or discarding according to different synchronization statuses. This can effectively reduce storage redundancy and synchronization algorithm complexity, and accurately identify and handle various scenarios such as frame synchronization, new frames, timestamp rollback failures, and uncertain attribution, thereby ensuring the accurate alignment and real-time synchronization of image frames from multiple cameras on the time scale. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a simplified flowchart of a multi-camera image frame synchronization method provided by an exemplary embodiment of this disclosure; Figure 2 This is a block diagram illustrating the principle of a multi-camera image frame synchronization method provided by an exemplary embodiment of this disclosure; Figure 3 This is a detailed flowchart of a multi-camera image frame synchronization method provided by an exemplary embodiment of this disclosure; Figure 4 This is a schematic diagram of the preparation area timestamp comparison area provided in an exemplary embodiment of this disclosure; Figure 5 This is a schematic diagram of the ready area timestamp comparison region provided in an exemplary embodiment of this disclosure; Figure 6 This is a schematic diagram of a visual perception process corresponding to a vehicle visual perception scenario provided by an exemplary embodiment of this disclosure; Figure 7 This is a schematic diagram of the functional modules of the multi-camera image frame synchronization system provided in an exemplary embodiment of this disclosure; Figure 8 This is a schematic diagram of a vehicle structure provided by an exemplary embodiment of this disclosure. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] "A and / or B" includes the following three combinations: A only, B only, and a combination of A and B.

[0021] The use of "applies to" or "configured to" in this application implies open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0022] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0023] In Advanced Driver Assistance Systems (ADAS), visual perception is a crucial component. It primarily utilizes images or video streams captured by multiple cameras, employing computer vision and deep learning technologies to identify and understand the environment surrounding the vehicle in real time, detecting and classifying various objects on the road (such as vehicles, pedestrians, cyclists, traffic signs, and traffic lights). Multi-camera frame synchronization is key to achieving accurate environmental perception.

[0024] Multi-camera image frame synchronization refers to aligning image frames captured by different cameras at the same physical moment on a time scale, thereby eliminating positional misalignment caused by vehicle movement or dynamic targets (such as pedestrians or vehicles) in asynchronous images. Ideally, each set of synchronized data contains image frames from all cameras at the same physical moment. However, due to limitations in their own clock precision, the timestamp assigned to the image frame by the camera at exposure time is not exactly the same as the actual physical time, resulting in a certain error. In other words, even several image frames captured at the same moment will have different timestamps. In addition, camera malfunctions or unexpected events during image frame transmission may lead to delayed arrival, image frame loss, or even timestamp rollback. Furthermore, performance requirements for the synchronization process may exist in practical applications, such as ADAS applications, where real-time performance and stability of the entire process must be guaranteed.

[0025] Based on this, on the one hand, this embodiment provides a multi-camera image frame synchronization method, applied to vehicles equipped with multiple cameras and advanced driver assistance systems (ADAS), to cooperate with the ADAS to achieve autonomous or assisted driving of the vehicle, such as... Figure 1 As shown, it includes the following steps: Step 100: Acquire image frames from multiple cameras and store them in the same image frame queue, then extract the target image frame sequentially from the image frame queue and read the timestamp of the target image frame.

[0026] Specifically, multiple cameras are installed on vehicles equipped with advanced driver assistance systems (ADAS) to support autonomous driving functions such as automatic parking, automatic emergency braking, blind spot monitoring, and adaptive cruise control. There is no specific limit to the number of cameras; for example, ten or more are acceptable, as long as the vehicle's autonomous driving requirements are met.

[0027] Considering that if each camera's data were stored in a separate queue, the frame synchronization algorithm would need to sequentially retrieve one image frame from each queue for synchronization. Maintaining each stream queue individually would lead to space redundancy and management complexity. Therefore, in this embodiment, after initiating the multi-camera image frame synchronization process, all image frames acquired by participating cameras are aggregated and stored in the same image frame queue. This image frame queue serves as a centralized receiving and temporary storage medium for all image frames. Then, image frames to be processed are sequentially extracted from this image frame queue according to time sequence as target image frames, while simultaneously reading the timestamp assigned to the target image frame during acquisition and exposure.

[0028] Step 200: Determine the first difference between the timestamp of the target image frame and the reference timestamp of the preset preparation area, which is used to cache the set of image frames being synchronized.

[0029] Specifically, a buffer, known as the preparation area, is pre-defined to cache a set of image frames undergoing synchronization. A reference timestamp is set for the preparation area, initially set to an invalid value to indicate that there is no valid synchronization data in the preparation area. After extracting the target image frame and its timestamp, the difference between the target image frame's timestamp and the reference timestamp in the preparation area is calculated. This difference is defined as the first difference, used to initially determine whether the target image frame and the set of image frames being synchronized in the preparation area belong to the same physical moment. Initially, the preparation area is empty and set to an invalid value for its reference timestamp. The first image frame added to the preparation area is considered to be synchronized with the preparation area, and its timestamp is set as the reference timestamp for the preparation area.

[0030] Step 300: Based on the relationship between the first difference and the preset first error threshold and the preset second error threshold, determine the synchronization status of the target image frame; the first error threshold is used to determine the synchronization of the image frame, the second error threshold is used to distinguish different image frames, and the second error threshold is greater than the first error threshold.

[0031] Specifically, two error thresholds are pre-configured. The first error threshold is used to determine whether two image frames belong to the same synchronized image frame at the same physical moment, and the second error threshold is used to determine whether two image frames belong to different synchronized image frames at different physical moments. The value of the second error threshold is greater than the first error threshold. Then, the first difference calculated in step 200 is compared with the first and second error thresholds respectively. Combined with whether the reference timestamp in the preparation area is invalid, the synchronization status of the target image frame is determined. The synchronization status includes types such as synchronized with the image frames in the preparation area, belonging to a new image frame, having a timestamp backslip fault, uncertain attribution, and being able to supplement synchronization with already synchronized image frames.

[0032] Step 400: Based on the synchronization state, perform the corresponding synchronization action.

[0033] Specifically, based on the type of synchronization status of the target image frame, the corresponding synchronization action is executed. If it is determined that the target image frame is synchronized with the image frame in the preparation area, or the reference timestamp in the preparation area is invalid (i.e., the preparation area is empty), the target image frame is stored in the preparation area. If the preparation area has collected image frames from all cameras at the corresponding time, the data migration in the preparation area is triggered. If it is determined that it belongs to a new image frame, the synchronized image frames in the preparation area are first migrated to another preset buffer, i.e., the ready area, to cache the synchronized image frames. Then, the target image frame is stored in the preparation area and the reference timestamp in the preparation area is updated. If it is determined that there is a timestamp backflip fault, the preparation area is cleared and its reference timestamp is reset to an invalid value to ensure that subsequent image frame synchronization proceeds normally. If the timestamp of the target image frame is between the first error threshold and the second error threshold, it cannot be clearly classified into the currently synchronized image frame set in the preparation area, nor can it be directly determined as a new image frame or a historical old image frame, and there is ambiguity in its classification, then it is classified as having an uncertain classification.

[0034] In this embodiment, by storing image frames from multiple cameras in the same queue, the space waste and management complexity caused by multiple independent queues can be avoided. The first difference between the timestamp of the target image frame and the reference timestamp in the preparation area is calculated, and the synchronization state is determined by combining this with two preset error thresholds. Finally, based on different synchronization states, corresponding actions such as storing in the preparation area, migrating to the ready area, clearing the preparation area, or discarding are performed. This effectively reduces storage redundancy and synchronization algorithm complexity, and accurately identifies and handles various scenarios such as frame synchronization, new frames, timestamp rollback failures, and uncertain attribution. This ensures accurate alignment and real-time synchronization of multi-camera image frames on the time scale, thereby improving the stability and practicality of the synchronization process. Simultaneously, it adapts to the needs of different numbers of cameras, providing reliable image data for advanced driver assistance functions such as automatic parking and automatic emergency braking.

[0035] In some embodiments, to avoid directly discarding potentially valid delayed image frames among target image frames with uncertain attribution, the difference between the timestamp of the target image frame and the reference timestamp of the ready area, i.e., the second difference, is calculated to determine whether the timestamp of the target image frame belongs to the physical time corresponding to the set of synchronized image frames in the ready area. If the absolute value of the second difference is less than the first error threshold, it indicates that although the target image frame has not been synchronized with the preparation area in time, it belongs to the delayed arrival data of the synchronized image frame set in the ready area and is still valid. In this case, it is stored in the ready area to supplement and improve the synchronized image frame set at that physical time, ensuring data integrity and providing a more comprehensive environmental image at the same time for downstream modules. If the absolute value of the second difference is greater than the first error threshold, it indicates that the target image frame does not belong to the currently synchronized image frame in the preparation area, nor to the synchronized image frame in the ready area. It has no clear attribution and no utilization value. To avoid it occupying storage resources or causing misjudgments in subsequent environmental perception, target detection, and other functions, the image frame is directly discarded. As can be seen, this embodiment, for target image frames whose synchronization status cannot be clearly determined, filters out the delayed-arriving valid image frames and improves the data in the ready area by comparing them with the reference timestamp of the ready area, while discarding worthless invalid image frames. This avoids the loss of valid data and prevents erroneous data from interfering with the synchronization accuracy.

[0036] In this embodiment, by using a unified frame queue to receive data from multiple cameras, dual buffers to cache synchronized and already synchronized frames, a refined comparison logic combining dual error thresholds and timestamp back-judgment, and a secondary filtering mechanism for frames with uncertain attribution, the complexity of managing multiple queues and spatial redundancy can be avoided. This also effectively addresses issues such as timestamp errors, timestamp back-judgments, frame delays, and frame loss. Furthermore, it eliminates the dependence on image content features. While ensuring the accuracy of multi-camera frame synchronization, it achieves lightweight operation and high real-time output in the synchronization process. Moreover, by flexibly adapting to the number of cameras and being compatible with hardware synchronization errors, it improves the scalability and stability of the solution, providing reliable data for accurate environmental perception in advanced driver assistance systems.

[0037] As an exemplary example, the principle block diagram of the multi-camera image frame synchronization method is as follows: Figure 2As shown, firstly, n image frames are acquired through cameras 1, 2, ..., n (where n is a positive integer), and then uniformly merged into a single frame queue, replacing the traditional design of multiple independent queues to simplify management and reduce redundancy. Then, a frame synchronization module reads frame data from this queue, reading the first frame at a time and using a frame synchronization algorithm for synchronization. Specifically, the frame synchronization module has two buffers: a preparation area and a ready area. Each buffer contains n data streams within a synchronization time frame, along with their respective reference timestamps for synchronization. The preparation area buffers data currently being synchronized, while the ready area buffers data that has already been synchronized. For each read image frame, a preset frame synchronization algorithm is used to determine whether it is synchronized with the preparation or ready area based on its timestamp. If synchronized, it is placed in the corresponding area; otherwise, the frame is discarded. Ideally, once the preparation area is full, all n data streams within the synchronization time frame have been synchronized, triggering a migration of the preparation area and updating the ready area. Therefore, the ready area always stores the latest set of synchronized frames, and downstream modules can directly obtain the latest synchronized frames from the ready area. This ensures synchronization accuracy while avoiding overall blocking caused by single-stream frame delays, achieving lightweight and high real-time multi-stream camera frame synchronization.

[0038] In some embodiments, step 300, which determines the synchronization state of the target image frame based on the relationship between the first difference and preset first error thresholds and preset second error thresholds, includes: Step 310: If the absolute value of the first difference is less than or equal to the first error threshold, determine that the target image frame is synchronized with the image frame in the preparation area.

[0039] Specifically, first, it is confirmed whether the reference timestamp in the preparation area is valid, i.e., not an initial invalid value. Then, the first difference between the target image frame timestamp and the reference timestamp in the preparation area is calculated and its absolute value is taken. When this absolute value is less than or equal to a preset first error threshold, it indicates that the physical time corresponding to the target image frame and the set of image frames being synchronized in the preparation area is consistent, and the two belong to the synchronization data of the same image frame. Therefore, it is determined that the target image frame is synchronized with the image frames in the preparation area. If the reference timestamp in the preparation area is an initial invalid value, it indicates that the preparation area is empty and has not yet started caching the synchronized image frame at a certain moment. In this case, the target image frame extracted at this time will be used as the starting data of the synchronized image frame set at that moment, and it will also be determined to be synchronized with the image frames in the preparation area. Subsequently, the reference timestamp in the preparation area will be updated with the timestamp of the target image frame.

[0040] Step 320: If the first difference is positive and greater than the second error threshold, or if the first difference is negative and its absolute value is greater than a preset multiple of the second error threshold, it is determined that the target image frame is out of sync with the image frames in the preparation area.

[0041] Specifically, after calculating the first difference between the timestamp of the target image frame and the reference timestamp of the preparation area, if the first difference is positive and greater than a preset second error threshold, and the second error threshold is greater than the first error threshold, it indicates that the timestamp of the target image frame is much later than the physical time corresponding to the reference timestamp of the preparation area, and there is no synchronous relationship between the two. Therefore, it is determined that the target image frame is out of sync with the image frames in the preparation area, and the target image frame belongs to the image frame of the new physical time. In addition, if the calculated first difference is negative and its absolute value is greater than a preset multiple of the second error threshold, it indicates that the absolute value of the first difference is much greater than the second error threshold, that is, the timestamp of the target image frame is much earlier than the reference timestamp of the preparation area. Combined with the normal timing characteristics of timestamps in the vehicle scenario, it can be determined that a timestamp rollback fault has occurred, indicating that the target image frame is out of sync with the image frames in the preparation area due to the timestamp abnormality.

[0042] Step 330: If the first difference is positive and greater than the first error threshold and less than or equal to the second error threshold, or if the first difference is negative and its absolute value is greater than the first error threshold and less than or equal to a preset multiple of the second error threshold, the synchronization state of the target image frame is determined to be uncertain.

[0043] Specifically, if the calculated first difference is positive and greater than the first error threshold, but less than or equal to the second error threshold, or if the first difference is negative and its absolute value is greater than the first error threshold and less than or equal to a preset multiple of the second error threshold, then, since the first error threshold is the boundary for determining synchronized image frames and the second error threshold is the boundary for distinguishing different image frames, the timestamp of the target image frame is in a fuzzy range that may belong to the current synchronized image frame set, the delayed data of the previous image frame, or the advanced data of the next image frame. Considering the stringent accuracy requirements of multi-camera synchronization in vehicles, it is impossible to accurately define its belonging through these two error thresholds. Forcibly classifying it into a certain set may lead to synchronization errors, thereby affecting subsequent environmental perception and decision-making. Therefore, the synchronization state of the target image frame is determined to be uncertain.

[0044] In this embodiment, by setting two levels of judgment criteria—a first error threshold and a second error threshold—and combining the absolute value of the first difference with the comparison results of the two thresholds, the target image frame is divided into three synchronization states: synchronized with the preparation area, asynchronous with the preparation area, and with uncertain synchronization status. This not only clarifies the basis for determining the ownership of frames at the same physical moment, but also defines the distinction boundary of new frames and identifies frames with ambiguous ownership to be verified. This avoids the synchronization misjudgment or loss of valid frames caused by traditional single threshold judgment, thereby providing an accurate and reliable judgment basis for performing synchronization actions and ensuring the orderliness and accuracy of the multi-camera frame synchronization process.

[0045] In some embodiments, step 400 involves performing a corresponding synchronization action based on the synchronization state, including: Step 410: If the target image frame is determined to be synchronized with the image frame in the preparation area, place the target image frame into the preparation area.

[0046] Specifically, once the target image frame is determined to be synchronized with the image frames in the preparation area, the target image frame is directly placed into the preparation area, forming a complete set of synchronized image frames together with the image frames from other cameras already cached in the preparation area at the same time. If the reference timestamp of the preparation area is initially invalid before placement, i.e., the preparation area is empty, the reference timestamp of the preparation area is immediately updated with the timestamp of the target image frame after placement, establishing the time reference for the synchronized image frame set. After the target image frame is placed into the preparation area, it is checked whether the preparation area has collected all the image frames corresponding to the cameras participating in the synchronization at the same time. If they have been collected, it indicates that the image frame synchronization at that moment is complete, and the preparation area migration operation is triggered to migrate the complete set of image frames in the preparation area to the preset ready area. At the same time, the reference timestamp of the ready area is updated to the original reference timestamp of the preparation area, and the reference timestamp of the preparation area is reset to an invalid value. The preparation area is then cleared to prepare for receiving the image frame synchronization of the next moment.

[0047] Step 420: If it is determined that the target image frame is out of sync with the image frames in the preparation area, and the timestamp of the target image frame is greater than the reference timestamp of the preparation area, the operation of migrating the set of image frames in the preparation area to the preset ready area is performed, and the target image frame is placed into the preparation area. The ready area is used to cache the set of image frames that have been synchronized.

[0048] Specifically, if the target image frame is determined to be out of sync with the image frames in the preparation area, and the timestamp of the target image frame is greater than the reference timestamp of the preparation area, it indicates that the target image frame belongs to image data from a new physical moment, and the synchronization operation of the current preparation area must be completed first. First, a ready area is pre-set to cache the set of synchronized image frames, ensuring that the ready area is in an empty state. Then, a preparation area migration operation is performed, completely migrating the cached synchronization data in the preparation area that has not yet collected all camera image frames at the current moment to the ready area for caching, thus clearing the preparation area and ensuring that synchronized valid data is not lost. The reference timestamp of the ready area is also updated synchronously to the original reference timestamp of the preparation area. After the migration is complete, the reference timestamp of the preparation area is reset to an invalid value and the preparation area is cleared. Then, the new target image frame is placed into the preparation area, and the reference timestamp of the preparation area is reset with the timestamp of the target image frame, initiating the image frame synchronization process for the new moment. This ensures that other camera image frames extracted at the same time can be accurately assigned to the preparation area.

[0049] In some embodiments, step 400, which involves performing a corresponding synchronization action based on the synchronization state, further includes step 430, which specifically includes: If the target image frame is out of sync with the image frames in the preparation area, the timestamp of the target image frame is less than the reference timestamp of the preparation area, and the absolute value of the first difference is greater than a preset multiple of the second error threshold, the reference timestamp of the preparation area is determined to be abnormal, and the operation of clearing the preparation area is executed.

[0050] Specifically, when it is determined that the target image frame is out of sync with the image frames in the preparation area, and the timestamp of the target image frame is less than the reference timestamp of the preparation area, while the absolute value of the first difference exceeds a preset multiple of the second error threshold, combined with the normal timing pattern of timestamps in the vehicle scenario, it can be determined that a timestamp rollback fault has occurred. The current reference timestamp in the preparation area is in an abnormal state, and continuing to synchronize based on this reference timestamp will cause subsequent valid image frames to be misjudged and discarded. To eliminate the impact of this anomaly on the synchronization process, the preparation area will be cleared, and the reference timestamp in the preparation area will be reset to its initial invalid value, restoring the preparation area to its initial state where it can receive new image frames. This ensures that subsequent valid image frames with the rolled-back timestamp can enter the preparation area normally, re-establish a new synchronization time reference, and ensure the continuous and stable operation of the entire multi-camera image frame synchronization process.

[0051] In some embodiments, step 400, which involves performing a corresponding synchronization action based on the synchronization state, further includes: Step 440: If the synchronization status of the target image frame is uncertain, calculate the second difference between the timestamp of the target image frame and the reference timestamp of the ready area.

[0052] Specifically, when the synchronization status of the target image frame is determined to be uncertain, in order to avoid directly discarding any potentially valid delayed image frames, the reference timestamp of the ready area is retrieved. This reference timestamp corresponds to the time reference of the set of synchronized image frames in the ready area. The difference between the timestamp of the target image frame and the reference timestamp of the ready area is calculated, and this difference is defined as the second difference. The second difference is used to determine whether the target image frame and the set of synchronized image frames in the ready area belong to the same physical time.

[0053] Step 450: If the absolute value of the second difference is less than or equal to the first error threshold, place the target image frame into the ready area.

[0054] Specifically, if the absolute value of the second difference is less than or equal to the preset first error threshold, it indicates that although the target image frame cannot match the currently synchronized image frame set in the preparation area, it is consistent with the physical time corresponding to the synchronized image frame set in the ready area, and belongs to the delayed valid data in the synchronized image frame set. To improve the synchronized image frame set in the ready area and ensure data integrity, the target image frame is placed in the ready area and integrated with other cached image frames from other cameras at the same time, so that the ready area always maintains the most complete synchronized image frame data at that physical time for downstream modules to call.

[0055] Step 460: If the absolute value of the second difference is greater than the first error threshold, discard the target image frame.

[0056] Specifically, if the absolute value of the calculated second difference is greater than the first error threshold, it indicates that the target image frame cannot match the currently synchronized image frame set in the preparation area, nor does it belong to the physical moment corresponding to the image frame set that has been synchronized in the ready area. Its ownership is unclear and it has no effective utilization value. Considering the stringent requirements for data accuracy in multi-camera synchronization in vehicles, including this unowned image frame in any synchronization set may lead to misjudgments in subsequent environmental perception, target detection, and other functions, thereby affecting driving safety. Therefore, the target image frame is directly discarded to avoid invalid data occupying storage resources or interfering with the synchronization process.

[0057] As an exemplary example, a detailed flowchart of multi-camera image frame synchronization via a frame synchronization module is provided, such as... Figure 3 As shown, the specific steps are as follows: 1) After the multi-camera image frame synchronization process begins, a first error threshold, diff_min, and a second error threshold, diff_max, are set. Diff_min is used to determine whether two images belong to the same frame, and diff_max is used to distinguish image frames at different physical times.

[0058] 2) Define two buffers: a preparation buffer and a ready buffer. The preparation buffer is used to buffer a frame that is being synchronized, while the ready buffer is used to store a frame that has been synchronized. Each buffer has its own base timestamp. Initialize the base timestamp of the preparation buffer to pBaseTsn and the base timestamp of the ready buffer to rBaseTsn. Set the initial value of both base timestamps to 0. A value of 0 indicates that the timestamp is invalid and there is no data in the corresponding area.

[0059] The synchronization of each image frame is achieved by comparing the image's timestamp with two reference timestamps. The detailed timestamp comparison logic is as follows: Figure 4 and Figure 5As shown, based on the reference timestamp of the preparation area (pBaseTsn), the reference timestamp of the ready area (rBaseTsn), the first error threshold (diff_min), and the second error threshold (diff_max), five regions—block a, block b, block c, block d, and block e—are divided. Among them, as shown... Figure 4 As shown, the preparation area is divided into regions centered on its reference timestamp pBaseTsn. Block a is the range on both sides of pBaseTsn, covering the first error threshold diff_min, corresponding to frames whose timestamp and pBaseTsn difference is within diff_min; Block b is the range outside block a but within the second error threshold diff_max, corresponding to frames whose difference is between diff_min and diff_max. Specifically, the range of block b to the right of block a is between diff_min and diff_max, and the range of block b to the left of block a is between diff_min and m*diff_max; Block c is the range to the right of pBaseTsn exceeding diff_max, corresponding to frames whose difference is greater than diff_max; Block d is the range to the left of pBaseTsn exceeding a preset multiple (m×diff_max), corresponding to frames whose absolute difference is greater than m×diff_max. Figure 5 As shown, the ready area is divided into regions centered on its reference timestamp rBaseTsn, and contains only block e, that is, the range of the first error threshold diff_min on both sides of rBaseTsn, corresponding to frames whose timestamp and rBaseTsn difference is within diff_min.

[0060] 3) Retrieve an image frame from the frame queue and read its timestamp tsn. Calculate the deviation between tsn and pBaseTsn, using this as the first difference value: a) If the absolute value of the first difference is less than diff_min, it indicates that tsn is close to pBaseTsn, corresponding to region a. This means the image frame is synchronized with the data in the preparation area, i.e., belongs to the same frame, and is therefore placed in the preparation area. Alternatively, if pBaseTsn is 0, it means the preparation area is temporarily empty. In this case, the image frame is also placed in the preparation area, and pBaseTsn is updated with the current frame's timestamp tsn, i.e., pBaseTsn = tsn. After the data is placed in the preparation area, if the preparation area is full, it means the frame's data synchronization is complete. The data in the preparation area is migrated to the ready area, and the base timestamps pBaseTsn and rBaseTsn in both the preparation and ready areas are updated synchronously, setting rBaseTsn = pBaseTsn and pBaseTsn = 0.

[0061] b) If the absolute value of the first difference is greater than diff_min, it indicates that tsn is not close to pBaseTsn, corresponding to block b, block c, or block d. Further judgment is needed based on the first difference, including: If tsn is greater than pBaseTsn and the first difference is greater than diff_max, it indicates that tsn is far from pBaseTsn. Corresponding to block c in the figure, it is determined that the timestamp tsn of the image frame is relatively new, and the deviation between the image frame and pBaseTsn is also relatively large. It can be clearly stated that the image frame is not synchronized with the data in the preparation area, that is, it does not belong to the same frame of data, but is a new frame of data. Therefore, the preparation area migration is triggered, the synchronized data in the preparation area is migrated to the ready area, and the new frame of data is added to the preparation area. The base timestamps pBaseTsn and rBaseTsn in the preparation area and the ready area are updated synchronously, and rBaseTsn=pBaseTsn and pBaseTsn=0 are set.

[0062] When tsn is less than pBaseTsn and the absolute value of the first difference is greater than the preset multiple m for diff_max, corresponding to block d in the diagram, it indicates that the timestamp is very old and the error is very large. This excludes delayed frames and confirms a timestamp rollback fault. The timestamps of subsequent frames will revert to a past point in time. At this point, the current frame in the preparation area cannot be synchronized normally and is in an abnormal state. Therefore, this abnormal frame is discarded, the preparation area is cleared, and pBaseTsn is set to 0 to ensure the normal synchronization process of subsequent timestamp-rollback frames, minimizing the impact of timestamp rollback on the entire system. To account for timestamp rollback, the preset multiple m can be set to a large integer based on the actual application scenario to effectively distinguish between delayed frame arrival and timestamp rollback, ensuring that image frames experiencing timestamp rollback can be synchronized normally, thus minimizing the impact of timestamp rollback on the entire system.

[0063] If tsn is greater than pBaseTsn, and the first difference is greater than diff_min and less than or equal to diff_max, or if tsn is less than pBaseTsn, and the absolute value of the first difference is greater than diff_min and less than or equal to m*diff_max, it means that tsn is neither close to nor far from pBaseTsn. In this case, corresponding to block b, it is impossible to determine the ownership of the image frame. It may belong to the current frame, i.e., synchronized with the data in the preparation area, or it may belong to the next frame. Considering its uncertainty, the image frame is discarded because, for scenarios with multiple cameras, the impact of discarding a frame from one camera is much smaller than synchronizing an incorrect frame. In particular, a special case needs to be considered: if this frame happens to be synchronized with the ready area, it does not need to be discarded. Therefore, it is necessary to further determine the deviation between tsn and the reference timestamp rBaseTsn of the ready area as the second difference. If the absolute value of the second difference is less than diff_min, corresponding to region e, it indicates that the image frame is close to the ready area. This image frame has only arrived with a short delay and can still be synchronized with the data in the ready area. Therefore, the image frame is placed in the ready area. As can be seen, this example can effectively distinguish between the delayed arrival of the previous frame and other invalid frames, discard the invalid frames, and complete the synchronization of the delayed frames.

[0064] 4) Continuously read image frames from the frame queue and repeat step 3) until all image frames are synchronized.

[0065] In summary, this application embodiment, as a supplement and connection to hardware synchronization, can successfully complete frame synchronization even with minor timestamp errors (including hardware synchronization errors and link transmission errors). When hardware synchronization is incomplete, there will be a certain error between the frame timestamp and the actual image exposure time, affecting the frame synchronization determination. This application embodiment sets two error thresholds: a first error threshold and a second error threshold. If the difference between the timestamps of two frames is within the first error threshold, corresponding to block a, they are considered synchronized. If the error is large, corresponding to blocks c and d, they are considered definitely out of sync. If the difference is between the two, corresponding to block b, it cannot be determined, and in most cases, a discard operation will be performed. This effectively addresses the timestamp error problem caused by incomplete hardware synchronization. In addition to setting only one error threshold in related technologies, this application embodiment adds an error threshold, introducing a new situation during frame synchronization determination where the frame's affiliation cannot be determined. The frame might belong to the current time frame or another time frame. Unless the image frame happens to be synchronized with the ready area, it will be directly discarded. This is because inserting an incorrect frame can significantly impact the recognition result, and may even overwrite an existing correct frame. Therefore, this application embodiment addresses the potential problems arising from hardware synchronization at the software level, effectively optimizing the frame synchronization process.

[0066] This application's embodiments do not rely on the content features contained in the image frames themselves, and can adapt to various complex driving environments such as lighting, weather, and roads. In some complex driving environments, due to the influence of factors such as lighting, weather, and the surrounding environment, the effective information contained in the acquired images is very limited. The accuracy of synchronization based on image content feature matching is very low. However, this application's embodiments rely on timestamps to achieve frame synchronization, and do not depend on the content of the images themselves. Therefore, they can adapt well to various complex driving environments.

[0067] This application embodiment can effectively identify and handle most timestamp rollback situations caused by system time synchronization, and can reduce the impact of frame delay or frame loss. In implementation, this application embodiment uses a dual-buffer structure of a preparation area and a ready area. The preparation area buffers data being synchronized, while the ready area buffers data that has completed synchronization. When the preparation area is full or a new frame arrives, the synchronization of the current time frame ends, and the migration of the preparation area is triggered, moving the synchronized data to the ready area. Therefore, this application embodiment can simultaneously use the preparation area to synchronize the data of the current frame and use the ready area to supplement the synchronization of delayed frames, minimizing the drop rate of valid frames. Even if a frame arrives late or is lost, it will not affect the synchronization of subsequent frames, effectively handling timestamp rollback situations. Moreover, frame delay and loss do not block the entire synchronization process, thereby improving the real-time performance of image frame synchronization and meeting the real-time requirements of advanced driver assistance systems.

[0068] Compared to related technical solutions that equip each camera with a separate queue, which easily leads to space redundancy and waste, this application embodiment uses only one queue to simultaneously receive video streams from multiple cameras. Furthermore, in the frame synchronization module, this application embodiment uses only simple timestamp comparison logic, and only uses a preparation area and a ready area to store the frame being synchronized and the frame that has completed synchronization, while releasing other invalid frames. Therefore, this application embodiment has extremely low requirements for computing and storage resources, achieving a lightweight process. For example, in automotive embedded systems, computing and storage resources are limited, requiring a lightweight application.

[0069] The embodiments of this application exhibit good scalability and stability. Firstly, these embodiments are compatible with varying numbers of cameras, even more than ten. When the number of cameras changes, a single queue is still used for receiving and processing; only the sizes of the preparation and ready areas need to be adjusted. Simultaneously, a synchronized frame of data is always placed in the ready area. Even if downstream modules change, only the frame delivery format needs to be renegotiated, without concern for changes in reading time, frequency, or purpose. This means that when upstream and downstream modules change, these embodiments require only minor adjustments to be applicable, allowing for flexible adaptation to many different autonomous driving scenarios. They possess excellent scalability, and most failures of upstream and downstream modules will not affect the operation of image frame synchronization, demonstrating strong stability.

[0070] As an application example, the multi-camera image frame synchronization method proposed in this application is applied to a vehicle visual perception scenario, such as... Figure 6 As shown. For the visual perception module, camera data, vehicle positioning data, chassis data, and inertial measurement unit (IMU) data are used as inputs. Through the visual perception algorithm, the final output is information such as detected lanes, pedestrians, and obstacles. Among them, the camera data includes nine channels of driving camera data, including forward wide-angle data, forward narrow-angle data, right front blind spot data, right rear blind spot data, left front blind spot data, left rear blind spot data, rear view data, face data, and blind spot data displayed in the vehicle cockpit. The frame synchronization module involved in this application synchronizes the video stream data of the nine cameras, which is then read by the visual perception module to complete the entire visual perception process.

[0071] In the implementation of this solution, some technical details need to be determined based on the actual application scenario. Firstly, the values ​​of the first error threshold `diff_min` and the second error threshold `diff_max` need to be set, which are usually related to the error of the camera hardware synchronization. For example, if the hardware synchronization error is small, these two values ​​can be set relatively small. Based on experience, setting `diff_min` to 3-5 times the hardware error and `diff_max` to 6-10 times the hardware error is appropriate. Through measurement, the camera hardware error in this scenario is approximately within 2ms. In this case, setting `diff_min` and `diff_max` to 6ms and 15ms respectively yields a relatively good synchronization effect. Furthermore, when dealing with the timestamp rollback problem, 30 times `diff_max` is used as a comparison error to determine whether a timestamp rollback has occurred. This accurately identifies the timestamp rollback fault and avoids mistaking delayed frames for timestamp rollback frames, thus smoothly and quickly achieving multi-camera image frame synchronization in this scenario, thereby verifying the effectiveness and feasibility of the method in this application.

[0072] On the other hand, this embodiment provides a multi-camera image frame synchronization system, such as Figure 7 As shown, it includes: The data acquisition module 701 is used to acquire image frames from multiple cameras and store them in the same image frame queue, and to extract the target image frame sequentially from the image frame queue and read the timestamp of the target image frame. The timestamp difference determination module 702 is used to determine the first difference between the timestamp of the target image frame and the reference timestamp of the preset preparation area. The preparation area is used to cache the set of image frames being synchronized. The synchronization state determination module 703 is used to determine the synchronization state of the target image frame based on the relationship between the first difference and a preset first error threshold and a preset second error threshold; the first error threshold is used to determine the synchronization of the image frame, the second error threshold is used to distinguish different image frames, and the second error threshold is greater than the first error threshold. The synchronization execution module 704 is used to execute corresponding synchronization actions based on the synchronization state.

[0073] This embodiment also provides a vehicle, such as Figure 8 As shown, it includes a memory and a processor. In a specific example, the memory stores a computer program, which, when executed by the processor, implements the method of any of the above embodiments.

[0074] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps of any of the methods in the above embodiments.

[0075] In the embodiments of this application, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0076] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0077] The foregoing has provided a detailed description of a multi-camera image frame synchronization method, system, vehicle, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-camera image frame synchronization method, characterized in that, include: Image frames from multiple cameras are acquired and stored in the same image frame queue. Target image frames are then extracted sequentially from the image frame queue, and the timestamps of the target image frames are read. Determine a first difference between the timestamp of the target image frame and the reference timestamp of a preset preparation area, wherein the preparation area is used to cache a set of image frames being synchronized; Based on the relationship between the first difference and the preset first error threshold and the preset second error threshold, the synchronization status of the target image frame is determined; The first error threshold is used to determine image frame synchronization, the second error threshold is used to distinguish different image frames, and the second error threshold is greater than the first error threshold. Based on the synchronization state, perform the corresponding synchronization action.

2. The multi-camera image frame synchronization method of claim 1, wherein, Based on the relationship between the first difference and preset first and second error thresholds, the synchronization state of the target image frame is determined, including: If the absolute value of the first difference is less than or equal to the first error threshold, it is determined that the target image frame is synchronized with the image frame in the preparation area; If the first difference is positive and greater than the second error threshold, or if the first difference is negative and its absolute value is greater than a preset multiple of the second error threshold, it is determined that the target image frame is out of sync with the image frames in the preparation area. If the first difference is positive and greater than the first error threshold and less than or equal to the second error threshold, or if the first difference is negative and its absolute value is greater than the first error threshold and less than or equal to a preset multiple of the second error threshold, the synchronization state of the target image frame is determined to be uncertain.

3. The multi-camera image frame synchronization method of claim 2, wherein, The execution of corresponding synchronization actions based on the synchronization state includes: If it is determined that the target image frame is synchronized with the image frame in the preparation area, the target image frame is placed into the preparation area; If it is determined that the target image frame is out of sync with the image frames in the preparation area, and the timestamp of the target image frame is greater than the reference timestamp of the preparation area, the operation of migrating the set of image frames in the preparation area to a preset ready area is performed, and the target image frame is placed into the preparation area. The ready area is used to cache the set of image frames that have been synchronized.

4. The multi-camera image frame synchronization method of claim 2, wherein, The step of performing the corresponding synchronization action based on the synchronization state further includes: If it is determined that the target image frame is out of sync with the image frames in the preparation area, the timestamp of the target image frame is less than the reference timestamp of the preparation area, and the absolute value of the first difference is greater than a preset multiple of the second error threshold, the reference timestamp of the preparation area is determined to be abnormal, and the operation of clearing the preparation area is performed.

5. The multi-camera image frame synchronization method of claim 3, wherein, The step of performing the corresponding synchronization action based on the synchronization state further includes: If the synchronization state of the target image frame is uncertain, calculate a second difference between the timestamp of the target image frame and the reference timestamp of the ready area; If the absolute value of the second difference is less than or equal to the first error threshold, the target image frame is placed into the ready area; If the absolute value of the second difference is greater than the first error threshold, the target image frame is discarded.

6. The multi-camera image frame synchronization method of claim 3, wherein, The step of migrating the set of image frames in the preparation area to the ready area includes: Clear the ready area and move all image frames in the preparation area to the ready area; Assign the reference timestamp value of the preparation area to the reference timestamp of the ready area; And, clear the preparation area.

7. The multi-camera image frame synchronization method of any of claims 1-6, wherein, Also includes: In the initial state, the reference timestamp of the preparation area is set to an invalid value; If the preparation area is empty, the timestamp of the first image frame placed in the preparation area is set as the reference timestamp of the preparation area.

8. A multi-camera image frame synchronization system, comprising: include: The data acquisition module is used to acquire image frames from multiple cameras and store them in the same image frame queue, and to extract target image frames sequentially from the image frame queue and read the timestamps of the target image frames. The timestamp difference determination module is used to determine the first difference between the timestamp of the target image frame and the reference timestamp of the preset preparation area, wherein the preparation area is used to cache the set of image frames being synchronized. The synchronization state determination module is used to determine the synchronization state of the target image frame based on the relationship between the first difference and a preset first error threshold and a preset second error threshold. The first error threshold is used to determine image frame synchronization, the second error threshold is used to distinguish different image frames, and the second error threshold is greater than the first error threshold. The synchronization execution module is used to perform corresponding synchronization actions based on the synchronization state.

9. A vehicle characterized by comprising: It includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the method according to any one of claims 1-7.

Images