Surround view video real-time stitching system and method for vehicle situation awareness

Abstract

The present application relates to a kind of omnidirectional video real-time splicing system and method for vehicle situation awareness, belong to vehicle video processing technical field, solve the problems such as the poor time consistency of multiple video, the big load fluctuation of splicing processing of omnidirectional video splicing.Therein, the system includes: multiple video acquisition module, for collecting and processing the video stream of vehicle multi-direction, generates omnidirectional video stream by each video stream with frame level time mark and frame serial number;Vehicle communication network, for transmitting omnidirectional video stream to video receiving and alignment module;Video receiving and alignment module, for parsing received omnidirectional video stream, according to frame level time mark and frame serial number, each video stream is sorted, synchronous cache, and the multiple groups of synchronous frames of current scheduling period are aligned and output;Static splicing processing module, for receiving the multiple groups of synchronous frames of current scheduling period, for each group of synchronous frames, execute geometric mapping and splicing area fusion processing, generate omnidirectional splicing image.

Description

Technical Field

[0001] This invention relates to the field of vehicle video processing technology, and in particular to a real-time panoramic video stitching system and method for vehicle situational awareness. Background Technology

[0002] With the continuous improvement of resolution and frame rate of vehicle-mounted sensors, panoramic video systems exhibit typical characteristics of multi-channel, high bandwidth, and strong real-time performance. During the acquisition, transmission, and processing of multiple video streams, latency and jitter are inevitably introduced. If video frames from different perspectives cannot maintain temporal consistency, it will directly lead to problems such as splicing misalignment, image tearing, and display jitter, thereby weakening situational awareness and even affecting operator judgment. In practical applications, the continuity, stability, and predictability exhibited by panoramic video systems place more stringent demands on end-to-end real-time performance and system reliability, rather than pursuing top-tier image quality or complex visual effects.

[0003] Existing video stitching research largely focuses on image processing and computer vision algorithms, improving stitching quality through feature matching, dynamic stitching seams, or adaptive fusion. However, these methods typically rely on complex computational processes, are highly sensitive to scene changes, and exhibit significant fluctuations in computational load and processing latency with environmental variations, making long-term stable operation on resource-constrained automotive platforms difficult. Furthermore, some studies assume that multiple video streams are naturally synchronized in time when designing stitching algorithms, ignoring the temporal uncertainties introduced by video transmission and scheduling in real-world systems, leading to a significant gap between algorithm performance and engineering implementation.

[0004] Therefore, the problem of panoramic video stitching for vehicle situational awareness cannot be studied in isolation from the perspective of algorithm optimization alone. Instead, it should be considered holistically at the system level, taking into account the common constraints of temporal consistency and real-time performance on all stages of video acquisition, transmission, and processing. Exploring a stable, controllable, and easily deployable panoramic video stitching method while meeting the requirements of the in-vehicle engineering environment and mission is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a real-time panoramic video stitching system and method for vehicle situational awareness, in order to solve the problems of poor time consistency of multiple videos and large fluctuations in stitching processing load in existing panoramic video stitching methods.

[0006] On one hand, this invention provides a real-time panoramic video stitching system for vehicle situational awareness, the system comprising a multi-channel video acquisition module, an in-vehicle communication network, a video receiving and alignment module, and a static stitching processing module; wherein, The multi-channel video acquisition module is used to acquire and process video streams from multiple directions of the vehicle, generating a panoramic video stream composed of various video streams with frame-level time stamps and frame sequence numbers. The vehicle-mounted communication network is used to transmit panoramic video streams to the video receiving and alignment module according to transmission priority, transmission period and scheduling window. The video receiving and alignment module is used to parse the received panoramic video stream, sort and synchronize each video stream according to the frame-level time stamp and frame sequence number, and align and output multiple sets of synchronized frames in the current scheduling cycle. The static stitching processing module is used to receive multiple sets of synchronization frames in the current scheduling period, perform geometric mapping and stitching region fusion processing on each set of synchronization frames, and generate a panoramic stitched image.

[0007] Based on the above solution, the present invention also makes the following improvements: the system further includes a display output module; The display output module is used to output the panoramic stitched image at a fixed refresh rate.

[0008] Furthermore, the video receiving and alignment module includes a synchronization buffer output unit; wherein, The synchronous buffer output unit is used to determine whether the time alignment condition is met based on the frame-level time stamp of each video frame with the same frame number in the current scheduling cycle. If the condition is met, the multiple video frames with the same frame number are output as a group of synchronous frames.

[0009] Furthermore, the time alignment condition is implemented according to the following logic in the frame arrival buffering mechanism: For the same frame sequence number, if a certain video frame arrives before other video frames, the video frame is buffered at a limited depth; after all the other video frames have arrived, it is checked whether the time deviation corresponding to the frame-level time stamp of the video frame with the corresponding frame sequence number in each video stream is within the allowable range. If so, output multiple video frames corresponding to the same frame number as a group of synchronized frames.

[0010] Furthermore, the time alignment condition also executes a timeout exception handling mechanism according to the following logic: If a video frame with the same frame number fails to arrive after a preset waiting time, the system will trigger an exception handling mechanism.

[0011] Furthermore, the static stitching processing module includes an image mapping unit and a region division unit; wherein, The image mapping unit is used to perform coordinate transformation on the corresponding video frames in each group of synchronization frames according to the geometric mapping parameters of each camera component, and map them to the corresponding stitching area of ​​the unified reference plane. The region division unit is used to determine the affiliation of each spliced ​​region in the unified reference plane.

[0012] Furthermore, the static stitching processing module also includes an image fusion generation unit, which is used to perform image fusion based on fixed stitching seams on each group of synchronized frames mapped to a unified reference plane to generate a corresponding panoramic stitched image.

[0013] Furthermore, in the image fusion generation unit, image fusion based on fixed stitching seams is performed as follows: For non-overlapping regions, the image fusion generation unit directly uses the corresponding pixels of the video source specified by the region division unit; For overlapping areas, the image fusion generation unit determines the pixel source according to the fixed stitching seam rule.

[0014] On the other hand, the present invention also provides a real-time panoramic video stitching method for vehicle situational awareness, the method comprising: The system acquires and processes video streams from multiple directions of the vehicle, generating a panoramic video stream composed of various video streams with frame-level time stamps and frame sequence numbers; and transmits the panoramic video stream according to transmission priority, transmission period, and scheduling window. The received panoramic video stream is parsed, and each video stream is sorted and synchronized according to the frame-level time stamp and frame sequence number. Multiple sets of synchronized frames in the current scheduling period are aligned and output. Receive multiple sets of synchronization frames in the current scheduling period, perform geometric mapping and splicing region fusion processing on each set of synchronization frames, and generate a panoramic spliced ​​image.

[0015] Furthermore, the method also includes: The panoramic stitched image is output at a fixed refresh rate.

[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: This invention relates to a real-time panoramic video stitching system for vehicle situational awareness. The system addresses the comprehensive requirements of panoramic video stitching in vehicle situational awareness, considering real-time performance, stability, and engineering deployability. First, from an end-to-end perspective, it constrains and models frame-level time alignment errors, end-to-end latency limits, and computational load stability in the stitching process, thereby constructing a timing and traffic characteristic model of multiple in-vehicle video streams. Next, in the communication section, a deterministic transmission and scheduling mechanism, represented by Time-Sensitive Networking (TSN), is introduced to limit the transmission latency and jitter of multiple video streams, providing a controllable timing basis for frame-level alignment at the receiving end. In the processing stage, a static stitching method with fixed geometric mapping and fixed stitching seams is adopted to prevent computational fluctuations and structural abrupt changes caused by dynamic content-aware optimization, thus achieving stable and predictable real-time panoramic video stitching display.

[0017] This invention also proposes a real-time panoramic video stitching method for vehicle situational awareness. This method addresses the stringent requirements of frame-level timing consistency, computational load controllability, and display stability in the multi-channel vehicle video stitching process, meeting the real-time display needs of panoramic situational awareness in complex maneuvers and environments. Based on the stability requirements of the continuous operation of the panoramic video stitching system, this method constructs frame-level time alignment constraints, end-to-end latency limits, and traffic characteristic models for multiple video streams. Based on this, a periodic scheduling mechanism using a time-sensitive network is introduced to provide deterministic transmission and jitter control support for the vehicle video stream. In terms of processing, a static stitching strategy combining fixed geometric mapping and fixed stitching seams is employed to avoid computational load fluctuations and stitching structure jumps caused by dynamic content-aware optimization, achieving consistent stitching of multiple video frames in both time and space dimensions. Experimental results show that the proposed scheme can stably control the end-to-end latency within 82.7ms, while the maximum deviation of the arrival time of multiple video frames does not exceed 4.6ms. The collaborative design of this system achieves stable and predictable real-time stitching of panoramic video while maintaining low computational complexity. It has good engineering deployability and is suitable for vehicle situational awareness systems with high requirements for reliability and long-term stable operation.

[0018] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0019] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is the overall architecture diagram of the real-time panoramic video stitching system for vehicle situational awareness provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the TSN periodic scheduling configuration algorithm driven by splicing requirements provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the demand-constrained deterministic transmission and static splicing process provided in Embodiment 1 of the present invention; Figure 4 This is a flowchart of a real-time panoramic video stitching method for vehicle situational awareness provided in Embodiment 2 of the present invention. Detailed Implementation

[0020] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0021] Specific embodiment 1 of the present invention discloses a real-time panoramic video stitching system for vehicle situational awareness. The overall architecture diagram of the system is shown below. Figure 1 As shown, the system includes a multi-channel video acquisition module, a vehicle-mounted communication network, a video receiving and alignment module, a static stitching processing module, and a display output module. The panoramic video real-time stitching system provided in this embodiment operates as an end-to-end processing flow with the video frame period as its basic rhythm. Based on the system architecture, this embodiment focuses on the execution order, timing constraints, and the connection relationships between each processing stage during system operation, thereby clearly depicting the real-time implementation mechanism of the stitching system. The specific implementation method is described below.

[0022] (1) Multi-channel video acquisition module The multi-channel video acquisition module is used to acquire and process video streams from multiple directions of the vehicle, generating a panoramic video stream composed of various video streams with frame-level time stamps and frame sequence numbers.

[0023] In this embodiment, the multi-channel video acquisition module includes a multi-camera assembly, a time stamping unit, a frame numbering unit, and a data encapsulation unit. Detailed explanations follow.

[0024] 1) Multi-camera assembly Each camera module is used to capture video streams from its corresponding direction.

[0025] In practice, each camera module is installed at different locations on the vehicle, covering the front, rear, left, right, and other areas to be monitored. The multiple camera modules operate at a fixed frame rate, with each module's frame rate set to the same or approximately the same. To facilitate subsequent frame-level alignment, it is preferable that each camera module triggers acquisition or completes time stamping based on a unified time reference.

[0026] 2) Time stamp unit The time stamping unit is used to perform frame-level time stamping on each video stream based on a unified time reference.

[0027] That is, in this embodiment, the time stamp unit is used to write acquisition time information for each video frame in each video stream based on a unified time reference.

[0028] 3) Frame numbering unit The frame numbering unit is used to assign a unique frame number to each video frame of each video stream. 4) Data encapsulation unit The data encapsulation unit is used to encapsulate all video frames in the video streams and their corresponding frame-level time stamps and frame numbers into a panoramic video stream.

[0029] In this embodiment, the multi-channel video acquisition module periodically acquires a panoramic video stream of the vehicle through multiple cameras. During system operation, each camera periodically acquires video data (video stream) at a fixed frame rate and adds a time stamp to each frame of video based on a unified time reference. The acquired video frames are then encapsulated into a periodic panoramic video stream and sent to the vehicular communication network according to a pre-configured scheduling strategy. Because the panoramic video stream has undergone traffic modeling and access configuration at the communication layer, its network transmission behavior is predictable in the time dimension.

[0030] (2) Vehicle-mounted communication network The vehicle-mounted communication network is used to transmit panoramic video streams to the video receiving and alignment module according to transmission priority, transmission period, and scheduling window.

[0031] In this embodiment, the vehicular communication network is a key infrastructure connecting the multi-channel video acquisition module and the video receiving and alignment module. Its core function is to provide a stable and predictable transmission environment for the panoramic video stream. Since panoramic video stitching has strict requirements for frame-level timing consistency and end-to-end latency upper bounds, the vehicular communication network must effectively constrain the transmission latency and jitter of the multi-channel video data streams to avoid introducing additional timing deviations due to transmission uncertainties. Therefore, this embodiment introduces a vehicular communication network with deterministic transmission capabilities at the communication layer, laying a stable timing foundation for subsequent stitching processing. This network transmits the panoramic video stream to the video receiving and alignment module by controlling transmission priority, transmission period, and scheduling window, ensuring the controllability of latency and jitter during transmission.

[0032] In this embodiment, the vehicle-mounted communication network includes a time synchronization unit, a service classification unit, a queue mapping unit, a periodic scheduling unit, and a switching and forwarding unit. Detailed explanations follow.

[0033] 1) Time synchronization unit The time synchronization unit provides a unified clock reference for the multi-channel video acquisition module and the video receiving and alignment module.

[0034] 2) Business Classification Unit The service classification unit is used to mark panoramic video streams as real-time transmission services.

[0035] 3) Queue Mapping Unit The queue mapping unit is used to map each video stream in the panoramic video stream to a specified send queue.

[0036] 4) Periodic scheduling unit The periodic scheduling unit is used to allocate a defined transmission window to each video stream in the panoramic video stream within each scheduling period according to the preset transmission priority and transmission period, so as to perform periodic scheduling.

[0037] In this embodiment, the queue mapping unit determines the transmission queue to which the video stream belongs, and the periodic scheduling unit determines the open time interval of that queue within the scheduling period. The periodic scheduling unit pre-allocates independent transmission windows for each video stream in the panoramic video stream, allowing multiple video streams to be transmitted within their respective windows, thereby reducing competition with other services. Preferably, the transmission period is consistent with or an integer multiple of the video frame period, thus ensuring a unified rhythm for the acquisition, transmission, and reception stages. Based on the pre-configured scheduling period, queue mapping relationship, and time window parameters, the periodic scheduling unit controls the transmission of each video stream within its corresponding time slot, effectively reducing competition and transmission jitter among multiple video streams in the network.

[0038] In the TSN environment of this embodiment, periodic services are typically scheduled using a time-aware shaping mechanism. The core idea is to allocate a fixed transmission window to critical service flows within each scheduling period. Based on this mechanism, the periodic scheduling unit can satisfy the TSN scheduling constraints driven by splicing requirements.

[0039] To address the transmission requirements of multiple periodic video streams in a panoramic video stitching system, the system scheduling cycle is set to be... , No. The transmission window allocated to the video stream in the current period is .

[0040] To ensure that each video stream completes the transmission of a single frame of data within one scheduling cycle and to avoid scheduling conflicts, the following TSN scheduling constraints (scheduling feasibility constraints) must be satisfied: (1) in, , They represent the first The start and end times of the corresponding transmission window for each video stream within the current scheduling period; Indicates the first The effective transmission duration allocated to each video stream within the current scheduling period; Represented as the first The guard band reserved for the video stream is used to offset the uncertainties caused by factors such as link switching, scheduling switching, forwarding timing deviation, and clock synchronization error; This indicates the scheduling period set by the system; This indicates the number of all video streams participating in the periodic transmission scheduling.

[0041] The above formula gives the basic feasibility conditions for the scheduling configuration on the TSN side. Whether these conditions are met directly determines the feasibility of the periodic sending window allocation step in the algorithm.

[0042] In addition, this embodiment also sets the following communication-side jitter upper bound and frame-level alignment conditions: Under the premise of adopting a periodic scheduling mechanism, the main uncertainty introduced by multiple video streams in the communication process is reflected in the residual deviation of frame arrival time.

[0043] Let the maximum frame arrival error introduced by the communication system be... In a TSN scheduling environment, this deviation can be defined by both the scheduling protection interval and the time synchronization error: (2) in, This represents the residual synchronization error caused by the system's time synchronization mechanism. To ensure that the splicing processing end can complete frame-level time alignment, the following further requirements need to be met: (3) Based on this, the receiving end can absorb residual minor jitter during communication by introducing an alignment buffer of finite depth. Let the alignment buffer depth be... (Based on frame count), the project configuration must meet the following requirements: (4) The above formula provides a direct basis for the design of the receiver buffer size, enabling frame-level alignment requirements to be met without significantly increasing end-to-end latency.

[0044] Based on the aforementioned constraint modeling, to transform the timing requirements of the stitching system into deployable TSN scheduling parameters, a stitching requirement-driven TSN periodic scheduling configuration algorithm is further presented. This algorithm takes video stream parameters and stitching alignment requirements as input, checks the feasibility of the scheduling window, and generates a periodic transmission window configuration that satisfies the constraints. A schematic diagram of the stitching requirement-driven TSN periodic scheduling configuration algorithm is shown below. Figure 2 As shown in the diagram. Through the above algorithm, the system can complete the feasibility check and configuration generation of TSN scheduling parameters during the deployment phase, thereby ensuring that the communication side can continuously meet the timing consistency and real-time requirements of the splicing process during the operation phase.

[0045] 5) Switching and forwarding unit The switching and forwarding unit is used to transmit according to preset transmission priorities, transmission periods, and scheduling windows. The network nodes of the vehicle-mounted communication network perform deterministic forwarding of each video stream in the panoramic video stream, so as to transmit it to the video receiving and alignment module.

[0046] (3) Video receiving and alignment module The video receiving and alignment module is used to parse the received panoramic video stream, sort and synchronize each video stream according to the frame-level time stamp and frame sequence number, and align and output multiple sets of synchronized frames in the current scheduling period.

[0047] The video receiving and alignment module includes a data receiving interface, a parsing and recognition unit, a sorting unit, and a synchronous buffer output unit. Details are as follows.

[0048] 1) Data receiving interface The data receiving interface is used to receive various video streams from the vehicle communication network.

[0049] 2) Analysis and Recognition Unit The parsing and recognition unit is used to parse the source identifier, frame-level time stamp, and frame sequence number corresponding to each video frame in each video stream.

[0050] 3) Sorting unit The sorting unit is used to sort the video frames in each video stream parsed by the parsing and recognition unit according to the frame number and frame-level time stamp order.

[0051] 4) Synchronous buffer output unit The synchronous buffer output unit is used to determine whether the time alignment condition is met based on the frame-level time stamp of each video frame with the same frame number in the current scheduling cycle. If the condition is met, the multiple video frames with the same frame number are output as a group of synchronous frames.

[0052] In this embodiment, the time alignment condition is executed according to the following logic: a) Frame arrival buffering mechanism: For the same frame sequence number, if a certain video frame arrives before other video frames, the video frame is buffered at a limited depth; after all the other video frames arrive, it is checked whether the time deviation corresponding to the frame-level time stamp of the video frame with the corresponding frame sequence number in each video stream is within the allowable range; if so, the multiple video frames corresponding to the same frame sequence number are output as a group of synchronous frames.

[0053] b) Timeout exception handling mechanism: For the same frame sequence number, if a certain video frame has not arrived after the preset waiting time, the system triggers the exception handling mechanism to ensure that the overall output rhythm is not interrupted.

[0054] In this embodiment, the exception handling mechanism may include at least one of the following methods: 1) calling the corresponding video frame from the previous cycle to participate in the stitching; 2) calling a preset default image to fill the corresponding area; 3) recording the exception status and continuing to execute the output.

[0055] It is important to note that the following key judgment logic should be followed during the time alignment condition determination process of the synchronous buffer output unit: Early / Late Arrival Criteria: The criteria for determining "early / late arrival" is the order of arrival time of the current frame relative to the target alignment time or the arrival time of the reference frame.

[0056] Handling within the deviation range: If the late arrival magnitude is still within the maximum frame-level time deviation allowable range, then short-term waiting and alignment absorption are performed through a finite-depth buffer.

[0057] Handling outside the deviation range: If the maximum frame-level time deviation has been exceeded, it is determined to be a timeout frame, and the output stability of the current cycle is maintained by using the previous cycle buffered frame, the default compensation frame, or the region preservation method.

[0058] It should be noted that in this embodiment, the maximum frame-level time deviation is only responsible for determining "whether it can still participate in the same stitching cycle"; the determination of "early / late arrival" does not rely on the threshold itself, but is based on "the order of the arrival time of the current frame relative to the target alignment time or the arrival time of the reference frame". "Late arrival" does not necessarily mean exceeding the maximum frame-level time deviation, but rather that a certain video frame arrives later than the reference alignment time. If the late arrival is still within the allowable range of the maximum frame-level time deviation, it can be temporarily waited for and aligned by a finite-depth buffer; if it has exceeded the maximum frame-level time deviation, it means that the video frame no longer meets the frame-level time alignment constraint of the current stitching cycle. At this time, it is no longer necessary to rely on the buffer for forced compensation, but is determined to be a timeout frame, and the output stability of the current cycle is maintained by using the buffered frame of the previous cycle, the default compensation frame, or the region preservation method.

[0059] Therefore, in this embodiment, to solve the frame-level time mismatch problem caused by slight early arrival, slight late arrival, and occasional timeouts of multiple video frames during network transmission, an overall processing mechanism for the above-mentioned frame-level time mismatch problem is adopted, which mainly involves the following technologies: Time stamping and periodic grouping: Time stamping and periodic grouping of video frames from various channels are performed based on a unified time reference; Minor jitter absorption: For video frames that are earlier or later than the reference alignment time but are still within the maximum frame-level time deviation range, a short wait is performed through a finite-depth buffer to absorb minor jitter in the transmission link; Timeout frame handling: For video frames that exceed the maximum frame-level time deviation, they will no longer be included in the current stitching cycle. Instead, the current stitching output will be completed by calling the cached frames of the previous cycle, the default compensation frames, or the region preservation results.

[0060] Through the above mechanism, while ensuring frame-level time alignment constraints, excessive end-to-end latency is avoided.

[0061] In practical implementation, after each video frame arrives, the video receiving and alignment module first enters the receiving and buffering stage based on its timestamp and frame number. The video receiving and alignment module sorts and aligns video frames from different cameras to determine whether the videos with the same frame number meet the frame-level time alignment constraints: when the arrival time deviation between video frames is within the allowable range, the system directly enters the subsequent stitching stage; when there are slight early or late arrivals, the system uses a limited-depth buffer to wait briefly to absorb residual minor jitter from the communication transmission process. The buffer depth is set to meet the frame-level time alignment constraints while minimizing the introduction of additional end-to-end latency.

[0062] In this embodiment, the video receiving and alignment module sorts and aligns data from different cameras according to the timestamps of the video frames, and performs necessary buffer management while meeting timing constraints. Through reasonable design of buffer depth and alignment strategy, residual minor jitter during transmission can be absorbed without significantly increasing system latency, providing time-consistent video frame input for the static stitching processing module.

[0063] (4) Static splicing processing module The static stitching processing module is used to receive multiple sets of synchronization frames in the current scheduling period, perform geometric mapping and stitching region fusion processing on each set of synchronization frames, and generate a panoramic stitched image.

[0064] In this embodiment, the static stitching processing module is responsible for spatial mapping and region fusion of time-aligned multi-channel video frames to generate a panoramic stitching result from a unified perspective. As the core functional module of the system, its core task is to complete the spatial registration and fusion display of multi-channel video while ensuring real-time performance. Considering the limited computing resources of the vehicle platform and the requirements for engineering reliability, this embodiment adopts a static stitching strategy that combines fixed geometric mapping and fixed stitching seams—the relevant geometric relationships and stitching parameters are determined through calibration and offline calculation during the system deployment phase and remain unchanged during operation, thereby avoiding the computational load fluctuations and system instability problems that may be caused by dynamic stitching algorithms. A schematic diagram of the demand-constrained deterministic transmission and static stitching process is shown below. Figure 3 As shown.

[0065] The static stitching processing module includes an image mapping unit, a region segmentation unit, and a fusion generation unit. The specific implementation is described below.

[0066] 1) Image mapping unit The image mapping unit is used to perform coordinate transformation on the corresponding video frames in each group of synchronization frames according to the geometric mapping parameters of each camera component, and map them to the corresponding stitching area of ​​the unified reference plane.

[0067] In this embodiment, the geometric mapping parameters corresponding to each camera component are pre-acquired and stored through offline calibration. During system operation, the image mapping unit directly calls the pre-stored mapping parameters to perform coordinate transformation operations on each frame of input video data, without needing to recalculate the mapping relationship during real-time processing, thereby effectively reducing computational overhead and ensuring the real-time performance of stitching processing.

[0068] 2) Regional division units The region division unit is used to determine the affiliation of each spliced ​​region in the unified reference plane.

[0069] Specifically, the region division unit divides the output image into multiple stitching regions according to the pre-designed reference plane layout, and clarifies the mapping relationship between each region and the corresponding video source, providing region attribution information for the subsequent image fusion generation unit.

[0070] Preferably, the image mapping unit and the region division unit work together: the image mapping unit is responsible for transforming each video frame from its own camera coordinate system to a unified reference plane coordinate system; the region division unit is responsible for defining the boundaries and affiliation of each stitching region in the unified reference plane, and the two work together to complete the spatial mapping from the original video frame to the stitching region.

[0071] 3) Image fusion generation unit The image fusion generation unit is used to perform image fusion based on fixed stitching seams on each video frame mapped to a unified reference plane for each group of synchronized frames, and generate the corresponding panoramic stitched image.

[0072] In this embodiment, the image fusion generation unit adopts a fusion strategy based on fixed stitching seams. The fixed stitching seam rule is defined as follows: within the same output area, the pixel source remains unchanged during system operation, or switches only according to a limited number of pre-set modes, without adaptively searching based on real-time image content.

[0073] Specifically, for non-overlapping areas, the image fusion generation unit directly uses the corresponding pixels of the video source specified by the region division unit; for overlapping areas, the image fusion generation unit determines the pixel source according to the fixed stitching seam rules, and the position of the stitching seam remains fixed during system operation, avoiding dynamic searching of stitching boundaries based on scene content during the operation phase.

[0074] By adopting a fixed stitching seam strategy, this embodiment significantly reduces the computational complexity during the runtime phase while ensuring the visual continuity of the stitched images, thereby improving the real-time performance and stability of the system.

[0075] In the static stitching processing module, the region attribution information output by the region division unit serves as the basis for the image fusion generation unit to determine the pixel source; the coordinate-transformed video frames output by the image mapping unit serve as the input data for the image fusion generation unit. The three units form a pipelined processing architecture of "mapping-division-fusion," with each unit having independent functions and clear interfaces, facilitating modular implementation and system expansion.

[0076] Specifically, this embodiment designs the following real-time video stitching method. This method mainly involves two parts: static geometric mapping and stitching structure design, fixed stitching seam strategy, and display stability analysis, as detailed below.

[0077] (1) Static geometric mapping and splicing structure design Spatial consistency in video stitching is fundamental to stable display, and it is primarily determined by the installation position and orientation of the cameras. In automotive applications, these geometric relationships remain unchanged during operation. Therefore, the spatial mapping relationship between each video stream and the reference plane can be pre-determined during deployment through offline calibration and modeling, and static geometric mapping can be used to complete the stitching during operation.

[0078] Let the first The imaging plane coordinates corresponding to the road camera are The coordinates of its corresponding stitching reference plane (such as the vehicle's panoramic view plane or virtual display plane) are as follows: Neglecting large-scale nonlinear distortion, the mapping relationship between the two can be approximated as a two-dimensional homography transformation: (5) in, Indicates the first Homography matrix from the road camera to the reference plane, symbol It represents the equivalence relation in the sense of homogeneous coordinates.

[0079] Homography matrix The parameters are jointly determined by the camera's intrinsic and extrinsic parameter matrices and the definition of the reference plane. During system deployment, the geometric parameters of each camera can be obtained through calibration, and the corresponding parameters can be calculated offline based on the vehicle structure and panoramic display requirements. During operation, This keeps the process unchanged, thus avoiding the computational load and uncertainty associated with online estimation or dynamic updates.

[0080] Based on the aforementioned geometric mapping relationship, multiple video streams can be uniformly mapped to the same reference plane. On this plane, the areas covered by different cameras overlap to some extent. Therefore, a stitching region segmentation strategy is introduced, dividing the reference plane into several sub-regions, each sub-region receiving pixel data only from a designated video source. Let the pixels on the reference plane be... Belongs to the region Then its pixel value is determined by the first... Road video confirmed: (6) in, , They represent the first The input video and the pixel value function of the planar stitching result.

[0081] By analyzing the region With proper design, a complete and continuous panoramic display structure can be formed on the reference plane. Since the area division is determined during the system design phase, there is no need for dynamic area adjustment or seam searching during the splicing process, thus ensuring the stability of the spliced ​​structure over time.

[0082] (2) Fixed splice seam strategy and display stability analysis After completing the geometric mapping of multiple video streams to a unified reference plane, the stitching system still needs to address the issue of selecting the pixel source within the overlapping region. This problem is typically solved using a seam, which defines a boundary within the overlapping region so that pixels on either side of the boundary originate from different video sources. The selection of the seam directly affects the visual continuity of the stitched result and the stability of the system operation.

[0083] Existing research commonly employs stitching seam strategies, including dynamic stitching seam search methods based on graph cut, energy minimization, or content awareness. These methods typically adjust the stitching seam position dynamically within each frame or across frames based on information such as image content, gradient changes, or texture differences, to reduce visual discontinuities in the seam area. However, dynamic stitching seam methods generally rely on complex optimization processes, resulting in high computational complexity and sensitivity to changes in image content. In real-time automotive systems, this can easily introduce problems such as fluctuating computational load and display instability.

[0084] In engineering applications such as vehicle situational awareness, the system prioritizes the continuity, stability, and predictability of display behavior over pursuing locally optimal visual effects. Based on these considerations, this embodiment employs a fixed stitching seam strategy to process overlapping areas, building upon static geometric mapping. Specifically, during the system design phase, the stitching boundary within the overlapping area is pre-determined based on the field-of-view coverage of each camera and the mapping results on the reference plane, and the position of this stitching seam remains unchanged during system operation.

[0085] Let the overlapping region on the reference plane be ,in, , They represent the first Road, First The area covered by the video feed. A fixed seam can be represented as a predefined boundary curve. The overlapping region is divided into two sub-regions: (7) in, , Corresponding to the first Road, First The video feed provides a region of pixels.

[0086] During the stitching process, for any pixel located within the overlapping area The source of its pixels is uniquely determined by a fixed stitching seam strategy: (8) Due to the seam Maintaining consistency over time, there are no jumps in the stitching boundary positions during the multi-frame video stitching process, thus avoiding the time inconsistency issues introduced by dynamic stitching seams. This feature is particularly important for the continuous display of panoramic video, effectively suppressing visual flicker and jitter caused by changes in the stitching structure.

[0087] (3) Calculate load stability From the perspective of system stability, the fixed splice seam strategy also has significant computational advantages. Let the dynamic splice seam method be... The computational complexity of the frame is Its value fluctuates with changes in image content; while the computational complexity of the fixed seam strategy can be approximated as a constant. ,Right now: (9) and It does not change over time. Therefore, the fixed seam strategy helps maintain the stability of the computational load of the static stitching processing module, ensuring the system's real-time performance and predictability.

[0088] After frame-level time alignment is completed, multiple video frames are simultaneously fed into the static stitching processing module. The static stitching module maps each video frame to a unified reference plane according to a defined static geometric mapping relationship. Then, based on a predefined fixed stitching seam strategy, it determines the source of pixels within the overlapping area, generating a complete panoramic stitching result. Since the geometric mapping parameters and stitching structure remain unchanged during system operation, the execution path and computational complexity of the stitching processing are consistent in the time dimension. The improvement of the above technical solution lies in changing the runtime panoramic stitching from dynamic content-driven processing to deterministic processing based on static geometric mapping and fixed stitching seams. This makes the execution path and computational load of each frame stitching essentially constant, thus making it more suitable for real-time stable output in automotive scenarios. Furthermore, it forms the methodology for the entire system, not just the stitching step. This solution can be fully implemented in hardware. If only the stitching effect is considered, without considering real-time performance and stability, then AI + dynamic stitching is definitely better.

[0089] Furthermore, considering the limited computing resources of the vehicle platform, the computational load of the static stitching processing module should remain stable to avoid significant changes with video content or temporal fluctuations. Therefore, this embodiment adopts a static stitching strategy combining fixed geometric mapping and fixed stitching seams, which can keep the computational complexity of a unit video frame approximately constant. Frame splicing computation for: (10) in, The fixed computational load determined during the system design phase.

[0090] It is important to note that the core prerequisite for real-time panoramic video stitching is that multiple video frames must meet the requirements of temporal and spatial consistency. Spatial consistency can be guaranteed through offline calibration and geometric mapping, while temporal consistency is directly affected by multiple stages such as video acquisition, transmission, and processing. If there are deviations in frame-level timing among multiple video streams, it will lead to problems such as misalignment, tearing, or display jitter in the stitched area. Therefore, it is necessary to formally describe the timing constraints in the panoramic video stitching process.

[0091] The multi-channel video acquisition module is deployed in a total of Street camera, number The first video stream captured by the road camera Video frames are denoted as The corresponding video frame capture time is ,in, , Indicates the video frame number.

[0092] Video frames need to undergo communication transmission and buffering processing from the acquisition end to the static stitching processing module. The first in the video stream The time from video frame acquisition to arrival at the static stitching processing module Represented as: (11) in, Indicates the first The first in the video stream The transmission latency of video frames in vehicular communication networks. Indicates the first The first in the video stream The processing latency of video frames in the video receiving and alignment module (introduced by buffering and scheduling).

[0093] Ideally, multiple video streams should maintain strict synchronization under the same frame number, that is: (12) However, in practical systems, due to unavoidable latency fluctuations in communication transmission and processing, the above conditions are difficult to fully satisfy. To characterize the temporal consistency of multiple video streams at the frame level, a frame-level time alignment error is introduced. Frame-level time alignment error Represented as: (13) To avoid splicing misalignment and display jitter, frame-level time alignment error should be limited to a threshold range that the system can tolerate, i.e.: (14) in, This represents the maximum allowable frame-level time deviation for the splicing system, and its value is related to factors such as the video frame period, the width of the splicing area, and the display refresh rate. In engineering implementation, It is usually less than a certain proportion of a single frame period.

[0094] Besides frame-level time consistency, end-to-end latency is also a crucial indicator affecting real-time panoramic video display. End-to-end latency represents the time it takes for video frames to be captured, stitched together, and displayed. The first in the video stream End-to-end latency of video frames Represented as: (15) in, Indicates each route The time it takes for the splicing result of the video frames to be output to the output display unit.

[0095] To meet the real-time requirements of vehicle situational awareness, the end-to-end latency must satisfy the upper bound constraint: (16) in, This represents the maximum acceptable display latency for the splicing system, and its value is determined by specific task requirements and human-computer interaction characteristics.

[0096] Furthermore, decomposing the end-to-end delay into two parts—communication transmission and fixed processing—can be written as follows: (17) in, Indicates the first The fixed processing latency of video frames in a network, excluding communication transmission, mainly includes the fixed processing latency introduced by buffering, splicing, and display output. The above formula provides a clear boundary for the latency budget on the communication side.

[0097] In summary, a panoramic video real-time stitching system must simultaneously meet frame-level time alignment constraints, end-to-end latency upper bound constraints, and computational load stability constraints.

[0098] The aforementioned constraints collectively determine the performance requirements that the vehicle communication system must meet in terms of transmission latency, jitter, and scheduling predictability, providing a formal basis for subsequent video traffic modeling and the collaborative design of communication and splicing systems.

[0099] Next, we will conduct in-vehicle video traffic modeling driven by stitching requirements. In order to meet the constraints of frame-level timing consistency and end-to-end latency for panoramic video stitching at the communication system level, it is necessary to formally model the data stream characteristics of in-vehicle multi-channel video, so as to provide a basis for video stream access control and scheduling configuration.

[0100] No. Video frame period of the video stream output by the road camera Represented as: (18) in, Indicates the first The frame rate of the video stream. In a panoramic video system, each camera typically uses the same or similar frame rate configuration; therefore, in the subsequent analysis, we can assume... . This is the standard video frame period.

[0101] Let the first The first in the video stream The data size of a video frame is Its size is determined by factors such as video resolution, encoding method, and compression parameters. In engineering implementation, to ensure the predictability of system behavior, video encoding parameters are usually kept fixed, thus making the amount of data per frame approximately constant in the time dimension, i.e.: (19) in, Indicates the first The amount of data per frame in a video stream.

[0102] Under the above assumptions, the first A video stream can be abstracted as a data stream with periodic arrival characteristics. Average bandwidth requirements of video streams It can be represented as: (20) Total video streaming bandwidth requirements when multiple video streams are transmitted simultaneously Represented as: (twenty one) To ensure stable system operation, It should not exceed the effective bandwidth limit of the vehicle communication system that can be used for video services.

[0103] Besides average bandwidth, the arrival characteristics of the video stream in the time dimension also have a significant impact on the splicing system. Because video frames arrive periodically... Generate, the The first in the video stream Ideal arrival time of video frames in a communication system It can be represented as: (twenty two) in, This is the system's reference start time.

[0104] In this embodiment, combining the frame-level time alignment error constraint defined above, to ensure the stability of the splicing system, multiple video streams must maintain consistent arrival at the receiving end, satisfying this constraint. This places requirements on the transmission jitter control of the communication system. Furthermore, combining the end-to-end delay constraint and the communication-side delay budget, the communication system needs to complete unified planning during the deployment phase and remain stable during operation to ensure the predictability of system behavior.

[0105] (5) Display output module The display output module is used to output the panoramic stitched image at a fixed refresh rate.

[0106] The display unit outputs panoramic stitched images according to a fixed refresh cycle to form a continuous and stable panoramic display of the vehicle, which is then displayed to the operator to ensure that the operator obtains continuous, stable and low-latency panoramic situation information.

[0107] The display output module includes an image receiving unit, a refresh control unit, and a display interface unit.

[0108] 1) Image receiving unit The image receiving unit is used to receive panoramic stitched images.

[0109] 2) Refresh control unit The refresh control unit is used to drive the panoramic stitching image according to a fixed refresh frequency (beat). 3) Display Interface Unit The display interface unit is used to output panoramic spliced ​​images to the vehicle-mounted display terminal.

[0110] In this embodiment, the stitched video frames are sent to the display output module and displayed according to a fixed refresh cycle. The display output module does not reorder the stitched results or dynamically adjust the rhythm; instead, it maintains the same output rhythm as the static stitching processing module to ensure the continuity and stability of the display process. To guarantee a stable display rhythm, the system does not add any additional frame reordering or dynamic adjustment mechanisms in the output stage, thereby preventing inconsistencies between the display rhythm and the stitching processing rhythm. Through this processing flow, the system can maintain a stable video running rhythm throughout the end-to-end path, achieving a continuous, low-jitter panoramic display effect.

[0111] Furthermore, the system may also include an offline configuration unit, which is used to complete camera calibration, reference plane construction, geometric mapping parameter calculation, stitching area division, transmission scheduling parameter configuration, and cache threshold configuration during the system deployment phase, and distribute the configuration results to each running unit so that the system can perform real-time stitching according to fixed parameters during the running phase.

[0112] Preferably, the unified time base is provided by the master clock in the vehicle network, and both the multi-channel acquisition end and the stitching processing end complete time synchronization based on the master clock. Preferably, the setting of the sending window is consistent with the video frame period, so that each video stream has a fixed sending opportunity within each period. Preferably, the video receiving and alignment module uses a finite-depth buffer to absorb only minor jitter during transmission without performing complex dynamic rearrangement optimization. Preferably, the static stitching processing module uses a pre-stored mapping lookup table or a pre-calculated coordinate mapping relationship to reduce the amount of computation during the running phase. Preferably, the display output module and the static stitching processing module use the same beat-driven method to avoid screen jitter caused by changes in display rhythm.

[0113] In summary, the panoramic video real-time stitching system for vehicle situational awareness proposed in this embodiment differs from existing research that mainly focuses on image processing or stitching algorithms. It takes a system collaboration perspective and explicitly transforms the inherent requirement for the stability of the stitching structure into constrained and configurable engineering conditions for the communication and processing layers. This provides an overall solution for the long-term stable operation of panoramic video and can meet the comprehensive requirements of real-time stitching of multiple videos in vehicle situational awareness applications for frame-level temporal consistency, display stability, and engineering deployability.

[0114] In terms of scheme design, this embodiment analyzes the key constraints of panoramic video stitching in the temporal and spatial dimensions, and establishes a constraint model centered on frame-level time alignment, end-to-end latency upper bound, and computational load stability. Based on this, a stitching strategy combining static geometric mapping and fixed stitching seams is adopted to avoid the computational load fluctuations and temporal instability problems introduced by content-aware optimization and structural adjustments in dynamic stitching methods, ensuring consistency and predictability of the stitched structure in the temporal dimension.

[0115] In terms of system implementation, this embodiment further maps the splicing requirements back to the communication system level, introducing a deterministic communication configuration mechanism based on periodic scheduling to constrain the transmission latency and jitter of multiple video streams. Through the collaborative design of the splicing system and the communication system, the system can meet the frame-level synchronization and end-to-end latency constraints under static configuration conditions, reducing the dependence on online scheduling optimization and dynamic structure reconstruction mechanisms, and improving the system's engineering feasibility and operational stability.

[0116] Specific embodiment 2 of the present invention provides a real-time panoramic video stitching method for vehicle situational awareness, the flowchart of which is as follows: Figure 4 As shown. The method includes the following steps: Step S1: Acquire and process video streams from multiple directions of the vehicle to generate a panoramic video stream composed of various video streams with frame-level time stamps and frame sequence numbers; and transmit the panoramic video stream according to transmission priority, transmission period and scheduling window. Step S2: Parse the received panoramic video stream, sort and synchronize each video stream according to the frame-level time stamp and frame sequence number, and align and output multiple sets of synchronized frames in the current scheduling period. Step S3: Receive multiple sets of synchronization frames in the current scheduling period, perform geometric mapping and splicing region fusion processing on each set of synchronization frames, and generate a panoramic spliced ​​image.

[0117] Step S4: Output the panoramic stitched image at a fixed refresh rate.

[0118] A specific embodiment 3 of the present invention provides a real-time panoramic video stitching method for vehicle situational awareness, the method comprising the following steps: Step S1: Acquire video data collected by multiple cameras around the vehicle, and add a time stamp and frame number under a unified time reference to each video frame.

[0119] Step S2: Send each video frame to the vehicle communication network according to the preset periodic scheduling rules, and complete the video frame reception at the receiving end.

[0120] In step S2, each video stream adopts a periodic transmission method based on a unified time base and occupies network bandwidth resources according to a pre-planned time window, thereby reducing queuing competition and random jitter of multiple video streams during transmission.

[0121] Step S3: Sort and synchronize multiple video frames according to the timestamp and frame number. Perform alignment output on video frames that meet the conditions, and perform short-term buffering for video frames that do not meet the conditions but are within the allowable range.

[0122] In step S3, the alignment judgment criteria include at least the consistency of frame sequence number and whether the time deviation is less than a preset threshold. When a certain video frame has not arrived after the allowed waiting time, the cached frame from the previous moment or the default compensation frame is used to participate in the splicing to maintain the stability of the output beat.

[0123] Step S4: Perform fixed geometry mapping on the time-aligned multi-channel video frames to map each video to a unified reference plane.

[0124] In step S4, the fixed geometric mapping parameters are obtained from the offline calibration results during the system deployment phase and are not dynamically updated during operation.

[0125] Step S5: Based on the pre-defined splicing area division and fixed splicing seam rules, select and merge the pixels of each area in the reference plane to generate the target panoramic spliced ​​image.

[0126] In step S5, the splicing area division and fixed splicing seam rules are predetermined during the deployment phase and remain unchanged during operation, so as to keep the processing path and computational load of each video frame basically consistent.

[0127] Step S6: Output the target panoramic stitched image to the display terminal according to a fixed refresh rate.

[0128] From a real-time implementation perspective, the above-mentioned stitching process has the following characteristics: First, video acquisition, transmission, reception, and stitching processing all exhibit periodic behavior, and the system as a whole can be regarded as a quasi-time-triggered structure, which is beneficial for timing analysis and resource planning; Second, the computational complexity of the static stitching processing module is fixed, avoiding fluctuations in computational load caused by changes in image content; Third, the caching and scheduling mechanisms are only used to meet frame-level time alignment requirements and do not participate in complex optimization processes, thereby reducing the system implementation complexity and improving operational stability.

[0129] Specific embodiment 4 of the present invention also discloses a real-time panoramic video stitching method for vehicle situational awareness. This method can be divided into an offline configuration stage and an online operation stage.

[0130] (a) Offline configuration stage Step A1: Camera installation and calibration.

[0131] Multiple cameras are installed at predetermined locations on the vehicle, and the intrinsic and extrinsic parameters of each camera are acquired. A unified reference plane coordinate system is established based on the vehicle's structural dimensions, camera installation orientation, and target display requirements.

[0132] Step A2: Generate geometric mapping parameters.

[0133] Based on the calibration results, the mapping parameters from each camera image to the reference plane are calculated, and the mapping parameters are written into the mapping parameter storage module.

[0134] Step A3: Configuration of splicing area and fixed splicing seam.

[0135] Based on the coverage of multiple cameras on the reference plane, the output areas responsible for each camera are divided. For areas with overlap, the pixel selection boundaries are predetermined to form fixed stitching seams, and the area division parameters and stitching seam parameters are saved.

[0136] Step A4: Configure transmission parameters.

[0137] Based on the number of cameras, resolution, frame rate, and video data volume, configure the transmission period, priority, queue mapping relationship, and transmission window parameters of multiple video streams, and write them into the deterministic transmission unit.

[0138] Step A5: Receive cache parameter configuration.

[0139] Configure the buffer depth, waiting threshold, and exception handling strategy of the video receiving and alignment module according to the allowable jitter range of the transmission link and the overall system latency requirements.

[0140] After the offline configuration phase is completed, the system enters the online operation phase. This process is consistent with the content of the paper.

[0141] (II) Online Operation Phase Step B1: Multi-channel video acquisition.

[0142] Multiple cameras capture images of the vehicle's surroundings at a set frame rate; each time a video frame is generated, a time stamp and frame number are added to the frame, and it is encapsulated into a video data unit to be transmitted.

[0143] Step B2: Deterministic transmission of video data.

[0144] The acquisition side writes each video data unit into the corresponding transmission queue; when the transmission window corresponding to the current video stream is open, the deterministic transmission unit sends the video data to the vehicle communication network and transmits it to the splicing processing end.

[0145] Step B3: Video frame reception and parsing.

[0146] The splicing processing unit receives video data from various sources, parses out the source identifier, time stamp, and frame number corresponding to each frame, and writes them into the corresponding input buffer.

[0147] Step B4: Multi-channel video frame synchronization judgment.

[0148] The video receiving and alignment module extracts video frames with the same frame number or within the same splicing period from each input buffer and compares their time deviations; when the time deviation is less than a preset threshold, it determines that the group of video frames meets the splicing conditions.

[0149] Step B5: Cache wait and exception compensation.

[0150] If a video frame has not yet arrived but is still within the allowed waiting range, a short-term buffer wait is executed; if the wait times out, the exception handling mechanism is invoked to replace the missing frame with a buffered frame from the previous cycle or a default compensation frame to ensure that the output beat is not interrupted.

[0151] Step B6: Fixed geometry mapping.

[0152] For each of the multiple video frames that have completed the synchronization judgment, the corresponding mapping parameters are called to perform image coordinate transformation so that each video frame is projected onto a unified reference plane.

[0153] Step B7: Fixed area splicing.

[0154] For non-overlapping areas in the reference plane, the corresponding video source pixels are directly written; for overlapping areas, the pixels of the specified video source are selected according to the fixed stitching seam rules to obtain a complete panoramic stitched image.

[0155] Step B8: Output and display the splicing result.

[0156] The panoramic stitched image is sent to the display output module and output to the display terminal according to a fixed refresh cycle to form a continuous panoramic situational image.

[0157] Compared with existing technologies, this invention has the following advantages: This invention does not simply address the video stitching problem from an image algorithm perspective, but rather designs video acquisition, communication transmission, receiving alignment, image stitching, and display output as a unified end-to-end system. On one hand, by unifying the time base, periodically scheduling transmission, and aligning the receiver buffer, the temporal consistency of multiple video inputs can be improved. On the other hand, by using fixed geometric mapping and fixed stitching seam design, computational fluctuations and changes in the stitching structure caused by variations in image content can be reduced, thereby improving the system's real-time performance, stability, and engineering deployability.

[0158] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0159] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time panoramic video stitching system for vehicle situational awareness, characterized in that, The system includes a multi-channel video acquisition module, a vehicle-mounted communication network, a video receiving and alignment module, and a static stitching processing module; among which... The multi-channel video acquisition module is used to acquire and process video streams from multiple directions of the vehicle, generating a panoramic video stream composed of various video streams with frame-level time stamps and frame sequence numbers. The vehicle-mounted communication network is used to transmit panoramic video streams to the video receiving and alignment module according to transmission priority, transmission period and scheduling window. The video receiving and alignment module is used to parse the received panoramic video stream, sort and synchronize each video stream according to the frame-level time stamp and frame sequence number, and align and output multiple sets of synchronized frames in the current scheduling cycle. The static stitching processing module is used to receive multiple sets of synchronization frames in the current scheduling period, perform geometric mapping and stitching region fusion processing on each set of synchronization frames, and generate a panoramic stitched image.

2. The panoramic video real-time stitching system for vehicle situational awareness according to claim 1, characterized in that, The system also includes a display output module; The display output module is used to output the panoramic stitched image at a fixed refresh rate.

3. The real-time panoramic video stitching system for vehicle situational awareness according to claim 1 or 2, characterized in that, The video receiving and alignment module includes a synchronization buffer output unit; wherein... The synchronous buffer output unit is used to determine whether the time alignment condition is met based on the frame-level time stamp of each video frame with the same frame number in the current scheduling cycle. If the condition is met, the multiple video frames with the same frame number are output as a group of synchronous frames.

4. The real-time panoramic video stitching system for vehicle situational awareness according to claim 3, characterized in that, The time alignment condition is executed according to the following logic in the frame arrival buffering mechanism: For the same frame sequence number, if a certain video frame arrives before other video frames, the video frame is buffered at a limited depth; after all the other video frames have arrived, it is checked whether the time deviation corresponding to the frame-level time stamp of the video frame with the corresponding frame sequence number in each video stream is within the allowable range. If so, output multiple video frames corresponding to the same frame number as a group of synchronized frames.

5. The real-time panoramic video stitching system for vehicle situational awareness according to claim 4, characterized in that, The time alignment condition also executes a timeout exception handling mechanism according to the following logic: If a video frame with the same frame number fails to arrive after a preset waiting time, the system will trigger an exception handling mechanism.

6. The real-time panoramic video stitching system for vehicle situational awareness according to claim 5, characterized in that, The static stitching processing module includes an image mapping unit and a region division unit; wherein... The image mapping unit is used to perform coordinate transformation on the corresponding video frames in each group of synchronization frames according to the geometric mapping parameters of each camera component, and map them to the corresponding stitching area of ​​the unified reference plane. The region division unit is used to determine the affiliation of each spliced ​​region in the unified reference plane.

7. The panoramic video real-time stitching system for vehicle situational awareness according to claim 6, characterized in that, The static stitching processing module also includes an image fusion generation unit, which is used to perform image fusion based on fixed stitching seams on each group of synchronous frames mapped to a unified reference plane to generate a corresponding panoramic stitched image.

8. The real-time panoramic video stitching system for vehicle situational awareness according to claim 7, characterized in that, In the image fusion generation unit, image fusion based on fixed stitching seams is performed as follows: For non-overlapping regions, the image fusion generation unit directly uses the corresponding pixels of the video source specified by the region division unit; For overlapping areas, the image fusion generation unit determines the pixel source according to the fixed stitching seam rule.

9. A real-time panoramic video stitching method for vehicle situational awareness, characterized in that, The method includes: The system acquires and processes video streams from multiple directions of the vehicle, generating a panoramic video stream composed of various video streams with frame-level time stamps and frame sequence numbers; and transmits the panoramic video stream according to transmission priority, transmission period, and scheduling window. The received panoramic video stream is parsed, and each video stream is sorted and synchronized according to the frame-level time stamp and frame sequence number. Multiple sets of synchronized frames in the current scheduling period are aligned and output. Receive multiple sets of synchronization frames in the current scheduling period, perform geometric mapping and splicing region fusion processing on each set of synchronization frames, and generate a panoramic spliced ​​image.

10. The real-time panoramic video stitching method for vehicle situational awareness according to claim 9, characterized in that, The method further includes: The panoramic stitched image is output at a fixed refresh rate.

Images