A vehicle-mounted video delay elimination method and system based on multi-thread optimization

Abstract

The application provides a vehicle-mounted video delay elimination method and system based on multi-thread optimization, comprising the following steps: acquiring original video frames through a video capture thread, and performing encoding processing on the original video frames through an encoder; performing network transmission operation on the encoded data through an independent network output thread; monitoring whether an error notification signal indicating network transmission is received during the running of the video capture thread; if not, the video capture thread and the network output thread continue to perform video capture, encoding and transmission operation; if yes, the video capture thread actively destroys the currently running network output thread, and creates and starts a new network output thread to restore data transmission. The application solves the video capture delay caused by the increase of the image processor load, and the resulting encoding and transmission data backlog, thereby solving the display delay problem of the front panel display, and realizing the fast self-recovery of the system in the event of transmission error.

Description

Technical Field

[0001] This invention relates to the field of video transmission technology, and more specifically, to a method and system for eliminating in-vehicle video latency based on multi-threading optimization. Background Technology

[0002] In existing in-vehicle systems, navigation interfaces or video content typically need to be transmitted from the main system to the front panel display (FPK) for display in real time. Current technologies mostly employ single-threaded or simple multi-threaded architectures to handle video capture, encoding, and transmission. In this architecture, the video capture module, encoding module, and network transmission module are tightly coupled. However, when the image processor (GPU) of the in-vehicle human-machine interface experiences a sudden increase in load due to image rendering, the performance of the video capture module degrades significantly, with the time to capture a single frame potentially increasing dramatically from the normal 5 milliseconds to over 150 milliseconds. In this traditional tightly coupled architecture, because network encoding and transmission functions are tied to the slow video capture process on the same processing link, the encoder does not have enough time to complete its task before the next capture loop begins. This causes encoded data to accumulate in the buffer, failing to be transmitted to the front panel display in a timely manner, ultimately resulting in a visible delay at the display end, severely impacting the user's real-time experience. Furthermore, existing solutions have weak error handling mechanisms; when anomalies occur in the network transmission or encoding process, the system struggles to recover quickly, potentially leading to prolonged video transmission interruptions. Summary of the Invention

[0003] This invention aims to solve the problem of video capture delay caused by increased image processor load, which in turn leads to backlog of encoded transmission data and causes display delay on the front panel display, and to achieve rapid self-recovery of the system in the event of transmission errors.

[0004] To address the aforementioned problems, this invention provides a multi-threaded optimized method for eliminating in-vehicle video latency, comprising the following steps: acquiring original video frames through a video capture thread and encoding the original video frames using an encoder; performing network transmission of the encoded data through an independent network output thread; during operation, the video capture thread monitors whether it receives an error notification signal indicating network transmission; if not, the video capture thread and the network output thread continue to perform video capture, encoding, and transmission operations; if so, the video capture thread actively destroys the currently running network output thread and creates and starts a new network output thread to restore data transmission.

[0005] Compared with existing technologies, the technical effects achieved by adopting this solution are as follows: By decoupling the network transmission function into a dedicated network output thread, its operation is separated from the video capture thread, which may slow down due to high GPU load. Even if the video capture interval is extended, the network output thread can still transmit encoded data in a timely manner according to its own rhythm, avoiding the backlog of encoded data in the buffer, thus fundamentally eliminating the display latency problem of FPK and ensuring the real-time performance of video transmission. A proactive error handling mechanism based on error notification signals, led by the video capture thread, is introduced. When the network transmission thread fails, the system can quickly detect and trigger a destroy-rebuild process, rather than simply retransmitting data or waiting for a timeout. This dynamic recovery capability greatly shortens the system's recovery time from anomalies, significantly improving the robustness and availability of the entire vehicle-mounted video transmission system. Thread collaboration is based on normal and abnormal condition judgment logic. Under normal conditions, each thread runs asynchronously, transmitting data only when it is ready, reducing unnecessary thread synchronization overhead, lowering CPU load, and making the system run more efficiently.

[0006] Furthermore, the original video frames are obtained through the video capture thread, specifically by the video capture thread calling a screen capture tool to perform screen capture and obtain the original video frames.

[0007] Compared to existing technologies, the technical benefits of this solution are as follows: It encapsulates the specific, high-frequency low-level operation of screen capture within a video capture thread, and implements it by calling a dedicated screen capture tool. This makes the thread's responsibilities clear and its functions focused. This design avoids mixing capture logic with other processing logic, conforming to the principles of high cohesion and low coupling in software engineering, and improving code maintainability. By defining a standard interface for "calling the screen capture tool," the specific capture implementation is separated from the upper-level multi-threaded scheduling and error handling logic. This allows the system to flexibly adapt to different screen capture technologies or low-level graphics interfaces without modifying the core thread management and error recovery architecture, enhancing the system's portability and adaptability.

[0008] Furthermore, the original video frames are encoded using an encoder, including the following steps: preprocessing the original video frame data by color space conversion using a color converter, and then encoding the preprocessed original video frames using an encoder.

[0009] Compared with existing technologies, the technical effects achieved by adopting this solution are as follows: By adding a preprocessing step of color space conversion before encoding, the captured raw video frames are uniformly converted into a format that the encoder can process more efficiently. This reduces the computational overhead of format conversion within the encoder, allowing it to focus more on the compression algorithm itself. This improves encoding speed while avoiding color distortion or quality loss that might occur due to the encoder's own conversion, thus ensuring the display quality of the output video.

[0010] Furthermore, the network transmission operation of the encoded data is performed through an independent network output thread, including the following steps: the network output thread continuously polls the encoder output to obtain the encoded usable data, adds a timestamp to the data, and sends it to the target device.

[0011] Compared to existing technologies, the technical effects achieved by adopting this solution are as follows: the network output thread actively acquires data from the encoder through continuous polling, rather than passively waiting for the preceding thread to transmit data. This design makes its operation completely independent of the potentially obstructed video capture thread. Even if upstream capture slows down, the transmission thread can still continuously process the ready-to-encode data. This is the core mechanism for achieving latency elimination, fundamentally avoiding data backlog caused by blocked transmission. By adding a timestamp to each frame of data, the receiving end can reconstruct the original timing of the video stream based on the timestamp information rather than the simple order of reception. This is crucial for compensating for jitter that may occur during network transmission, ensuring audio-visual synchronization, and achieving smooth playback, significantly improving the viewing experience on the display terminal.

[0012] Furthermore, the network output thread adjusts the data transmission strategy based on the current network conditions to ensure reliable data transmission.

[0013] Compared with existing technologies, the technical effects achieved by adopting this solution are as follows: By dynamically adjusting the strategy, the system does not need to always operate in the most conservative manner. When network conditions are good, a more aggressive strategy with lower latency can be adopted to pursue real-time performance; when network conditions deteriorate, it automatically switches to a more reliable strategy with stronger packet loss resistance. This intelligent trade-off ensures that optimal transmission performance is achieved under any network conditions, realizing the optimal utilization of system resources.

[0014] Furthermore, the network transmission operation of the encoded data includes the following steps: the encoded data is transmitted from the encoder to the network output thread, and asynchronous communication is performed through the shared file descriptor established by the video capture thread at the beginning.

[0015] Compared to existing technologies, the technical effects achieved by this solution are as follows: By using a shared file descriptor (FD) established by the video capture thread during initialization as an asynchronous communication channel between the encoder and the network output thread, the data production (encoding) and data consumption (transmission) processes are completely decoupled. The encoder can write data to the FD immediately after encoding, without waiting for the network thread to be ready; the network output thread can obtain data by polling the FD, without relying on direct scheduling by the encoder. This "producer-consumer" model allows the two key components to run independently and asynchronously at their respective maximum rates, which is the core architectural design to avoid data backlog and eliminate latency. The shared file descriptor is a lightweight inter-process communication (IPC) mechanism at the operating system level, and its data transfer is usually completed in kernel space, avoiding the overhead of multiple copies of data in user space. Compared to traditional solutions using message queues or shared memory with locking mechanisms, this method greatly reduces the cost of thread synchronization and context switching, achieving extremely low-latency data transfer, making it particularly suitable for video streaming scenarios with extremely high real-time requirements.

[0016] Furthermore, monitoring whether an error notification signal indicating network transmission is received includes: the network output thread generating an error notification signal when a transmission error is detected, and passing it to the video capture thread through a preset network output error event file descriptor.

[0017] Compared to existing technologies, the technical benefits of this solution are as follows: Error detection is delegated to the network output thread, which has the best understanding of the transmission status, while error recovery decisions and execution are handled by the video capture thread, which acts as the core system manager. This separation of responsibilities architecture is clear and reasonable, avoiding the bloat of a single thread while ensuring globally optimal error response decisions. The two communicate via a dedicated Event File Descriptor (Event FD), a highly efficient asynchronous signaling mechanism that ensures error information is delivered promptly and reliably, laying the foundation for rapid recovery.

[0018] Furthermore, monitoring whether error notification signals indicating network transmission are received includes: the video capture thread receiving error notification signals by monitoring network output error event file descriptors.

[0019] Compared with existing technologies, the technical effects achieved by adopting this solution are as follows: The video capture thread receives error signals by polling or listening to specific network output error event file descriptors, which is an event-driven asynchronous notification mechanism. Compared with the traditional method of having the capture thread actively and frequently query the network thread status (active query), this solution greatly reduces meaningless CPU resource consumption and inter-thread communication overhead. The capture thread is only triggered and processed when an error event actually occurs (i.e., the FD becomes readable), allowing the system to concentrate computing resources on core tasks such as video capture during normal operation.

[0020] Furthermore, the video capture thread actively destroys the currently running network output thread and creates and starts a new network output thread to resume data transmission. This includes the following steps: pausing the current video capture loop; destroying the network output thread instance that encountered the error and releasing its related resources; reinitializing the network connection parameters and creating a new network output thread instance; resuming the video capture loop and handing over subsequent data to the new network output thread instance for transmission.

[0021] Compared with existing technologies, the technical effects achieved by adopting this solution are as follows: Existing technologies typically handle network transmission errors by resetting connections and retransmitting data, which makes it difficult to guarantee that the internal state or residual resources of the faulty thread have been completely cleared, potentially leading to recurring problems. This solution achieves complete isolation and removal of the faulty unit through a series of rigorous steps: "pause-destroy-release-rebuild-recover." By destroying the old thread instance and releasing all its resources, it ensures that any abnormal states that caused the error are completely cleared. Subsequently, a completely new thread instance is created, fundamentally avoiding error accumulation or interference from residual problems, greatly improving the success rate and reliability of recovery.

[0022] To address the aforementioned problems, this invention provides a multi-threaded optimized in-vehicle video delay cancellation system. The in-vehicle video delay cancellation system includes the multi-threaded optimized in-vehicle video delay cancellation method provided by any of the above technical solutions, comprising: a capture thread module, an encoder module, and a network output thread module; the capture thread module is used to perform screen capture, manage the encoding process, and receive error notification signals; the encoder module is communicatively connected to the capture thread module and is used to encode video frames; the network output thread module operates independently, is communicatively connected to the encoder module, is used for network transmission, and generates error notification signals when transmission errors are detected; wherein, when no error notification is received, the capture thread module works collaboratively with the network output thread module; when an error notification signal is received, the capture thread module can destroy and reconstruct the network output thread module.

[0023] Compared with existing technologies, the technical effects achieved by adopting this solution are as follows: The system is clearly divided into three core components: a capture thread module, an encoder module, and a network output thread module, with clearly defined interfaces through communication connections. The network output thread module is designed to operate independently, completely separating its workflow from the capture module, which may be affected by GPU load. This thorough modular decoupling ensures that even if the capture module slows down for some reason, the network transmission module can continue to process the encoded data, thereby eliminating processing chain blockage and data backlog caused by coupling at the system architecture level and achieving a fundamental elimination of latency. When the transmission module fails, the system can automatically and quickly restart the module, achieving closed-loop automation from failure to service recovery, greatly improving the reliability of long-term stable operation in complex automotive environments.

[0024] In summary, the technical solutions described above in this application can have one or more of the following advantages or beneficial effects: i) By decoupling video capture, encoding, and network transmission functions into independent threads / modules and adopting an asynchronous communication mechanism, the network transmission link is no longer affected by fluctuations in front-end capture performance. Even when high GPU load causes a drop in the capture frame rate, the system can ensure that encoded data is sent in a timely manner, fundamentally avoiding data backlog in the buffer, thereby eliminating display latency on the front panel display (FPK) and ensuring the real-time performance of the video stream. ii) When a transmission thread fails, the system can quickly detect and automatically trigger a "destroy-rebuild" process, rather than relying on simple retransmission or timeout waiting. This proactive, module-level self-healing capability greatly shortens the system's recovery time from anomalies and significantly enhances the reliability of continuous and stable operation in complex automotive environments. iii) The polling and event notification-based collaboration mode between threads reduces unnecessary synchronization waiting and context switching overhead. The modular architecture allows each thread to focus on its core tasks, achieving efficient utilization of computing resources such as the CPU and improving the overall operating efficiency of the system. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a multi-threaded optimization-based in-vehicle video latency elimination method according to an embodiment of the present invention.

[0026] Figure 2 This is a module architecture diagram of the in-vehicle video delay cancellation system based on multi-threading optimization in an embodiment of the present invention;

[0027] Figure 3 This is a flowchart illustrating the normal operation of the in-vehicle video delay cancellation system based on multi-threading optimization in this embodiment of the invention.

[0028] Figure 4 This is a flowchart illustrating the anomaly recovery process of the in-vehicle video delay elimination system based on multi-threading optimization in this embodiment of the invention. Detailed Implementation

[0029] The purpose of this invention is to provide a method and system for eliminating in-vehicle video latency based on multi-threading optimization, which can eliminate display latency caused by image processor load and achieve rapid self-recovery in the event of transmission errors.

[0030] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0031] See Figures 1-4 This invention provides a method for eliminating in-vehicle video latency based on multi-threading optimization, comprising the following steps:

[0032] S1: Obtain the original video frames through the video capture thread, and encode the original video frames through the encoder;

[0033] S2: Performs network transmission operations on encoded data through a separate network output thread;

[0034] S3: During the video capture thread's operation, it monitors whether it receives an error notification signal indicating network transmission errors.

[0035] S4: If not, the video capture thread and the network output thread continue to perform video capture, encoding, and transmission operations;

[0036] S5: If so, the video capture thread actively destroys the currently running network output thread and creates and starts a new network output thread to resume data transmission.

[0037] Specifically, after system startup, the video capture thread (CaptureThread) begins operation. It calls screen capture tools like WesonCapture to obtain raw video frame buffers from the display interface of the in-vehicle navigation system (ARNavi). This thread then passes the acquired raw frame data to the encoder for compression encoding. Simultaneously, the network output thread (NetworkOutputThread) is created and started as a separate thread. It is responsible for continuously processing the encoded data and sending it over the network to the front panel display (FPK). During system operation, the video capture thread has a monitoring role. It polls the encoder error event file descriptor (EncoderErrorEventFD) and the network error event file descriptor (NetworkErrorEventFD) to check for error notification signals indicating network transmission failures. If no error notification signal is received, the video capture, encoding, and network transmission processes continue to operate normally and collaboratively. If an error notification signal is received (via NetworkErrorEventFD), the video capture thread immediately takes recovery action: first, it proactively destroys the currently malfunctioning network output thread instance, cleans up its occupied resources, and then immediately creates and starts a completely new network output thread instance. The new thread will re-establish the network connection, thereby taking over the subsequent data transmission tasks and restoring the system to normal operation.

[0038] See Figure 1 and Figure 3 The original video frames are obtained through the video capture thread, specifically by the video capture thread calling the screen capture tool to perform screen capture and obtain the original video frames.

[0039] Specifically, in the capture loop, a specific screen capture tool is invoked to perform the actual screen capture operation, thereby obtaining the raw video frame buffer data.

[0040] See Figure 1 and Figure 3 The encoding process of the original video frames by the encoder includes the following steps: preprocessing the original video frame data by color space conversion using a color converter, and then encoding the preprocessed original video frames using the encoder.

[0041] Specifically, the system includes a preprocessing step before encoding. After obtaining the raw video frame data, the video capture thread first uses a color converter to perform preprocessing operations such as color space conversion. Only after preprocessing is completed is the processed data pushed to the encoder for formal encoding.

[0042] See Figure 3 The network transmission operation of the encoded data is performed through an independent network output thread, including the following steps: the network output thread continuously polls the encoder output to obtain the encoded usable data, adds a timestamp to the data and sends it to the target device.

[0043] Specifically, the network output thread runs independently, polling the encoder frame output file descriptor (EncoderFrameOutputFD) established by the video capture thread during initialization to obtain available encoded data. When new data is detected, the pushFrame() function is called to retrieve the frame data and push it to the timestamp encoder (TsEncoder) to add timestamp information to the data. Finally, the data is sent to the target device (FPK).

[0044] See Figure 3 The network output thread adjusts the data transmission strategy according to the current network conditions to ensure reliable data transmission.

[0045] Specifically, during transmission, the network output thread automatically adjusts its transmission strategy based on real-time network conditions to ensure reliable data transmission under various network conditions.

[0046] See Figure 3 The network transmission operation of the encoded data includes the following steps: the encoded data is transmitted from the encoder to the network output thread, and asynchronous communication is carried out through the shared file descriptor established by the video capture thread at the beginning.

[0047] Specifically, during the initialization phase, the video capture thread creates and passes the encoder frame output file descriptor (EncoderFrameOutputFD). The encoded data is then transmitted asynchronously between the encoder and the network output thread via this shared file descriptor. The network output thread retrieves data by polling this FD, without requiring the encoder to actively push it.

[0048] See Figure 4 The system monitors whether it receives an error notification signal indicating network transmission. Specifically, the network output thread generates an error notification signal when it detects a transmission error and passes it to the video capture thread through a preset network output error event file descriptor.

[0049] Specifically, when the network output thread detects a network transmission error, it generates and sends an error notification signal by performing a write operation to a preset network output error event file descriptor (NetworkErrorEventFD).

[0050] See Figure 4The system monitors whether error notification signals indicating network transmission are received. Specifically, the video capture thread receives error notification signals by monitoring the network output error event file descriptor.

[0051] Specifically, the video capture thread periodically polls the NetworkErrorEventFD (and the EncoderErrorEventFD) in its main capture loop to listen for and receive error notification signals.

[0052] See Figure 4 The video capture thread actively destroys the currently running network output thread and creates and starts a new network output thread to resume data transmission. Specifically, this includes the following steps: pausing the current video capture loop; destroying the network output thread instance that encountered the error and releasing its related resources; reinitializing the network connection parameters and creating a new network output thread instance; resuming the video capture loop and handing over subsequent data to the new network output thread instance for transmission.

[0053] Specifically, upon receiving an error notification signal, the video capture thread (CaptureThread) immediately stops the current video capture operation (if in progress) and executes error handling logic. The video capture thread destroys the problematic network output thread (NetworkOutputThread), releases related resources, and then creates a new network output thread (NetworkOutputThread). After the new network output thread is successfully created, the video capture thread resumes video capture operations and continues to pass subsequently acquired video frame data to the encoder for encoding and transmission, thereby restoring the system to normal operation.

[0054] See Figure 2 This invention provides a multi-threaded optimized in-vehicle video delay cancellation system, including the multi-threaded optimized in-vehicle video delay cancellation method provided by any of the above-mentioned technical solutions. The in-vehicle video delay cancellation system includes: a capture thread module, an encoder module, and a network output thread module; the capture thread module is used to perform screen capture, manage the encoding process, and receive error notification signals; the encoder module is communicatively connected to the capture thread module and is used to encode video frames; the network output thread module operates independently, is communicatively connected to the encoder module, is used for network transmission, and generates error notification signals when a transmission error is detected; wherein, when the capture thread module does not receive an error notification, it works in conjunction with the network output thread module; when an error notification signal is received, the capture thread module can destroy and rebuild the network output thread module.

[0055] Specifically, the capture thread module is responsible for screen capture, managing the encoding process, and receiving error notification signals by monitoring the Network Error Event File Descriptor (NetworkErrorEventFD). The encoder module communicates with the capture thread module and is responsible for encoding video frames; its output is shared through the EncoderFrameOutputFD. The network transmission thread module runs as an independent thread, polling the EncoderFrameOutputFD to obtain encoded data for transmission, and generating error notification signals by writing to the Network Error Event File Descriptor (NetworkErrorEventFD) when a transmission error is detected. The system's operating logic is as follows: during normal operation, the modules work asynchronously and collaboratively through file descriptors; once the capture thread module receives an error notification signal through the FD, it can dynamically destroy and rebuild the network output thread module, thereby achieving rapid self-recovery of the system.

[0056] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A method for removing video delay based on multi-thread optimization in a vehicle, characterized in that, The method comprises the following steps: The original video frame is obtained by a video capture thread, and the original video frame is encoded by an encoder, specifically including: the video capture thread performs screen capture by calling a screen capture tool to obtain the original video frame; the original video frame data is preprocessed by color space conversion through a color converter, and then the original video frame after preprocessing is encoded by the encoder; The network transmission operation of the encoded data is performed by an independent network output thread, including the following steps: the network output thread continuously polls the output of the encoder to obtain the available data after encoding, and sends the data to a target device after adding a timestamp; the network transmission operation of the encoded data includes the following steps: the encoded data is transmitted from the encoder to the network output thread, and asynchronous communication is performed through a shared file descriptor established by the video capture thread at the initial time; The video capture thread monitors whether an error notification signal indicating network transmission is received during running; If not, the video capture thread and the network output thread continue to perform video capture, encoding and transmission operations; If yes, the video capture thread actively destroys the currently running network output thread, and creates and starts a new network output thread to restore data transmission; The error notification signal is generated by the network output thread when a transmission error is detected, and is transmitted to the video capture thread through a preset network output error event file descriptor; The video capture thread actively destroys the currently running network output thread, and creates and starts a new network output thread to restore data transmission, specifically including the following steps: pause the current video capture loop; destroy the network output thread instance that has an error and release its related resources; reinitialize network connection parameters and create a new network output thread instance; resume the video capture loop and transfer subsequent data to the new network output thread instance for transmission.

2. The in-vehicle video delay cancellation method of claim 1, wherein, The network output thread adjusts the data transmission strategy according to the current network condition to ensure reliable transmission of data.

3. The in-vehicle video delay cancellation method of claim 1, wherein, The monitoring whether an error notification signal indicating network transmission is received specifically includes: The video capture thread receives the error notification signal by monitoring the network output error event file descriptor.

4. A multi-threading optimization based in-vehicle video delay cancellation system, characterized by, The vehicle-mounted video delay elimination system for implementing the vehicle-mounted video delay elimination method based on multi-thread optimization according to any one of claims 1 to 3 comprises: A capture thread module for performing screen capture, managing encoding process, and receiving error notification signals; An encoder module in communication connection with the capture thread module for encoding video frames; A network output thread module independently running in communication connection with the encoder module for network transmission and generating error notification signals when detecting transmission errors; The capture thread module cooperates with the network output thread module when no error notification is received, and is capable of destroying and rebuilding the network output thread module when an error notification signal is received.

Images