Parallel driving low-latency streaming method and system based on delay estimation

Abstract

The present invention provides a parallel driving low-latency streaming method and system based on delay estimation, including: Step 1: deploying a camera on the vehicle side to collect audio and video data of the surrounding environment; Step 2: compressing the collected audio and video data and pushing it to the remote cab display; Step 3: initially allocating each stream based on the network bandwidth and the importance of the video stream, and reconfiguring the number of streams and the bit rate based on the media stream delay fed back by the simulated driving side and the stream priority; Step 4: deploying a signaling server and a streaming media server on the public network to establish a link between the vehicle side and the simulated driving side, forwarding the streaming media data pushed by the vehicle side, and being responsible for the interaction of flow control information between the vehicle side and the simulated driving side. The present invention directly estimates the delay on an end-to-end basis, and the server only forwards the media data. It can evaluate the network delay status on the entire link, making the network delay judgment of the entire system more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of parallel driving of sanitation vehicles, and in particular to a low-latency streaming method and system for parallel driving based on delay estimation. Background Art

[0002] The idea for parallel driving was initially conceived in the mid-1990s by Professor Wang Feiyue, then director of the Robotics Laboratory at the University of Arizona. The goal was to leverage the concept of "local simplicity, remote complexity" by leveraging agents to fully utilize autonomous vehicle technology, transforming it into an online "software robot" system to assist human drivers. In 2005, Professor Wang formally proposed the concept of parallel driving at the IEEE International Conference on Automotive Electronics and Safety (ICVES) in Xi'an. This application of the concept of virtual-reality interaction between artificial and real systems to the driving field formed the prototype of today's parallel driving theory.

[0003] In recent years, with the development of technologies such as the internet, big data, cloud computing, the Internet of Things, and artificial intelligence, data has become a crucial resource. Software-defined components, processes, and systems have gradually gained traction, further driving the development of parallel driving. Against this backdrop, parallel driving technology has garnered significant industry attention. Simply put, the concept of parallel driving involves designing a virtual vehicle in cyberspace, transforming it into an intelligent system that assists autonomous vehicle operation, with virtual and real-world systems running parallel to each other. This allows simple decision-making to be handled "inside the vehicle," while complex, computationally intensive tasks are handled "outside the vehicle." This allows limited resources to be shared and integrated, enabling vehicle-road interaction, multi-vehicle collaboration, parallel operation, and safe driving. Regardless of the application scenario for these unmanned vehicles, remote control, virtual-reality integration, and human-vehicle collaboration are the core elements of parallel driving. These three elements require low-latency, real-time road imagery transmission to ensure smooth autonomous driving and control. To ensure safe, reliable, and stable operation in parallel driving scenarios, major manufacturers are actively developing parallel driving solutions that provide highly reliable, low-latency, real-time messaging, audio, and video services for remote unmanned vehicles.

[0004] Patent document CN113079347A (application number: CN202110275338.8) discloses a method and device for implementing remote driving, which includes: encoding and packaging the currently collected video data according to the default bit rate; uploading the encoded and packaged video data to the server; obtaining the packet loss rate and delay sent by the server; obtaining the rules and dynamic bit rate of the intelligent hybrid automatic retransmission request based on the packet loss rate and delay; encoding and packaging the currently collected video data according to the current dynamic bit rate and the rules of the intelligent hybrid automatic retransmission request, and uploading it to the server. However, this patent does not evaluate the bandwidth and network conditions of the server and the stream pulling end, cannot achieve end-to-end bit rate evaluation, and there is a problem of inaccurate delay calculation. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a low-latency streaming method and system for parallel driving based on delay estimation.

[0006] In a first aspect, the present invention provides a low-latency streaming method for parallel driving based on delay estimation, comprising:

[0007] Step 1: Deploy multiple cameras on the vehicle to collect audio and video data of the surrounding environment;

[0008] Step 2: The collected audio and video data is compressed by the codec module of the GPU in the vehicle controller, and then the compressed audio and video data is pushed to the remote cockpit display via streaming media using network equipment;

[0009] Step 3: Initially allocate each stream based on network bandwidth and the importance of the video stream. Based on the media stream delay reported by the simulated driver and the stream priority, the number of streams and bitrate are reconfigured.

[0010] Step 4: Deploy a signaling server and a streaming media server on the public network. The signaling server is used to establish a link between the vehicle and the driving simulator, including controlling signaling forwarding, vehicle-side information reporting, and streaming media distribution. The streaming media server is responsible for forwarding streaming media data pushed by the vehicle and for exchanging flow control information between the vehicle and the driving simulator.

[0011] The vehicle-side information includes gear position, vehicle battery level, speed and lighting status;

[0012] The simulated driving end is responsible for collecting information from the simulated driving equipment, including the throttle, brake, steering wheel, light control, and operation buttons, and is responsible for sending control instructions from the driving end and receiving and displaying information on the vehicle end; according to the information from the signaling server, it pulls the media stream on the streaming media server, generates a delay estimation data packet for the corresponding streaming media, and feeds it back to the vehicle end that pushes the stream through the streaming media server.

[0013] Preferably, the vehicle side sends an RTSP request, the streaming media server forwards the data packet allocated by the GPU to the simulated driving side via the RTP protocol via the RTSP protocol, and then the simulated driving side transmits it to the vehicle side;

[0014] During an RTP session, each connector monitors the performance of the underlying network and places relevant information into RTCP packets, periodically transmitting RTCP packets to notify the sender. The sender uses RTCP packets to provide information about each session. The packet contains the number of data packets sent and the number of data packets lost. The server uses this information to dynamically change the transmission rate and payload type.

[0015] Preferably, the delay estimation is specifically as follows: the transmitting end sends data packets at a fixed frame rate, the receiving end uses the first received data packet as a reference, and then subtracts the difference between the time of the next received data packet and the time of the last received data packet as the basis for determining the delay estimation;

[0016] The delay is estimated using the time of n GOPs as Te, the time to receive the first data packet of the first I frame as Tes, and the time to receive the first data packet of the nth I frame as Tee. Then, Te = Tee - Tes. The parameter time is recorded as Tr, where Tr = n*GOP / 20. The evaluation reference formula is: の = |(Te-Tr) / Tr|. When の is greater than the set threshold, the bit rate is adjusted to reduce the bit rate based on the current bit rate evaluation, and the bit rate during video transmission is limited.

[0017] Preferably, among the cameras deployed on the vehicle side, the priority of the front-view camera is set to 2, the priority of the surround-view camera is set to 1, and the priority of the left and right front-view cameras is set to 0. The higher the priority, the more important it is. When the vehicle status and network bandwidth are not sufficient to meet the push of all streams, the push of the lower priority bitrate is stopped first.

[0018] Preferably, the end-to-end bitrate evaluation is split, firstly the bitrate estimation is performed on the push end and the server separately, and then the bitrate adjustment is performed on the push end; at the same time, the bitrate estimation is performed on the server and the pull end separately, and then the encoding and decoding bitrate adjustment is performed during the server transfer, so as to evaluate the bitrate on the entire link.

[0019] In a second aspect, the present invention provides a parallel driving low-latency streaming system based on delay estimation, comprising:

[0020] Module M1: Multiple cameras are deployed on the vehicle to collect audio and video data of the surrounding environment;

[0021] Module M2: compresses the collected audio and video data through the codec module of the GPU in the vehicle controller, and then uses network equipment to push the compressed audio and video data to the remote cab display through streaming media;

[0022] Module M3: Initially allocates each stream based on network bandwidth and the importance of the video stream. Based on the media stream delay reported by the simulated driver, the number of streams and bitrate are reconfigured in combination with the stream priority.

[0023] Module M4: Deploy a signaling server and a streaming media server on the public network. The signaling server is used to establish a link between the vehicle and the driving simulator, including controlling signaling forwarding, vehicle-side information reporting, and streaming media distribution. The streaming media server is responsible for forwarding streaming media data pushed by the vehicle and for exchanging flow control information between the vehicle and the driving simulator.

[0024] The vehicle-side information includes gear position, vehicle battery level, speed and lighting status;

[0025] The simulated driving end is responsible for collecting information from the simulated driving equipment, including the throttle, brake, steering wheel, light control, and operation buttons, and is responsible for sending control instructions from the driving end and receiving and displaying information on the vehicle end; according to the information from the signaling server, it pulls the media stream on the streaming media server, generates a delay estimation data packet for the corresponding streaming media, and feeds it back to the vehicle end that pushes the stream through the streaming media server.

[0026] Preferably, the vehicle side sends an RTSP request, the streaming media server forwards the data packet allocated by the GPU to the simulated driving side via the RTP protocol via the RTSP protocol, and then the simulated driving side transmits it to the vehicle side;

[0027] During an RTP session, each connector monitors the performance of the underlying network and places relevant information into RTCP packets, periodically transmitting RTCP packets to notify the sender. The sender uses RTCP packets to provide information about each session. The packet contains the number of data packets sent and the number of data packets lost. The server uses this information to dynamically change the transmission rate and payload type.

[0028] Preferably, the delay estimation is specifically as follows: the transmitting end sends data packets at a fixed frame rate, the receiving end uses the first received data packet as a reference, and then subtracts the difference between the time of the next received data packet and the time of the last received data packet as the basis for determining the delay estimation;

[0029] The delay is estimated using the time of n GOPs as Te, the time to receive the first data packet of the first I frame as Tes, and the time to receive the first data packet of the nth I frame as Tee. Then, Te = Tee - Tes. The parameter time is recorded as Tr, where Tr = n*GOP / 20. The evaluation reference formula is: の = |(Te-Tr) / Tr|. When の is greater than the set threshold, the bit rate is adjusted to reduce the bit rate based on the current bit rate evaluation, and the bit rate during video transmission is limited.

[0030] Preferably, among the cameras deployed on the vehicle side, the priority of the front-view camera is set to 2, the priority of the surround-view camera is set to 1, and the priority of the left and right front-view cameras is set to 0. The higher the priority, the more important it is. When the vehicle status and network bandwidth are not sufficient to meet the push of all streams, the push of the lower priority bitrate is stopped first.

[0031] Preferably, the end-to-end bitrate evaluation is split, firstly the bitrate estimation is performed on the push end and the server separately, and then the bitrate adjustment is performed on the push end; at the same time, the bitrate estimation is performed on the server and the pull end separately, and then the encoding and decoding bitrate adjustment is performed during the server transfer, so as to evaluate the bitrate on the entire link.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) The present invention directly estimates the delay on an end-to-end basis. The server only forwards media data. This allows the network delay status of the entire link to be assessed, making the network delay judgment of the entire system more accurate.

[0034] (2) The present invention uses relative time gradients to estimate delay, so that the delay estimation of each terminal no longer needs to rely on a unified time reference, circumventing the limitations of inaccurate network time and the need for network time synchronization for each terminal, and thus has wider applications.

[0035] (3) The present invention sorts the cameras according to their importance, giving priority to successfully pushing high-priority images, thereby improving the driving experience and the safety of parallel driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0037] Figure 1 This is a flow chart of the overall solution of the present invention;

[0038] Figure 2 This is the architecture diagram of the vehicle side, driving side and server side. DETAILED DESCRIPTION

[0039] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0040] Example 1:

[0041] The present invention proposes a low-latency streaming method for parallel driving based on delay estimation, such as Figure 1 Shown, including:

[0042] Deploy multiple cameras (preferably 4 surround view + 3 front view) on the vehicle to collect audio and video data of the surrounding environment;

[0043] The collected audio and video data is compressed (for example, using H265 encoding compression) by the codec module of the GPU in the ADU (vehicle controller), and then pushed to the remote cab display via streaming media using network equipment. On the one hand, each bitstream is initially allocated based on the network bandwidth and the importance of the video stream. On the other hand, the number of bitstreams and the bitrate are reconfigured based on the media stream delay fed back by the simulated driving end and the bitstream priority.

[0044] like Figure 2 As shown, a signaling server and a streaming media server are deployed on the public network. The signaling server is used to establish a link between the vehicle side and the simulated driving side, including controlling signaling forwarding and vehicle side information reporting, as well as streaming media distribution; the streaming media server is responsible for forwarding streaming media data pushed by the vehicle side, and is responsible for the interaction of flow control information between the vehicle side and the simulated driving side.

[0045] Vehicle-side information reporting includes gear position, vehicle battery level, speed, lighting status, etc.

[0046] Simulated driving terminal: responsible for collecting information from simulated driving equipment, including accelerator, brake, steering wheel, light control, operation buttons, etc., responsible for sending control instructions from the driving terminal (sent through control signaling) and receiving and displaying information on the vehicle side; according to the signaling server information, it pulls the media stream on the streaming media server, generates the corresponding streaming media delay estimation data packet and feeds it back to the vehicle side through the streaming media server.

[0047] Management platform / person: used for vehicle and driver management, including registration, deregistration, basic information entry, query, modification, deletion, vehicle and driver binding (for example, a single driver can be bound to up to 5 vehicles) and unbinding, etc.

[0048] First, the vehicle sends an RTSP request. The streaming server forwards the data packet allocated by the GPU to the simulated driving terminal via the RTP protocol through the RTSP protocol, and then the simulated driving terminal transmits it to the vehicle.

[0049] RTP operates in one-to-one or one-to-many transmission scenarios, providing time information and stream synchronization. RTP itself cannot provide a reliable delivery mechanism for sequential data packets; it relies on RTCP to provide flow control and congestion control services. During an RTP session, each connected peer monitors the performance of the underlying network and embeds this information in RTCP packets, which are periodically transmitted to notify the sender. The sender can also use RTCP packets to provide session-by-session information, including statistics such as the number of packets sent and the number of packets lost. The server can use this information to dynamically adjust the transmission rate and even the payload type. The combination of RTP and RTCP optimizes transmission efficiency through effective feedback and minimal overhead.

[0050] The streaming media server only forwards data between the vehicle and the simulated driving end, and does not involve functions such as encoding and decoding and bit rate control. The delay estimation and bit rate estimation are both completed on the simulated driving end. The simulated driving end will count the data received per unit time as the bit rate estimate on the link.

[0051] The time delay estimation method is as follows: the transmitter sends data packets at a fixed frame rate, such as 20fps. The receiver uses the first received data packet as the benchmark (which is reset when the link is reestablished and after the cycle time). The delay estimation is based on the difference between the time of the next received data packet and the time of the last received data packet.

[0052] The sending period is mainly calculated based on the period of the image frame GOP. For example, if the GOP is 30, then IP...P{29 P frames}IP...P.

[0053] Latency is estimated using the time taken to receive the first packet of the first I-frame, Tes, and the time taken to receive the first packet of the nth I-frame, Tee. This means: Te = Tee - Tes. The parameter time is denoted as Tr, where Tr = n*GOP / 20. The reference evaluation formula is: の = |Te - Tr / Tr|. A value greater than 20% indicates high latency. 20% is a set threshold that can be adjusted. If exceeded, the bitrate is adjusted to reduce based on the current bitrate assessment. The bitrate is also continuously evaluated during video transmission and is subject to certain limits, such as a maximum bitrate of 2Mbps.

[0054] In the default settings, the priority of the front view camera is 2, the priority of the surround view camera is 1, and the priority of the left and right front view cameras is 0; the higher the priority, the more important it is. When the vehicle status and network bandwidth are insufficient to meet the push of all streams, the push of the lower priority bitrate will be stopped first.

[0055] The end-to-end bitrate evaluation is split into two parts: first, the bitrate of the push end and the server is estimated separately, and then the bitrate of the push end is adjusted. At the same time, the bitrate of the server and the pull end is estimated separately, and then the encoding and decoding bitrate are adjusted during the server transfer, thereby evaluating the bitrate of the entire link.

[0056] Example 2:

[0057] The present invention also provides a parallel driving low-latency streaming system based on delay estimation. The parallel driving low-latency streaming system based on delay estimation can be implemented by executing the process steps of the parallel driving low-latency streaming method based on delay estimation. That is, those skilled in the art can understand the parallel driving low-latency streaming method based on delay estimation as a preferred implementation of the parallel driving low-latency streaming system based on delay estimation.

[0058] The parallel driving low-latency streaming system based on delay estimation provided by the present invention includes: module M1: deploying multiple cameras on the vehicle side to collect audio and video data of the surrounding environment; module M2: compressing the collected audio and video data through the codec module of the GPU in the vehicle controller, and then using the network equipment to push the compressed audio and video data to the remote cab display through streaming media; module M3: initially allocating each stream according to the network bandwidth and the importance of the video stream, and reconfiguring the number of streams and the bit rate according to the media stream delay fed back by the simulated driving side and the stream priority; module M4: deploying a signaling server and a streaming media server on the public network, wherein the signaling server is used to establish a link between the vehicle side and the simulated driving side, including controlling signaling forwarding and vehicle side information reporting, as well as streaming media distribution; the streaming media server is responsible for forwarding the streaming media data pushed by the vehicle side, and is responsible for the interaction of flow control information between the vehicle side and the simulated driving side;

[0059] The vehicle-side information includes gear position, vehicle battery level, speed and lighting status;

[0060] The simulated driving end is responsible for collecting information from the simulated driving equipment, including the throttle, brake, steering wheel, light control, and operation buttons, and is responsible for sending control instructions from the driving end and receiving and displaying information on the vehicle end; according to the information from the signaling server, it pulls the media stream on the streaming media server, generates a delay estimation data packet for the corresponding streaming media, and feeds it back to the vehicle end that pushes the stream through the streaming media server.

[0061] Preferably, the vehicle side sends an RTSP request, the streaming media server forwards the data packet allocated by the GPU to the simulated driving side via the RTP protocol via the RTSP protocol, and then the simulated driving side transmits it to the vehicle side;

[0062] During an RTP session, each connector monitors the performance of the underlying network and places relevant information into RTCP packets, periodically transmitting RTCP packets to notify the sender. The sender uses RTCP packets to provide information about each session. The packet contains the number of data packets sent and the number of data packets lost. The server uses this information to dynamically change the transmission rate and payload type.

[0063] Preferably, the delay estimation is specifically as follows: the transmitting end sends data packets at a fixed frame rate, the receiving end uses the first received data packet as a reference, and then subtracts the difference between the time of the next received data packet and the time of the last received data packet as the basis for determining the delay estimation;

[0064] The delay is estimated using the time of n GOPs as Te, the time to receive the first data packet of the first I frame as Tes, and the time to receive the first data packet of the nth I frame as Tee. Then, Te = Tee - Tes. The parameter time is recorded as Tr, where Tr = n*GOP / 20. The evaluation reference formula is: の = |(Te-Tr) / Tr|. When の is greater than the set threshold, the bit rate is adjusted to reduce the bit rate based on the current bit rate evaluation, and the bit rate during video transmission is limited.

[0065] Preferably, among the cameras deployed on the vehicle side, the priority of the front-view camera is set to 2, the priority of the surround-view camera is set to 1, and the priority of the left and right front-view cameras is set to 0. The higher the priority, the more important it is. When the vehicle status and network bandwidth are not sufficient to meet the push of all streams, the push of the lower priority bitrate is stopped first.

[0066] Preferably, the end-to-end bitrate evaluation is split, firstly the bitrate estimation is performed on the push end and the server separately, and then the bitrate adjustment is performed on the push end; at the same time, the bitrate estimation is performed on the server and the pull end separately, and then the encoding and decoding bitrate adjustment is performed during the server transfer, so as to evaluate the bitrate on the entire link.

[0067] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0068] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A low-latency streaming method for parallel driving based on delay estimation, characterized in that: include: Step 1: Deploy multiple cameras on the vehicle to collect audio and video data of the surrounding environment; Step 2: The collected audio and video data is compressed by the codec module of the GPU in the vehicle controller, and then the compressed audio and video data is pushed to the remote cockpit display via streaming media using network equipment; Step 3: Initially allocate each stream based on network bandwidth and the importance of the video stream. Based on the media stream delay reported by the simulated driver and the stream priority, the number of streams and bitrate are reconfigured. Step 4: Deploy a signaling server and a streaming media server on the public network. The signaling server is used to establish a link between the vehicle and the simulated driving terminal, including controlling signaling forwarding, vehicle-side information reporting, and streaming media distribution. The streaming media server is responsible for forwarding the streaming media data pushed by the vehicle side, and is responsible for the interaction of flow control information between the vehicle side and the simulated driving side; The vehicle-side information includes gear position, vehicle battery level, speed and lighting status; The simulated driving terminal is responsible for collecting information from the simulated driving equipment, including the accelerator, brake, steering wheel, light control, and operation buttons, and is responsible for sending control commands from the driver end and receiving and displaying information on the vehicle end. It pulls the media stream from the streaming server based on the signaling server information, generates a delay estimation data packet for the corresponding streaming media, and feeds it back to the vehicle end through the streaming server. Specifically, the delay estimation is as follows: the transmitting end sends data packets at a fixed frame rate, and the receiving end uses the first received data packet as a benchmark, and then subtracts the difference between the time of the next received data packet and the last received data packet as the basis for determining the delay estimation; The delay is estimated using the time of n GOPs, denoted as Te. The time it takes to receive the first packet of the first I frame is denoted as Tes, and the time it takes to receive the first packet of the nth I frame is denoted as Tee. Thus, Te = Tee - Tes. The parameter time is denoted as Tr, where Tr = n*GOP / 20. The reference evaluation formula is: の = |(Te - Tr) / Tr|. When の is greater than the set threshold, the bitrate is adjusted to reduce the bitrate based on the current bitrate assessment, while also limiting the bitrate during video transmission. For cameras deployed on the vehicle, set the priority of the front-view camera to 2, the priority of the surround-view camera to 1, and the priority of the left and right front-view cameras to 0. The higher the priority, the more important it is. When the vehicle status and network bandwidth are insufficient to push all streams, the lower-priority bitrate push is stopped first. The end-to-end bitrate evaluation is split into two parts: first, the bitrate of the push end and the server is estimated separately, and then the bitrate of the push end is adjusted. At the same time, the bitrate of the server and the pull end is estimated separately, and then the encoding and decoding bitrate are adjusted during the server transfer, thereby evaluating the bitrate of the entire link.

2. The parallel driving low-latency streaming method based on delay estimation according to claim 1 is characterized in that: The vehicle sends an RTSP request, and the streaming server forwards the data packet allocated by the GPU to the simulated driving terminal via the RTP protocol through the RTSP protocol, and then the simulated driving terminal transmits it to the vehicle. During an RTP session, each connector monitors the performance of the underlying network and places relevant information into RTCP packets, periodically transmitting RTCP packets to notify the sender. The sender uses RTCP packets to provide information about each session. The packet contains the number of data packets sent and the number of data packets lost. The server uses this information to dynamically change the transmission rate and payload type.

3. A parallel driving low-latency streaming system based on delay estimation, characterized in that: include: Module M1: Multiple cameras are deployed on the vehicle to collect audio and video data of the surrounding environment; Module M2: compresses the collected audio and video data through the codec module of the GPU in the vehicle controller, and then uses network equipment to push the compressed audio and video data to the remote cab display through streaming media; Module M3: Initially allocates each stream based on network bandwidth and the importance of the video stream. Based on the media stream delay reported by the simulated driver, the number of streams and bitrate are reconfigured in combination with the stream priority. Module M4: Deploy a signaling server and a streaming media server on the public network. The signaling server is used to establish a link between the vehicle and the simulated driving terminal, including controlling signaling forwarding and vehicle-side information reporting, as well as distributing streaming media. The streaming media server is responsible for forwarding the streaming media data pushed by the vehicle side, and is responsible for the interaction of flow control information between the vehicle side and the simulated driving side; The vehicle-side information includes gear position, vehicle battery level, speed and lighting status; The simulated driving terminal is responsible for collecting information from the simulated driving equipment, including the accelerator, brake, steering wheel, light control, and operation buttons, and is responsible for sending control commands from the driver end and receiving and displaying information on the vehicle end. It pulls the media stream from the streaming server based on the signaling server information, generates a delay estimation data packet for the corresponding streaming media, and feeds it back to the vehicle end through the streaming server. Specifically, the delay estimation is as follows: the transmitting end sends data packets at a fixed frame rate, and the receiving end uses the first received data packet as a benchmark, and then subtracts the difference between the time of the next received data packet and the last received data packet as the basis for determining the delay estimation; The delay is estimated using the time of n GOPs, denoted as Te. The time it takes to receive the first packet of the first I frame is denoted as Tes, and the time it takes to receive the first packet of the nth I frame is denoted as Tee. Thus, Te = Tee - Tes. The parameter time is denoted as Tr, where Tr = n*GOP / 20. The reference evaluation formula is: の = |(Te - Tr) / Tr|. When の is greater than the set threshold, the bitrate is adjusted to reduce the bitrate based on the current bitrate assessment, and the bitrate during video transmission is also limited. For cameras deployed on the vehicle, set the priority of the front-view camera to 2, the priority of the surround-view camera to 1, and the priority of the left and right front-view cameras to 0. The higher the priority, the more important it is. When the vehicle status and network bandwidth are insufficient to push all streams, the lower-priority bitrate push is stopped first. The end-to-end bitrate evaluation is split into two parts: first, the bitrate of the push end and the server is estimated separately, and then the bitrate of the push end is adjusted. At the same time, the bitrate of the server and the pull end is estimated separately, and then the encoding and decoding bitrate are adjusted during the server transfer, thereby evaluating the bitrate of the entire link.

4. The parallel driving low-latency streaming system based on delay estimation according to claim 3 is characterized in that: The vehicle sends an RTSP request, and the streaming server forwards the data packet allocated by the GPU to the simulated driving terminal via the RTP protocol through the RTSP protocol, and then the simulated driving terminal transmits it to the vehicle. During an RTP session, each connector monitors the performance of the underlying network and places relevant information into RTCP packets, periodically transmitting RTCP packets to notify the sender. The sender uses RTCP packets to provide information about each session. The packet contains the number of data packets sent and the number of data packets lost. The server uses this information to dynamically change the transmission rate and payload type.

Images