A full-link image transmission delay test method and system for remote control driving system

Abstract

The present invention discloses a full-link image transmission delay test method and system for a remote control driving system, including: synchronizing the time base of the vehicle and the remote cockpit; obtaining the image pixel timestamp data reading delay; generating vehicle camera environment perception simulation data, injecting the first timestamp; creating a TCP message and injecting the first timestamp, correcting the second timestamp; injecting the corrected third timestamp and sending it to the cloud server; the remote cockpit receives data and injects the corrected fourth timestamp; sending the TCP message with timestamp data to the smart camera, and injecting the corrected fifth timestamp; the smart camera synchronizes with the display frequency, captures the display image and records the sixth timestamp; the computer verifies the TCP message data and calculates the difference between the timestamps to obtain the delay. The present invention can accurately measure the transmission delay of the entire transmission link of the image data from the vehicle to the remote cockpit, ensuring the reliability and safety of the remote driving system in various environments and scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of data transmission delay testing, and in particular relates to a full-link image transmission delay testing method and system for a remote control driving system. Background Art

[0002] With the development of remote driving technology, the real-time transmission of data has become a key factor affecting driving safety and operational response speed. In the remote driving system, image data must be transmitted from the vehicle to the remote cockpit in real time and displayed to the driver quickly. Any transmission delay may affect the driver's decision-making and operating efficiency, and thus affect the driving safety of the vehicle. It is crucial to accurately measure and evaluate the transmission delay of image data. At present, the test technology for remote driving image transmission delay is fragmented, that is, it only covers part of the transmission link, and lacks a comprehensive test method for the entire transmission link. This incomplete test coverage may lead to blind spots in the evaluation of system performance and stability. Further research and development of full-link test methods are needed to ensure the reliability and safety of remote driving systems in various environments and scenarios.

[0003] Existing image transmission delay test technologies mainly focus on delay tests between remote cockpits, cloud servers, and vehicles, and evaluate delays by recording and comparing the transmission time of data packets. However, these methods can often only cover part of the transmission link and cannot fully reflect the delay of the entire process from image data generation to display. In addition, the existing time synchronization methods are not accurate enough in complex network environments, resulting in inaccurate delay measurement results. To address these issues, remote driving systems require a high-precision delay test method that covers the entire transmission link to ensure the real-time and reliability of the system. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the purpose of the present invention is to provide a full-link image transmission delay testing method and system for a remote-controlled driving system, so as to solve the problem that the time synchronization method in the existing remote driving system is insufficiently accurate in a complex network environment, resulting in inaccurate delay measurement results.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A full-link image transmission delay test method for a remote control driving system of the present invention is applied to a remote driving scenario of a vehicle, wherein the scenario includes a vehicle, a cloud server, a remote cockpit, a display screen, and a computer. The cloud server is respectively connected to the vehicle and the remote cockpit through a network, the remote cockpit is connected to the display screen through a data line, and a smart camera is provided in front of the display screen for photographing the display screen, and the smart camera data is connected to the computer. The method steps are as follows:

[0007] 1) Synchronize the time base of the vehicle and the remote cockpit, and obtain the image pixel timestamp data reading delay of the vehicle and the remote cockpit;

[0008] 2) Generate vehicle camera environment perception simulation data, inject a first timestamp into the image and pixel data of the simulation data respectively, and then send the image data to the vehicle data receiving interface;

[0009] 3) The vehicle data receiving interface receives the image data with the first timestamp. When the vehicle receives the image data, it creates a TCP message, injects the first timestamp into the TCP message, and corrects the second timestamp.

[0010] 4) Transmitting the TCP message with the first timestamp, the corrected second timestamp and the image data to the vehicle data transmission interface, and the vehicle data transmission interface injects the corrected third timestamp into the TCP message when receiving the image data;

[0011] 5) Send the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp and the image data to the cloud server;

[0012] 6) The cloud server sends the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp and the image data to the remote cockpit. When the remote cockpit receives the image data, it injects the corrected fourth timestamp into the TCP message;

[0013] 7) The remote cockpit sends a TCP message with a first timestamp, a corrected second timestamp, a corrected third timestamp, and a corrected fourth timestamp to the smart camera, and injects the corrected fifth timestamp into the TCP message at the time of sending, and sends the image data to the display screen for display;

[0014] 8) Synchronize the frequency of the smart camera and the display screen. The smart camera receives the TCP message, captures the display screen image and records the sixth timestamp of the current moment when capturing. It recognizes the first timestamp in the image and injects the sixth timestamp into the corresponding TCP message and then sends it to the computer.

[0015] 9) The computer calculates the difference between the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to obtain each part of the delay, and calculates the difference between the first timestamp and the sixth timestamp to obtain the total delay value.

[0016] Furthermore, the step 1) specifically includes:

[0017] 11) Configuration of remote cockpit GPS timing equipment;

[0018] The GPS timing device installed on the remote cockpit obtains time information by receiving GPS signals, and synchronizes the time information to the remote cockpit via the Ethernet interface in the form of gptp, so that all devices in the remote cockpit refer to the same time standard;

[0019] 12) Configuration of vehicle GPS timing equipment;

[0020] The GPS timing device installed on the vehicle converts the received GPS signal into the master clock signal of the vehicle system, thereby achieving time synchronization in the entire vehicle system;

[0021] 13) Obtain image reading delay;

[0022] An image with the same format and size as the output image of the video injection device is stored in the vehicle and the remote cockpit respectively. Before the test starts, the vehicle and the remote cockpit read the stored image respectively, obtain timestamp A when the reading starts, and obtain timestamp B when the reading is completed. The total time of reading the image is obtained by subtracting timestamp A from timestamp B, and then the time of reading 64 bits of image data is calculated according to the image size. The time for the vehicle to read 64 bits of image data is recorded as t64, and the time for the remote cockpit to read 64 bits of image data is recorded as t'64. The time of reading 64 bits of image data = total time of reading the image × 64 / the number of all image data bits.

[0023] Furthermore, the step 2) specifically includes:

[0024] 21) Generation of environmental perception simulation data;

[0025] In the scene simulation software, by simulating the characteristics of lighting and object reflection in the actual scene, environmental perception simulation data that is highly similar to the images captured by the actual camera is generated;

[0026] 22) Transmission and processing of environmental perception simulation data;

[0027] The generated environmental perception simulation data is transmitted to the video injection device through the PCIe bus. The video injection device obtains the time information of the vehicle's GPS timing device, decodes the received environmental perception simulation data, encodes the image data according to the data format received by the vehicle, and embeds the first timestamp T1 of the current moment into the image pixel when the encoding is completed (replace the pixel value of the first 8 bytes of the image data with the value of the first timestamp T1, and write it to the corresponding position in hexadecimal). At the same time, the first timestamp T1 is superimposed on the original image in the form of a string image and sent to the vehicle data receiving interface.

[0028] Furthermore, the first timestamp T1 is the moment when the video injection device generates all the image data.

[0029] Furthermore, the step 3) specifically includes:

[0030] The vehicle data receiving interface receives image data with a first timestamp T1, reads the first 8 bytes of pixel values ​​of the image data to obtain the first timestamp T1, records the current second timestamp T2 after reading the first timestamp T1, calculates and corrects the second timestamp t2=T2-t64, and creates a TCP message. The TCP message includes an identification bit, a check bit and a data bit, and the subsequent TCP message format is consistent with it; the identification bit uses the first timestamp T1 value as an identification bit, writes the first timestamp T1 and the corrected second timestamp t2 into the TCP message data bit in sequence, and stores the value obtained by t2-T1 into the check bit as verification data.

[0031] Furthermore, the second timestamp T2 is the time when the vehicle data receiving interface completes reading the first timestamp T1.

[0032] Furthermore, the corrected second timestamp t2 is the moment when the vehicle data receiving interface receives the image data.

[0033] Furthermore, the step 4) specifically includes:

[0034] The vehicle data sending interface receives a TCP message and image data with a first timestamp and a corrected second timestamp. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. At the end of the reading, the third timestamp T3 at the current moment is recorded, and the corrected third timestamp t3=T3-t64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected third timestamp t3 is added to the first timestamp T1 and the corrected second timestamp t2 in the TCP message, and the value of the check bit is added to the corrected third timestamp t3.

[0035] Furthermore, the third timestamp T3 is the time when the vehicle data sending interface completes reading the first timestamp T1.

[0036] Furthermore, the corrected third timestamp t3 is the moment when the vehicle data sending interface receives the image data.

[0037] Furthermore, the step 6) specifically includes:

[0038] The remote cockpit data receiving interface receives TCP messages and image data with a first timestamp, a corrected second timestamp, and a corrected third timestamp. When receiving image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. At the same time, the fourth timestamp T4 at the current moment is recorded after the reading is completed, and the corrected fourth timestamp t4=T4-t'64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected fourth timestamp t4 is added to the corrected third timestamp t3 in the TCP message, and the value of the check bit is subtracted from the corrected fourth timestamp t4.

[0039] Furthermore, the fourth timestamp T4 is the time when the remote cockpit data receiving interface completes reading the first timestamp T1.

[0040] Furthermore, the corrected fourth timestamp t4 is the moment when the remote cockpit data receiving interface receives the image data.

[0041] Furthermore, the step 7) specifically includes:

[0042] The remote cockpit data sending interface receives the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp and the received image data, and sends it to the display screen. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. After the reading is completed, the fifth timestamp T5 at the current moment is recorded, and the corrected fifth timestamp t5=T5-t'64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected fifth timestamp t5 is added to the corrected fourth timestamp t4 in the TCP message, and the value of the check bit is added to the corrected fifth timestamp t5, and the TCP message is sent to the smart camera.

[0043] Furthermore, the fifth timestamp T5 is the time when the remote cockpit data sending interface completes reading the first timestamp T1.

[0044] Furthermore, the corrected fifth timestamp t5 is the moment when the remote cockpit data sending interface receives the image data.

[0045] Furthermore, the step 8) specifically includes:

[0046] 81) Adjust the sampling frequency of the smart camera to synchronize with the refresh frequency of the display screen;

[0047] 82) The smart camera receives the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, and the corrected fifth timestamp, and captures the image on the display screen, obtains the first timestamp T1 printed on the image in the form of a character string image through image recognition, and generates a sixth timestamp T6 at the time of capturing, and after adding the sixth timestamp T6 to the corrected fifth timestamp t5 in the TCP message with the first timestamp T1, the value of the check digit is subtracted from the sixth timestamp T6;

[0048] 83) Outputting the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to the computer.

[0049] Furthermore, the step 9) specifically includes:

[0050] The computer extracts the data bits and check bits of the TCP message output by the smart camera, calculates the value of T1-t2+t3-t4+t5-T6 and compares it with the check bit value. If they are consistent, it means that the data is valid and the delay calculation can be performed. The matrix multiplication is used to calculate the delay of each part. The calculation formula is:

[0051] r=M*x

[0052] In the formula, x is a column vector, M is a transformation matrix, and r is an array of delay calculation results;

[0053]

[0054]

[0055] r[1]=-1×T1+1×t2=t2-T1

[0056] r[2]=-1×t2+1×t3=t3-t2

[0057] r[3]=-1×t3+1×t4=t4-t3

[0058] r[4]=-1×t4+1×t5=t5-t4

[0059] r[5]=-1×t5+1×T6=T6-t5

[0060] r[6]=1×T1+0×t2+0×t3+0×t4+0×t5-1×T6=T6-T1

[0061] Among them, r[1] represents the delay of image data transmission from the video injection device to the vehicle data receiving interface, r[2] represents the delay of the vehicle processing image data, r[3] represents the delay of image data from the vehicle to the remote cockpit through the cloud server, r[4] represents the delay of image data processing in the remote cockpit, r[5] represents the delay of image data display through the display screen, and r[6] represents the total delay of the entire link.

[0062] A full-link image transmission delay test system for a remote control driving system of the present invention comprises:

[0063] A time base synchronization module, used to synchronize the time bases of the vehicle and the remote cockpit;

[0064] Delayed reading module, used to obtain the image pixel timestamp data reading delay of the vehicle and remote cockpit;

[0065] A simulation data generation module, used for generating vehicle camera environment perception simulation data, injecting a first timestamp into the image and pixel data respectively, and then sending the image data to the vehicle data receiving interface;

[0066] A TCP message creation module, used for creating a TCP message when the vehicle receives image data with a first timestamp, injecting the first timestamp into the TCP message, and correcting the second timestamp;

[0067] A first timestamp injection module, configured to inject a corrected third timestamp into a TCP message when the vehicle data transmission interface receives image data with a first timestamp and a corrected second timestamp;

[0068] A second timestamp injection module, configured to inject a corrected fourth timestamp into a TCP message when the remote cockpit receives image data with the first timestamp, the corrected second timestamp, and the corrected third timestamp;

[0069] A third timestamp injection module is used to inject the corrected fifth timestamp into the TCP message when the remote cockpit sends the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, and the corrected fourth timestamp to the smart camera;

[0070] A frequency synchronization module is used to synchronize the frequency of the smart camera with the display screen;

[0071] A fourth timestamp injection module, used to identify the first timestamp in the display screen image captured by the smart camera, and inject the sixth timestamp of the shooting moment into the corresponding TCP message;

[0072] The delay calculation module is used to calculate the difference between the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to obtain each part of the delay, and calculate the difference between the first timestamp and the sixth timestamp to obtain the total delay value.

[0073] Beneficial effects of the present invention:

[0074] 1. The present invention uses video injection technology to inject pixel timestamps and character string timestamps of the generation time when generating image data, and can measure the time delay between the camera output image time and the vehicle data receiving interface receiving time.

[0075] 2. The present invention adopts the TCP synchronous transmission timestamp mechanism, and writes the timestamp operation in the TCP message after reading the image timestamp, so as to avoid writing the timestamp in the image data and causing an increase in the image data transmission delay.

[0076] 3. The present invention calculates the timestamp reading time by reading the image data in advance, and adds the timestamp reading time to the calculation in the subsequent full-link delay test, thereby reducing the error of the test result.

[0077] 4. The present invention uses the same frequency technology of the smart camera and the display screen to obtain the timestamp of the display screen display moment, ensuring that the full-link time delay of the remote driving image data transmission is effectively verified and the time when the human eye obtains the image is obtained.

[0078] 5. The TCP message delayed transmission in the present invention has an identification bit and a check bit to ensure data accuracy, and can identify bit errors in the data transmission process to avoid invalid data from participating in calculations.

[0079] 6. The present invention adopts a matrix multiplication calculation method to calculate all required delay data at one time, thereby reducing the number of analysis steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0081] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0082] Reference Figure 1As shown, a full-link image transmission delay test method for a remote control driving system of the present invention is applied to a remote driving scenario of a vehicle, wherein the scenario includes a vehicle, a cloud server, a remote cockpit, a display screen and a computer. The cloud server is respectively connected to the vehicle and the remote cockpit through a network, the remote cockpit is connected to the display screen through a data cable, a smart camera is provided in front of the display screen for shooting the display screen, and the smart camera data is connected to the computer; the method steps are as follows:

[0083] 1) Synchronize the time base of the vehicle and the remote cockpit, and obtain the image pixel timestamp data reading delay of the vehicle and the remote cockpit; specifically include:

[0084] 11) Configuration of remote cockpit GPS timing equipment;

[0085] The GPS timing device installed on the remote cockpit obtains time information by receiving GPS signals, and synchronizes the time information to the remote cockpit via the Ethernet interface in the form of gptp, so that all devices in the remote cockpit refer to the same time standard;

[0086] 12) Configuration of vehicle GPS timing equipment;

[0087] The GPS timing device installed on the vehicle converts the received GPS signal into the master clock signal of the vehicle system, thereby achieving time synchronization in the entire vehicle system;

[0088] 13) Obtain image reading delay;

[0089] An image with the same format and size as the output image of the video injection device is stored in the vehicle and the remote cockpit respectively. Before the test starts, the vehicle and the remote cockpit read the stored image respectively, obtain timestamp A when the reading starts, and obtain timestamp B when the reading is completed. The total time of reading the image is obtained by subtracting timestamp A from timestamp B, and then the time of reading 64 bits of image data is calculated according to the image size. The time for the vehicle to read 64 bits of image data is recorded as t64, and the time for the remote cockpit to read 64 bits of image data is recorded as t'64. The time of reading 64 bits of image data = total time of reading the image × 64 / the number of all image data bits.

[0090] 2) Generate vehicle camera environment perception simulation data, inject a first timestamp into the image and pixel data of the simulation data, and then send the image data to the vehicle data receiving interface; specifically including:

[0091] 21) Generation of environmental perception simulation data;

[0092] In the scene simulation software, by simulating the characteristics of lighting and object reflection in the actual scene, environmental perception simulation data that is highly similar to the images captured by the actual camera is generated;

[0093] 22) Transmission and processing of environmental perception simulation data;

[0094] The generated environment perception simulation data is transmitted to the video injection device through the PCIe bus. The video injection device obtains the time information of the vehicle's GPS timing device, decodes the received environment perception simulation data, encodes the image data according to the data format received by the vehicle, and embeds the first timestamp T1 of the current moment into the image pixel when the encoding is completed (replace the pixel value of the first 8 bytes of the image data with the value of the first timestamp T1, and write it to the corresponding position in hexadecimal), and at the same time superimposes the first timestamp T1 on the original image in the form of a string image and sends it to the vehicle data receiving interface;

[0095] The first timestamp T1 is the moment when the video injection device generates all image data.

[0096] 3) The vehicle data receiving interface receives the image data with the first timestamp. When the vehicle receives the image data, it creates a TCP message, injects the first timestamp into the TCP message, and corrects the second timestamp. Specifically, it includes:

[0097] The vehicle data receiving interface receives image data with a first timestamp T1, reads the first 8 bytes of pixel values ​​of the image data to obtain the first timestamp T1, records the current second timestamp T2 after reading the first timestamp T1, calculates and corrects the second timestamp t2=T2-t64, and creates a TCP message. The TCP message includes an identification bit, a check bit and a data bit, and the subsequent TCP message format is consistent with it; the identification bit uses the first timestamp T1 value as an identification bit, writes the first timestamp T1 and the corrected second timestamp t2 into the TCP message data bit in sequence, and stores the value obtained by t2-T1 into the check bit as verification data.

[0098] The second timestamp T2 is the time when the vehicle data receiving interface completes reading the first timestamp T1.

[0099] The corrected second timestamp t2 is the time when the vehicle data receiving interface receives the image data.

[0100] 4) Transmitting the TCP message with the first timestamp, the corrected second timestamp and the image data to the vehicle data transmission interface, and injecting the corrected third timestamp into the TCP message when the vehicle data transmission interface receives the image data; specifically including:

[0101] The vehicle data sending interface receives a TCP message and image data with a first timestamp and a corrected second timestamp. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. At the end of the reading, the third timestamp T3 at the current moment is recorded, and the corrected third timestamp t3=T3-t64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected third timestamp t3 is added to the first timestamp T1 and the corrected second timestamp t2 in the TCP message, and the value of the check bit is added to the corrected third timestamp t3.

[0102] The third timestamp T3 is the time when the vehicle data sending interface completes reading the first timestamp T1.

[0103] The corrected third timestamp t3 is the time when the vehicle data sending interface receives the image data.

[0104] 5) Send the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp and the image data to the cloud server;

[0105] 6) The cloud server sends the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp and the image data to the remote cockpit, and the remote cockpit injects the corrected fourth timestamp into the TCP message when receiving the image data; specifically including:

[0106] The remote cockpit data receiving interface receives TCP messages and image data with a first timestamp, a corrected second timestamp, and a corrected third timestamp. When receiving image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. At the same time, the fourth timestamp T4 at the current moment is recorded after the reading is completed, and the corrected fourth timestamp t4=T4-t'64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected fourth timestamp t4 is added to the corrected third timestamp t3 in the TCP message, and the value of the check bit is subtracted from the corrected fourth timestamp t4.

[0107] The fourth timestamp T4 is the time when the remote cockpit data receiving interface completes reading the first timestamp T1.

[0108] The corrected fourth timestamp t4 is the moment when the remote cockpit data receiving interface receives the image data.

[0109] 7) The remote cockpit sends a TCP message with a first timestamp, a corrected second timestamp, a corrected third timestamp, and a corrected fourth timestamp to the smart camera, and injects the corrected fifth timestamp into the TCP message at the time of sending, and sends the image data to the display screen for display; specifically including:

[0110] The remote cockpit data sending interface receives the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp and the received image data, and sends it to the display screen. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. After the reading is completed, the fifth timestamp T5 at the current moment is recorded, and the corrected fifth timestamp t5=T5-t'64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected fifth timestamp t5 is added to the corrected fourth timestamp t4 in the TCP message, and the value of the check bit is added to the corrected fifth timestamp t5, and the TCP message is sent to the smart camera.

[0111] The fifth timestamp T5 is the time when the remote cockpit data sending interface completes reading the first timestamp T1.

[0112] The corrected fifth timestamp t5 is the moment when the remote cockpit data sending interface receives the image data.

[0113] 8) Synchronize the frequency of the smart camera and the display screen, the smart camera receives the TCP message, shoots the display screen image and records the sixth time stamp of the current moment when shooting, identifies the first time stamp in the image and injects the sixth time stamp into the corresponding TCP message and then sends it to the computer; specifically includes:

[0114] 81) Adjust the sampling frequency of the smart camera to synchronize with the refresh frequency of the display screen;

[0115] 82) The smart camera receives the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, and the corrected fifth timestamp, and captures the image on the display screen, obtains the first timestamp T1 printed on the image in the form of a character string image through image recognition, and generates a sixth timestamp T6 at the time of capturing, and after adding the sixth timestamp T6 to the corrected fifth timestamp t5 in the TCP message with the first timestamp T1, the value of the check digit is subtracted from the sixth timestamp T6;

[0116] 83) Outputting the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to the computer.

[0117] 9) The computer calculates the difference between the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to obtain each part of the delay, and calculates the difference between the first timestamp and the sixth timestamp to obtain the total delay value; specifically including:

[0118] The computer extracts the data bits and check bits of the TCP message output by the smart camera, calculates the value of T1-t2+t3-t4+t5-T6 and compares it with the check bit value. If they are consistent, it means that the data is valid and the delay calculation can be performed. The matrix multiplication is used to calculate the delay of each part. The calculation formula is:

[0119] r=M*x

[0120] In the formula, x is a column vector, M is a transformation matrix, and r is an array of delay calculation results;

[0121]

[0122]

[0123] r[1]=-1×T1+1×t2=t2-T1

[0124] r[2]=-1×t2+1×t3=t3-t2

[0125] r[3]=-1×t3+1×t4=t4-t3

[0126] r[4]=-1×t4+1×t5=t5-t4

[0127] r[5]=-1×t5+1×T6=T6-t5

[0128] r[6]=1×T1+0×t2+0×t3+0×t4+0×t5-1×T6=T6-T1

[0129] Among them, r[1] represents the delay of image data transmission from the video injection device to the vehicle data receiving interface, r[2] represents the delay of the vehicle processing image data, r[3] represents the delay of image data from the vehicle to the remote cockpit through the cloud server, r[4] represents the delay of image data processing in the remote cockpit, r[5] represents the delay of image data display through the display screen, and r[6] represents the total delay of the entire link.

[0130] The present invention also provides a full-link image transmission delay test system for a remote control driving system, comprising:

[0131] A time base synchronization module, used to synchronize the time bases of the vehicle and the remote cockpit;

[0132] Delayed reading module, used to obtain the image pixel timestamp data reading delay of the vehicle and remote cockpit;

[0133] A simulation data generation module, used for generating vehicle camera environment perception simulation data, injecting a first timestamp into the image and pixel data respectively, and then sending the image data to the vehicle data receiving interface;

[0134] A TCP message creation module, used for creating a TCP message when the vehicle receives image data with a first timestamp, injecting the first timestamp into the TCP message, and correcting the second timestamp;

[0135] A first timestamp injection module, configured to inject a corrected third timestamp into a TCP message when the vehicle data transmission interface receives image data with a first timestamp and a corrected second timestamp;

[0136] A second timestamp injection module, configured to inject a corrected fourth timestamp into a TCP message when the remote cockpit receives image data with the first timestamp, the corrected second timestamp, and the corrected third timestamp;

[0137] A third timestamp injection module is used to inject the corrected fifth timestamp into the TCP message when the remote cockpit sends the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, and the corrected fourth timestamp to the smart camera;

[0138] A frequency synchronization module is used to synchronize the frequency of the smart camera with the display screen;

[0139] A fourth timestamp injection module, used to identify the first timestamp in the display screen image captured by the smart camera, and inject the sixth timestamp of the shooting moment into the corresponding TCP message;

[0140] The delay calculation module is used to calculate the difference between the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to obtain each part of the delay, and calculate the difference between the first timestamp and the sixth timestamp to obtain the total delay value.

[0141] The present invention has many specific application paths. The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements can be made without departing from the principle of the present invention. These improvements should also be regarded as the protection scope of the present invention.

Claims

1. A full-link image transmission delay test method for a remote control driving system, characterized in that: In the remote driving scenario applied to vehicles, the scenario includes a vehicle, a cloud server, a remote cockpit, a display screen and a computer. The cloud server is connected to the vehicle and the remote cockpit data through the network respectively, the remote cockpit is connected to the display screen through a data cable, and a smart camera is provided in front of the display screen for shooting the display screen. The smart camera data is connected to the computer. The method steps are as follows: 1) Synchronize the time base of the vehicle and the remote cockpit, and obtain the image pixel timestamp data reading delay of the vehicle and the remote cockpit; 2) Generate vehicle camera environment perception simulation data, inject a first timestamp into the image and pixel data of the simulation data respectively, and then send the image data to the vehicle data receiving interface; 3) The vehicle data receiving interface receives the image data with the first timestamp, and when the vehicle receives the image data, it creates a TCP message, injects the first timestamp into the TCP message, and corrects the second timestamp; 4) transmitting the TCP message with the first timestamp, the corrected second timestamp and the image data to the vehicle data transmission interface, and the vehicle data transmission interface injects the corrected third timestamp into the TCP message when receiving the image data; 5) Sending the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp and the image data to the cloud server; 6) The cloud server sends the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp and the image data to the remote cockpit, and the remote cockpit injects the corrected fourth timestamp into the TCP message when receiving the image data; 7) The remote cockpit sends a TCP message with a first timestamp, a corrected second timestamp, a corrected third timestamp, and a corrected fourth timestamp to the smart camera, and injects the corrected fifth timestamp into the TCP message at the time of sending, and sends the image data to the display screen for display; 8) The smart camera is frequency synchronized with the display screen, the smart camera receives the TCP message, shoots the display screen image and records the sixth timestamp at the current moment when shooting, identifies the first timestamp in the image and injects the sixth timestamp into the corresponding TCP message and then sends it to the computer; 9) The computer calculates the difference between the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to obtain each part of the delay, and calculates the difference between the first timestamp and the sixth timestamp to obtain the total delay value; The step 3) specifically includes: The vehicle data receiving interface receives the image data with the first timestamp T1, reads the first 8 bytes of pixel value of the image data to obtain the first timestamp T1, records the second timestamp T2 at the current moment after reading the first timestamp T1, and calculates and corrects the second timestamp t2=T2-t64, creates a TCP message, and the TCP message includes an identification bit, a check bit and a data bit, and the subsequent TCP message format is consistent with it; the identification bit uses the first timestamp T1 value as an identification, writes the first timestamp T1 and the corrected second timestamp t2 into the TCP message data bit in sequence, and stores the value obtained by t2-T1 into the check bit as verification data; The step 4) specifically includes: The vehicle data transmission interface receives a TCP message and image data with a first timestamp and a corrected second timestamp. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. When the reading is completed, the third timestamp T3 at the current moment is recorded, and the corrected third timestamp t3=T3-t64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected third timestamp t3 is added to the first timestamp T1 and the corrected second timestamp t2 in the TCP message, and the value of the check bit is added to the corrected third timestamp t3; The step 6) specifically includes: The remote cockpit data receiving interface receives the TCP message and image data with the first timestamp, the corrected second timestamp, and the corrected third timestamp. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. At the same time, the fourth timestamp T4 at the current moment is recorded after the reading is completed, and the corrected fourth timestamp t4=T4-t'64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected fourth timestamp t4 is added to the corrected third timestamp t3 in the TCP message, and the value of the check bit is subtracted from the corrected fourth timestamp t4; The step 7) specifically includes: The remote cockpit data transmission interface receives the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp and the received image data, and sends them to the display screen. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. After the reading is completed, the fifth timestamp T5 at the current moment is recorded, and the corrected fifth timestamp t5=T5-t'64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected fifth timestamp t5 is added to the corrected fourth timestamp t4 in the TCP message, and the value of the check bit is added to the corrected fifth timestamp t5, and the TCP message is sent to the smart camera; Wherein, t64 is the time when the vehicle reads 64 bits of image data, and t'64 is the time when the remote cockpit reads 64 bits of image data.

2. The full-link image transmission delay test method of the remote control driving system according to claim 1 is characterized in that: The step 1) specifically includes: 11) Configuration of remote cockpit GPS timing equipment; The GPS timing device installed on the remote cockpit obtains time information by receiving GPS signals, and synchronizes the time information to the remote cockpit via the Ethernet interface in the form of gptp, so that all devices in the remote cockpit refer to the same time standard; 12) Configuration of vehicle GPS timing equipment; The GPS timing device installed on the vehicle converts the received GPS signal into the master clock signal of the vehicle system, thereby achieving time synchronization in the entire vehicle system; 13) Obtain image reading delay; An image with the same format and size as the output image of the video injection device is stored in the vehicle and the remote cockpit respectively. Before the test starts, the vehicle and the remote cockpit read the stored image respectively, obtain timestamp A when reading starts, and obtain timestamp B when reading is completed. The total time of reading the image is obtained by subtracting timestamp A from timestamp B, and then the time of reading 64 bits of image data is calculated according to the image size. The time of reading 64 bits of image data = total time of reading the image * 64 / number of all image data bits.

3. The full-link image transmission delay test method of the remote control driving system according to claim 2 is characterized in that: The step 2) specifically includes: 21) Generation of environmental perception simulation data; In the scene simulation software, by simulating the characteristics of lighting and object reflection in the actual scene, environmental perception simulation data that is highly similar to the images captured by the actual camera is generated; 22) Transmission and processing of environmental perception simulation data; The generated environmental perception simulation data is transmitted to the video injection device through the PCIe bus. The video injection device obtains the time information of the vehicle's GPS timing device, decodes the received environmental perception simulation data, encodes the image data according to the data format received by the vehicle, and embeds the first timestamp T1 of the current moment into the image pixel when the encoding is completed. At the same time, the first timestamp T1 is superimposed on the original image in the form of a string image and sent to the vehicle data receiving interface.

4. The full-link image transmission delay test method of the remote control driving system according to claim 1 is characterized in that: The step 8) specifically includes: 81) Adjust the sampling frequency of the smart camera to synchronize with the refresh frequency of the display screen; 82) The smart camera receives the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, and the corrected fifth timestamp, and captures the image on the display screen, obtains the first timestamp T1 printed on the image in the form of a character string image through image recognition, and generates a sixth timestamp T6 at the time of capturing, and after adding the sixth timestamp T6 to the corrected fifth timestamp t5 in the TCP message with the first timestamp T1, the value of the check digit is subtracted from the sixth timestamp T6; 83) Output the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to the computer.

5. The full-link image transmission delay test method of the remote control driving system according to claim 4 is characterized in that: The step 9) specifically includes: The computer extracts the data bits and check bits of the TCP message output by the smart camera, calculates the value of T1-t2+t3-t4+t5-T6 and compares it with the check bit value. If they are consistent, it means that the data is valid and the delay calculation can be performed. The matrix multiplication is used to calculate the delay of each part. The calculation formula is: r=M*x In the formula, x is a column vector, M is a transformation matrix, and r is an array of delay calculation results; r[1]=-1*T1+1*t2=t2-T1 r[2]=-1*t2+1*t3=t3-t2 r[3]=-1*t3+1*t4=t4-t3 r[4]=-1*t4+1*t5=t5-t4 r[5]=-1*t5+1*T6=T6-t5 r[6]=1*T1+0*t2+0*t3+0*t4+0*t5-1*T6=T6-T1 Among them, r[1] represents the delay of image data transmission from the video injection device to the vehicle data receiving interface, r[2] represents the delay of the vehicle processing image data, r[3] represents the delay of image data from the vehicle to the remote cockpit through the cloud server, r[4] represents the delay of image data processing in the remote cockpit, r[5] represents the delay of image data display through the display screen, and r[6] represents the total delay of the entire link.

6. A full-link image transmission delay test system for a remote control driving system, characterized in that: include: A time base synchronization module, used to synchronize the time bases of the vehicle and the remote cockpit; Delayed reading module, used to obtain the image pixel timestamp data reading delay of the vehicle and remote cockpit; A simulation data generation module, used for generating vehicle camera environment perception simulation data, injecting a first timestamp into the image and pixel data respectively, and then sending the image data to the vehicle data receiving interface; A TCP message creation module, used for creating a TCP message when the vehicle receives image data with a first timestamp, injecting the first timestamp into the TCP message, and correcting the second timestamp; A first timestamp injection module, configured to inject a corrected third timestamp into a TCP message when the vehicle data transmission interface receives image data with a first timestamp and a corrected second timestamp; A second timestamp injection module, configured to inject a corrected fourth timestamp into a TCP message when the remote cockpit receives image data with the first timestamp, the corrected second timestamp, and the corrected third timestamp; A third timestamp injection module is used to inject the corrected fifth timestamp into the TCP message when the remote cockpit sends the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, and the corrected fourth timestamp to the smart camera; A frequency synchronization module is used to synchronize the frequency of the smart camera with the display screen; A fourth timestamp injection module, used to identify the first timestamp in the display screen image captured by the smart camera, and inject the sixth timestamp of the shooting moment into the corresponding TCP message; A delay calculation module, used to calculate the difference between the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp, the corrected fifth timestamp, and the sixth timestamp to obtain each part of the delay, and to calculate the difference between the first timestamp and the sixth timestamp to obtain the total delay value; The vehicle data receiving interface receives the image data with the first timestamp T1, reads the first 8 bytes of pixel values ​​of the image data to obtain the first timestamp T1, records the second timestamp T2 at the current moment after reading the first timestamp T1, calculates and corrects the second timestamp t2=T2-t64, and creates a TCP message. The TCP message includes an identification bit, a check bit and a data bit, and the subsequent TCP message format is consistent with it; The identification bit uses the first timestamp T1 value as the identification, writes the first timestamp T1 and the corrected second timestamp t2 into the TCP message data bit in sequence, and stores the value obtained by t2-T1 into the check bit as the check data; The vehicle data transmission interface receives a TCP message and image data with a first timestamp and a corrected second timestamp. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. When the reading is completed, the third timestamp T3 at the current moment is recorded, and the corrected third timestamp t3=T3-t64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected third timestamp t3 is added to the first timestamp T1 and the corrected second timestamp t2 in the TCP message, and the value of the check bit is added to the corrected third timestamp t3; The remote cockpit data receiving interface receives the TCP message and image data with the first timestamp, the corrected second timestamp, and the corrected third timestamp. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. At the same time, the fourth timestamp T4 at the current moment is recorded after the reading is completed, and the corrected fourth timestamp t4=T4-t'64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected fourth timestamp t4 is added to the corrected third timestamp t3 in the TCP message, and the value of the check bit is subtracted from the corrected fourth timestamp t4; The remote cockpit data transmission interface receives the TCP message with the first timestamp, the corrected second timestamp, the corrected third timestamp, the corrected fourth timestamp and the received image data, and sends them to the display screen. When receiving the image data, the first timestamp T1 in the image pixel is read and compared with the TCP message identification bit. After the reading is completed, the fifth timestamp T5 at the current moment is recorded, and the corrected fifth timestamp t5=T5-t'64 is calculated. If the first timestamp T1 is consistent with the TCP message identification bit, the corrected fifth timestamp t5 is added to the corrected fourth timestamp t4 in the TCP message, and the value of the check bit is added to the corrected fifth timestamp t5, and the TCP message is sent to the smart camera; Wherein, t64 is the time when the vehicle reads 64 bits of image data, and t'64 is the time when the remote cockpit reads 64 bits of image data.

Images