Automatic driving automobile Touling-like double-blind-person-machine confrontation type driving test system and method

Abstract

The invention provides an automatic driving vehicle Tailing double blind person machine confrontation type driving test system and method, and the system comprises a road network, one or more target vehicles, a test vehicle, one or more communication networks, a target vehicle driving facility, and a Tailing test vehicle test facility. The method comprises a starting phase, a capturing phase, a testing phase, a checking phase and an evaluating phase. The brand-new test method is created, the method is simple and easy to implement, and the problems that the safety of automatic driving cannot be evaluated by comparing human driving in an existing test field test, and a public road test is not practical and cannot provide enough evidence to prove the safety of the automatic driving automobile are perfectly solved.

Description

Turing-like double-blind human-machine adversarial driving test system and method for autonomous vehicles

[0001] This application claims priority to a U.S. provisional application, entitled “Systems and Methods of the Turing-Like Double Blind Machine-versus-Human Driving Test of Autonomous Vehicles,” with application number 63544179 and filing date October 15, 2023. Technical Field

[0002] The present invention relates to an autonomous driving vehicle testing system and method, as well as a remote reality system, a remote reality driving unmanned vehicle, and a robot. Background Art

[0003] Since their introduction, road vehicles have greatly facilitated human mobility, a vital component of modern society. However, human-driven road vehicles have resulted in significant casualties. In the United States, approximately 32,000 people die and over 2 million are injured each year in traffic accidents. The economic and societal costs of motor vehicle accidents in the United States alone may exceed $800 billion. Furthermore, over 90% of crashes are caused by human error, such as speeding, misjudging other drivers' behavior, and driving under the influence, distraction, and fatigue. Autonomous vehicles have the potential to eliminate many of the common mistakes made by human drivers and significantly alleviate this public health crisis. However, autonomous vehicles may not completely eliminate traffic accidents. For example, inclement weather and complex driving environments pose challenges to both autonomous vehicles and human drivers. Until the underlying technology matures, autonomous vehicles may perform worse than human drivers. Given the high stakes, the key question is: How safe should autonomous vehicles be before they can be approved for consumer use? This leads to two fundamental questions: How good is autonomous driving compared to human driving? And how safe is autonomous driving compared to human driving?

[0004] These questions need to be answered through driving tests of autonomous vehicles. For a century, the automotive industry has built various test sites, tracks, and facilities to test the various functions and performance of traditional vehicles. However, traditional test sites were not built for driving tests. Therefore, new dedicated test sites have been built around the world to test autonomous driving, including:

[0005] Mcity (Ann Arbor, Michigan, USA): Mcity is a simulated urban environment for testing autonomous vehicles. It replicates a city landscape, including streets, traffic signs, and other infrastructure, to evaluate the performance of autonomous vehicles in real-world scenarios.

[0006] Waymo's Castle (Central Valley, California): Waymo, a subsidiary of Alphabet (formerly Google), has used Castle as a testbed for its self-driving cars. It features a variety of challenging driving conditions, including rural roads and highways.

[0007] GoMentum Test Site (Concord, California, USA): The GoMentum Test Site is the largest connected autonomous driving proving ground in the United States. It has 20 miles of paved roads, tunnels, bridges and other infrastructure, and is used by many companies to test autonomous driving technology.

[0008] Autonomous Vehicle Proving Ground (Ypsilanti, Michigan, U.S.): This facility, which includes the American Center for Mobility (ACM) and the University of Michigan's Mcity, offers a range of testing environments, including urban and highway settings.

[0009] Cruise Automation San Francisco Test Zone (San Francisco, California, USA): General Motors subsidiary Cruise Automation conducts extensive testing in the complex and challenging urban environment of San Francisco.

[0010] Nürburg, Germany: Known as the "Green Hell," Nürburg is a legendary racing track where automakers test their vehicles, including autonomous driving prototypes, under demanding driving conditions.

[0011] UK Automated Driving Testbed (Milton Keynes and Coventry, UK): The UK Automated Driving Project involves testing on public roads in Milton Keynes and Coventry to evaluate various autonomous driving technologies.

[0012] K-City (Hwaseong, South Korea): K-City is a large-scale testing ground designed to simulate urban and suburban driving conditions, including streets, highways, and a parking lot for autonomous vehicle testing.

[0013] CityMobil2 (Europe): The CityMobil2 project includes several test sites in Europe, such as La Rochelle (France) and Trikala (Greece), to test autonomous vehicles in real urban environments.

[0014] National Advanced Driving Simulator (NADS) (Iowa City, Iowa, U.S.): NADS provides a high-tech simulation environment for testing autonomous vehicles and their interactions with human-driven vehicles in a variety of scenarios.

[0015] Singapore Autonomous Vehicle Testing Centre (Singapore): SAVeT is a government-backed project that provides a controlled testing environment for autonomous vehicles in Singapore's urban landscape.

[0016] NICE-City (Shanghai, China) National Intelligent Connected Vehicle Test Area: This facility serves as a key hub for research, development, and real-world testing of smart and intelligent technologies. Connected vehicles are designed to support advancements in autonomous driving technology, and NICE City provides a wide range of controlled environments and test scenarios to support the development of autonomous vehicles and connected vehicle systems.

[0017] These testing grounds play an important role in the development and validation of autonomous vehicle technology, helping companies and researchers evaluate the performance, safety, and reliability of autonomous vehicles in a diverse and controlled environment.

[0018] To recreate specific traffic scenarios, existing autonomous driving test sites deploy automatically controlled target vehicles or equipment, as well as target vehicles or robots with pre-programmed driving paths and speeds, to simulate the pedestrians, cyclists, and vehicles surrounding the test vehicle. These target vehicles and equipment are designed to be "crash-safe," typically lightweight, easily disassembled, and reassembled without damage, or with an inflatable vehicle body or human model mounted on a mobile, ground-level platform. If an autonomous vehicle fails a scenario test and collides with a target vehicle or equipment, the target vehicle or equipment will either disassemble easily or the inflatable human model atop it will be pushed away by the impact, causing no damage or injury to the autonomous vehicle or its occupants.

[0019] A lot of research has been done on how to generate various test scenarios. Traditionally, the test field mechanically repeats the supported test scenarios and does not change. Since autonomous vehicles have artificial intelligence, they can easily identify these repeated test scenarios and further give preset "standard" driving decisions and controls. Therefore, these tests will not reveal the true driving capabilities and performance of autonomous vehicles in the real world. Recently, new research has proposed randomizing the pre-programmed parameters in the scenario to create an extended series of test scenarios for each scenario with parameters distributed according to probability. In a scenario where an autonomous vehicle turns right and hits a pedestrian crossing the road, when the autonomous vehicle is set to repeat its starting position, driving direction and the same right turn driving task, the pedestrian crossing the road is pre-programmed to start crossing after a randomized starting delay, or walk a randomized distance, or hesitate for a randomized time. This results in a series of test scenario examples that are similar but at least not completely repeated to challenge the autonomous vehicle [1].

[0020] However, from a methodological perspective, the testing methods used in existing test fields cannot compare the driving and safety levels of autonomous driving with those of human driving, and therefore cannot answer two basic questions: no matter how many such mechanically repetitive test scenarios an autonomous vehicle passes, whether they remain fixed or introduce random changes, it cannot be concluded that the autonomous driving being tested is as good and as safe as human driving, and therefore can be issued a qualified "driving license" for large-scale market application.

[0021] Due to the various limitations of autonomous vehicle testing sites, the industry relies on public road driving tests to assess safety, particularly for Level 3 to Level 5 autonomous driving. This involves testing autonomous vehicles in real traffic on public roads to observe their performance and evaluate and improve their systems. A human driver typically serves as a safety officer, behind the wheel, ready to take control in the event of an imminent autonomous driving failure or system exit. Many autonomous driving test zones have been established on public roads, including in California, USA, and Shanghai, China.

[0022] But is it feasible to evaluate the safety of self-driving cars through public road testing? The safety of human drivers is a key benchmark for judging the safety of autonomous vehicles. While human drivers experience a high number of crashes, injuries, and fatalities, the incidence of these failures is low compared to the number of miles driven by humans. Americans collectively drive nearly 3 trillion miles annually (2015). In 2013, 2.3 million injuries were reported, equivalent to 77 injuries per 100 million miles. In 2013, 32,719 fatalities were reported, equivalent to 1.09 fatalities per 100 million miles. It's no secret that after more than a decade and $100 billion in investment in the autonomous vehicle industry, the technology is still far from mature, and large-scale deployment remains elusive. Even assuming that the autonomous vehicles being tested have achieved mature driving performance (the average human driver in the United States), [2] shows that for a round of driving tests (e.g., without patch repairs or retesting), autonomous vehicles must drive 8.8 billion miles to demonstrate with 95% confidence that their failure rate does not exceed 20% of the true failure rate (1.09 deaths per 100 million miles), achieving a safety level comparable to human driving. For example, a test fleet of 100 autonomous vehicles, driving 24 hours a day, 365 days a year at an average speed of 25 miles per hour, would take 400 years to complete such a huge test mileage. In addition, assuming that autonomous vehicles have perfected driving performance (e.g., without patch repairs or retesting) and can match the target driving safety level of 382 total collisions, 103 total injuries, and 1.09 deaths per 100 million miles, a round of driving tests lasting 8.8 billion miles would also result in 33,616 accidents, 9,064 injuries, and 96 deaths. This is clearly unrealistic.

[0023] Since existing testing grounds cannot evaluate the safety of autonomous driving by comparing it to human driving, and public road testing is impractical and cannot provide sufficient evidence to prove the safety of autonomous vehicles, new testing methods must be created.

[0024] Theoretically, autonomous driving is the specialized application of artificial intelligence (AI) to vehicle driving skills, so the questions involved in autonomous driving testing fall under the purview of AI testing. The Turing test, proposed by Alan Turing, the father of AI, in 1950, was designed to answer the historic question, "Can machines think?" The Turing test is widely used to test whether a machine can exhibit intelligent behavior comparable to or indistinguishable from human intelligence, as shown in Figure 5. Turing proposed that machine A, designed to produce human-like responses, be evaluated by a human evaluator, C, in a natural language conversation between a human, B, and machine A. Evaluator C knows that the two interlocutors, one a machine and the other a human, are isolated and invisible to each other. The conversation is limited to text channels, such as a computer keyboard and screen, so the result does not depend on the machine's ability to convert text into speech. If evaluator C cannot reliably distinguish between machine A and human B, machine A is deemed to have passed the test. The principle is that the test result depends not on the machine's ability to correctly answer questions, but only on how closely its responses resemble those of humans.

[0025] The Turing test has been the classic standard for testing artificial intelligence for more than 70 years since its invention. However, the Turing test, a classic standard for testing whether general artificial intelligence (knowing something) has reached human-level performance, is not suitable for driving tests, which test professional skills in doing something, because it is a conversation-based test.

[0026] AI tests of professional skill have also been successful. Go is a complex board game that requires intuition, creativity, and strategic thinking. Human Go players have varying skill levels, leading to a rating system that ranks players using a dan system. For example, 9th dan is the highest rank for professional players. Go has long been considered a formidable challenge for artificial intelligence. In 2015, Google DeepBrain successfully developed AlphaGo, a revolutionary AI Go player. AlphaGo's algorithm combines machine learning and search tree techniques, coupled with extensive training, including both human and computer games. Its Go-playing ability was tested through matches between AlphaGo and human players. In October 2015, AlphaGo defeated European Champion and 2nd dan professional Go player Fan Hui with a score of 5.0. Furthermore, in a five-game match between AlphaGo and Lee Se-ok, the world's top 9th dan professional Go player, also known as the Google DeepBrain Challenge, AlphaGo won all but the fourth game, with the opponent resigning in each match. After the game, the Korean Go Association awarded AlphaGo the highest level of Grandmaster - "9th Dan Honor".

[0027] AlphaGo's matches against humans demonstrate a successful example of how human expertise can be used to test the expertise of artificial intelligence.

[0028] Summary of the Invention

[0029] The purpose of the present invention is to address the problems existing in the prior art and to provide a Turing-like double-blind human-machine adversarial driving test system and method for autonomous vehicles, so as to solve the problem that existing test field tests cannot compare with human driving to evaluate the safety of autonomous driving, while public road tests are impractical and cannot provide sufficient evidence to prove the safety of autonomous vehicles.

[0030] The object of the present invention is achieved through the following technical solutions:

[0031] A Turing-like double-blind human-machine adversarial driving test system for an autonomous vehicle is characterized in that the system includes a road network, one or more target vehicles, a test vehicle, one or more communication networks, a target vehicle driving facility, and a Turing-like test vehicle testing facility.

[0032] As a further technical solution, the target vehicle is a remote reality driving vehicle, including a vehicle platform and a remote reality driving device that can achieve no electrical-to-electrical delay or nearly no electrical-to-electrical delay, the vehicle platform including various vehicles, various pedestrians and various animals with external driving interfaces; the various vehicles are vehicles that can be driven remotely and are equipped with simulated drivers and passengers in the vehicle, and the simulated drivers and passengers in the vehicle are collision test dummies equipped with sensors to simulate human occupants and collect data on the consequences of collision on the occupants; the various pedestrians are robots that can be driven remotely and are equipped with human-shaped bodies, and are equipped with sensors to simulate human pedestrians and collect data on the consequences of collision on pedestrians; the various animals are robots that can be driven remotely and are equipped with bodies in various animal shapes, and are equipped with sensors to simulate animals and collect data on the consequences of collision on animals.

[0033] As a further technical solution, the test vehicle has both autonomous driving functions and remote reality driving functions, and is a two-in-one vehicle of autonomous driving vehicles and remote reality driving vehicles. The test vehicle includes: an autonomous driving vehicle platform and a remote reality driving device that can achieve no electrical-to-electrical delay or nearly no electrical-to-electrical delay; the autonomous driving vehicle platform includes various autonomous driving vehicles and various autonomous walking robots with external driving interfaces and in-vehicle mode switches; the various autonomous driving vehicles are autonomous driving vehicles equipped with simulated in-vehicle drivers and passengers, and the simulated in-vehicle drivers and passengers are collision test dummies equipped with sensors to simulate human occupants and collect data on the consequences of collisions on the occupants; the various autonomous walking robots are autonomous walking robots equipped with humanoid bodies, animal-shaped bodies or other shapes, and the robots are equipped with collision sensors to collect data on the consequences of collisions on the robots.

[0034] As a further technical solution, the in-vehicle mode switch selects whether the vehicle platform is driven by the autonomous driving system to provide autonomous driving functions, or is driven by driving signals received from an external driving interface to provide remote reality driving functions.

[0035] As a further technical solution, the communication network includes one or more base stations, a wireless network between all traffic participants and the base stations, one or more roaming stations, a trunk network between the base stations and the roaming stations, and a trunk network between the roaming stations and the target vehicle driving facilities and the Turing-like test vehicle test facilities.

[0036] As a further technical solution, the communication network also includes a driving mode switch, which selects the remote driving station in room A or B to be connected to the test car: when the remote driving station in room A is connected to the test car, the test car is in automatic driving mode; when the remote driving station in room B is connected to the test car, the test car is in remote driving mode and is remotely driven by human driver B.

[0037] As a further technical solution, the target vehicle driving facility includes one or more remote driving stations, each remote driving station includes a display device for displaying a current view of the remote reality, a perspective input device and a vehicle control input device, each remote driving station is equipped with a human driver and is connected to a target vehicle so that the human driver can drive the target vehicle.

[0038] As a further technical solution, the Turing-like test car testing facility is a facility consisting of three closed rooms, which are called Room A, Room B and Room C; in Room A, there is a remote driving station but no human driver; in Room B, there is a remote driving station equipped with human driver B; the driving mode switch determines whether the remote driving station in Room A or Room B is connected to the test car; all test results are sent anonymously to Room C, where they are evaluated by human evaluator C.

[0039] As a further technical solution, the target vehicle includes a remote reality driving device capable of achieving zero or near zero electrical-to-electrical delay. The remote reality driving device includes multiple cameras, a stitching engine, a display engine, a drive recorder, and a communication engine. The stitching engine stitches all camera videos together to generate a horizontal 360° surround video around the remote reality driving vehicle with zero or near zero delay. The display engine projects the horizontal 360° surround video onto a designated display area in a mathematical model of the remote driving station's display device based on a viewing angle control command from the remote driving station, and generates a video of the current view. The drive recorder records all generated videos as well as vehicle status and vehicle control commands. The communication engine includes a wireless video transmitter, a two-way wireless data transceiver, and an auxiliary wireless transceiver.

[0040] As a further technical solution, the test vehicle includes a remote reality driving device capable of achieving zero or near zero electrical-to-electrical delay. The remote reality driving device includes multiple cameras, a stitching engine, a display engine, a drive recorder, and a communication engine. The stitching engine stitches all camera videos together to generate a horizontal 360° surround video around the remote reality driving vehicle with zero or near zero delay. The display engine projects the horizontal 360° surround video onto a designated display area in a mathematical model of the remote driving station's display device based on a viewing angle control command from the remote driving station, and generates a video of the current view. The drive recorder records all generated videos as well as vehicle status and vehicle control commands. The communication engine includes a wireless video transmitter, a two-way wireless data transceiver, and an auxiliary wireless transceiver.

[0041] As a further technical solution, the optical axes of the multiple cameras are placed on a horizontal plane of the vehicle, and the optical axes intersect at one point.

[0042] As a further technical solution, the stitching engine and display engine are replaced by a stitching and display two-in-one engine to achieve delay-free or nearly delay-free stitching and display joint processing.

[0043] As a further technical solution, the wireless video transmitter adopts a delay-free and uncompressed video transmission method.

[0044] A testing method based on a Turing-like double-blind human-machine adversarial driving test system for autonomous vehicles, the method comprising:

[0045] Startup phase: The system randomly selects a test car to connect to Room A or Room B in the Turing-like test car test facility; through one or more safety scenarios, one or more target cars are brought close enough to the test car;

[0046] Capture phase: starts when the test vehicle is close enough and the target vehicle starts taking action to seize the expected vehicle position relative to the test vehicle. If the target vehicle successfully seizes the expected vehicle position, the expected capture scenario is reached. The capture phase ends when the test vehicle is captured in the expected capture scenario. If the target vehicle fails to seize the expected vehicle position relative to the test vehicle and misses the capture opportunity, the expected capture scenario is not reached. The capture phase ends when the test vehicle is not captured in the expected capture scenario.

[0047] Testing phase: After the test vehicle is captured in the expected capture scenario, the target vehicle begins to move to create a test scenario for the test vehicle. Once a test scenario is reached, the test vehicle's driving will be tested;

[0048] Inspection phase: Begins after the test phase ends. Regardless of whether the test vehicle safely passes the test scenario, all equipment is inspected and all test results are collected, including all video recordings, information records, and collision reports of the test vehicle and all target vehicles involved in the test run.

[0049] Evaluation phase: It starts after obtaining or selecting a pair of test results from two test runs. The pair of test results are anonymously sent to Room C in the Turing-like test car testing facility. Human evaluator C compares and evaluates the driving skills of the two drivers without knowing their identities and decides which set of test results has better driving skills and wins, or which set has the same driving skills and is tied, or no evaluation can be performed and no evaluation conclusion is given.

[0050] As a further technical solution, a pair of test results from both test runs were obtained by a human driver remotely driving the test car in reality, to compare and evaluate the driving skills of the two human drivers.

[0051] As a further technical solution, a pair of test results from two test runs were obtained by the test car's autonomous driving and the human driver's remote real-life driving, respectively, to compare and evaluate the driving skills of the autonomous driver and the human driver.

[0052] As a further technical solution, comparative evaluation of driving skills includes evaluating how good autonomous driving is compared to human driving and evaluating how safe autonomous driving is compared to human driving.

[0053] As a further technical solution, a method for evaluating how well autonomous driving compares to human driving includes the following steps:

[0054] Set up a rating system: Establish a rating system based on the driver rank system. Based on the evaluation results of human evaluator C, calculate the win rate of each driver and rate them according to the rank system.

[0055] Evaluate human drivers: Rating the human drivers of the test vehicles and obtaining their respective ranks;

[0056] Rating autonomous drivers: After human drivers are ranked, autonomous drivers are rated based on the same rating system to obtain their rank, which quantitatively measures their driving skills compared to human drivers.

[0057] As a further technical solution, the establishment of a driver ranking system includes the following steps:

[0058] A driver who has just obtained a driver's license is classified as stage 0;

[0059] Using the formula The given win rate determines each rank, where: P dd It represents the winning rate of driver B1 with low rank d1 against driver B2 with high rank d2, the rank difference Δd=d2-d1, and a is a constant.

[0060] As a further technical solution, a method for evaluating human drivers and automated drivers includes: finding a standard Segment 0 driver by comparing the win rate between Segment 0 drivers; when the win rate of the driver to be evaluated against the standard Segment 0 driver is higher than an upper limit set value, the driver is rated as Segment 1, and the evaluation continues; if the win rate is lower than a lower limit set value, the driver is rated as -1, and the evaluation ends; otherwise, the driver is rated as Segment 0, and the evaluation ends;

[0061] The standard 1st stage driver is found by the win rate between 1st stage drivers. When the win rate of the driver to be evaluated against the standard 1st stage driver is higher than an upper limit set value, he is rated as 2nd stage and the evaluation continues. If it is not higher, he remains at 1st stage and the evaluation ends.

[0062] The same applies to higher ranks.

[0063] As a further technical solution, when the constant a=1, the upper limit setting value is 76% and the lower limit setting value is 24%.

[0064] As a further technical solution, the method for evaluating how safe autonomous driving is compared to human driving includes:

[0065] Rating the test scenarios: First, set the scenario win / failure conditions as follows: During the evaluation phase, if the human evaluator C determines that the test car successfully and safely passes the scenario, the scenario fails; if the evaluator determines that the test car fails to safely pass the scenario, the scenario wins; if the evaluator determines that the test car passes half and fails half in terms of safety, the scenario is tied with the driver; then rate the test scenarios: Step 1: Compared with the standard 0-segment driver, if the scenario's win rate is lower than a lower limit set value, the scenario has a negative segment; otherwise, the scenario has a non-negative segment; Step 2: If the scenario has a non-negative segment, first set it to segment 0, and then if its win rate is higher than an upper limit set If the win rate is higher than the second lower limit set value, the scenario is promoted by 1 stage, and the rating continues, comparing it with the standard driver of the next stage until its highest stage. If the scenario has a negative stage, one evaluation method for the negative stage scenario is: if its win rate is higher than the second lower limit set value, it is evaluated as -1 stage and the rating ends, otherwise it is compared with the third lower limit set value until its lowest stage. Another evaluation method for the negative stage scenario is: use the standard 0 stage driver to find and evaluate all -1 stage scenarios, then use the standard -1 stage scenario to find and evaluate all -2 stage scenarios, and so on. Then, starting from the -1 stage standard scenario, the scenario to be evaluated is compared with each negative stage standard scenario until its lowest stage is obtained.

[0066] Determine the safety level: The safety level of an autonomous vehicle passing through a scene is determined by its segment difference to the scene. In the rating system set, the failure rate P of driver B with a grade of d0 who safely passes the scenario S with a grade of d1 is ds (Δd), according to the formula Given; when the constant a = 1, if the autonomous vehicle has the same level as the scene, its safety level is level 0, and the autonomous vehicle has a 50% chance of winning and a 50% chance of collision; if the autonomous vehicle is 1 level higher than the scene, its safety level is level 1, and the autonomous vehicle has a 91% chance of winning and a 9% chance of collision; if the autonomous vehicle is 2 levels higher than the scene, its safety level is level 2, and the autonomous vehicle has a 99% chance of winning or a 1% chance of collision, and so on.

[0067] A remote reality driving vehicle for a Turing-like double-blind human-machine adversarial driving test system for autonomous vehicles, the vehicle consisting of a vehicle platform and a remote reality driving device capable of achieving zero or nearly zero electrical-to-electrical delay.

[0068] As a further technical solution, when the remote reality driving vehicle is used as a test vehicle, the vehicle platform includes: an automatic driving system, an external driving interface, and an in-vehicle mode switch.

[0069] As a further technical solution, when a remote reality driving vehicle is used as a target vehicle, the vehicle platform includes various vehicles and robots with an external driving interface.

[0070] As a further technical solution, when a remote reality driving vehicle is used as an unmanned racing car in an unmanned racing sports system, the vehicle platform includes a racing car with an external driving interface, and a human racer can remotely drive the remote reality driving vehicle at high speed and speed to perform racing sports by implementing a remote reality driving device with no electrical to electrical delay or nearly no electrical to electrical delay.

[0071] As a further technical solution, when the remote reality driving vehicle is used as an unmanned tank-type remote reality driving vehicle in a new type of unmanned military-style sports system, the vehicle platform includes an armored vehicle with an external driving interface, and the driving signal received from the external driving interface includes an armored vehicle driving signal and a combat control signal. The armored vehicle includes a tank. Human racers can achieve high-speed and rapid remote driving of the remote reality driving armored vehicle and control the combat equipment by realizing a remote reality driving device with no electrical to electrical delay or nearly no electrical to electrical delay to conduct combat-type military sports.

[0072] As a further technical solution, when the remote reality driving vehicle is used as an unmanned heavy machinery remote reality driving vehicle, the vehicle platform includes a ground heavy vehicle or an aerial flying machinery with an external driving interface. The driving signals received from the external driving interface include ground vehicle driving signals, flight driving signals and operation control signals. The ground heavy vehicles include construction vehicles, mining vehicles, and garbage disposal vehicles. The aerial flying machinery includes drones. Workers can remotely drive and remotely operate in a safe and healthy environment by implementing remote reality driving equipment with no electrical to electrical delay or nearly no electrical to electrical delay, avoiding working in person under various unsafe and unhealthy field conditions.

[0073] As a further technical solution, the remote reality driving device includes multiple cameras, a stitching engine, a display engine, a driving recorder, and a communication engine. The stitching engine stitches all camera videos together to generate a horizontal 360° surround video around the remote reality driving vehicle with no or near-no delay. The display engine projects the horizontal 360° surround video onto a designated display area in a mathematical model of the remote driving station's display device according to the remote driving station's viewing angle control command, and generates a video of the current view. The driving recorder records all generated videos as well as vehicle status and vehicle control commands. The communication engine includes a wireless video transmitter, a two-way wireless data transceiver, and an auxiliary wireless transceiver.

[0074] As a further technical solution, the wireless video transmitter adopts a delay-free and uncompressed video transmission method.

[0075] As a further technical solution, the stitching engine and the display engine are combined into a two-in-one stitching and display engine. After the stitching engine completes the stitching process and obtains the stitched spherical image, the display processing directly projects the stitched spherical image onto the mathematical model of the display surface, thereby realizing delay-free or near-delay-free joint stitching and display processing.

[0076] As a further technical solution, the vehicle is used for other remote driving and remote operation systems, including unmanned racing sports systems, new unmanned military-style sports systems, and unmanned heavy machinery remote reality driving systems.

[0077] A remote realistic driving robot for a Turing-like double-blind human-machine adversarial driving test system for autonomous vehicles is provided, wherein the robot is a robot equipped with a humanoid body or an animal-shaped body, and the robot is equipped with sensors to simulate human pedestrians or animals and collect data on the consequences of collisions on pedestrians or animals.

[0078] Compared with the existing technology, the present invention creates a brand-new testing method that is simple and easy to implement. It perfectly solves the problem that existing test field tests cannot evaluate the safety of autonomous driving by comparing with human driving, while public road tests are impractical and cannot provide sufficient evidence to prove the safety of autonomous vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] FIG1 shows an embodiment of the Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle of the present invention.

[0080] FIG2 shows an embodiment of the present invention of a remote reality driving vehicle and robot with no or near no delay.

[0081] FIG3 shows an embodiment of a remote reality driving device according to the present invention.

[0082] FIG4 shows an embodiment of a remote reality driving device with a stitching and display two-in-one engine according to the present invention.

[0083] FIG5 illustrates the Turing test.

[0084] FIG6 shows an embodiment (a) of the Turing-like double-blind human-machine adversarial driving test method for an autonomous vehicle of the present invention.

[0085] FIG7 shows an embodiment (b) of the Turing-like double-blind human-machine adversarial driving test method for an autonomous vehicle of the present invention.

[0086] FIG8 shows an embodiment (c) of the Turing-like double-blind human-machine adversarial driving test method for an autonomous driving vehicle of the present invention.

[0087] FIG9 shows an embodiment (d) of the Turing-like double-blind human-machine adversarial driving test method for an autonomous vehicle of the present invention.

[0088] FIG10 shows an embodiment (e) of the Turing-like double-blind human-machine adversarial driving test method for an autonomous vehicle of the present invention.

[0089] FIG11 shows an embodiment (f) of the Turing-like double-blind human-machine adversarial driving test method for an autonomous driving vehicle of the present invention.

[0090] FIG12 shows an embodiment (g) of the Turing-like double-blind human-machine adversarial driving test method for an autonomous vehicle of the present invention.

[0091] FIG13 shows an embodiment (h) of the Turing-like double-blind human-machine adversarial driving test method for an autonomous vehicle of the present invention. DETAILED DESCRIPTION

[0092] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0093] Example

[0094] The principles and embodiments of the present invention will now be described in detail with reference to the accompanying drawings. The accompanying drawings provide exemplary embodiments to enable those skilled in the art to practice the invention. It should be noted that the drawings and the following embodiments are not intended to limit the scope of the present invention to a single embodiment. Other embodiments may be created by interchanging some or all of the described or illustrated parts. Where convenient, identical reference numerals in the drawings refer to identical or similar parts. Certain portions of these embodiments may utilize known components. This specification will describe how to use known components to facilitate understanding of the present invention, but details of known components may not be provided or omitted entirely to avoid obscuring the present invention. Furthermore, unless expressly stated otherwise, embodiments depicting a singular component in this specification should not be construed as limited to the singular; rather, the present invention is intended to encompass other embodiments comprising multiple identical components, and vice versa. Furthermore, unless expressly stated otherwise, applicants do not assign a unique or special meaning to any term in the specification or claims. Furthermore, steps or components illustrated in this specification encompass currently or future-known equivalents.

[0095] For the sake of brevity, this specification will describe in detail some of the basic methods of the present invention mentioned above, without describing all derived methods.

[0096] This Turing-like double-blind human-machine adversarial driving test system for autonomous vehicles consists of a road network (road network), one or more target vehicles (TV), a test vehicle (VUT), one or more communication networks, a target vehicle driving facility (TV facility) and a Turing-like test vehicle test facility (VUT facility).

[0097] The road network of the system of the present invention includes all elements in the scene, but does not include traffic participants, including but not limited to: one or more streets and street buildings, one or more levels of roads, one or more highways, one or more tunnels, one or more bridges, one or more traffic intersections, one or more traffic lights and one or more traffic signs. The road network also includes roads under different weather conditions, such as rain, snow, fog, etc. The road network also includes stationary obstacles on the road, such as fallen trees and rocks, broken-down vehicles and parked vehicles on the roadside. Generally, the road network consists of all the static elements of the scene, which change but do not move. As described below, since real car accidents may occur in the testing of the present invention, in a typical embodiment, in order to protect public safety, the road network is permanently or at least temporarily closed or isolated during testing, becoming an uninhabited area.

[0098] The road network includes roads within the test site and selected public road networks. To effectively and safely test the safety of automated driving, the present invention employs a scenario-based testing approach. Given the aforementioned target driving safety level, the number of scenarios included in the 8.8 billion-mile test is estimated under the following assumptions: 1) All collisions in different 8.8 billion-mile driving test rounds occur in the same set of scenarios; and 2) All collisions in each 8.8 billion-mile drive round occur in different scenarios. Assumption 1 holds true if the distribution of test mileage on various road networks under various conditions is consistent with the aforementioned national statistics. Assumption 2 is a worst-case scenario and provides an upper limit on the required test scenarios, as multiple collisions may occur in the same scenario within a single test round. Based on these assumptions, and based on the aforementioned 8.8 billion-mile driving test, the scenario-based testing approach would require a total of 33,616 collision scenarios, including 9,064 injury scenarios and 96 fatality scenarios, to equal the 8.8 billion-mile road test. The combined road network of all test sites may not be able to cover all 33,616 total crash scenarios, especially those scenarios on specific road networks that may be too expensive to build and include in the test sites. Therefore, some scenarios will be tested on the public road network. It should be noted that in the present invention, the scenario testing on the public road network is different from the existing public road testing method because the scenario testing on the public road network of the present invention can actively create dangerous test traffic flow, rather than simply relying on public traffic flow to accidentally create crash scenarios as done by the existing public road testing method.

[0099] In a typical embodiment, the target vehicles (TVs) of the present system include all traffic participants that generate scenarios for the vehicle under test (VUT). Target vehicles (TVs) include, but are not limited to, various vehicle TVs, including cars, trucks, buses, construction vehicles, agricultural vehicles, ambulances, fire trucks, police cars, motorcycles, bicycles, scooters, and the like; various pedestrian TVs, including groups of pedestrians crossing, walking, or running, young pedestrians sprinting, elderly pedestrians staggering, pedestrians pushing shopping carts, pedestrians walking their dogs, and the like; and various animal TVs, including pets, farm animals, and wild animals.

[0100] The vehicle TV in a typical embodiment of the system of the present invention is a remote reality driving vehicle (RRDV) equipped with simulated in-car drivers and passengers, which are crash test dummies equipped with sensors to simulate human occupants and collect data on the consequences of collisions on occupants. In combination with real car collisions and crash test dummies, the test of the present invention is a combination of driving tests and crash tests, which can quantitatively measure the severity of the collision in terms of property damage, human injury, and the number of deaths in driving tests, just like public road tests, but without any real human injury or loss of life. In short, the system of the present invention is able to test driving skills across the full spectrum from safe traffic flow, to collisions with only property damage, or collisions with physical injury or even death. In contrast, existing test sites that use non-destructive collision testing equipment are unable to distinguish between collisions or measure the severity of collisions.

[0101] The pedestrian TV in an exemplary embodiment of the present system is a remote reality driving robot (RRDR) equipped with a humanoid body. This humanoid body also serves as a crash test dummy, equipped with sensors to simulate a human pedestrian and collect data on the injury outcomes of collisions with pedestrians. Therefore, in collisions involving pedestrians in the present system, the number of pedestrian injuries or fatalities can be quantitatively measured.

[0102] The animal TV in a typical embodiment of the present invention's system is also a RRDR, with a body in the shape of a variety of animals, which also serves as a crash test dummy. It is equipped with sensors to simulate an animal and collect data on the impact of the animal. Therefore, in crashes involving animals in the present invention's test, the body injuries or fatalities of the animals can be quantitatively measured.

[0103] The test vehicle (VUT) of the present invention's system is a two-in-one autonomous vehicle and remote reality driving vehicle—that is, a vehicle with both autonomous and remote reality driving capabilities. Its autonomous driving system (figuratively, referred to as an autopilot) typically includes sensors, a trained artificial neural network, and vehicle dynamics and control. Therefore, the VUT has two driving modes. In one mode, the autonomous driving function is activated and the remote reality driving function is disabled, effectively acting as an autonomous vehicle driven by its own autonomous driving system. In the other mode, the autonomous driving function is disabled and the remote reality driving function is enabled, effectively acting as a remote reality driving vehicle driven remotely by a human driver via remote reality. Similarly, in both modes, the onboard driver and passenger in the VUT are replaced by sensor-equipped crash test dummies to simulate human occupants and collect data on collision injury outcomes for the driver and passenger. VUT types include the same types as the aforementioned VT vehicle, as well as new autonomous service robots, including but not limited to certain package delivery vehicles that participate in traffic as bicycles (i.e., occupying bike lanes) and certain food delivery robots that participate in traffic as pedestrians (i.e., occupying sidewalks and crosswalks). These VUTs are dual autonomous vehicles and remote reality driving robots. For simplicity, all vehicles in the present invention also include robots unless otherwise explicitly stated.

[0104] In a typical embodiment of the system, all traffic participants are equipped with crash sensors. After a collision occurs, all sensor data is collected and all dummies are inspected for damage. The severity of the collision is fully measured and incorporated into the test results to assess the driving skills of the VUT driver (autonomous or human).

[0105] In an advanced embodiment of the system of the present invention, public transportation is incorporated into the scenario in addition to the target vehicle (TV) and the test vehicle (VUT). When potential collisions between public transportation and the target vehicle (TV) and the test vehicle (VUT) are eliminated by safety separation, it can be included in the traffic flow within the scenario. For example, during testing of a public road scenario with two-way safety separation, half of the road is closed for a driving test, while the other half remains open. The other half of the public transportation flow is visible to the test vehicle (VUT) and is also considered part of the test traffic flow (background traffic flow), further increasing the attention burden or distracting the test vehicle (VUT) driver (autonomous or human). In another advanced embodiment of the system of the present invention, safety separation walls are installed between lanes of same-direction traffic flow so that the public transportation flow in these safety-separated same-direction lanes is intentionally incorporated into the test traffic flow as background traffic flow, further increasing the attention burden or distracting the test vehicle (VUT) driver.

[0106] The communication network in a typical embodiment of the system of the present invention consists of one or more base stations (BSs), a wireless network between all RRDV traffic participants and the BSs, one or more roaming stations (RSs), a backbone network between the BSs and RSs, and a backbone network between the RSs and VT / VUT facilities. The wireless network is a roaming vehicle network, conceptually similar to a public wireless roaming network. It connects all RRDVs roaming at vehicle speeds to one or more base stations, and further to RSs for delay-free or near-delay-free wireless roaming reception or transmission. The backbone network may include, but is not limited to, fiber optic networks, cable networks, microwave networks, and laser networks. The backbone network can utilize existing digital backbone connections, offering extremely high throughput and zero latency over distances of several to over 100 miles (primarily determined by the speed of light). The communication network also includes a driving mode switch that randomly selects either the DS in room A or B to connect to the VUT. When the DS in room A connects to the VUT, the VUT is in autonomous driving mode, driven by the DS. When the DS in room B connects to the VUT, the VUT is in remote driving mode, driven remotely by human driver B.

[0107] In a typical embodiment of the system of the present invention, a target vehicle driving facility is a facility for a human driver to drive a target vehicle (TV), which includes one or more remote driving stations (DS), each DS including, but not limited to, a display device for displaying the current view of the remote reality, a perspective input device, and a vehicle control input device, and is connected to a remote reality driving vehicle (RRDV). The display device includes a monocular display or a stereoscopic display, including an external monitor, a projector, or a head-mounted display (e.g., a mixed reality head-mounted display). The perspective input device can be a button or a detector for the angle of the human head facing or the angle of the eye viewing (also known as a head tracking or eye tracking device), typically integrated into the head-mounted display. The vehicle control input device includes, but is not limited to, a steering wheel, a brake pedal, an accelerator pedal, and, in the case where the RRDV is actually a remote reality driving robot, other robot arm control input devices, including motion sensors worn by the driver or operator, and human movement or action recognition and measurement devices based on computer vision. The human driver views the remote reality view of the remote reality driving vehicle (RRDV) to remotely observe the real situation on the scene and operates the remote reality driving vehicle (RRDV) by inputting perspective control commands and vehicle control commands.

[0108] The Turing-like vehicle under test (VUT) testing facility of the present invention system, in a typical embodiment, comprises three enclosed rooms, designated Room A, Room B, and Room C, similar to a Turing test. Room A houses a remote driving station (DS) without a human driver; Room B houses a remote driving station (DS) with human driver B. A random switch determines whether the remote driving station (DS) in Room A or Room B will connect to the test vehicle (VUT). When the remote driving station (DS) in Room A connects to the test vehicle (VUT), the test vehicle (VUT) is in autonomous driving mode, driven by its own autonomous driving system. When the remote driving station (DS) in Room B connects to the test vehicle (VUT), the test vehicle (VUT) is in remote reality driving vehicle (RRDV) mode, driven by human driver B. Each driving test will be repeated at least twice: once with the test vehicle (VUT) in autonomous driving mode and once with the test vehicle (VUT) in remote reality driving vehicle (RRDV) mode. Both sets of test results, including all video recordings, data recordings, and crash severity inspection reports, are anonymously sent to Room C, where a human evaluator C compares the two sets of test results side by side or alternately and decides which set (i.e., which driver) wins, with better driving skills, or if the two sets are tied, with the same driving skills.

[0109] In a typical embodiment of the system of the present invention, each target vehicle (VT) is paired with a remote driving station (DS) in a target vehicle driving facility, while the vehicle under test (VUT) is paired with a remote driving station (DS) in room A or B. The basic function of the communication network is to provide a driving link between each pair of remote reality driving vehicle (RRDV) and remote driving station (DS). The driving link includes a remote reality driving vehicle (RRDV) to remote driving station (DS) video transmission link, which spans the wireless network and the backbone network to transmit video from the remote reality driving vehicle (RRDV), whether it is the target vehicle (VT) or the test vehicle (VUT)) to its paired remote driving station (DS); a remote reality driving vehicle (RRDV) to remote driving station (DS) data transmission link, which spans the wireless network and the backbone network to transmit vehicle status from the remote reality driving vehicle (RRDV) to its paired remote driving station (DS); and a remote driving station (DS) to remote reality driving vehicle (RRDV) reverse data transmission link, which spans the backbone network and the wireless network to transmit view control commands, vehicle control commands, and robot control commands (when the remote reality driving vehicle (RRDV) is a remote reality driving robot (RRDR)) from the paired remote driving station (DS) back to the remote reality driving vehicle (RRDV). All three links are latency-critical, that is, they are required to be latency-free or nearly latency-free. In the present invention, a delay of less than 1 millisecond is referred to as zero delay, while a delay greater than 1 millisecond but still close to or comparable to 1 millisecond is referred to as near zero delay. Since the delay of the backbone network is very small and can usually be ignored, the delay of all links is mainly determined by the delay of the wireless link. In traditional wireless networks (such as public cellular wireless networks), videos are transmitted with heavy compression. There is an inherent long delay in video compression and decompression. In a preferred embodiment, in order to eliminate this long delay, after the remote reality driving vehicle (RRDV) generates a video output representing the current view in the original uncompressed format, the remote reality driving vehicle (RRDV) uses the method of the previous series of inventions [3][4][5] to quasi-continuously modulate the uncompressed video of the remote reality driving vehicle (RRDV) with moderate bandwidth and transmit it on the wireless network with high image quality and no delay. Similarly, in a preferred embodiment, the wireless data link uses a dedicated digital wireless network to ensure zero delay and high quality, and is not interfered with by network congestion that often occurs in public wireless networks.

[0110] It should be noted that one feature of the present invention is that each driving test of the present invention is double-blind to ensure that it is conducted fairly for both the autonomous driver and the human driver. First, since the test vehicle (VUT) is randomly set to autonomous driving mode or remote reality driving vehicle (RRDV) mode, and the Turing-like test facility is isolated from the target vehicle (TV) driver, the target vehicle (TV) driver does not know whether the test vehicle (VUT) is driven by an autonomous driver or a human driver. Secondly, the human evaluator C is isolated from Room A and Room B, so the human evaluator C also does not know which set of test results of the test vehicle (VUT) is driven by the autonomous driver and which set is driven by the human driver. The double-blind test of the present invention ensures that the autonomous driver is fairly challenged by the human driver (the human driver of the target vehicle (TVs)) during the test, and is further fairly evaluated with the human driver (the human driver of the test vehicle (VUT)) to accurately judge the test results.

[0111] It should also be noted that another feature of the present invention is that each driving test of the present invention is a dual human-machine confrontation to evaluate the driving skill level and safety level of autonomous driving compared with human driving. First, the surrounding test traffic flow that confronts the autonomous driving (machine) in the present invention is the same as the traffic flow in the public road test, which is generated by vehicles driven by humans (people), and is different from the test traffic flow in the existing test field, which is generated by automated equipment. Secondly, the test results of the automatic driver (machine) are compared with the test results of the human driver (human) on the same test vehicle (VUT) in the same scenario. The human-machine confrontation test of the present invention ensures that the driving skills of the artificial intelligence are compared and measured with the driving skills of humans, and can answer the above two target questions.

[0112] The system of the present invention has many variations. The system used for driving tests can be remotely driven or remotely operated, allowing unsafe or unhealthy field operations to be performed remotely and safely. In one embodiment of an unmanned racing sport, all test vehicles (VTs) are remote reality driving vehicles (RRDVs), and no Turing-like testing facilities are required. Human racers remotely drive the RRDVs to compete against each other. Unmanned racing will promote the rapid development of the sport by eliminating human safety constraints (preventing driver injury and death). In another embodiment of an unmanned military-style sport, all test vehicles (VTs) are unmanned tank-type remote reality driving vehicles (RRDVs), and no Turing-like testing facilities are required. Human tank crews remotely drive and operate the unmanned tank-type remote reality driving vehicles (RRDVs), competing against each other as individuals or teams. This creates an armored combat sport, a new type of military-style sport that safely integrates military-style vehicles into real-world combat. In contrast, existing military-style sports, such as real-person counter-attack, cannot support armored vehicle combat. For example, the famous soft air survival game, originally popular in Japan, is a team shooting game in which participants tag and eliminate opposing players by shooting spherical plastic pellets with low-power air guns, and has spread globally as a true real-world combat military-style sport, but it only supports infantry players. In another embodiment, all test vehicles (VTs) are unmanned heavy machinery remote reality driving vehicles (RRDVs), widely including construction remote reality driving vehicles (RRDVs), mining remote reality driving vehicles (RRDVs), garbage disposal remote reality driving vehicles (RRDVs) and other ground vehicles or aerial flying machinery (especially drones). These remote reality driving vehicles (RRDVs) allow workers to remotely drive and work remotely in a safe and healthy environment, rather than working in person under various unsafe and unhealthy field conditions.

[0113] The Remote Reality Driving Vehicle (RRDV) of the present invention provides a remote driving experience that is the same as or approximately the same as the in-car driving experience at highway speeds or above, and is composed of a vehicle platform and a Remote Reality Driving Device (RRDD).

[0114] The vehicle platform is a conventional vehicle with an external driving interface, which can be driven and controlled by external electrical signals by sending drive and control commands of electrical signals to the external driving interface. In a preferred embodiment, the external driving interface is an external wire-controlled driving interface, which generally includes a traditional CAN interface, an Ethernet interface, etc. In another embodiment, the external driving interface also includes a mechanical arm and a mechanical leg for mechanically controlling the steering wheel and pedals. However, since these mechanical arms and mechanical legs greatly increase the execution delay, the agility and maneuverability of the remote reality driving vehicle (RRDV) is significantly reduced.

[0115] The remote reality driving device (RRDD) in a typical embodiment of the present invention mainly consists of the following parts: multiple cameras, a stitching engine, a display engine, a driving recorder and a communication engine.

[0116] In a preferred embodiment, the cameras of a remote reality driving device (RRDD) are identical in structure and evenly mounted around the circumference of the RRDV's horizontal plane, with the optical axes of all cameras intersecting at the center of the circle. Ideally, the cameras are distortion-free, and the stitching engine of the present invention stitches all camera videos together to produce a 360° horizontal surround view of the RRDV without latency. In practice, due to distortion that needs to be corrected in wide-angle or fisheye lenses, the stitching engine of the present invention can produce a 360° horizontal surround view with minimal latency, achieving near-zero latency when stitching all camera videos together. In a typical embodiment, the 360° horizontal surround view is represented in the video as a spherical image, which is then mapped back to a rectangular image in an equirectangular format (similar to how the Earth's surface is mapped to a rectangular two-dimensional world map). In contrast, in other surround view camera systems where the cameras are not all mounted horizontally, for example, some existing surround view cameras place six or more cameras across the entire sphere, stitching these camera videos together will result in a delay of one or more video frames, making it impossible to achieve the zero or near-zero latency stitching of the present invention with such an architecture. After stitching, the display engine projects the horizontal 360° surround video onto a designated display area in the mathematical model of the remote driving station's (DS) display device, based on the remote driving station's (DS) viewing angle control commands. This generates a current-view video that reflects the portion of reality in the direction the remote driver is currently viewing. Simultaneously, the camera video, the stitched surround-view video, the current-view video, along with vehicle status (e.g., vehicle speed, throttle percentage, braking percentage), remote reality driving vehicle (RRDV) vehicle control commands, and other supporting information (e.g., GPS location, navigation, etc.) are recorded by a drive recorder and later sent to assessor C for inclusion in the test result set to be evaluated. The drive recorder, equivalent to a flight recorder in the aviation industry, serves as a vehicle's "black box," recording all driving data and typically surviving a vehicle crash. In a preferred embodiment, the stitching engine and display engine are combined to eliminate the long latency inherent in a standalone display engine. The camera-side display engine also reduces the bandwidth required to transmit the remote reality video that provides 360° surround view.

[0117] The communication engine of the remote reality driving device (RRDD) includes, but is not limited to, the following parts: a wireless video transmitter, a two-way wireless data transceiver, and other auxiliary wireless transceivers. The wireless video transmitter transmits the current view video via a wireless network in a delay-free or near-delay-free manner when the remote reality driving vehicle (RRDV) is moving at or above highway speed. In a preferred embodiment, the video transmitter adopts the method [3] [4] [5] in the previous series of inventions to directly transmit uncompressed video without video compression. Therefore, the long delay inherent in video compression in the transmitter and video decompression in the receiver is eliminated, and delay-free video transmission (i.e., transmission delay is less than 1 millisecond) is obtained. The two-way wireless data transceiver receives view control commands and vehicle control commands, and transmits the vehicle status back in a delay-free manner. Although the latest public wireless communication networks are designed to provide a delay of less than 1 millisecond, actual public wireless communication networks are not reliable in terms of coverage and QoS, and are insufficient to meet the key requirements of high-speed vehicle driving. In a preferred embodiment, as opposed to public wireless network data flow, the bidirectional data transceiver uses a dedicated digital wireless transceiver to provide guaranteed delay-free and interference-free public network data flow as well as reliable coverage and reliable QoS to meet the key requirements of high-speed vehicle driving.

[0118] It's important to note that the system and remote reality driving vehicle (RRDV) of the present invention are latency-critical, meaning they are designed to minimize latency. In theory, the system and remote reality driving vehicle (RRDV) of the present invention are remote versions of visual feedback control systems. The forward path is from the remote reality world to the current view displayed in front of the driver. The driver is the visual feedback controller, generating control commands based on their vision by reacting to input devices. The reverse path is from the driver's reactions to the live remote reality world, which is further modified by the feedback control commands. This forms a closed-loop control system. As with any closed-loop control system, performance and stability degrade as closed-loop latency increases. If closed-loop latency is too long, the control system becomes unstable. In remote driving, this means the driver can't even control the remote reality driving vehicle (RRDV) to move straight forward, but instead sways from side to side, further losing control and drifting out of lane. In contrast, for remote reality systems without feedback control (where viewing angle is not considered because it doesn't modify the real world), latency is not critical at all. The viewer can only perceive the remote reality world but cannot alter it in any way. In such viewing-only systems, the display engine resides on the viewer's end and can alter the current view based on the viewer's perspective. However, the remote reality itself remains unchanged. In extreme cases, recorded remote reality programs and sports games can be replayed for the viewer. Viewers cannot alter the remote reality programs or sports games. These viewing-only systems have only a forward path and no closed loop. Therefore, latency is insignificant. Typically, when their content is streamed over the internet, these viewing-only remote reality systems experience latency in the order of seconds, making them completely unsuitable for high-speed vehicle driving.

[0119] The test vehicle (VUT) faces varying degrees of collision risk in different scenarios. To illustrate this, the present invention defines spatial degrees of freedom (SDoF), which refers to the number of free spaces in which the test vehicle (VUT) can move forward, backward, left, and right without collision. For example, assume a road has three lanes running in the same direction, and the test vehicle (VUT) is driving in the middle lane. If no cars are surrounding the test vehicle (VUT), its spatial degrees of freedom (SDoF) is the maximum, at 4. If four cars are surrounding the test vehicle (VUT), preventing it from moving forward, backward, left, or right and forcing it to follow the surrounding vehicles in the same lane, its spatial degrees of freedom (SDoF) is the minimum, at 0. Time to collision (T2C) refers to the time it takes for the test vehicle (VUT) to crash while maintaining its current driving state. Even if the number of free spaces (SDoF) for the test vehicle (VUT) is 0, if all vehicles are traveling at the same speed, the time to collision (T2C) is infinite, and the test vehicle (VUT) has no collision risk.

[0120] In an exemplary embodiment of the present invention, a Turing-like double-blind human-machine adversarial driving test method for an autonomous vehicle includes the following steps: a startup phase, a capture phase, a test phase, a check phase, and an evaluation phase. A single cycle of the process, including the startup phase, the capture phase, the test phase, and the check phase, is referred to as a test run. Typically, a driving test includes one or more test runs, followed by an evaluation phase. Each driving test always begins with the first test run, as follows:

[0121] The startup phase is the first step at the beginning of each test. First, the system randomly selects the test vehicle (VUT) to connect to Room A or Room B in the Turing-like VUT test facility. If the test vehicle (VUT) is connected to Room A, the test vehicle (VUT) is driven by the autonomous driving system, otherwise it is driven by human driver B. In either case, the test vehicle (VUT) starts from a certain location, whether off-road (e.g., a parking lot, garage, pickup area) or on the road, and approaches a certain location on the road. The target vehicle (VT) driver does not know who is actually driving the test vehicle (VUT). All target vehicles (VT) are driven by their human drivers and are arranged to create a safe scenario for the test vehicle (VUT). The startup phase includes one or more scenarios that are safe and do not show any signs of potential risks, so that one or more target vehicles are close enough to the test vehicle. Multiple test runs that are adjacent in time usually start from different scenarios so that the test vehicle (VUT) cannot be sure of the upcoming test scenario.

[0122] The capture phase begins when the VUT is sufficiently close and the target vehicle (VT) begins taking actions to capture the expected vehicle position relative to the VUT (e.g., the positions of vehicles surrounding the VUT). The capture phase consists of one or more scenarios that are safe and do not indicate any potential risk. However, as the capture phase progresses, the VUT's spatial degrees of freedom (SDoF) and time to crash (T2C) typically decrease. If the VT successfully captures the expected vehicle position relative to the VUT, the expected capture scenario is reached, and the capture phase ends with the VUT being captured in the expected capture scenario. If the VT fails to capture the expected vehicle position relative to the VUT and misses the capture opportunity, the expected capture scenario is not reached, and the capture phase ends with the VUT not being captured in the expected capture scenario, and the test run ends without any test results.

[0123] The test phase begins when the test vehicle (VUT) is captured in a predetermined capture scene and the target vehicle (VT) begins to act to create a test scene for the test vehicle (VUT). Once a certain test scene is reached, the driving of the test vehicle (VUT) will be tested. Many test scenes have different degrees of collision risk. In a preferred embodiment, after reaching the capture scene, multiple subsequent test scenes are arranged, and one of them is selected based on a certain randomness for the target vehicle (VT) driver to actually perform. This increases the blindness of the test scene to the test vehicle (VUT), making it impossible for the test vehicle (VUT) to predict the upcoming test scene and unable to make a preset driving response. Therefore, the test reveals the true driving skills of the test vehicle (VUT), just as achieved in public road testing.

[0124] In an advanced embodiment, to increase the blindness of the test vehicle (VUT), more target vehicles (VT) than necessary are involved in the driving test during the startup, acquisition, and test phases. The required target vehicles (VT) are those that are commanded to take action during the test, while the remaining target vehicles (VT) serve as backup vehicles for the action vehicles or simply appear as background traffic. This prevents the test vehicle (VUT) from even pre-identifying the target vehicle (VT) in the traffic scene and, of course, from making the intended driving response, further increasing the blindness of the test scene to the test vehicle (VUT), just as achieved during public road testing.

[0125] The inspection phase begins after the test phase. Regardless of whether the test vehicle (VUT) safely passes the test scenario, all test results are collected, including but not limited to all video and information records of the test vehicle (VUT) and all target vehicles (VTs) participating in this test run, as well as any other reports. For example, if a collision occurs, all vehicles and test dummies involved will be inspected and a collision report will be generated, which includes estimated property damage, personal injury and fatality costs, just as if the collision were a car accident occurring on a public road, thereby deriving the severity of the collision. This collision report is an important part of the test results of the present invention. It can distinguish and quantify the severity of the collision, thereby allowing an accurate quantitative assessment of driving skills.

[0126] After collecting a complete set of test results from the first test run, a second test run is assumed. In one embodiment, the selection of the test vehicle (VUT) driver for this run is not random, but rather reversed from the selection in the first test run. Therefore, after two test runs, a pair of test result sets for the same test scenario are collected: one set from the test run with the autonomous test vehicle (VUT) and the other from the test run with a human. In another embodiment, to increase the blindness of the test to the target vehicle (VT) driver (i.e., it is not even certain whether an autonomous driver is driving the test vehicle (VUT)), the test vehicle (VUT) driver is randomly selected in each test run. This test run is then repeated multiple times until multiple sets of test results from the autonomous test vehicle (VUT) and multiple sets of test results from the human-driven test vehicle (VUT) are collected. Then, a pair of test result sets for the same test scenario are randomly selected from the collected test result sets: one set from the autonomous test vehicle (VUT) and the other set from the human-driven test vehicle (VUT). This also increases the blindness of the driving test to the human driver of the test vehicle (VUT). Even if the human driver of the test vehicle (VUT) knows that he is driving a certain test vehicle (VUT) to participate in a certain driving test, he is not sure whether his test result set will be selected and whether it will be evaluated. In another embodiment, a pair of test result sets are randomly selected from the collected test result sets of the same test scenario, each of which may come from a test run of the autonomous driving test vehicle (VUT) or a human-driven test vehicle (VUT). The pair of test result sets may include all three combinations of test vehicle (VUT) drivers (autonomous driving + autonomous driving, autonomous driving + human driving, human driving + human driving). This further increases the blindness of the driving test to assessor C, because he is not even sure whether there is a test result set from the autonomous driving test vehicle (VUT) included that he needs to evaluate. Maximizing blindness ensures that the driving skills and safety of autonomous driving are fairly and accurately evaluated compared to humans.

[0127] The evaluation phase begins after obtaining or selecting a pair of test result sets. The pair of test result sets are anonymously sent to Room C in the Turing-like test vehicle (VUT) test facility and displayed side by side or alternately. Without knowing the identity information of the two drivers, human evaluator C compares and decides which set has better driving skills and wins, or the two sets have the same driving skills and are tied, or no evaluation can be performed (for example, the test scenarios in the two test runs are not similar enough to be compared), and thus no evaluation conclusion is given. In a preferred embodiment, the same pair of test result sets are evaluated by multiple human evaluators, and the evaluation results of all evaluators are combined according to certain rules. This can improve the fairness of the evaluation by offsetting the personal preferences of human evaluators.

[0128] In a typical embodiment of the present invention, a method for evaluating how good autonomous driving is compared to human driving includes, but is not limited to, the following steps: setting up a rating system, evaluating human drivers, and evaluating autonomous drivers.

[0129] In one embodiment, the rating system is set up as follows: similar to Go, each driver is rated in terms of rank.

[0130] Given a specific level of driving skill, there are various methods to assess its rank. In one embodiment, an initial driver who has just obtained a driver's license is rated as 0 dan. After meeting certain conditions, the driver will be re-rated, for example, if a driver has participated in the driving test of all required scenes and obtained enough evaluation results in the last 12 months. In one embodiment, the scoring of each evaluation result is as follows: if a driver wins, 1 point; if the driver loses, 0 points; if two drivers tie, each gets 0.5 points. One evaluation result corresponds to the result of a Go game. In one embodiment, the driver's score is combined as follows: the driver's winning rate is calculated as the total score of all his evaluation results divided by the percentage of his total number of evaluations, and the credibility of the driver's winning rate is calculated as the total number of test scenes he has participated in the driving test and obtained the evaluation divided by the number of all required scenes. In one embodiment, when the driver's winning rate credibility reaches a certain level (that is, the coverage requirement for all required test scenes), the driver is re-rated. If a driver's win rate against other drivers of the same rank is above a certain upper limit, he will be promoted by 1 rank; if a driver's win rate against other drivers of the same rank is below a certain lower limit, he will be demoted by 1 rank.

[0131] Assessing human drivers. Human drivers have different levels of driving skills. Before evaluating or measuring the driving skill level and safety level of autonomous driving, the human drivers of the test vehicles (VUTs) need to be rated, just like human chess players must be rated before playing against AlphaGo. Based on the set rating system, all human drivers are rated and re-rated, and their ranks are updated accordingly. In one embodiment, in the rating system, all initial drivers are rated as 0th rank. All non-initial drivers are rated according to 0th rank drivers as follows: find a 1st rank driver and rate him with the 0th rank driver; find a 2nd rank driver and rate him with the 1st rank driver, and so on. In a rating system, as an empirical estimate, after 1 year of daily but non-professional driving, it is expected that most initial drivers (e.g., 70%) will reach 1st rank; after 2 years of daily but non-professional driving, it is expected that a large number of initial drivers (e.g., 50%) will reach 2nd rank, and so on. It can generally be expected that everyday but non-professional drivers usually have low to medium levels of driving skills (e.g., 0 to 6); professional drivers usually have medium levels (e.g., 4 to 7); and professional racing drivers have high levels of driving skills (e.g., 7 to 9).

[0132] Rating autonomous drivers. After human drivers are ranked, autonomous drivers can now be ranked against human drivers based on the same rating system as non-initial human drivers. For example, in one embodiment, assuming the promotion threshold is a 90% win rate, if an autonomous vehicle's autonomous driver is evaluated against a 2-dan human driver in all required scenarios and achieves a 100% win rate, the autonomous driver is ranked 3-dan and requires further testing and ranking against a 3-dan human driver. In another example, if an autonomous vehicle's autonomous driver is evaluated against a 9-dan human driver in all required scenarios and achieves an 80% win rate, the autonomous driver is ranked 9-dan. This is similar to the rating system used by AlphaGo.

[0133] The method of the present invention for evaluating how safe autonomous driving is compared to human driving, in a typical embodiment, includes but is not limited to: rating the test scenarios and determining the safety level.

[0134] The test scenarios are rated. The test scenarios have different driving difficulties and different degrees of collision risk. After the human driver is rated, in one embodiment, all test scenarios are rated as follows: the test result set of each human-driven test vehicle (VUT) is evaluated. If the human assessor C determines that the test vehicle (VUT) successfully and safely passes the scenario (i.e., has an acceptable level of safety), the scenario fails and gets 0 points; if the assessor determines that the test vehicle (VUT) fails to safely pass the scenario (i.e., a collision occurs or has an unacceptable level of risk), the scenario wins and gets 1 point; if the assessor determines that the test vehicle (VUT) passes and fails half in terms of safety, the scenario is tied with the driver and gets 0.5 points. Further, the winning rate of the test scenario is calculated as the sum of the scores of the scenario divided by the percentage of the total number of times the same-level driver drives the test vehicle (VUT) to test the scenario. The winning rate credibility of the test scenario is calculated as the percentage of the number of driving tests performed by the same-level driver in the scenario divided by the required number of driving tests. The test scenario is then ranked against test drivers of the same rank in the rating system based on the win rate, just as if the scenario were a non-novice driver. Therefore, the test scenario is also ranked and assigned a rank. In one embodiment, since human driver ratings start at rank 0, simple scenarios have negative ranks. For example, if a scenario has a 0.01% probability of causing a collision when a novice driver navigates it, the scenario is rated -4.

[0135] Determining the safety level. In one embodiment, the safety level of a driver (autonomous driver or human driver) driving a vehicle under test (VUT) through a scenario is determined by the difference in their skill level, also referred to as the driver's safety level. For example, in a certain rating system, if the autonomous vehicle (AV) has the same skill level as the scenario, its safety level is level 0, and the autonomous vehicle (AV) has a 50% chance of successfully completing the scenario, meaning it has a 50% chance of colliding with the scenario. If the autonomous vehicle (AV) is one level higher than the scenario, its safety level is level 1, with a 90% chance of successfully completing the scenario and a 10% chance of colliding. If the autonomous vehicle (AV) is two levels higher than the scenario, its safety level is level 2, with a 99% chance of successfully completing the scenario and a 1% chance of colliding with the scenario, and so on. It is important to note that a driver's driving safety level and driving skill level are different. As a general rule of thumb, drivers with lower driving skills can improve their safety level by avoiding high-risk scenarios.

[0136] The technical solution of the present invention is described in further detail below with reference to the accompanying drawings.

[0137] FIG1 shows an embodiment of a Turing-like double-blind human-machine adversarial driving test system for an autonomous vehicle according to the present invention.

[0138] The system shown in the figure includes a road network 1100, which includes a northbound lane 1110, a southbound lane 1130, and lane dividers 1120. On this road segment, vehicles can cross lane dividers 1120, which are shown as two parallel dashed lines.

[0139] The system shown in the figure includes a test vehicle (VUT) 1500 traveling in the northbound lane 1110, and three target vehicles (TVs) 1210, 1220, and 1230. The target vehicle (TV) 1210 follows the test vehicle (VUT) 1500 in the northbound lane 1110; the target vehicle (TV) 1220 is ahead of the test vehicle (VUT) 1500 in the same lane; and the target vehicle (TV) 1230 is traveling in the southbound lane 1130 and heading toward the test vehicle (VUT) 1500. This scenario is designed to test whether the test vehicle (VUT) 1500 can overtake the target vehicle (TV) 1220 in front of it by using the southbound lane 1130, that is, crossing the lane divider 1120, driving in the opposite direction in the southbound lane 1130, overtaking the target vehicle (TV) 1220, and then crossing the lane divider 1120 again to return to the northbound lane 1110 and continue traveling north.

[0140] The system in the figure includes a target vehicle (TV) driving facility 1400, which includes a remote driving station (DS) 1410 and its equipped human driver D1, a remote driving station (DS) 1420 and its equipped human driver D2, and a remote driving station (DS) 1413 and its equipped human driver D3.

[0141] The system shown includes a Turing-like vehicle under test (VUT) test facility 1600, which includes a remote driving station (DS) 1610 without a human driver in room A (the upper left room), a remote driving station (DS) 1620 with a human driver B in room B (the upper right room), and a dual display device in room C (the lower room) and its equipped human evaluator C.

[0142] The system shown includes a communication network 1300 that connects vehicles (a target vehicle (VT) and a test vehicle (VUT)) with remote driving stations (DS) in two facilities. Communication network 1300 includes base stations (BS) 1310, 1320, and 1330, and wireless networks between the vehicles and the base stations (BS), including wireless connections 1211, 1221, 1231, and 1501, connecting target vehicles (VT) 1210, 1220, and 1230 and test vehicle (VUT) 1500 to base stations (BS) 1310, 1320, 1330, and 1310, respectively. The communication network 1300 also includes a roaming station (RS) 1340 and a fiber optic trunk network including fiber optic connections 1311, 1321, and 1331 connecting the base stations (BSs) 1310, 1320, and 1330 to the roaming station (RS) 1340, and another fiber optic trunk network including fiber optic connections 1341, 1342, and 1343 connecting the remote driving stations (DSs) 1410, 1420, and 1430 to the roaming station (RS) 1340. The communication network 1300 also includes a driving mode switch 1510 and a fiber optic trunk network including fiber optic connection 1345 connecting the driving mode switch 1510 to the roaming station (RS) 1340, and fiber optic connections 1511 and 1512 connecting the driving mode switch 1510 to the remote driving stations (DSs) 1610 and 1620, respectively.

[0143] Three target vehicles (VTs), 1210, 1220, and 1230, are paired with three remote driving stations (DSs) 1410, 1420, and 1430, and the communication network 1300 provides three driving links to connect each target vehicle (VT)-remote driving station (DS) pair. The first link connects the target vehicle (VT) 1210 with the remote driving station (DS) 1410 and includes a wireless connection 1211, a base station (BS) 1310, an optical fiber connection 1311, a roaming station (RS) 1340, and an optical fiber connection 1341. The second link connects the target vehicle (VT) 1220 with the remote driving station (DS) 1420 and includes a wireless connection 1221, a base station (BS) 1320, an optical fiber connection 1321, a roaming station (RS) 1340, and an optical fiber connection 1342. The third link connects the target vehicle (VT) 1230 with the remote driving station (DS) 1430 and includes a wireless connection 1231, a base station (BS) 1330, an optical fiber connection 1331, a roaming station (RS) 1340, and an optical fiber connection 1343. These three links provide human drivers D1, D2, and D3 with a remote driving experience that is the same as, or nearly the same as, their driving experience in the vehicle (VT).

[0144] However, depending on the switch 1510, the test vehicle (VUT) 1500 is paired with the remote driving station (DS) 1610 or the remote driving station (DS) 1620, and the communication network 1300 provides a driving link to connect the test vehicle (VUT) and the remote driving station (DS) pair. When the switch 1510 selects to connect the optical fiber connection 1345 to 1511, the test vehicle (VUT) 1500 is in the autonomous driving mode. The driving link of the test vehicle (VUT) connects the test vehicle (VUT) 1500 to the remote driving station (DS) 1610 in room A (upper left corner) without a human driver, including the wireless connection 1501, the base station (BS) 1310, the optical fiber connection 1311, the rover station (RS) 1340, the optical fiber connection 1345, the switch 1510, and the optical fiber connection 1511; when the switch 1510 selects to connect the optical fiber connection 1345 to 1511, the test vehicle (VUT) 1500 is in the autonomous driving mode. The driving link of the test vehicle (VUT) connects the test vehicle (VUT) 1500 to the remote driving station (DS) 1610 in room A (upper left corner) without a human driver. 10 When the fiber optic connection 1345 is selected to be connected to 1512, the test vehicle (VUT) 1500 is in the remote reality driving vehicle (RRDV) mode, and the driving link of the test vehicle (VUT) connects the test vehicle (VUT) 1500 to the remote driving station (DS) 1620 equipped with human driver B in room B (upper right corner), including wireless connection 1501, base station (BS) 1310, fiber optic connection 1311, roaming station (RS) 1340, fiber optic connection 1345, switch 1510 and fiber optic connection 1512.

[0145] Through the four driving links described above, human drivers D1, D2, and D3, as well as human driver B when the test vehicle (VUT) is in remote reality driving vehicle (RRDV) mode, can perceive the remote reality surrounding the live vehicle on the display device in front of them, make driving decisions, operate input devices, and generate feedback control, including view control commands and vehicle control commands. The functions of the driving test system in this embodiment will be described in detail later together with the driving test method.

[0146] FIG2 illustrates an embodiment of a zero-latency or near-zero-latency remote reality driving vehicle and robot according to the present invention. Because both the test vehicle (VUT) and the target vehicle (VT) are remote reality driving vehicles (RRDVs), and the test vehicle (VUT) is structurally a superset of the target vehicle (VT), the embodiment shows a remote reality driving vehicle (RRDV) for the test vehicle (VUT). The RRDV is a two-in-one autonomous vehicle and remote reality driving vehicle, consisting of a vehicle platform 2100 and a remote reality driving device (RRDD) 2200.

[0147] In one embodiment, the vehicle platform 2100 includes an in-vehicle human driving system, including a steering wheel 2110, an accelerator pedal 2120, a brake pedal 2130, and other devices for in-vehicle human driving. In one embodiment, the vehicle platform 2100 also includes an automated driving system 2150 (ADS, also known as an automatic pilot). The ADS 2150 typically includes various sensors, including cameras, lidar, radar, GPS locators, and ultrasonic sensors, to observe the surrounding environment; recognition and decision-making devices, typically including various artificial neural networks and expert systems, to make driving decisions and plans; and vehicle dynamics and control systems to monitor vehicle status and generate vehicle control commands. In one embodiment, the vehicle platform 2100 also includes an external driving interface 2140, to which remote driving commands can be sent and from which vehicle status can be retrieved. In one embodiment, the vehicle platform 2100 also includes a VCU 2170, which executes various received vehicle control commands and reports vehicle status, as well as an in-vehicle mode switch 2160. When the in-car mode switch 2160 connects the in-car human driving system (lowest switch position) to the VCU 2170, the test vehicle (VUT) is in the traditional in-car human driving mode and is driven by the in-car human driver; when the in-car mode switch 2160 connects the automatic driving system 2150 (middle switch position) to the VCU 2170, the test vehicle (VUT) is in the automatic driving mode and is driven by its automatic driving system 2150 (automatic driver); when the in-car mode switch 2160 connects the external driving interface 2140 (highest switch position) to the VCU 2170, the test vehicle (VUT) is in the remote driving mode and is driven by the human driver in its paired remote driving station (DS). For remote reality driving vehicles (RRDV), the in-car human driving system is not necessary. It is shown in the figure to better describe the automatic driving system 2150 or the external driving interface 2140 in parallel. In addition, if the remote reality driving vehicle (RRDV) is only used for the target vehicle (VT), the automatic driving system 2150 is not needed.

[0148] The remote reality driving device (RRDD) 2200 with a stitched display two-in-one engine, as described later, receives perspective control commands from the wireless connection 2202, generates a current view under the received perspective, and sends it to the wireless connection 2202; the remote reality driving device (RRDD) 2200 also receives vehicle control commands from the wireless connection 2202 and sends them to the external driving interface 2140 via the wired connection 2201. When in the remote reality driving vehicle (RRDV) mode, it further controls the vehicle through the switch 2160 and the VCU 2170, and returns a vehicle status report and sends it to the wireless connection 2202.

[0149] FIG3 illustrates an embodiment of a remote reality driving device according to the present invention, comprising a surround vision imaging device 2210, a display engine 2220, a driving recorder 2230, and a communication engine 2240. The surround vision imaging device 2210 includes cameras 2211, 2212, 2213, 2214, 2215, and 2216, all of which are of identical structure and mounted on a horizontal circumference. All cameras are synchronized. The optical axes of all cameras intersect at the center of the circumference. All cameras have negligible or low distortion. Each camera can be a monocular or stereo camera. As described above, this is the structure of a surround vision system with the lowest stitching latency. Furthermore, the surround vision imaging device 2210 includes a stitching engine 2217, which receives all video outputs from the cameras and stitches the frame images of each video into a horizontal 360° surround vision image with zero or near-zero latency. In one embodiment, the stitched surround view image is mathematically located on a sphere, but does not necessarily need to cover the entire sphere (i.e., a full spherical image) because vehicle driving does not require a remote reality view in a vertical upward direction (i.e., toward the North Pole) or downward direction (towards the South Pole). In another embodiment, the stitched image can be located on a 360° cylindrical surface. The stitched image can be monocular or stereoscopic. To be compatible with existing planar video interfaces, the stitched spherical image is mapped to a rectangular image using an equirectangular format and sent out via wired connection 2218.

[0150] The display engine 2220 receives a video representing a stitched horizontal 360° surround view from the surround vision imaging device 2210 via a wired connection 2218. In one embodiment, the equirectangular image is mapped back into a spherical image. Based on the view control commands received from the communication engine 2240 via a wired connection 2243 and known parameters of the display device in the remote driving station (DS), including but not limited to the horizontal field of view (HFOV), the vertical field of view (VFOV), and the display surface type (flat or curved), the spherical image is projected onto a mathematical model of the display surface, thereby generating a video representing the current view of the driver of the remote driving station (DS) paired with the remote reality driving vehicle (RRDV) and transmitting it via a wired connection 2221. The current view can be monocular or stereoscopic.

[0151] The driving recorder 2230 records all videos and all other information known to the remote reality driving device (RRDD), including but not limited to single camera videos, surround vision videos generated by stitching, current view videos generated by projection, received viewing angle control commands, vehicle control commands, vehicle status, and other information, including GPS location, etc. When the remote reality driving device (RRDD) is operating normally, the records in the driving recorder 2230 are usually sent out in non-real time via the wired connection 2231; when the remote reality driving device (RRDD) is damaged (for example, a car accident occurs during a driving test) and can no longer operate normally, the driving recorder needs to survive the collision, so the saved records can be manually retrieved. The obtained records are very necessary for analyzing the car accident collision.

[0152] The communication engine 2240 includes a two-way wireless data transceiver and a wireless video transmitter. To minimize latency, all videos in the remote reality driving vehicle (RRDV) are uncompressed, except for the video recorded in the drive recorder 2230. In conventional remote reality driving vehicles, the communication engine does not include a dedicated wireless video transmitter, but only a two-way wireless data transceiver, such as a data modem for a public wireless network. The video is also sent over the wireless data network. Due to the limited bandwidth of the public wireless network, the video is heavily compressed to reduce the bandwidth. However, video compression results in long and variable delays and cannot meet the requirements of delay-critical systems (e.g., remote reality driving vehicles (RRDVs) at highway speeds). The communication engine 2240 uses a dedicated uncompressed wireless video transmitter that uses the previous series of inventions [3][4][5] to significantly reduce the redundant energy of the video signal by applying non-causal predictive coding (but not compression) and directly modulating the DCT spectrum onto OFDM subcarriers or singular value subchannels without digital constellation. This provides delay-free and uncompressed video transmission up to highway speeds or above.

[0153] FIG4 illustrates an embodiment of a remote reality driving device of the present invention having a two-in-one stitching and display engine. The display engine 2220 in FIG3 still generates considerable internal latency because it needs to remap the input video in a flat rectangular format back into a spherical image. To eliminate the internal latency in the display engine, the stitching engine 2217 and the display engine 2220 are combined into a two-in-one stitching and display engine 2227, as shown in FIG4. As a feature of this two-in-one engine, after the stitching process is completed and the stitched spherical image is obtained, the display process directly projects the stitched spherical image onto a mathematical model of the display surface, eliminating the calculations required to convert the stitched spherical image into a flat equirectangular video format and then back into a spherical image, as well as the long latency associated with these calculations. All other aspects of the embodiment in FIG4 are identical to those in FIG3, and therefore, for the sake of brevity, their description will not be repeated.

[0154] It should be noted that unlike conventional remote reality systems, where the display engine is located on the viewer side, the display engine of the remote reality driving device (RRDD) of the present invention is located on the camera side. As another feature of this two-in-one engine, no matter which direction the driver is looking, this two-in-one engine only transmits the video of the current view instead of the entire surround vision video, so that the video transmission bandwidth of the surround vision system is greatly reduced, and there is no video compression and its inherent long delay or quality loss. For example, assuming that the horizontal field of view (HFOV) of the driver on the display device side is 120°, no matter which direction the driver is looking, only about 1 / 3 of the 360° surround vision needs to be transmitted. If the stitched surround vision has a larger vertical field of view (VFOV) than the display device, the transmission bandwidth is further reduced exponentially.

[0155] It should be noted that the display engine of the remote reality driving device (RRDD) of the present invention is located at the camera end, allowing the stitching engine and the display engine to be merged into a two-in-one stitching and display engine, thereby eliminating the delay of the independent display engine and enabling the remote reality driving device (RRDD) to achieve zero or near zero delay.

[0156] In remote reality driving systems, latency is crucial for reducing driver visual perception errors. Specifically, optical-to-optical (O2O) latency refers to the total end-to-end delay from the optical image of the surrounding scene in front of the remote reality driving vehicle (RRDV) camera to the optical image generated in front of its remote driving station (DS) display device. Similarly, electrical-to-electrical (E2E) latency refers to the total end-to-end delay from the stitched front camera video output of the remote reality driving vehicle (RRDV) to the video input of its remote driving station (DS) display device. Obviously, optical-to-optical (O2O) latency includes camera latency (the delay from the camera's optical image input to the video output); display device latency (the delay from the display device's video input to the generated optical image); and electrical-to-electrical (E2E) latency. Assuming that in a highway driving test scenario, the remote reality driving vehicle (RRDV) may be traveling at 80 miles per hour (128.8 kilometers per hour), this is equivalent to 72.9 feet (35.8 meters) per second. Due to latency, the remote reality vehicle (RRDV) seen by the driver in the displayed image always lags behind the live RRDV. The distance between the live RRDV and the displayed RRDV is called the driver's visual perception error. Assuming an optical-to-optical (O2O) delay of 1 second, the driver's visual perception error is as high as 72.9 feet (35.8 meters). With such a large perception error, the remote driver effectively has no idea where the vehicle is, making remote driving practically inoperable. Assuming an O2O delay of 0.1 seconds, the driver's visual perception error is still as high as 7.29 feet (3.58 meters), equivalent to the size of a lane or car. During driving tests, to capture the vehicle under test (VUT) in a collision-risk scenario, the target vehicle (VT) driver typically drives the VT in a race car-like manner, quickly identifying a gap and then accurately and swiftly cutting in. Perception errors comparable to lane or car size make accurate and rapid maneuvering of target vehicles (VTs) difficult or even impractical. With optical-to-optical (O2O) latency of less than 10 milliseconds, the driver's visual perception error is less than 0.73 feet (0.36 meters), which is generally acceptable for racing-style driving at highway speeds. In the worst-case scenario, assuming the driving test includes oncoming vehicles on a two-lane road, the perception errors of relative speed and oncoming vehicles can even double. Therefore, short optical-to-optical (O2O) latency is key to providing the remote driver with the ability to accurately and rapidly maneuver a remote reality driven vehicle (RRDV), achieving a remote driving experience that is identical or nearly identical to the in-car driving experience.

[0157] Camera latency and display device latency are relatively easy to control and can usually be reduced by increasing the frame rate. For example, high-speed traffic monitoring cameras can reach 480 frames per second and game displays can reach 500 frames per second, thereby reducing both delays to approximately 2 milliseconds. Therefore, the current focus is on how to minimize electrical to electrical (E2E) delay. In a preferred embodiment of the system of the present invention, the stitching engine and display engine of the remote reality driving device (RRDD) are combined to eliminate the delay of the independent display engine, and the optical cable trunk delay is small and negligible, and the electrical to electrical (E2E) delay is composed of the delay of the stitching display two-in-one engine and the delay of the wireless video transmission. Through the remote reality driving device (RRDD) of the present invention, the delay of the stitching display two-in-one engine can be reduced to no delay or nearly no delay. By adopting the existing invention series [3][4][5], the communication engine of the remote reality driving device (RRDD) adopts an uncompressed video transmitter, which can transmit video with high fidelity and no delay, eliminating video compression and decompression delays. In combination, the present invention achieves no or nearly no electrical to electrical (E2E) delay.

[0158] The prior invention [6] discloses a remote control based on virtual reality (VR) display, which can take over the autonomous driving vehicle when the autonomous driving system cannot pass through certain abnormal environments without violating road traffic regulations (for example, a dump truck blocks the street, the autonomous driving vehicle (AV) stops and cannot pass through the solid dividing line in the middle of the street to overtake). The remote reality driving vehicle (RRDV) of the present invention is substantially different from the prior invention in the following aspects: a) The remote reality driving vehicle (RRDV) of the present invention has its own dedicated surround vision imaging equipment and other sensors required for remote driving, all of which are included in the remote reality driving device (RRDD), and does not include any sensors (for example, cameras) or any components of the autonomous driving system (ADS) for remote driving. The prior invention relies on the sensors (for example, cameras) and other components of the autonomous driving system (ADS) for remote driving. The autonomous driving system (ADS) includes a long-range narrow-angle camera (using a telephoto lens), a medium-range wide-angle camera (using a wide-angle lens) and a short-range surround camera (using a fisheye lens). Existing automotive surround-view cameras provide a short-range (e.g., a few feet) bird's-eye view of the vehicle's surroundings, typically used for parking and low-speed maneuvers around obstacles, and are unsuitable for normal medium- and high-speed driving. In contrast, automated driving systems (ADS) rely on telephoto cameras for forward driving at medium and high speeds and wide-angle cameras for lane changes. In contrast to existing technologies, the surround-view imaging device of the present invention's remote reality driving vehicle (RRDV) combines the functionality of long-range, wide-angle, and surround-view cameras to provide a remote reality view that is identical or similar to the in-vehicle human vision required for remote high-speed human driving. b) The present invention discloses a specific structure for the surround-view system to achieve zero or near-zero latency video stitching, which is not present in existing technologies. In fact, the fisheye lens used in existing surround-view cameras can cause severe distortion, and their fisheye correction results in extremely long latency. c) The present invention discloses a specific structure for the display engine on the vehicle side, further integrating it with the stitching engine to minimize latency in the display engine (which projects the video to generate the current view), thereby achieving zero or near-zero latency. Existing technologies traditionally use a display engine on the driver's station side and fail to eliminate the inherent long latency of the display engine. d) The present invention uses a preferred proprietary method to transmit video of the viewed view without compression, achieving delay-free transmission. However, existing inventions use public wireless data networks to transmit videos, which requires heavy video compression. This results in a serious increase in the total electrical-to-electrical (E2E) delay, which includes video compression, public network routing and transmission, and video decompression delay, from a fraction of a second to several seconds. This causes huge perception errors and is not suitable for high-speed long-distance driving.The Remote Realistic Driving Vehicle (RRDV) of the present invention minimizes latency and is invented for latency-critical applications, such as autonomous vehicle testing at highway speeds or above, racing, etc., while existing inventions produce long delays and are not used for high-speed remote driving, but for low-speed remote illegal detours around abnormal obstacles when the autonomous driving system fails to exit.

[0159] The prior invention [7] discloses an autonomous vehicle system comprising an autonomous vehicle having a first-person view (FPV) camera and a remote vehicle control device. The two are connected via the Internet. The remote vehicle control device receives first-person view (FPV) images from the first-person view (FPV) camera on the vehicle and allows a remote user to take over autonomous driving in the event of a malfunction, hijacking, theft, etc. of the autonomous vehicle (AV). The remote reality driving vehicle (RRDV) of the present invention is substantially different from the prior invention in the following aspects: a) The remote reality driving vehicle (RRDV) of the present invention has its own dedicated surround vision imaging device and other sensors to provide 360° surround vision to meet the requirements of high-speed remote driving (especially racing-style driving: high-speed road car speed cutting, interception, etc.), all of which are included in the remote reality driving device (RRDD), and does not use any sensors (such as cameras) and any components of any autonomous driving system (ADS) for remote driving. The prior art relies on the front-facing camera of the automated driving system (ADS) to produce a first-person perspective (FPV) image, which does not cover a 360° environment and is therefore incapable of providing full-scale high-speed remote driving (such as freeway-speed cut-ins and interceptions). It was invented for low-speed control to allow for autonomous driving to be taken over in the event of a failure or illegal situation. b) The present invention discloses a preferred proprietary method for transmitting video of the viewed view without compression, achieving zero-latency transmission, whereas the prior art uses an internet connection to transmit video with heavy video compression. This results in a significantly increased total electrical-to-electrical (E2E) latency, including video compression, public network routing and transmission, and video decompression delays, ranging from fractions of a second to several seconds, causing significant perceptual errors and making it unsuitable for high-speed remote driving. The present invention's Remote Reality Driving Vehicle (RRDV) minimizes latency and is designed for latency-critical applications, such as autonomous vehicle testing at or above freeway speeds, racing, etc., whereas the prior art incurs long latency and is not intended for high-speed remote driving, but rather for low-speed autonomous driving to be taken over in the event of a failure or illegal situation.

[0160] Figures 6 to 13 illustrate an embodiment of the Turing-like double-blind human-machine adversarial driving test method for autonomous vehicles of the present invention. In addition to the system shown in the embodiment of Figure 1 , the system shown in the embodiments of Figures 6 to 13 also includes a northbound lane 1105 in the road network, an additional target vehicle (TV) 1215 and its paired remote driving station (DS) 1415, which is equipped with a human driver D4 in the target vehicle (TV) driving facility 1400, and a driving link between the target vehicle (VT) 1215 and its remote driving station (DS) 1415. For the sake of simplicity, the communication network, target vehicle (TV) driving facility, and Turing-like test vehicle (VUT) testing facility are not shown in Figures 6 to 13, but they are all included. If the driving mode switch 1510 is in remote driving mode, the target vehicles (VT) 1210, 1215, 1220, and the test vehicle (VUT) 1500 are all driven by a human driver. For the sake of brevity, all target vehicles (VTs) and test vehicles (VUTs) in remote driving mode are described without mentioning their human drivers.

[0161] The test scenarios in Figures 6 to 13 are example scenarios for testing whether an autonomous driver or a human driver has the driving skills to overtake the vehicle in front of it in the opposite lane.

[0162] When the first test run begins, the driving mode switch 1510 performs a random selection. Assuming autonomous driving mode is selected, switch 1510 connects to wired connection 1511, establishing a driving link between the vehicle under test (VUT) 1500 and a remote driving station (DS) 1610, located in Room A of the Turing-like vehicle under test (VUT) test facility 1600, which lacks a human driver. Accordingly, the in-vehicle mode switch 2160 is set to autonomous driving mode, and the VCU 2170 is connected to the autonomous driving system 2150. The vehicle under test (VUT) 1500 drives autonomously during this test run. By design, the driver of the target vehicle (VT) remains unaware of this.

[0163] However, since the remote driving station (DS) 1610 in room A is not operated by a human driver, there is no human driver to input the view angle control command. In one embodiment, a simulator is used to generate the view angle control command. In one embodiment, the view angle is generated based on the steering command of the test vehicle (VUT) 1500 generated by the autonomous driving system 2150 and sent to the VCU 2170, and then reported back to the remote driving station (DS) 1610 as part of the vehicle state information. This points the current view in the direction the current vehicle is turning, which is the same as what a human driver usually does. However, the generated view angle in the embodiment may still be different from the view angle of a human driver. For example, when the test vehicle (VUT) is about to change lanes, the generated view angle will not turn to the side in advance and look for an empty space in the adjacent lane before the steering command is issued, which a human driver would usually do.

[0164] The startup phase of the first test run begins with the scenario shown in Figure 6. In this scenario, the vehicle under test (VUT) 1500 is traveling northbound in the left lane 1110 at the speed limit. All target vehicles (VTs) are traveling northbound in the right lane 1105 at speeds below the speed limit. The VUT has four spatial degrees of freedom (SDoF) and an infinite time to collision (T2C). The VUT can now proceed normally and safely. The startup phase can also begin with any other scenario as long as the desired capture scenario is ultimately achieved. Different initial scenarios may require different capture strategies to achieve the desired capture scenario.

[0165] The capture phase begins when the test vehicle (VUT) 1500 approaches and passes the target vehicle (VT) 1210. At this point, the target vehicle (VT) 1210 "happens" (actually, the target vehicle (VT) intends to do so) to turn into lane 1110 and immediately follows the test vehicle (VUT) 1500 at the same speed, as shown in Figure 7. The test vehicle (VUT) now has three spatial degrees of freedom (SDoF) and an infinite time to collision (T2C), allowing it to safely proceed. This is the first scenario in the capture phase.

[0166] Just as test vehicle (VUT) 1500 approaches and drives alongside target vehicle (VT) 1215, target vehicle (VT) 1220 "happens" to turn into lane 1220 and drive immediately in front of test vehicle (VUT) 1500, as shown in Figure 8. All target vehicles (VTs) and test vehicle (VUT) 1500 are traveling at the same speed, the speed limit. This is the capture scenario (i.e., the last scenario of the capture phase), and the capture phase ends. Test vehicle (VUT) 1500 now has one spatial degree of freedom (SDoF) and an infinite time to collision (T2C). Test vehicle (VUT) 1500 can still drive normally and safely.

[0167] With reference to the terminology of motorsports, the target vehicle (VT) can directly attack the test vehicle (VUT) by creating a scenario with the risk of colliding with the test vehicle (VUT). This may require accurate and rapid cutting in front of the test vehicle (VUT), sudden braking in front of the test vehicle (VUT), rapid lateral approach to the test vehicle (VUT), rapid rear approach to the test vehicle (VUT), etc. The target vehicle (VT) can also indirectly attack the test vehicle (VUT) by destroying the driving plan of the test vehicle (VUT) and creating a risk scenario in which the test vehicle (VUT) collides with other target vehicles (VT). This may require strategic blocking, seizing the space that the test vehicle (VUT) needs to occupy, etc. The driving test requires the target vehicle (VTs) to carry out accurate and rapid driving in a racing style to create dangerous scenarios, and the test vehicle (VUT) also needs to drive so as to escape from dangerous scenarios. It is precisely this vehicle that carries out accurate and rapid maneuvering at highway speeds that makes the system of the present invention a delay-critical system.

[0168] The testing phase can begin after the test vehicle (VUT) 1500 is captured in the capture scene. To perform a blind test on the test vehicle (VUT) 1500 (i.e., without the test vehicle (VUT) 1500 anticipating or preparing for the next scenario), multiple scenarios are planned to initiate a test after the capture scene. For example, the first scenario involves the target vehicle (VT) 1220 smoothly and gradually decelerating and stopping (even with emergency lights flashing). If the test vehicle (VUT) consistently follows the target vehicle (VT) 1220, decelerating, and stopping, this indicates that the test vehicle (VUT) lacks the driving skills to overtake or is inadequately skilled and prefers not to overtake. The second scenario involves the target vehicle (VT) 1220 decelerating to 15 mph below the speed limit. The third scenario involves the target vehicle (VT) 1220 suddenly applying full braking force, creating a scenario that places the test vehicle (VUT) at risk of a forward collision with the target vehicle (VT) 1220. This is one of the automatic emergency braking (AEB) test scenarios. Assume that scenario 3 is executed, becoming the first scenario of the test phase. The test vehicle (VUT) has one spatial degree of freedom (SDoF) and a time to collision (T2C) of several seconds. Since the southbound lane 1130 in the opposite direction is completely clear, it is now safer for the test vehicle (VUT) 1500 to drive in the opposite direction. Furthermore, assume that the test vehicle (VUT) possesses the driving skill to overtake in the opposite direction and decides to overtake the target vehicle (VT) 1220 to escape danger. This is the first scenario of the test phase, as shown in Figure 9. This begins the opposite direction overtaking test.

[0169] The vehicle under test (VUT) 1500 turns into the southbound lane 1130 in the opposite direction and overtakes the target vehicle (VT) 1120 at the speed limit. Simultaneously, the target vehicle (VT) 1120 accelerates to 15 mph below the speed limit, as shown in Figure 10. This is a typical scenario where a faster vehicle overtakes a slower vehicle. The vehicle under test (VUT) 1150 again has three spatial degrees of freedom (SDoF) and an infinite time to collision (T2C).

[0170] Next, several scenarios were planned. The first was that the scenario in Figure 10 continued unchanged; the second was that target vehicle (VT) 1230 appeared and approached from north to south in southbound lane 1130. Assume that the second scenario is executed. Because target vehicle (VT) 1230 is far away, the test vehicle (VUT) has a time to collision (T2C) of tens of seconds. Test vehicle (VUT) 1500 can safely continue and complete the overtaking of target vehicle (VT) 1220. Therefore, test vehicle (VUT) 1500 continues its driving plan to overtake target vehicle (VT) 1220, as shown in Figure 11.

[0171] Next, several scenarios are planned. The first is that the scenario in Figure 11 continues unchanged. The second is that the target vehicle (VT) 1220 accelerates to the speed limit and is at the same speed as the test vehicle (VUT) 1500. Assume that the second scenario is executed. Now, it is impossible for the test vehicle (VUT) 1500 to complete the overtaking within the current collision time (T2C). The overtaking driving plan of the test vehicle (VUT) has been destroyed. In addition, there is no need to overtake the target vehicle (VT) 1220 now. The test vehicle (VUT) 1500 finds that its original position is still an empty space, and now its driving plan is to slow down and fall behind, and then try to return to its original position, as shown in Figure 12.

[0172] Next, several scenarios are planned. The first is that the scenario in Figure 12 continues unchanged. The second is that the target vehicle (VT) 1215 "just happens" to turn to the left lane 1110, just before the test vehicle (VUT) 1500 returns, and seizes the original position of the test vehicle (VUT). The third is that the target vehicle (VT) 1210 "just happens" to accelerate and seize the original position of the test vehicle (VUT). Assume that the second scenario is executed. Now, the driving plan of the test vehicle (VUT) to return to its original position is disrupted again. The test vehicle (VUT) 1500 finds that there is an empty space behind the target vehicle (VT) 1210, and its driving plan now is to slow down further and try to return to the empty space behind the target vehicle (VT) 1210, as shown in Figure 13.

[0173] Next, several scenarios were planned. The first was a continuation of the scenario in Figure 13. The second involved target vehicle (VT) 1210 slowing down to prevent test vehicle (VUT) 1500 from returning to northbound lane 1110. Assuming the first scenario is executed, test vehicle (VUT) 1500 returns to northbound lane 1110 and slightly collides with target vehicle (VT) 1210 (corner-to-corner contact). This completes the test phase.

[0174] The inspection phase begins after the test phase. All video and information recordings are collected. All dummies and crash sensors are inspected, and a crash report is generated. During this test run, the vehicle under test (VUT) 1550, while attempting to steer into northbound lane 1110 to avoid a head-on collision with the target vehicle (VT) 1230, suffered a minor collision with the target vehicle (VT) 1210, as shown in Figure 13. The dummy is inspected and found to be in good condition, and the crash sensors are read. Minor property damage is reported, and the repair cost is estimated (e.g., $2,000). This includes all of the above, resulting in the first set of test results. The first test run is complete.

[0175] Assume that the second test run begins after the completion of the first test run. When the second test run begins, the driving mode switch 1510 performs another random selection. Assuming that remote driving mode is selected, the switch 1510 connects to the wired connection 1512, establishing a driving link and connecting the vehicle under test (VUT) 1500 to the remote driving station (DS) 1620, which is equipped with human driver B in room B of the Turing-like vehicle under test (VUT) test facility 1600. Therefore, the in-vehicle mode switch 2160 is set to remote driving mode, and the VCU 2170 is connected to the external driving interface 2140. The vehicle under test (VUT) 1500 is driven by human driver B in this test run. By design, the target vehicle (VT) driver is unaware of this. Assume that a similar and comparable test scenario to the first test run is conducted in the second test run (if not, additional test runs are required until comparable test runs are obtained), and test results are collected again. This results in a second set of test results. Assume that in the scenario shown in Figure 13, vehicle under test (VUT) 1550 collides moderately with target vehicle (VT) 1210 while attempting to steer into northbound lane 1110 to avoid a head-on collision with target vehicle (VT) 1230. The dummy is inspected and found to be in good condition, and the crash sensors are read. Moderate property damage is reported, and the repair cost is estimated (e.g., $8,000).

[0176] The evaluation phase begins after selecting a pair of test result sets. A pair of test result sets is two sets of test results collected from two test runs with comparable test scenarios. The pair of test result sets is sent anonymously to a human evaluator C. By design, human evaluator C does not know the identity of the driver in any test result set. Human evaluator C compares the pair of test result sets and decides which one has better driving skills and wins, or which two have the same driving skills and tie, or which two sets are incomparable and cannot be determined. In this embodiment, because the current view may reveal driver information (i.e., whether it is the autonomous driver or the human driver), it is excluded from the test result sets to be compared. In this embodiment, the human evaluator recreates the two test run events by playing back a recorded video representing a 360° surround view around the test vehicle (VUT) 1500, as if he were a passenger in the test vehicle (VUT) to experience the driving test (similar to what a road test examiner does in existing human driver license tests). All other recorded information is also displayed. The two sets can be displayed side by side (parallel) or one after another (alternating) to facilitate comparative studies.

[0177] Once the evaluation results are determined, the driving test method is complete, typically with a winner, loser, or a tie. In the example driving test, although both drivers failed to safely navigate the test scenario in Figure 13, the driver in the first set of test results, the automated driver, clearly possessed superior driving skills and outperformed the human driver in the second set of test results, as the first set of test results only included a minor collision.

[0178] There are several types of evaluations. The first is an autonomous driver versus a human driver evaluation, where one set of test results is collected from the autonomous driver and the other set is collected from the human driver, as described in the example above. The second is a human driver versus another human driver evaluation, where a pair of test result sets are collected from both human drivers. The third is a test scenario versus human driver evaluation, where one set of test results is collected from only one human driver. In this type of evaluation, if the human evaluator C decides that the driver of the vehicle under test (VUT) successfully and safely passes the test scenario, the driver wins and the scenario loses; if the human evaluator C decides that the driver fails to safely pass the test scenario, the driver loses and the scenario wins; if the human evaluator C decides that the driver successfully and safely passes the test scenario half and half, the driver and scenario tie. The fourth is a test scenario versus another test scenario evaluation, where a pair of test result sets are collected from the same human driver or two human drivers of the same level performing driving tests in two different scenarios. In this type of evaluation, if a human evaluator C decides that a scenario the driver passed more unsafely or failed more dangerously, then that scenario wins and the other scenario loses; if a human evaluator C decides that the driver passed both scenarios equally safely or failed equally dangerously, then the two scenarios tie.

[0179] The method of the present invention for evaluating how good autonomous driving is compared to human driving is described in detail in the embodiments using numerical examples.

[0180] Set up a rating system. In this example, assume that the win rate of driver B1 with low rank d1 against driver B2 with high rank d2 is

[0181] Among them, the rank difference Δd = d2-d1, a is a constant, which is set to 1 rank in the embodiment. For example, when two drivers are both 1 rank, Δd = 0, and the winning rate is P dd =50%; when the difference between the two drivers is 1 stage, that is, Δd = 1, the winning rate is P dd =9.09%.

[0182] Although the initial drivers are rated as level 0, they actually have different levels of driving skill, and the same is true for drivers with higher levels. Therefore, assuming there are 1,000 human drivers to be rated, their actual driving skills are evenly distributed among 50 clusters between -0.4 and 9.4 levels. That is to say, there are 100 initial drivers, 20 of whom are actually in the -0.4 segment, 20 are actually in the -0.2 segment, 20 are actually in the 0 segment, 20 are actually in the 0.2 segment, and 20 are actually in the 0.4 segment; there are 100 drivers, 20 of whom are actually in the 0.6 segment, 20 are actually in the 0.8 segment, 20 are actually in the 1 segment, 20 are actually in the 1.2 segment, 20 are actually in the 1.4 segment, and so on; finally, there are 100 drivers, 20 of whom are actually in the 8.6 segment, 20 are actually in the 8.8 segment, 20 are actually in the 9 segment, 20 are actually in the 9.2 segment, and 20 are actually in the 9.4 segment.

[0183] As previously mentioned, the rating system in the embodiment assigns all initial drivers a level of 0.

[0184] The first step in setting up the rating system is to find standard 0-band drivers—20 starting drivers whose actual driving skills are exactly 0-band. The identities of the 100 starting drivers are known, but the identities of these 20 drivers who are actually 0-band are unknown. Standard 0-band drivers are found by evaluating each starting driver against all other starting drivers. Standard 0-band drivers are those with a 50% win rate. This is because stronger starting drivers (0.2 and 0.4 bands) and weaker starting drivers (-0.2 and -0.4 bands) are symmetrically distributed around the standard 0-band drivers.

[0185] The second step in setting up the rating system is to assess 1st-dan drivers. In this example, non-initial drivers with actual 0.6 to 1.4 dan ratings are assessed as 1st-dan drivers as follows: non-initial drivers are compared and evaluated against standard 0-dan drivers; 0.5 dan is chosen as the lower limit of 1st-dan, and 1.5 dan is chosen as the upper limit. According to Formula 1, drivers with a win rate against standard 0-dan drivers ranging from 76% to 97% are assessed as 1st-dan drivers. 1st-dan drivers include the 100 drivers whose actual dan ratings range from 0.6 to 1.4.

[0186] The third step in setting up the rating system is to find a standard 1st dan driver. Similar to step 1, a standard 1st dan driver is one that has a 50% win rate against all other 1st dan drivers, as the actual rank of 1st dan drivers is also symmetrically distributed.

[0187] The fourth step in setting up the rating system is to evaluate the 2nd tier drivers. Similar to step 2, all remaining drivers are evaluated against the standard 1st tier driver to find a 2nd tier driver with the same win rate range as them.

[0188] This process continues until the last 100 drivers are rated as 9th Dan, the most skilled or excellent drivers.

[0189] Rating human drivers. After the rating system is set up, new human drivers can be added and rated. Each new human driver is first evaluated against a standard Segment 0 driver. If the new driver's win rate is less than 76%, he is rated as Segment 0 and his rating is completed; otherwise, he is rated as Segment 1 and further evaluated against a standard Segment 1 driver; if the new driver's win rate is less than 76%, he remains at Segment 1 and his rating is completed; otherwise, he is rated as Segment 2 and further evaluated against a standard Segment 2 driver. Rating continues until his highest segment is reached. In this way, every human driver who drives the test vehicle (VUT) 1500 for various driving tests will be evaluated and obtain his segment.

[0190] Ranked autonomous drivers. After human drivers are ranked, autonomous drivers (i.e., AVs) are evaluated against human drivers as new drivers. Autonomous drivers are evaluated against standard 0-segment drivers. If the autonomous driver's win rate exceeds 76%, it is ranked as 1st segment. Then, the autonomous driver is evaluated against standard 1st segment drivers. If the autonomous driver's win rate again exceeds 76%, it is ranked as 2nd segment. Rating continues until it reaches its highest segment. If the autonomous driver's win rate against a certain segment falls below 76%, it remains at that segment and the rating is completed. For example, if an autonomous driver (i.e., AV) has a 100% win rate against a standard 2nd segment driver, the autonomous driver (AV) is ranked as 2nd segment and the rating continues; if an autonomous driver (i.e., AV) has an 80% win rate against a standard 9th segment driver (not a top 9th segment driver), the autonomous driver (AV) is ranked as 10th segment and the rating continues.

[0191] Because the autonomous driving vehicle may have lower driving skills than the initial driver, that is, its level is lower than level 0, which is a negative level. In one embodiment, when the autonomous driver is compared and evaluated with the standard level 0 driver, if its winning rate is lower than 24%, it is rated as level -1 and the rating ends. This will rate all driving skills below level 0 as level -1, which means that it does not meet the minimum requirements for driving on the road. Its actual driving skills may be lower than level -1, and the rating system of the present invention can accurately assess its actual driving skills, but it is generally not necessary to quantify and subdivide the negative driving skills of unqualified autonomous driving vehicles in practice. This embodiment can also be used to assess human drivers whose driving skills are lower than those of the initial driver due to various reasons such as long-term lack of driving, physical decline, and illness.

[0192] Using this method, we can quantitatively measure how good an AV’s driving skills are compared to those of humans.

[0193] The method of the present invention for evaluating how safe autonomous driving is compared to human driving is described in detail in the embodiments using numerical examples.

[0194] Evaluate the test scenarios. Based on the rating system for evaluating drivers in the above method, each test scenario is compared and evaluated with the evaluated human drivers (i.e., determine whether the scenario wins or the human driver wins). Some simple and very low-risk test scenarios have negative segments (i.e., below segment 0), while complex and high-risk test scenarios have non-negative segments (i.e., segment 0 and above). Still assuming that there are 1,000 test scenarios to be evaluated, the actual segments of these test scenarios are evenly distributed between a negative segment (e.g., -4 segment) and a positive segment (e.g., 5 segment), and each segment has 5 clusters (the offset of each cluster to the standard segment is -0.4 segment, -0.2 segment, 0 segment, 0.2 segment, 0.4 segment), which is the same as the example of 1,000 drivers.

[0195] The first step in rating a test scenario is to evaluate the scenario against a standard 0-segment driver. According to the rating system set up above, if the win rate of the scenario is less than 24%, then the actual segment of the scenario is less than -0.5 segments, that is, the scenario has a negative segment; otherwise, the scenario has a non-negative segment. If the scenario has a non-negative segment, the remaining steps for rating the scenario are the same as those for rating a new driver using the rating system set up, as if the scenario were a new driver, and therefore will not be repeated here. If the scenario has a negative segment, since the lowest or worst-skilled human driver is 0-segment, a new method is needed to rate scenarios with negative segments, as shown below:

[0196] The second step in evaluating test scenarios with negative ratings is to evaluate all negative rating scenarios against a standard driver with a rating of 0. If the scenario has a win rate higher than 3% (i.e., -1.5 ratings), it is rated as -1 rating; otherwise, if the scenario has a win rate higher than 0.3% (i.e., -2.5 ratings), it is rated as -2 rating, and so on.

[0197] In another embodiment, step 2 of rating test scenarios with negative segments is to find and rate all -1 segment scenarios using a standard 0 segment driver, then find and rate all -2 segment scenarios using a standard -1 segment scenario, and so on.

[0198] Determine the safety level. In this example, based on Formula 1, the failure percentage of driver B with a grade of d0 to safely pass scenario S with a grade of d1, that is, the failure rate of driver B with a grade of d0 to safely pass scenario S with d1, is given as follows:

[0199] Wherein, the segment difference Δd=d0-d1, a is a constant, which is set to 1 segment in the embodiment. The driver's safety level is determined by his failure rate p through scenario S. ds To express.

[0200] For example, assume that the scenario in Figure 10 is rated Segment 0, the scenario in Figure 11 is rated Segment 1, the scenario in Figure 12 is rated Segment 2, and the scenario in Figure 13 is rated Segment 3. Furthermore, assume that a certain autonomous vehicle (AV) is rated Segment 2. Therefore, the autonomous vehicle (AV) has a 1% failure rate for safely passing the scenario in Figure 10, a 9% failure rate for safely passing the scenario in Figure 11, a 50% failure rate for safely passing the scenario in Figure 12, and a 91% failure rate for safely passing the scenario in Figure 13.

[0201] It's important to note that a driver's driving skill level is different from their safety level in navigating a scenario. Autonomous vehicles (AVs) can be designed to make safer decisions based on their safety level. For example, based on the aforementioned assumption that the autonomous test vehicle (VUT) has a 1% failure rate in safely navigating the scenario in Figure 10, if the probability of a forward collision between the autonomous test vehicle (VUT) 1500 and the target vehicle (VT) 1220 is less than 1% (e.g., 0%), the autonomous test vehicle (AV) in the second stage will not overtake the target vehicle (VT) 1220 (i.e., it will always follow the target vehicle (VT) 1220), as shown in Figure 9. If the probability of a forward collision between the autonomous test vehicle (VUT) 1500 and the target vehicle (VT) 1220 is greater than 1% (e.g., 20%), the autonomous test vehicle (VUT) in the second stage will begin to overtake the target vehicle (VT) 1220, as shown in Figure 9. That is, the AV always avoids dangerous scenarios (e.g., overtaking with a 1% failure rate) unless it must avoid a more dangerous scenario (e.g., a forward collision with a 20% speed limit). In this way, the AV makes the safest driving decisions, allowing lower-level AVs to achieve higher safety through conservative driving strategies.

[0202] The scenario distribution function of an autonomous vehicle (or human driver) is defined as the probability of each scenario occurring to the driver. Given a test scenario set {S i}, where i = 1, 2, ..., M, and M is the total number of test scenarios. The driver's scenario distribution function is expressed as P s (S i ), assuming the driver’s rank is d0, and the rank of each scene is d i , the driver's overall safety level is calculated by the failure rate of each scenario, which is given by the following formula:

[0203] For example, assume a specific commuter AV drives 2 hours per day, with an average scenario duration of 20 seconds. Over the course of a year, the AV will experience 131,400 scenarios. Assume these 131,400 scenarios are grouped into 1,000 distinct scenarios. Accordingly, a statistical approximation of the AV's scenario distribution function is the normalized frequency of each of these 1,000 scenarios within the total 131,400 scenarios.

[0204] The number of failures (crashes) for a particular autonomous vehicle or human driver is given by: d =P d N so Formula.4

[0205] where N so is the number of times the scene passes. Given the above example, the number of failures of the autonomous vehicle (AV) in one year is 131,400P d For example, assuming the same number of annual failures, in one embodiment, it can be expected that 5 segments of a self-driving private shuttle car are sufficient to match the driving skill and safety level of a human, while a self-driving taxi may need 6 segments to drive 10 times the miles per day while maintaining the same number of annual failures (crashes).

[0206] Furthermore, the average estimated cost of a driver's failure (i.e., the loss caused by failing to safely pass the scene) can be calculated from the crash report. Taking into account the severity of the crash, the failure cost of each scene can be calculated and the total failure cost of the driver over the year can be calculated.

[0207] It should be further noted that while the present invention is described with reference to the accompanying drawings, it should be understood that the invention is not limited to such exemplary embodiments. Those skilled in the art may make modifications and variations without departing from the spirit or scope of the invention as defined in the appended claims. The exemplary embodiments of the present invention are merely examples of applying the present invention to video transmission. The present invention has various embodiments. These embodiments are not described in detail because they can be derived by those skilled in the art.

[0208] References

[0209] [1] Daniel J. Fremont, Edward Kim, et al., Formal scenario-based testing of autonomous vehicles: From simulation to the real world, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), September 20–23, 2020.

[0210] [2] Nidhi Kalra, Susan M. Paddock, Driving to Safety, RAND Corporation, April 2016.

[0211] 【3】U.S. Patent US10225577B2, Chen Shidong, Method and system for non-causal image and video prediction.

[0212] 【4】China Patent WO2015180663A1, Chen Shidong, Method for transmitting high-definition video based on transform domain.

[0213] 【5】International Patent EP3512173B1, Chen Shidong, Method and device for transmitting video over a multiple-input multiple-output channel.

[0214] [6] U.S. Patent US11099558B2, Jen-Hsun Huang et al., remotely operated vehicles using an immersive virtual reality environment.

[0215] 【7】U.S. Patent US11079753B1, Matthew Roy, Autonomous driving vehicle with remote user supervision and temporary takeover.

Claims

1. Turing-like double-blind human-machine confrontation driving test system for autonomous driving vehicles, characterized by: The system includes a road network, one or more target vehicles, a test vehicle, one or more communication networks, a target vehicle driving facility and a Turing-like test vehicle testing facility.

2. The Turing-like double-blind human-machine confrontation driving test system for autonomous driving vehicles according to claim 1 is characterized in that: The target vehicle is a remote reality driving vehicle, including a vehicle platform and a remote reality driving device capable of achieving no electrical to electrical delay or nearly no electrical to electrical delay, wherein the vehicle platform includes various vehicles, various pedestrians and various animals with an external driving interface; the various vehicles are vehicles that can be driven by remote reality and are equipped with simulated in-vehicle drivers and passengers, and the simulated in-vehicle drivers and passengers are collision test dummies equipped with sensors to simulate human occupants and collect data on the results of collision injuries to the occupants; Various pedestrians are remotely drivable robots equipped with humanoid bodies, which are equipped with sensors to simulate human pedestrians and collect data on the consequences of collisions with pedestrians; various animals are remotely drivable robots equipped with various animal-shaped bodies, which are equipped with sensors to simulate animals and collect data on the consequences of collisions with animals.

3. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 1 is characterized in that: The test vehicle has both autonomous driving functions and remote reality driving functions, and is a two-in-one vehicle of an autonomous driving vehicle and a remote reality driving vehicle. The test vehicle includes: an autonomous driving vehicle platform and a remote reality driving device that can achieve no electrical to electrical delay or nearly no electrical to electrical delay; the autonomous driving vehicle platform includes various autonomous driving vehicles and various automatic walking robots with external driving interfaces and in-vehicle mode switches; the various autonomous driving vehicles are autonomous driving vehicles equipped with simulated in-vehicle drivers and passengers, and the simulated in-vehicle drivers and passengers are collision test dummies equipped with sensors to simulate human occupants and collect data on the results of collisions on the occupants; the various automatic walking robots are automatic walking robots equipped with humanoid bodies, animal-shaped bodies or other shapes, and the robots are equipped with collision sensors to collect data on the results of collisions on the robots.

4. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 3 is characterized in that: The in-vehicle mode switch selects whether the vehicle platform is driven by the autonomous driving system to provide an autonomous driving function, or is driven by a driving signal received from an external driving interface to provide a remote reality driving function.

5. The Turing-like double-blind human-machine confrontation driving test system for autonomous driving vehicles according to claim 1, characterized in that: The communication network includes one or more base stations, a wireless network between all traffic participants and the base stations, one or more roaming stations, a trunk network between the base stations and the roaming stations, and a trunk network between the roaming stations and the target vehicle driving facilities and the Turing-like test vehicle testing facilities.

6. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 5 is characterized in that: The communication network also includes a driving mode switch that selects whether the remote driving station in room A or B is connected to the test car: when the remote driving station in room A is connected to the test car, the test car is in automatic driving mode; when the remote driving station in room B is connected to the test car, the test car is in remote driving mode and is remotely driven by human driver B.

7. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 1 is characterized in that: The target vehicle driving facility includes one or more remote driving stations, each of which includes a display device for displaying a current view of the remote reality, a perspective input device, and a vehicle control input device. Each remote driving station is equipped with a human driver and is connected to a target vehicle so that the human driver can drive the target vehicle.

8. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 1 is characterized in that: The Turing-like test vehicle testing facility is a facility that includes three closed rooms, which are called Room A, Room B, and Room C; in Room A, there is a remote driving station but no human driver; in Room B, there is a remote driving station with human driver B; the driving mode switch determines whether the remote driving station in Room A or Room B is connected to the test vehicle; All test results are anonymously sent to Room C, where they are evaluated by a human evaluator C.

9. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 2 is characterized in that: The target vehicle includes a remote reality driving device capable of achieving zero electrical to electrical delay or nearly zero electrical to electrical delay, the remote reality driving device includes multiple cameras, a stitching engine, a display engine, a driving recorder and a communication engine, the stitching engine stitches all camera videos together to generate a horizontal 360° surround video around the remote reality driving vehicle with zero or nearly zero delay, the display engine projects the horizontal 360° surround video to a designated display area in a mathematical model of a display device of the remote driving station according to a viewing angle control command of the remote driving station, and generates a video of the current view, the driving recorder records all generated videos as well as vehicle status and vehicle control commands, and the communication engine includes a wireless video transmitter, a two-way wireless data transceiver and an auxiliary wireless transceiver.

10. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 3 is characterized in that: The test vehicle includes a remote reality driving device that can achieve no electrical to electrical delay or nearly no electrical to electrical delay. The remote reality driving device includes multiple cameras, a stitching engine, a display engine, a driving recorder and a communication engine. The stitching engine stitches all camera videos together to generate a horizontal 360° surround video around the remote reality driving vehicle with no delay or nearly no delay. The display engine projects the horizontal 360° surround video to a specified display area in a mathematical model of a display device of the remote driving station according to a viewing angle control command of the remote driving station, and generates a video of the current view. The driving recorder records all generated videos as well as vehicle status and vehicle control commands. The communication engine includes a wireless video transmitter, a two-way wireless data transceiver and an auxiliary wireless transceiver.

11. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 9 or 10, characterized in that: The optical axes of the multiple cameras are placed on a horizontal plane of the vehicle, and the optical axes intersect at one point.

12. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 9 or 10, characterized in that: The stitching engine and display engine are replaced by a stitching and display two-in-one engine, achieving delay-free or nearly delay-free stitching and display joint processing.

13. The Turing-like double-blind human-machine confrontation driving test system for an autonomous driving vehicle according to claim 9 or 10, characterized in that: The wireless video transmitter adopts a delay-free and uncompressed video transmission method.

14. The testing method based on the Turing-like double-blind human-machine confrontation driving test system for the automatic driving vehicle according to claim 1 is characterized in that: The method includes: Startup phase: The system randomly selects to connect the test car to Room A or Room B in the Turing-like test car test facility; through one or more safety scenarios, one or more target cars are brought close enough to the test car; Capture phase: starts when the test vehicle is close enough and the target vehicle starts to take action to seize the expected vehicle position relative to the test vehicle. If the target vehicle completes the seizure of the expected vehicle position, the expected capture scene is reached and the capture phase ends when the test vehicle is captured in the expected capture scene. If the target vehicle fails to complete the seizure of the expected vehicle position relative to the test vehicle and misses the capture opportunity, the expected capture scene is not reached and the capture phase ends when the test vehicle is not captured in the expected capture scene. Testing phase: After the test vehicle is captured in the expected capture scene, the target vehicle starts to move to create a test scene for the test vehicle. Once a certain test scene is reached, the driving of the test vehicle will be tested; Inspection phase: starts after the test phase. Regardless of whether the test vehicle has safely passed the test scenario, all equipment is inspected and all test results are collected, including all video records, information records and collision reports of the test vehicle and all target vehicles involved in this test run; Evaluation phase: starts after obtaining or selecting a pair of test results from two test runs, which are anonymously sent to room C in the Turing-like test car testing facility, where human evaluators C, without knowing the The system compares and evaluates the driving skills of the two drivers without the driver's identity information, and decides which set of test results has better driving skills and wins, or which two sets have the same driving skills and tie, or no evaluation can be performed and no evaluation conclusion is given.

15. The testing method according to claim 14, characterized in that: A pair of test results from both test runs were obtained with a human driver remotely driving the test car in reality to compare and evaluate the driving skills of the two human drivers.

16. The testing method according to claim 14, characterized in that: A pair of test results from the two test runs were obtained by the test car's autonomous driving and the human driver's remote real-life driving, respectively, to compare and evaluate the driving skills of the autonomous driver and the human driver.

17. The testing method according to claim 14, characterized in that: Comparative assessments of driving skills include evaluating how good autonomous driving is compared to human driving and evaluating how safe autonomous driving is compared to human driving.

18. The testing method according to claim 17, characterized in that: The method for evaluating how good autonomous driving is compared to human driving includes the following steps: Set up a rating system: Set up a rating system for driver rankings, calculate the win rate of each driver based on the evaluation results of human evaluator C, and rate them according to the ranking system; Assessing human drivers: Rating the human drivers of the test vehicles and obtaining the rank of each human driver; Rating automated drivers: After human drivers are ranked, automated drivers are ranked based on the same rating system to obtain the automated driver's rank, which quantitatively measures the automated driver's driving skills compared to human drivers.

19. The testing method according to claim 18, characterized in that: The said rating system for establishing the driver ranking system comprises the steps of: A driver who has just obtained a driver's license is classified as stage 0; Using the formula The given winning rate determines each rank, where: P dd It represents the winning rate of driver B1 with low rank d1 against driver B2 with high rank d2. The rank difference Δd=d2-d1, and a is a constant.

20. The testing method according to claim 19, characterized in that: The method for evaluating human drivers and automatic drivers includes: finding a standard 0-segment driver through the winning rate between 0-segment drivers; when the winning rate of the driver to be evaluated against the standard 0-segment driver is higher than an upper limit setting value, the driver is rated as 1-segment and the rating continues; if it is lower than a lower limit setting value, the driver is rated as -1-segment and the rating ends; otherwise, the driver is rated as 0-segment and the rating ends; Find the standard 1st stage driver through the winning rate between 1st stage drivers; when the winning rate of the driver to be evaluated against the standard 1st stage driver is higher than an upper limit set value, he will be rated as 2nd stage and the rating will continue; if it is not higher, he will remain at 1st stage and the rating will end; The same applies to higher ranks.

21. The testing method according to claim 20, characterized in that: When the constant a=1, the upper limit setting value is 76% and the lower limit setting value is 24%.

22. The testing method according to claim 19, characterized in that: Methods for assessing how safe autonomous driving is compared to human driving include: Rating the test scenarios: First, set the scenario win / failure conditions as follows: During the evaluation phase, if the human assessor C determines that the test car successfully and safely passes the scenario, the scenario fails; if the assessor determines that the test car fails to safely pass the scenario, the scenario wins; if the assessor determines that the test car passes half and fails half in terms of safety, the scenario is tied with the driver; then rate the test scenarios: Step 1: Compared with the standard 0-segment driver, if the scenario's win rate is lower than a lower limit setting value, the scenario has a negative segment; otherwise, the scenario has a non-negative segment; Step 2: If the scenario has a non-negative segment, first set it to 0 segment, and then if its win rate is higher than an upper limit setting If the winning rate is higher than the second lower limit set value, the scene will be upgraded by 1 segment, and the rating will continue, and the scene will be compared with the standard driver of the next segment until the highest segment. If the scene has a negative segment, one evaluation method for the negative segment scene is: if its winning rate is higher than the second lower limit set value, it will be evaluated as -1 segment and the rating will end, otherwise it will be compared with the third lower limit set value until the lowest segment. Another evaluation method for the negative segment scene is: use the standard 0 segment driver to find and evaluate all -1 segment scenes, and then use the standard -1 segment scene to find and evaluate all -2 segment scenes, and so on, and then start from the -1 segment standard scene, and compare the scene to be evaluated with each negative segment standard scene until the lowest segment is obtained. Determine the safety level: The safety level of an autonomous vehicle passing through a scene is determined by its segment difference to the scene. In the rating system set, the failure rate P of driver B with a rank of d0 to safely pass the scenario S with a rank of d1 is ds (Δd), according to the formula Given; when the constant a = 1, if the autonomous vehicle has the same level as the scene, its safety level is level 0, and the autonomous vehicle has a 50% chance of winning and a 50% chance of collision; if the autonomous vehicle is one level higher than the scene, its safety level is level 1, and the autonomous vehicle There is a 91% chance of winning and a 9% chance of collision; if the autonomous vehicle is 2 segments higher than the scenario, its safety level is 2 segments, and the autonomous vehicle has a 99% chance of winning or a 1% chance of collision, and so on.

23. A remote reality driving vehicle used in the Turing-like double-blind human-machine confrontation driving test system for the autonomous driving vehicle according to claim 1, characterized in that: The vehicle consists of a vehicle platform and a remote reality driving device capable of achieving no electrical to electrical delay or nearly no electrical to electrical delay.

24. The remote reality driving vehicle according to claim 23, characterized in that: When the remote reality driving vehicle is used as a test vehicle, the vehicle platform includes: an autonomous driving system, an external driving interface, and an in-vehicle mode switch.

25. The remote reality driving vehicle according to claim 23, characterized in that: When a remote reality driving vehicle is used as a target vehicle, the vehicle platform includes various vehicles and robots with an external driving interface.

26. The remote reality driving vehicle according to claim 23, characterized in that: When the remote reality driving vehicle is used as an unmanned racing car of an unmanned racing sports system, the vehicle platform includes a racing car with an external driving interface, and the human racer can remotely drive the remote reality driving vehicle at high speed and quickly to perform racing sports by implementing a remote reality driving device with no electrical to electrical delay or nearly no electrical to electrical delay.

27. The remote reality driving vehicle according to claim 23, characterized in that: When the remote reality driving vehicle is used as an unmanned tank-type remote reality driving vehicle in a new type of unmanned military-style sports system, the vehicle platform includes an armored vehicle with an external driving interface, and the driving signals received from the external driving interface include armored vehicle driving signals and combat control signals. The armored vehicle includes a tank, and human racers can remotely drive the remote reality driving armored vehicle and control the combat equipment at high speed and quickly to conduct combat-type military sports by implementing a remote reality driving device with no electrical to electrical delay or nearly no electrical to electrical delay.

28. The remote reality driving vehicle of claim 23, wherein: When the remote reality driving vehicle is used as an unmanned heavy machinery remote reality driving vehicle, the vehicle platform includes a ground heavy vehicle or an aerial flying machinery with an external driving interface, and the driving signals received from the external driving interface include ground vehicle driving signals, flight driving signals and operation control signals. The ground heavy vehicles include construction vehicles, mining vehicles, and garbage disposal vehicles, and the aerial flying machinery includes drones. Workers can remotely drive and remotely operate in a safe and healthy environment by implementing remote reality driving equipment with no electrical to electrical delay or nearly no electrical to electrical delay, avoiding working in person under various unsafe and unhealthy field conditions.

29. The remote reality driving vehicle of claim 23, wherein: The remote reality driving device includes multiple cameras, a stitching engine, a display engine, a driving recorder and a communication engine. The stitching engine stitches all camera videos together to generate a horizontal 360° surround video around the remote reality driving vehicle with no delay or nearly no delay. The display engine projects the horizontal 360° surround video to a specified display area in a mathematical model of a display device of the remote driving station according to a viewing angle control command of the remote driving station, and generates a video of the current view. The driving recorder records all generated videos as well as vehicle status and vehicle control commands. The communication engine includes a wireless video transmitter, a two-way wireless data transceiver and an auxiliary wireless transceiver.

30. The remote reality driving vehicle according to claim 29, characterized in that: The wireless video transmitter adopts a delay-free and uncompressed video transmission method.

31. The remote reality driving vehicle according to claim 30, characterized in that: The stitching engine and the display engine are combined into a two-in-one stitching and display engine. After the stitching process is completed and the stitched spherical image is obtained, the display process directly projects the stitched spherical image onto the mathematical model of the display surface, thereby realizing delay-free or nearly delay-free joint stitching and display processing.

32. The remote reality driving vehicle of claim 23, wherein: The vehicle is used in other remote driving, remote operation systems, including unmanned racing sports systems, new unmanned military-style sports systems, and unmanned heavy machinery remote reality driving systems.

33. A remote reality driving robot for a Turing-like double-blind human-machine confrontation driving test system for an autonomous vehicle, characterized in that: The robot is a robot equipped with a humanoid body or an animal-shaped body, and the robot is equipped with sensors to simulate a human pedestrian or an animal and collect data on the injury results of a collision to the pedestrian or the animal.

Images