PLUG-AND-PLAY HARDWARE-IN-THE-LOOP TEST BENCH FOR AUTONOMOUS VEHICLE DEVELOPMENT

Abstract

The systems and methods described herein relate to the use of multimodal foundation models. In one embodiment, a method comprises the following steps: providing a vehicle test bench capable of operating physical and virtual vehicle components, receiving a test plan for a set of vehicle components, determining test bench components and test bench connections, including virtual components, to implement the test plan on the vehicle test bench, generating the virtual components to execute the test plan, and implementing the test bench connections to enable testing of the vehicle components.

Description

TECHNICAL FIELD

[0001] The subject matter described here generally refers to strategies for testing autonomous vehicles where there is a need to test multiple components. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from U.S. Prov. App. No. 63 / 555,707, filed February 20, 2024. The above-referenced application is hereby incorporated by reference in its entirety and is to be considered a part of this specification. All priority claims set forth in the application data sheet or in any corrigendum thereto are hereby incorporated by reference in accordance with 37 CFR 1.57. BACKGROUND

[0003] With regard to the development of autonomous vehicles, it is advantageous to be able to test autonomous functions with respect to the vehicle hardware before these functions are released to the public. One approach to validating vehicle hardware is to simulate the behavior of a virtual vehicle under autonomous control in a virtual environment. However, virtual environments are limited in the information they can encompass for testing, and important edge cases can be missed when testing autonomous functions solely on virtual representations. Furthermore, the use of virtual vehicles or environments may provide an idealized representation of vehicle behavior, thus not fully testing the real-world behavior of an autonomously controlled vehicle. SUMMARY

[0004] In one embodiment, a system is disclosed. The system includes one or more processors and a memory communicatively coupled to the one or more processors. The memory stores an instruction module with instructions that, when executed by the one or more processors, cause the one or more processors to provide a vehicle test bench capable of operating physical and virtual vehicle components, receive a test plan for a set of vehicle components, determine test bench components and test bench connections, including virtual components, to implement the test plan on the vehicle test bench, create the virtual components to perform the test plan, and implement the test bench connections to enable testing of the vehicle components.

[0005] In one embodiment, a non-transitory computer-readable medium is disclosed containing instructions that, when executed by one or more processors, cause the one or more processors to perform one or more functions. The instructions include instructions for providing a vehicle test bench capable of operating physical and virtual vehicle components, receiving a test plan for a set of vehicle components, determining test bench components and test bench connections, including virtual components, to implement the test plan on the vehicle test bench, generating the virtual components to execute the test plan, and implementing the test bench connections to enable testing of the vehicle components.

[0006] In one embodiment, a method is disclosed. In one embodiment, the method comprises providing a test bench capable of operating physical and virtual vehicle components, receiving a test plan for a set of vehicle components, determining test bench components and test bench connections, including virtual components, to implement the test plan on the vehicle test bench, generating the virtual components to execute the test plan, and implementing the test bench connections to enable testing of the vehicle components. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are incorporated into the specification, illustrate various systems, methods, and other embodiments of the disclosure. The element boundaries depicted in the figures (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundaries. In some embodiments, an element may be configured as multiple elements or multiple elements may be configured as one element. In some embodiments, an element depicted as an internal component of another element may be implemented as an external component, and vice versa. Additionally, elements may not be drawn to scale. Fig. 1 shows an embodiment of a vehicle in which the systems and methods disclosed herein may be implemented. Fig. Figure 2 shows an embodiment of an autonomous driving system connected to multimodal foundation models. Fig. shows one embodiment of a cloud computing environment in which the systems and methods described herein may operate. Fig. 4 shows an embodiment of a test bench for autonomous driving. Fig. 5 shows an embodiment of a vehicle interface component. Fig. 6 shows an embodiment of a vehicle environment simulator. Fig. Figure 7A shows an embodiment of a software stack for autonomous driving. Fig. Figure 7B shows an example of a state transition diagram for three different operating states that can be executed by a safety controller. Fig. 8 shows an embodiment of an HMI hardware component. Fig. 9 shows an embodiment of a component on the left side. Fig. shows an embodiment of a right-hand component. Fig. 11 shows an example of the virtual representation of a vehicle in a test environment. Fig. illustrates an example of a procedure for autonomous vehicle testing that may include both a test driver and a safety-trained driver. Fig. Figure 13 shows an example of a method for handling vehicle control with respect to an implementation with both a driver to be tested and a safety-trained driver. Fig. Figure 14 shows an example of a method for handling multiple instances of production components, prototype components, and virtual components during testing. DETAILED DESCRIPTION

[0008] Systems, methods, and other embodiments related to a hardware-in-the-loop testbench with dual cockpit driver support for testing autonomous functions are described herein. With respect to previous approaches, various methods, including hardware-in-the-loop (HIL), software-in-the-loop (SIL), model-in-the-loop (MIL), simulator-in-the-loop (SimIL), process-in-the-loop (PIL), human-in-the-loop (HuIL), etc., have attempted to solve problems of verification and validation related to autonomous functions. However, these various approaches alone are not sufficient to effectively represent real-world scenarios for testing a driver under test (DUT), a trained safety driver (TSD), and an autonomous system (Guardian) operating simultaneously in a common vehicle environment ( ). In such an environment, for example, various control inputs may be mixed in a drive-by-wire vehicle environment (e.g.,Combined by weighting) to enable smooth and predictable transitions between a test object, a TSD, or a Guardian. Furthermore, these various approaches did not take into account that a test bench must be capable of highly adaptive testing of different vehicle components to test different combinations of systems from different vendors.

[0009] A test bench using a dual-cockpit design offers the advantage that a TSD can monitor the DUT and correct errors made by the DUT, the Guardian, or both while driving. This allows the TSD to (i) make corrections for which the Guardian is not sufficiently trained during development and (ii) also correct errors in the Guardian's training, which can then be iteratively incorporated into the further training of the autonomous functions.

[0010] Furthermore, it is important to be able to test autonomous functions at all stages of a vehicle's development. The test bench presented here can therefore test autonomous functions on virtual components (e.g., a proposed model of a braking system), prototype components (e.g., a prototype braking system), and production components (e.g., a production braking system), or a combination thereof, in real or virtual environments.

[0011] In Fig. 1, an example of a vehicle 100 is shown. As used herein, a "vehicle" is any form of motorized transportation. In one or more embodiments, the vehicle 100 is an automobile. While the arrangements described herein relate to motor vehicles, it should be understood that the embodiments are not limited to motor vehicles. In some embodiments, the vehicle 100 may be any robotic device or form of motorized transportation, including, for example, sensors for sensing aspects of the environment and thus benefiting from the functionality discussed herein associated with autonomous vehicle testing strategies that may include both a driver under test and a safety-trained driver.Furthermore, in this disclosure, vehicle 100 is generally described as a vehicle capable of traveling on a roadway with surrounding vehicles that are understood in a similar manner to vehicle 100 itself. That is, surrounding vehicles may include any vehicle that may encounter vehicle 100 on a roadway.

[0012] The vehicle 100 also includes various elements. It is understood that in various embodiments, it is not necessary for the vehicle 100 to include all of the Fig. 1. The vehicle 100 may comprise any combination of the various elements shown in Fig. 1. Furthermore, the vehicle 100 may have additional elements to those shown in Fig. 1. In some arrangements, the vehicle 100 may be provided without one or more of the Fig. 1. While the various elements in Fig. 1 as being located within the vehicle 100, one or more of these elements may also be located outside the vehicle 100. Furthermore, the illustrated elements may be spatially separated by large distances. As previously mentioned, for example, one or more components of the disclosed system may be implemented within a vehicle, while other components of the system may be implemented in a cloud computing environment or another system remote from the vehicle 100.

[0013] Some of the possible elements of the vehicle 100 are shown in Fig. 1 and are described together with the following figures. However, for the sake of brevity of this description, a description of many of the elements in Fig. 1 after discussion of the Fig. 2-12. In addition, for simplicity and clarity, reference numerals are repeated where appropriate throughout the several figures to indicate corresponding or analogous elements. Moreover, numerous specific details are described in the discussion to provide a thorough understanding of the embodiments described herein. However, those skilled in the art will understand that the embodiments described herein may be practiced with various combinations of these elements. In any event, the vehicle 100 includes a test bench system 170 implemented to perform the methods and other functions described herein. As will be explained in more detail below, in various embodiments, the test bench system 170 is implemented partially within the vehicle 100 and as a cloud-based service.For example, in one approach, the functionality associated with at least one module of the test bench system 170 is implemented in the vehicle 100, while additional functionality is implemented in a cloud-based computer system.

[0014] With reference to Fig. 2 shows an embodiment of the test bench system 170 from Fig. 1. The test bench system 170 is shown as including processor(s) 110 from the vehicle 100 of Fig. 1. Accordingly, the processor(s) 110 may be part of the test bench system 170, the test bench system 170 may include a separate processor from the processor(s) 110 of the vehicle 100, or the test bench system 170 may access the processor(s) 110 via a data bus or other communication path. In one embodiment, the test bench system 170 includes a memory 210 in which the detection module 220 and the command module 230 are stored. The memory 210 is random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable memory for storing the detection module 220 and the command module 230.The detection module 220 and the command module 230 are, for example, computer-readable instructions that, when executed by the processor(s) 110, cause the processor(s) 110 to perform the various functions disclosed herein.

[0015] The Fig. The test bench system 170 illustrated in Figure 2 is generally an abstracted form of the test bench system 170 as may be implemented between the vehicle 100 and a cloud computing environment. Accordingly, the test bench system 170 may be implemented at least partially in a cloud computing environment to perform the methods described herein.

[0016] With reference to Fig. 2, the detection module 220 generally includes instructions that control the processor(s) 110 to receive data inputs from one or more sensors of the vehicle 100. The inputs, in one embodiment, are observations of one or more objects in an environment proximate the vehicle 100, other aspects of the environment, or both. As provided herein, in one embodiment, the detection module 220 collects sensor data 250, including at least camera images. In further embodiments, the detection module 220 collects sensor data 250 from additional sensors such as radar 123, LiDAR 124, and other sensors suitable for identifying vehicles, vehicle locations, pavement markings, crosswalks, traffic signs, vehicle parking spaces, road surface types, curbs, vehicle barriers, etc.

[0017] Accordingly, in one embodiment, the detection module 220 controls the respective sensors to provide sensor data 250. Although the detection module 220 is described as controlling the various sensors to provide sensor data 250, in one or more embodiments, the detection module 220 may also employ other techniques for collecting sensor data 250 that are either active or passive. For example, the detection module 220 may passively read sensor data 250 from a stream of electronic information provided by the various sensors to other components in the vehicle 100. Furthermore, the detection module 220 may take various approaches to fusing data from multiple sensors in providing sensor data 250, sensor data collected via a wireless communication link (e.g., v2v) from one or more of the surrounding vehicles, or a combination thereof.Thus, in one embodiment, the sensor data 250 represents a combination of perceptions collected by multiple sensors.

[0018] In addition to the locations of surrounding vehicles, the sensor data 250 may also include, for example, odometry data, GPS data, or other location data. Furthermore, in one embodiment, the sensing module 220 controls the sensors to collect sensor data over an area encompassing 360 degrees around the vehicle 100, which may then be stored in the sensor data 250. In some embodiments, such area sensor data may be used to obtain a comprehensive assessment of the surroundings of the vehicle 100. Of course, in alternative embodiments, the sensing module 220 may only collect sensor data over a forward direction if, for example, the vehicle 100 is not equipped with further sensors to capture additional areas around the vehicle, or the additional areas are not scanned for other reasons (e.g., unnecessarily due to known current conditions).

[0019] In one embodiment, the test bench system 170 also includes a database 240. The database 240, in one embodiment, is an electronic data structure stored in the memory 210 or other data storage and configured with routines executable by the processor(s) 110 to analyze stored data, provide stored data, organize stored data, etc. Thus, in one embodiment, the database 240 stores data used by the detection module 220 and the command module 230 in performing various functions. In one embodiment, the database 240 includes sensor data 250 along with, for example, metadata characterizing various aspects of the sensor data 250. The metadata may include, for example, location coordinates (e.g.,Longitude and latitude), relative map coordinates or tile identifiers, time / date stamp from the generation of each sensor data 250, etc.

[0020] In one embodiment, the command module 230 generally includes instructions for controlling the processor(s) 110 or the collection of processors in the cloud computing environment 300, as shown in Fig. 3 shown.

[0021] With reference to Fig. 3, the vehicle 100 may be connected to a network 305 that enables communication between the vehicle 100 and cloud servers (e.g., cloud server 310), infrastructure devices (e.g., infrastructure device 340), other vehicles (e.g., vehicle 380), and other systems connected to the network 305. With respect to the network 305, such a network may use any form of communication or networking to exchange data, including, but not limited to, the Internet, Directed Short Range Communication (DSRC) service, LTE, 5G, millimeter wave (mmWave) communication, and so on.

[0022] Cloud server 310 is illustrated as including a processor 315, which may be part of test bench system 170 via network 305 and communications unit 335. In one embodiment, cloud server 310 includes memory 320 in which a communications module 325 is stored. Memory 320 is random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable storage for storing communications module 325. Communications module 325 is, for example, computer-readable instructions that, when executed by processor 315, cause processor 315 to perform the various functions disclosed herein. In one embodiment, cloud server 310 also includes a database 330.Database 330, in one embodiment, is an electronic data structure stored in memory 320 or other data storage and configured with routines executable by processor 315 to analyze stored data, provide stored data, organize stored data, etc.

[0023] The infrastructure device 340 includes a processor 345, which may be part of the test bench system 170 via the network 305 and the communication unit 370. In one embodiment, the infrastructure device 340 includes a memory 350 in which a communication module 355 is stored. The memory 350 is a random access memory (RAM), a read-only memory (ROM), a hard disk drive, a flash memory, or other suitable storage for storing the communication module 355. The communication module 355 is, for example, computer-readable instructions that, when executed by the processor 345, cause the processor 345 to perform the various functions disclosed herein. In one embodiment, the infrastructure device 340 also includes a database 360.Database 360, in one embodiment, is an electronic data structure stored in memory 350 or other data storage and configured with routines executable by processor 345 to analyze stored data, provide stored data, organize stored data, etc.

[0024] Accordingly, in addition to the information obtained from sensor data 250, test bench system 170 may receive information from cloud servers (e.g., cloud server 310), infrastructure devices (e.g., infrastructure device 340), other vehicles (e.g., vehicle 380), and any other systems connected to network 305. For example, cloud servers (e.g., cloud server 310) may be used to perform the same tasks as described herein with respect to command module 230.

[0025] Fig. shows an example of the autonomous driving test bench that can be implemented with the vehicle 100 and the test bench system 170. The autonomous driving test bench 400 can consist of a vehicle interface component 410, a vehicle environment simulator 420, an autonomous driving software stack 430, an HMI hardware component 440, a left-side head-up (LHS) component 450, and a right-side head-up (RHS) component 460. These components can be directly or indirectly connected to facilitate electronic communication between the various components of the autonomous driving test bench 400. As shown in Fig. 4, the vehicle interface component 410 may be coupled, for example, via a CAN bus to exchange messages with the vehicle environment simulator 420, the HMI hardware component 440, the LHS component 450, and the RHS component 460. Furthermore, the vehicle interface component 410 may be coupled via Ethernet to exchange messages with the autonomous driving software stack 430. Ethernet may also be used to connect the autonomous driving software stack 430 to the HMI hardware component 440. Likewise, a CAN bus may be used to connect both the LHS component 450 and the RHS component 460 to the vehicle environment simulator 420 and the HMI hardware component 440. In various embodiments, instead of the Fig. 4, or in addition to the connections shown, other forms of connection may be used. For example, a CAN bus may be replaced by an Ethernet network; one or more gateways may be implemented to allow CAN bus messages to be sent over Ethernet; wired connections may be replaced by wireless or optical connections; protocols other than Ethernet may be used, etc.

[0026] In Fig. 5 illustrates an example of a vehicle interface component 500 that may be used to provide the vehicle interface component 410 described above. The vehicle interface component 500 may consist of a vehicle control interface 510 and a prototype control interface 520. The vehicle control interface 510 may be configured to control a vehicle (e.g., the vehicle 100) or a portion thereof (e.g., production components). The prototype control interface 520 may be configured to control one or more prototype hardware components, which may be fully or only partially integrated into the vehicle (or a portion thereof).

[0027] For example, the vehicle control interface 510 may provide all vehicle functions of the vehicle 100 except for a steering system, while the prototype control interface 520 may provide the vehicle functions of a prototype steering system that interacts with the vehicle 100. In this way, the prototype steering system provided by the prototype control interface 520 may be configured to function as a steering system for the vehicle 100.

[0028] In some embodiments, the prototype control interface 520 may configure the prototype steering system as the steering system of the vehicle 100 by instructing the vehicle 100 to send all messages that would be received by a standard steering system of the vehicle 100 to the prototype steering system. Furthermore, the prototype control interface 520 may be configured to provide all the functionality required to enable the operation of the one or more prototype hardware components that the vehicle control interface 510 may not be able to provide (e.g., the operation of a cooling system for the prototype hardware component, which the vehicle control interface 510 is not designed to provide).

[0029] uf Fig. 6 depicts an example of a vehicle environment simulator 600 that may be used to provide the vehicle environment simulator 420 described above. The vehicle environment simulator 600 may consist of a vehicle communication agent 610 and a vehicle environment simulator 620. The vehicle communication agent 610 may enable one or more vehicle-based communication protocols (e.g., a vehicle CAN bus agent, Ethernet) to support the interconnection of hardware and software components within the test bench (e.g., the autonomous driving test bench 400). For example, the vehicle communication agent 610 may provide vehicle-based communication functionality to support various vehicle types, such as hybrid vehicles, internal combustion engine vehicles, electric vehicles, or others, allowing production components, prototype components, or simulation components to interoperate with each other.For example, the vehicle communication agent 610 can emulate the connections of multiple CAN buses simultaneously. The vehicle communication agent 610 can also translate messages so that production, prototype, or simulation components can communicate seamlessly with each other, even if they use incompatible communication protocols (e.g., translating CAN bus messages from one component into Ethernet messages that can be received by another component, and vice versa). The vehicle communication agent 610 can also duplicate messages, e.g., when multiple instances of production components, prototype components, or simulation components are to be tested.For example, the vehicle communication agent 610 may duplicate the brake control inputs from the LHS component 900 so that one or more braking vehicle components, one or more braking prototype components, one or more braking virtual components, or a combination thereof, all operate with the same brake control input messages. In some embodiments, when duplicating messages, the vehicle communication agent 610 may also adjust the timing of the duplicate messages so that multiple instances of production components, prototype components, or simulation components can operate synchronously (or alternatively, in a staggered arrangement, if desired).

[0030] The vehicle environment simulator 620 can provide simulation and evaluation functions based on information received from the production components, prototype components, or simulation components. As described in Fig. As shown in Figure 11, the information collected by the vehicle environment simulator 620 may, for example, be used to model a virtual representation of a vehicle in a test environment. The vehicle environment simulator 620 may also be capable of evaluating the performance of the virtual vehicle in the virtual environment, e.g., the effects of various vehicle controls in extreme driving scenarios (e.g., high-speed driving, signal losses, unstable communication, unexpected events). The vehicle environment simulator 620 may also compare its evaluations with data obtained outside the test bench (e.g., from external sensors on a test track) and may also enable adjustment of the evaluations based on such comparisons.

[0031] Fig. 7 shows an example of an autonomous driving software stack 700 that can be used to provide an autonomous driving software stack 430 as described above. The autonomous driving software stack 700 can provide all of the semi-autonomous or autonomous functions described herein, including simulating scenarios and performing perception, planning, and control functions. The autonomous driving software stack 700 can also include the safety controller 710. The safety controller 710 can be used to implement various operating states, where each operating state can specify how a vehicle (e.g., vehicle 100) is controlled between the LHS component 900, the RHS component 1000, the autonomous driving software stack 700, other sources of control inputs, or a combination thereof. For example, Fig. 7B is a state transition diagram of three different operating states that can be implemented by safety controller 710. In one such example, safety controller 710 has three different operating states provided.

[0032] In operating state S1, which may be the default state, the safety controller 710 may configure the test bench so that the vehicle drives exclusively under the control of the LHS component 900 (e.g., as if it were a conventional vehicle).

[0033] In operating state S2, which may be the test state, the safety controller 710 may configure the test bench so that the vehicle operates in substate S2a, S2b1, or S2b2.

[0034] In substate S2a, the vehicle may be operated exclusively under the control of the RHS component 1000. In some embodiments, substate S2a may enable joint control by the LHS component 900 and the RHS component 1000, which may also include a request that the RHS component 1000 override the LHS component 900 with respect to vehicle control if there is a difference between the two.

[0035] In substate S2b1, the vehicle may operate exclusively under the control of autonomous driving software stack 700. In some embodiments, substate S2b1 may allow LHS component 900, RHS component 1000, or both to provide control inputs to autonomous driving software stack 700, which may then be evaluated by autonomous driving software stack 700 in determining control inputs for the vehicle.

[0036] In substate S2b2, the vehicle may operate as described above with respect to substate S2b1, except that the RHS component 1000 may override all or part of the autonomous driving software 700, the LHS component 900, or both if a guardian interrupt mode (as described below) is activated. For example, an initial activation of the guardian interrupt mode may cause the RHS component 1000 to override the autonomous driving software stack 700, the LHS component 900, or both until the guardian interrupt becomes inactive.As another example, a second activation of Guardian Interrupt Mode may result in an operating condition that requires the steering and throttle control adjustments made by RHS component 1000 to be implemented, but still allows autonomous driving software stack 700, LHS component 900, or both to affect braking control if no contrary input has been received from RHS component 1000.

[0037] In operating state S0, which may be the emergency state, the vehicle may be operated to safely come to a stop, e.g., by decelerating to a stop and pulling over to the side of the road, disengaging the accelerator pedal, etc. State S0 may be a state that must be reached by the safety controller 710 from any other state when certain conditions are met (e.g., failure of a critical system). In such an embodiment, operating states that prevent a vehicle from reaching an emergency state when the predetermined conditions are met may be rejected as invalid by the safety controller 710.

[0038] The safety controller 710 can determine, based on operating conditions, the extent to which one or more components (e.g., autonomous driving software stack 700, LHS component 900, RHS component 1000) are permitted to operate the vehicle. Thus, the safety controller 710 can enable individual control or joint control between the LHS component 900, the RHS component 1000, or the autonomous driving software stack 700, depending on the operating condition. If an operating condition dictates joint control, the operating condition can determine which control inputs have priority (e.g., the autonomous driving software stack 700 always takes precedence over the LHS component 900 and the RHS component 1000). In some embodiments, the priority can be determined by weighting the control inputs.For example, an operating state may specify that the control signals of RHS component 1000 have twice the influence of the control signals of LHS component 900. As another example, an operating state may use a weighting function applied to one or more control inputs of the same type to form a combined control of that type.

[0039] In some embodiments, the safety controller 710 may establish concurrency rules regarding the visible state of the control input devices. For example, a simple concurrency rule is that the steering wheel of the RHS component 1000 and the steering wheel of the LHS component 900 must be equally positioned with respect to steering control inputs (e.g., if the RHS component 1000 generates control inputs, the LHS component 900 must be set to reflect those control inputs, or vice versa) or steering feedback (e.g., feedback signals from a steering system in the vehicle 100). In this way, the steering wheels may appear to respond equally due to control inputs originating from either driver, both drivers, the autonomous driving software stack 700, etc. In some embodiments, the positioning of the control input devices may be determined by a combined control input (e.g.,by a weighting function).

[0040] As another example, in a state of operation where the RHS component 1000 and LHS component 900 can jointly operate the vehicle and the LHS component 900 can override the RHS component 1000, the concurrency rule may require: (i) that the control input devices of the RHS component 1000 mimic the control input devices of the LHS component 900 when no drive inputs are applied to control input devices of the RHS component 1000; and (ii) that the control input devices of the LHS component 900 always mimic the control input devices of the RHS component 1000 when drive inputs are applied to control input devices of the RHS component 1000.In this way, the control input devices of the RHS component 1000 may reflect the actions of a first driver who controls the vehicle primarily through the control input devices of the LHS component 900, unless a second driver uses the control input devices of the RHS component 1000 to take control from the first driver.

[0041] In some embodiments, the safety controller 710 may also define transition rules for handling changes between operating states with different concurrency rules. For example, when the safety controller 710 transitions from an operating state in which only control of the LHS component 900 is possible ( ) to an operating state in which joint control by the LHS component 900 and the RHS component 1000 is possible, a transition rule may specify a ramp function to be applied to the safety controller 710's implementation of the concurrency rule of this operating state, which may make the transitions appear smoother to the driver when changing operating states.In some embodiments, such a transition rule may not only be aesthetically pleasing but may also provide safety benefits by limiting sudden changes that could startle or injure a driver due to the sudden implementation of a concurrency rule. In some embodiments, the transition rule may only affect how the implementation of a concurrency rule affects the physical position of control input devices of the LHS component 900 or the RHS component 1000. For example, even if it is desirable to create a smoother appearance of transitions as described above, entering an operational state may still require the immediate implementation of a concurrency rule, except with respect to the physical position of control devices.Such transition rules may be implemented, for example, when entering a new operating state causes the autonomous driving software stack 700 to perform a critical and extreme maneuver for safety purposes.

[0042] In Fig. 8, an example of an HMI hardware component 800 is shown: The HMI hardware component 800 may include a vehicle notification system 810, a guard interrupt component 820 (e.g., a pedal that can be actuated by a safety driver to assume full control of a vehicle), or other HMI components not provided by the LHS component 900 or the RHS component 1000, as described below.

[0043] The vehicle notification system 810 may support any type of audible, visual, haptic, or other warning or indicator elements that can be used to communicate with the vehicle operator. For example, the vehicle notification system 810 may use visual indicators (e.g., colored LEDs), audible indicators (e.g., various tones, voice prompts), vibrations, etc., to indicate the status of system components or the system as a whole, the operational state of the vehicle (e.g., manual, autonomously active), the activation of a Guardian suspend mode, the operator controlling the vehicle (e.g., DUT, TSD, Guardian), the operation of the vehicle in emergency situations, etc.

[0044] In some embodiments, the vehicle notification system 810 may use HMI rules to facilitate the translation of HMI messages. For example, various displays, steering wheels, or other HMI components may be used within the test bench, each having different HMI interfaces. Accordingly, the vehicle notification system 810 may store and use HMI rules that enable HMI to be implemented between the test bench and one or more HMI devices. In some embodiments, when a new component is added to the test bench, the vehicle notification system 810 may detect the change in available HMI interfaces, locate one or more HMI rules based on this detection, and then implement the one or more HMI rules.If a component is removed from the test bench, the vehicle notification system 810 can detect such a change in the HMI interfaces and discontinue the use of HMI rules that are no longer required.

[0045] The sentinel interrupt component 820 may enable a variety of input control devices (e.g., those used by a TSD via the RHS component 1000) to partially or fully control the vehicle in a manner that overrides all other driving inputs. For example, the sentinel interrupt component 820 may utilize a pedal, button, voice command, or other forms of HMI interface to allow a driver to activate or deactivate a sentinel interrupt mode.

[0046] When the watchdog mode is active, the safety controller 710 may enter an operating state in which one or more control signals take precedence over all other control signals. For example, if a TSD operating the RHS component 1000 activates the watchdog mode, the steering, brakes, throttle, and all other control inputs of the RHS component 1000 may take precedence over all control inputs of the LHS component 900, the autonomous driving software stack 700, or other sources. In some embodiments, such priority may suppress all control signals not permitted by the operating state triggered by the activation of the watchdog interrupt mode (e.g., RHS only, no LHS, no autonomous driving software stack 700). In some embodiments, such suppression may be limited to some controls (e.g.,only RHS steering and throttle, but LHS brake or autonomous driving software stack 700 can still activate braking systems).

[0047] In some embodiments, the guardian interrupt component 820 may provide multiple guardian interrupt modes with different priorities. For example, the LHS component 900 may have an LHS guard interrupt pedal that allows it to suppress control inputs from the autonomous driving software stack 700, while the RHS component 1000 may have an RHS guard interrupt pedal that allows it to suppress control inputs from the LHS component 900 or the autonomous driving software stack 700.

[0048] In some embodiments, guardian interrupt interfaces may be operated by test users other than a DUT or TSD. For example, a track observer may enable a "guardian interrupt" mode via a mobile application interface communicating with the test bench, which prioritizes the control inputs of the autonomous driving software stack 700 over the control inputs of the LHS component 900 or the RHS component 1000. Such a configuration may be useful when only one driver (e.g., the test subject) operates the test bench vehicle and the application of the autonomous driving software stack 700 is desired to ensure the safe operation of the vehicle.

[0049] In Fig. 9 illustrates an example of an LHS component 900. The LHS component 900 may consist of any components used to generate control inputs based on the actions of a left-hand drive vehicle. For example, the LHS component 900 may utilize an LHS brake pedal 910 with an LHS brake actuator 920 and an LHS brake ECU 930 to receive braking inputs from the LHS driver and generate LHS brake control inputs. The LHS component 900 may also utilize an LHS throttle pedal 940 with an LHS throttle actuator 950 and an LHS throttle ECU 960 to receive throttle inputs from the LHS driver and generate LHS throttle control inputs. Similarly, the LHS component 900 may also utilize an LHS steering wheel 970 with an LHS steering actuator 980 and an LHS steering ECU 990 to receive steering inputs from the LHS driver and generate LHS steering control inputs.In some embodiments, the LHS component 900 may include other arrangements of LHS devices for generating control inputs based on the actions of an LHS driver. For example, the LHS brake pedal 910, the LHS brake actuator 920, and the LHS brake controller 930 may be omitted when the throttle components provide both accelerator and brake functions (e.g., in single-pedal operation). As another example, LHS devices may be added to the LHS component 900 to generate control inputs based on the voice commands of an LHS driver.

[0050] In Fig. Figure 10 illustrates an example of an RHS component 1000. The RHS component 1000 may be identical in configuration to the LHS component 900, except that the devices of the RHS component 1000 may be used to generate RHS control inputs based on the actions of an RHS driver.

[0051] In some embodiments, the devices of RHS component 1000 may differ from those of LHS component 900. For example, LHS component 900 may use a prototype steering wheel, while RHS component 1000 may use a production steering wheel. As another example, LHS component 900 may use devices for generating LHS control inputs based on voice commands from the LHS driver, while RHS component 1000 may employ production control input devices (e.g., brake, accelerator, steering wheel). Even if RHS component 1000 uses the same devices as LHS component 900, the arrangement of these components between RHS component 1000 and LHS component 900 may be identical, reversed, mirrored, or otherwise arbitrarily arranged.

[0052] Although the examples refer to LHS and RHS, these references are merely exemplary. The LHS components and the LHS driver can be located anywhere, for example, inside a vehicle (e.g., on the left side of the vehicle, in the middle of the vehicle, on the right side of the vehicle, at the rear of the vehicle) or outside a vehicle (e.g., at a desk, a dual cockpit). Likewise, the location of the RHS components and the RHS driver can be anywhere. All examples given here with reference to left-hand drive vehicles are therefore also applicable to other vehicle configurations (e.g., center-drive, right-hand drive, remote-control).

[0053] In some embodiments, the autonomous driving test bench (400) may support testing any arrangement of one or more production components, one or more prototype components, one or more virtual components, or a combination thereof. For example, it may be desirable to perform a test that compares prototype brake components to an in-use production brake component and also to a virtual brake component (which may, for example, represent an ideal brake component). To perform such a test, any control signals, feedback signals, sensor signals, or other data communicated within the autonomous driving test bench 400 may be duplicated.For example, the brake control inputs of the autonomous driving software stack 700, the LHS component 900, the RHS component 1000, or a combination thereof may be duplicated so that each production brake component, prototype brake component, or virtual brake component receives the same set of brake control inputs.

[0054] In some cases, open-loop testing by duplicating control inputs or other data may be sufficient to test all components. For example, production, prototype, or virtual door-lock components can be tested to determine when each component might fail at extreme temperatures (e.g., by failing to lock or unlock within specified parameters). The test leverages the duplication of door-lock control signals and temperature sensor data. In virtual component testing, a virtual component may contain a deterministic or probabilistic model that demonstrates how such a virtual component might degrade, fail, or otherwise behave as the system test parameters change.

[0055] In some cases, closed-loop testing may be required to fully test a component. For example, the output signals of a braking component may affect the overall behavior of the vehicle in a way that results in changes to future braking control inputs. In such a situation, the autonomous driving test bench 400 may, among others, take the following approaches to test a set of production, prototype, or virtual components.

[0056] First, to the extent that the necessary resources for closed-loop testing are available, the autonomous driving test bench 400 may attempt to create as many instances of closed-loop testing as possible. Even if the autonomous driving test bench 400 only has access to one vehicle, such a vehicle may, for example, have four sets of braking systems, allowing closed-loop testing of up to four production, prototype, or virtual braking components at the same time. In such a situation, the autonomous driving test bench 400 may be provided with a selection of possible closed-loop testing options from which the user can choose (e.g., front brakes only, rear brakes only) to limit the closed-loop testing to specific configurations of interest.

[0057] Second, to the extent the test bench settings allow, the autonomous driving test bench (400) may attempt to schedule closed-loop testing for a sequence of subsets of the set of production, prototype, or virtual components according to time-based intervals. For example, closed-loop testing may proceed by testing two brake components at a time until completed. In some embodiments, the test sequence may be structured using time-division multiplexing, e.g., where closed-loop testing may be completed before setting a system test parameter to a new state (e.g., testing all brake components at -20°F before lowering the temperature to -25°F).

[0058] Third, in some embodiments, the autonomous driving test bench (400) may be instructed to perform closed-loop testing on a first subset of the set of production, prototype, or virtual components and to perform open-loop testing on a second subset of the set of production, prototype, or virtual components. Such an approach may be useful, for example, when a failure of an open-loop component test precludes the need for further closed-loop testing.

[0059] Fourth, in some embodiments, the autonomous driving test bench 400 may generate simulated feedback signals or other data so that a subset of the production, prototype, or virtual components that is incapable of performing closed-loop testing can still perform simulated closed-loop testing. Simulated feedback signals may be implemented, for example, by simulating in a virtual environment how the subset's outputs affect vehicle operation (e.g., via the vehicle environment simulator 620) to generate simulated feedback signals, simulated sensor data, or other simulated data. Such an approach may be useful, for example, when a failure of the simulated closed-loop testing of a component may preclude the need for further closed-loop testing of the component.In some embodiments, simulated closed-loop testing only requires simulating those physical resources that are insufficient for closed-loop testing. It should also be understood that in some embodiments, closed-loop testing may use virtual components (e.g., a software prototype), but unlike simulated closed-loop testing, there is no substitution of a simulated component with a production component, prototype component, virtual component, etc.

[0060] With regard to the bench tests described above, it should be understood that the production, prototype, or virtual components do not necessarily have to be located in the same vehicle or even in the same city. For example, a practical aspect of the autonomous driving test bench 400 is that it can enable component testing at remote locations. For example, if a mechanic at a car dealership wants to check whether a vehicle component is still reliable, they can access the autonomous driving test bench 400, connect the vehicle component in question to the bench, and then perform tests via the bench to evaluate the component's reliability. Such an approach can be useful when the reliability of a component cannot be fully determined without real testing in a vehicle or vehicle subsystem.Likewise, third parties developing a prototype component may access the test bench to perform testing on one or more prototype components. In some embodiments, a vehicle (e.g., vehicle 100) located at a location (or a portion of the vehicle) may be used as part of the autonomous driving test bench 400 to remotely test components.

[0061] Fig. 12 shows a flowchart of a method 1200 associated with strategies for autonomous vehicle testing that may include both a driver to be tested and a safety-trained driver. The method 1200 is described from the perspective of the test bench system 170 of Fig. 1 and Fig. 2. Although the method 1200 is discussed in combination with the test bench system 170, it should be understood that the method 1200 is not limited to being implemented within the test bench system 170, but rather is an example of a system that may implement the method 1200.

[0062] In step 1210, the command module 230 may initialize the test bench system 170 to provision a test bench (e.g., the autonomous driving test bench 400). Upon provisioning the test bench, the command module 230 may configure one or more production components, one or more prototype components, one or more virtual components, or a combination thereof to communicate with the test bench system 170 and also be capable of operating as a vehicle component for a test that may be performed on the test bench system 170. In some embodiments, the command module 230 may provide this with a plug-and-play functionality, where a component added to the test bench system 170 provides sufficient information for the command module 230 to configure the component for interoperability with the test bench system 170.

[0063] In step 1220, the command module 230 may receive instructions to perform a test with respect to a subset of the one or more production components, one or more prototype components, one or more virtual components, or a combination thereof. Upon receiving the instructions, the command module 230 may determine the testbench resources available to perform the test. For example, if a closed-loop test is requested, the command module 230 may determine the available resources to support the closed-loop test. If there are insufficient resources available for the closed-loop test, the command module 230 may offer alternative approaches, such as sequential closed-loop testing (e.g., performing the closed-loop test on components one at a time until the test is complete), closed-loop / open-loop testing (e.g.,Performing closed-loop testing of some components, performing open-loop testing for the remaining components), closed-loop / simulated closed-loop testing (e.g., virtually adding resources via simulation to perform a simulated closed-loop test for the components that cannot participate in the closed-loop test), and so on.

[0064] In step 1230, the command module 230 may create (and present to a test user) a customizable test implementation plan that demonstrates which types of tests can be performed by the test bench system 170. The command module 230 may then receive instructions that include modifications, selections, or other adjustments to the customizable test execution plan (e.g., selecting closed-loop / simulated closed-loop tests and specifying which components should be tested with closed-loop or simulated closed-loop tests, respectively).

[0065] In step 1240, the command module 230 may select one or more production components, one or more prototype components, one or more virtual components, or a combination thereof to be tested according to the adjustable test implementation plan and configure those components to perform the test. To configure the components to perform the test, the command module 230 may instruct the test bench system 170 regarding the arrangement of control signals, feedback signals, simulated signals, sensor data, etc., that must be communicated throughout the test bench system 170. For example, the command module 230 may instruct the LHS component 900 to use a prototype throttle component for disabled drivers instead of a standard production throttle component.In some embodiments, the command module 230 may generate simulation components, simulation environments, or both to facilitate simulated closed-loop testing as needed. Furthermore, instructions regarding the arrangement of control signals, feedback signals, simulated signals, sensor data, etc., may include specifications regarding the routing, duplication, generation, or simulation of information, or other adaptations necessary to enable the components under test.

[0066] In step 1250, the command module 230 may determine operating states for use with the adjustable test implementation plan. In some embodiments, the operating states may be automatically generated based on the components being tested or used by the adjustable test implementation plan. In some cases, the operating states may also be generated based on a selection of a predefined template. The operating states may also be further refined by a test bench user's changes to the operating states generated by the command module 230. In some embodiments, the command module 230 may generate the operating state based on the number of existing drivers or the functional availability of the autonomous driving software stack 700.For example, if it is determined that no driver is capable of operating the RHS component 1000, the operating states may be changed by removing states that are only used when the RHS component 1000 is used. Similarly, if the autonomous driving software stack 700 lacks the functionality to work with certain aspects of a test (e.g., because a TSD is overseeing the vehicle until such functionality is sufficiently developed), the operating states may be changed by removing states that would only be used if such functionality were present. The determination of the operating states by the command module 230 may also include the command module 230 determining concurrency rules, transition rules, weighting functions, etc. for each operating state.The command module 230 may also generate or modify operating states to incorporate criteria related to changes in test or environmental data that may also cause a transition from one operating state to another. For example, certain operating states may be restricted based on geolocation data (e.g., for use with test tracks only).

[0067] In step 1260, the command module 230 may perform the test, including tracking the operating states. While tracking the operating states, the command module 230 may adjust the source of the control signals, the position of the control devices, generate combined control signals, etc., based on the control inputs of the one or more drivers, the autonomous driving software stack 700, or other vehicle control sources.

[0068] In step 1270, the command module 230 may receive an activation signal for a Guardian Interrupt mode. After receiving the Guardian Interrupt mode, the command module 230 may transition to another operating state based on the activated Guardian Interrupt mode. While the Guardian Interrupt mode is active, the command module 230 may only follow certain control signals (e.g., those originating from the RHS component 1000) while ignoring others (e.g., from the autonomous driving software stack 700 or the LHS component 900). Furthermore, after deactivating the Guardian Interrupt mode, the command module 230 may transition to another operating state (e.g., to the state prior to the Guardian Interrupt mode activation).

[0069] Fig. 13 shows a flowchart of a method 1300 associated with handling the vehicle control system with respect to an implementation with both a driver to be tested and a safety-trained driver. The method 1300 is described from the perspective of the test bench system 170 of the Fig. 1 and Fig. 2. While the method 1300 is discussed in combination with the test bench system 170, it should be understood that the method 1300 is not limited to being implemented within the test bench system 170, but instead is an example of a system that may implement the method 1300.

[0070] In step 1310, the command module 230 may provide a test bench capable of receiving a first set of inputs from a first set of devices operated by a first driver, a second set of inputs from a second set of devices operated by a second driver, and a third set of inputs from an autonomous driving component. For example, the vehicle 100 may be configured to have controls on both the left and right sides of the front vehicle compartment. The left side may be configured for a device under test. The right side may be configured for a TSD that includes a monitor pedal for activating a monitor interrupt mode. The test bench system 170 of the vehicle 100 may be configured to provide the autonomous driving test bench 400 related to operation of the vehicle 100 by the device under test, the TSD, or an autonomous driving module.

[0071] In step 1320, the command module 230 may generate operating states, each operating state determining how the first, second, and third sets of inputs can control a vehicle. For example, one operating state may be such that the left side of the vehicle 100 can control the vehicle 100 without interference from the TSD or the autonomous driving module, and another operating state may be such that the left side of the vehicle 100 can control the vehicle 100 if overridden by the TSD or the autonomous driving module.

[0072] In step 1330, the command module 230 may determine a current operating state of the vehicle. For example, the command module 230 may determine the current operating state of the vehicle 100 based on the test being performed, the environmental parameters, the behavior of the device under test, the TSD or autonomous driving module, or other factors.

[0073] In step 1340, the command module 230 may configure the vehicle control through the first input set, the second input set, and the third input set based on the current operating state. For example, if the operating state of the vehicle changes, the command module 230 may adjust the ability of the device under test, the TSD, and the autonomous driving module to control the vehicle. Furthermore, the command module 230 may use concurrency rules or transition rules when implementing such a vehicle control configuration.

[0074] Fig. 14 shows a flowchart of a method 1400 associated with handling multiple instances of production components, prototype components, and virtual components during testing. The method 1400 is described from the perspective of the test bench system 170 of the Fig. discussed. Although the method 1400 is discussed in connection with the test bench system 170, it should be understood that the method 1400 is not limited to being implemented within the test bench system 170, but is merely one example of a system that may implement the method 1400.

[0075] In step 1410, the command module 230 may provide a vehicle test bench capable of operating physical and virtual vehicle components. For example, the test bench system 170 may initialize the autonomous driving test bench 400 and provide the plug-and-play functionality described herein. For example, if a prototype component is added to the test bench, the command module 230 may electronically receive an identifier from the prototype component that allows it to look up how the test bench can be configured to interact with the prototype component.

[0076] In step 1420, the command module 230 may receive a test plan for a set of vehicle components. A test plan may, for example, specify the test procedures, the components to be tested, the test environment, the test parameters, etc.

[0077] In step 1430, the command module 230 may determine test bench components and test bench connections, including virtual components, to implement the test plan on the vehicle test bench. For example, the test plan may include sufficient information about the components to be tested ( ) as well as the closed-loop components whose use is restricted during the test (e.g., only one component may use it at a time). In some embodiments, the type of test or the component may provide sufficient information for the command module 230 to determine which control loop components may be restricted. For example, if the type of test is an automated driving-based longitudinal control function with varying levels of aggressiveness, the command module 230 may determine that the braking and throttle components should be restricted.Once the constraints are identified, the command module 230 can determine if other components are available elsewhere (e.g., at a different location) or if the virtual simulation can be used as a substitute. If no substitute components are available, the command module 230 can analyze the test plan and suggest changes that enable the execution of a modified test plan. In some embodiments, the command module 230 can have settings that instruct it how to proceed with test changes without test bench user approval.

[0078] In step 1440, the command module 230 may generate the virtual components for executing the test plan. For example, if the command module 230 has determined that a test plan requires a virtual steering system that controls a virtual vehicle in a virtual environment, the command module 230 may construct such elements for use with the test plan.

[0079] In step 1450, the command module 230 may implement the test bench connections to enable testing of the vehicle components. For example, as the command module 230 progresses through the test plan, it may be necessary to change the components used for closed-loop testing or other types of tests. In this case, the command module 230 may adapt the test bench by changing the routing of the test bench signals through the test bench. For example, if a portion of a test was completed with a production throttle control module and needs to be continued with a prototype throttle control module, the command module 230 may reroute the throttle control inputs through the prototype throttle control module instead of the production throttle control module.

[0080] Fig. 1 will now be discussed in detail as an example of an environment in which the system and methods disclosed herein may operate. In some cases, the vehicle 100 is configured to selectively switch between different modes, such as an autonomous mode, one or more semi-autonomous modes of operation, a manual mode, etc. Such switching may be accomplished in any suitable manner now known or later developed. "Manual mode" means that all or a majority of the navigation / maneuvering of the vehicle is performed according to inputs from a user (e.g., a human driver). In one or more arrangements, the vehicle 100 may be a conventional vehicle configured to operate only in a manual mode.

[0081] In one or more embodiments, the vehicle 100 is an autonomous vehicle. As used herein, "autonomous vehicle" refers to a vehicle operating in an autonomous mode. "Autonomous mode" refers to the use of one or more computer systems to control the vehicle 100, such as providing navigation / maneuvering of the vehicle 100 along a travel route, with minimal or no input from a human driver. In one or more embodiments, the vehicle 100 is either highly automated or fully automated. In one embodiment, the vehicle 100 is configured with one or more semi-autonomous modes of operation in which one or more computer systems provide some of the navigation / maneuvering of the vehicle along a travel route, and a vehicle operator (i.e.the driver) provides inputs to the vehicle to perform part of the navigation / maneuvering of the vehicle 100 along a travel route.

[0082] The vehicle 100 may include one or more processors 110. In one or more arrangements, the processor(s) 110 may be a main processor of the vehicle 100. For example, the processor(s) 110 may be an electronic control unit (ECU). The vehicle 100 may include one or more data memories 115 for storing one or more types of data. The data memories 115 may comprise volatile memory, non-volatile memory, or both. Examples of suitable data memories 115 include random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, magnetic disks, optical disks, hard disks, or any other suitable storage medium or combination thereof.The data storage(s) 115 may be a component of the processor(s) 110, or the data storage 115 may be operatively connected to the processor(s) 110 for use by the processor(s). The term "operatively connected" as used in this specification may encompass direct or indirect connections, including connections without direct physical contact.

[0083] In one or more arrangements, the data storage(s) 115 may contain map data 116. Map data 116 may contain maps of one or more geographic areas. In some cases, the map data 116 may contain information or data about roads, traffic control devices, road markings, structures, features, landmarks, or any combination thereof within the one or more geographic areas. The map data 116 may be in any suitable form. In some cases, the map data 116 may contain aerial photographs of an area. In some cases, the map data 116 may contain ground views of an area, including 360-degree ground views. The map data 116 may contain measurements, dimensions, distances, information, or any combination thereof for one or more items contained in the map data 116.The map data 116 may also include measurements, dimensions, distances, information, or any combination thereof related to other elements contained in the map data 116. The map data 116 may include a digital map with information about the road geometry. The map data 116 may be high quality, highly detailed, or both.

[0084] In one or more arrangements, the map data 116 may include one or more terrain maps 117. The terrain map(s) 117 may include information about the ground, terrain, roads, surfaces, other features, or any combination thereof in one or more geographic areas. The terrain map(s) 117 may include elevation data in the one or more geographic areas. The terrain map(s) 117 may be high quality, highly detailed, or both. The terrain map(s) 117 may define one or more ground surfaces, which may include paved roads, unpaved roads, land, and other things that define a ground surface.

[0085] In one or more arrangements, the map data 116 may include one or more static obstacle maps 118. The static obstacle map(s) 118 may include information about one or more static obstacles located in one or more geographic areas. A "static obstacle" is a physical object whose position does not change, or changes only insignificantly, over a given period of time and whose size does not change, or changes only insignificantly, over a given period of time. Examples of static obstacles include trees, buildings, curbs, fences, railings, medians, utility poles, statues, monuments, signs, benches, furniture, mailboxes, large rocks, and hills. The static obstacles may be objects that extend above ground level.The static obstacles included in the static obstacle map(s) 118 may be associated with location data, size data, dimension data, material data, other data, or any combination thereof. The static obstacle map(s) 118 may contain measurements, dimensions, distances, information, or any combination thereof for one or more static obstacles. The static obstacle map(s) 118 may be high quality, highly detailed, or both. The static obstacle map(s) 118 may be updated to reflect changes within a mapped area.

[0086] The data storage(s) 115 may contain sensor data 119. In this context, "sensor data" means all information about the sensors with which the vehicle 100 is equipped, including the capabilities and other information about those sensors. As explained below, the vehicle 100 may include a sensor system 120. The sensor data 119 may relate to one or more sensors of the sensor system 120. In one or more arrangements, the sensor data 119 may, for example, contain information about one or more LIDAR sensors 124 of the sensor system 120.

[0087] In some cases, at least a portion of the map data 116 or sensor data 119 may be located in data storage devices 115 located on board the vehicle 100. Alternatively or additionally, at least a portion of the map data 116 or sensor data 119 may be stored in data storage devices 115 located remotely from the vehicle 100.

[0088] As previously mentioned, vehicle 100 may include a sensor system 120. Sensor system 120 may include one or more sensors. A "sensor" is a device, component, or system capable of detecting or sensing something. The one or more sensors may be configured to sense, detect, or both in real time. As used herein, the term "real time" means a level of processing responsiveness that a user or system perceives as sufficiently immediate for a particular process or determination, or that allows the processor to keep pace with an external process.

[0089] In arrangements where the sensor system 120 includes a plurality of sensors, the sensors may operate independently of one another. Alternatively, two or more of the sensors may operate in combination with one another. In such an embodiment, the two or more sensors may form a sensor network. The sensor system 120, the one or more sensors, or both may be operatively connected to the processor(s) 110, the data storage(s) 115, another element of the vehicle 100 (including one of the Fig. 1) or any combination thereof. The sensor system 120 may collect data from at least a portion of the external environment of the vehicle 100 (e.g., from nearby vehicles).

[0090] The sensor system 120 may include any suitable type of sensor. Various examples of different types of sensors are described herein. However, it should be understood that the embodiments are not limited to the specific sensors described. The sensor system 120 may include one or more vehicle sensors 121. The vehicle sensor(s) 121 may detect, determine, sample, or a combination thereof, information about the vehicle 100 itself. In one or more arrangements, the vehicle sensor(s) 121 may be configured to detect, sample, or a combination thereof, such as based on inertial acceleration, position and orientation changes of the vehicle 100.In one or more arrangements, the vehicle sensor(s) 121 may include one or more accelerometers, one or more gyroscopes, an inertial measurement unit (IMU), a dead reckoning system, a global navigation satellite system (GNSS), a global positioning system (GPS), a navigation system 147, other suitable sensors, or any combination thereof. The vehicle sensor(s) 121 may be configured to detect one or more characteristics of the vehicle 100, or to sense a combination thereof. In one or more arrangements, the vehicle sensor(s) 121 may include a speedometer to determine the current speed of the vehicle 100.

[0091] Alternatively or additionally, the sensor system 120 may include one or more environmental sensors 122 configured to collect, sense, or any combination thereof, driving environment data. "Driving environment data" includes data or information about the external environment in which an autonomous vehicle is located, or one or more portions thereof. For example, the environmental sensor(s) 122 may be configured to detect, quantify, sense, or any combination thereof, obstacles in at least a portion of the external environment of the vehicle 100, information / data about such obstacles, or any combination thereof. Such obstacles may consist of stationary objects, dynamic objects, or a combination thereof.Environmental sensor(s) 122 may be configured to detect, measure, quantify / sense, or any combination thereof, other things in the external environment of the vehicle 100, such as pavement markings, signs, traffic lights, traffic signals, lane lines, crosswalks, curbs near the vehicle 100, objects in the terrain, etc.

[0092] Various examples of sensors of the sensor system 120 are described herein. The example sensors may be part of the one or more environmental sensors 122, the one or more vehicle sensors 121, or both. However, it should be understood that the embodiments are not limited to the specific sensors described.

[0093] For example, the sensor system 120 may include, in one or more arrangements, one or more radar sensors 123, one or more LIDAR sensors 124, one or more sonar sensors 125, one or more cameras 126, or any combination thereof. In one or more arrangements, the camera(s) 126 may be high dynamic range (HDR) cameras or infrared (IR) cameras.

[0094] The vehicle 100 may include an input system 130. An "input system" includes any device, component, system, element, or assembly, or group thereof, that enables the input of information / data into a machine. The input system 130 may receive input from a vehicle occupant (e.g., a driver or a passenger). The vehicle 100 may include an output system 135. An "output system" includes any device, component, or assembly, or group thereof, that enables the presentation of information / data to a vehicle occupant (e.g., a person, a vehicle passenger, etc.).

[0095] The vehicle 100 may include one or more vehicle systems 140. Various examples of vehicle systems 140 are described in Fig. 1. However, vehicle 100 may include more, fewer, or different vehicle systems. It should be noted that although certain vehicle systems are defined separately below, each of the systems or portions thereof may be combined in other ways or separated by hardware, software, or a combination thereof within vehicle 100. Vehicle 100 may include a propulsion system 141, a braking system 142, a steering system 143, a throttle system 144, a transmission system 145, a signaling system 146, a navigation system 147, other systems, or any combination thereof. Each of these systems may include one or more devices, components, or combinations thereof now known or later developed.

[0096] Navigation system 147 may include one or more known or newly developed devices, applications, or combinations thereof configured to determine the geographic location of vehicle 100, a travel route for vehicle 100, or both. Navigation system 147 may include one or more mapping applications to determine a travel route for vehicle 100. Navigation system 147 may include a global positioning system, a local positioning system, a geolocation system, or any combination thereof.

[0097] The processor(s) 110, the test bench system 170, the automated driving module(s) 160, or any combination thereof may be operatively connected to communicate with various aspects of the vehicle system(s) 140 or individual components thereof. For example, to Fig. 1, the processor(s) 110, the automated driving module(s) 160, or a combination thereof may communicate with each other to send or receive information from various aspects of the vehicle system(s) 140 to control the movement, speed, maneuvering, course, direction, etc., of the vehicle 100. The processor(s) 110, the test bench system 170, the automated driving module(s) 160, or any combination thereof may control some or all of these vehicle systems 140 and thus be partially or fully autonomous.

[0098] The processor(s) 110, the test bench system 170, the automated driving module(s) 160, or any combination thereof may control the navigation or maneuvering of the vehicle 100 by controlling one or more of the vehicle systems 140 or their components. For example, when operating in an autonomous mode, the processor(s) 110, the test bench system 170, the automated driving module(s) 160, or any combination thereof may control the direction, speed, or both of the vehicle 100. The processor(s) 110, the test bench system 170, the automated driving module(s) 160, or any combination thereof may cause the vehicle 100 to accelerate (e.g., by increasing fuel supply to the engine), decelerate (e.g., by decreasing fuel supply to the engine, by applying the brakes), change direction (e.g.,by turning the two front wheels) or any combination thereof. As used herein, "cause" or "induce" means to effect, compel, force, direct, command, direct, enable, or any combination thereof, an event or act, or at least to be in a state where such an event or act may, directly or indirectly, occur.

[0099] The vehicle 100 may include one or more actuators 150. The actuator(s) 150 may be any element or combination of elements that can modify, adjust, alter, or operate one or more vehicle systems 140 or components thereof in response to receiving signals or other inputs from the processor(s) 110, the automated driving module(s) 160, or any combination thereof. Any suitable actuator may be used. The actuators 150 may include, for example, motors, pneumatic actuators, hydraulic pistons, relays, solenoids, and piezoelectric actuators, to name a few.

[0100] The vehicle 100 may include one or more modules, at least some of which are described herein. The modules may be implemented as computer-readable program code that, when executed by processor(s) 110, implements one or more of the processes described herein. One or more of the modules may be a component of the processor(s) 110, or one or more of the modules may be executed on or distributed among other processing systems to which the processor(s) 110 is / are operatively connected. The modules may include instructions (e.g., program logic) that may be executed by the processor(s) 110. Alternatively or additionally, the data storage(s) 115 may include such instructions.

[0101] In one or more arrangements, one or more of the modules described herein may include elements of artificial or computational intelligence, e.g., neural networks, fuzzy logic, or other machine learning algorithms. Furthermore, in one or more arrangements, one or more of the modules may be distributed among a plurality of the modules described herein. In one or more arrangements, two or more of the modules described herein may be combined into a single module.

[0102] The vehicle 100 may include one or more autonomous driving modules 160. The automated driving module(s) 160 may be configured to receive data from the sensor system 120 or any other type of system capable of sensing information about the vehicle 100, the external environment of the vehicle 100, or a combination thereof. In one or more arrangements, the automated driving module(s) 160 may use such data to create one or more driving scene models. The automated driving module(s) 160 may determine the position and speed of the vehicle 100. The automated driving module(s) 160 may determine the position of obstacles or other environmental features such as traffic signs, trees, bushes, neighboring vehicles, pedestrians, etc.

[0103] The automated driving module(s) 160 may be configured to receive, determine, or any combination thereof, location information for obstacles in the external environment of the vehicle 100, which may be used by the processor(s) 110, one or more of the modules described herein, or any combination thereof, to estimate a position or orientation of the vehicle 100; a vehicle position or orientation in global coordinates based on signals from multiple satellites or other geolocation systems; or any other data / signals that could be used to determine a position or orientation of the vehicle 100 with respect to its environment, either to create a map or to determine the position of the vehicle 100 with respect to map data.

[0104] The automated driving module(s) 160 may be configured, either independently or in combination with the test bench system 170, to determine the driving path(s), current autonomous driving maneuvers for the vehicle 100, future autonomous driving maneuvers, changes to the current autonomous driving maneuvers, etc. Such determinations by the automated driving module(s) 160 may be based on data collected by the sensor system 120, driving scene models, data from another suitable source, such as determinations from sensor data 250, or any combination thereof. In general, the automated driving module(s) 160 may operate to implement various levels of automation, including advanced driver assistance (ADAS) features, semi-autonomous features, and fully autonomous features.A "driving maneuver" is one or more actions that affect the movement of a vehicle. Examples of driving maneuvers include accelerating, decelerating, braking, turning, moving the vehicle 100 laterally, changing lanes, merging into a lane, and reversing, to name a few. The automated driving module(s) 160 may be configured to perform driving maneuvers. The automated driving module(s) 160 may directly or indirectly cause such autonomous driving maneuvers to be performed. As used herein, "cause" or "induce" means to cause, command, direct, enable, or any combination thereof, an event or action, or at least to be in a state where such an event or action may occur, either directly or indirectly.The automated driving module(s) 160 may be configured to perform various vehicle functions individually or in combination to transmit data to, receive data from, interact with, or control the vehicle 100 or one or more of its systems (e.g., one or more of the vehicle systems 140).

[0105] Detailed embodiments are disclosed herein. However, it is to be understood that the disclosed embodiments are intended only as examples. Therefore, specific structural and functional details disclosed herein are not to be considered limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art how to employ the aspects described herein in virtually any reasonable detailed structure. Furthermore, the terms and phrases used herein are not to be considered limiting, but rather are intended to provide a clear description of possible embodiments. In the Fig. Various embodiments are shown, but are not limited to the structure or application shown.

[0106] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code comprising one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions specified in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown in sequence may in reality execute substantially concurrently, or the blocks may sometimes execute in reverse order, depending on the functionality involved.

[0107] The systems, components, or methods described above may be implemented in hardware or a combination of hardware and software, and may be implemented centrally in a processing system or decentralized, i.e., with various elements distributed across multiple interconnected processing systems. Any type of processing system or other device suitable for carrying out the methods described herein is suitable. A typical combination of hardware and software may be a processing system having computer-usable program code that, when loaded and executed, controls the processing system to carry out the methods described herein. The systems, components, or processes may also be embedded in computer-readable memory, such asa computer program product or other data program storage device that can be read by a machine and that contains a program of instructions that can be executed by the machine to perform the methods and processes described herein. These elements may also be embedded in an application product that includes all the features that enable the methods described herein to be performed and that, when loaded into a processing system, can perform those methods.

[0108] Furthermore, the arrangements described herein may take the form of a computer program product embodied in one or more computer-readable media on which a computer-readable program c is embodied, e.g., stored. Any combination of one or more computer-readable media may be used. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The term "computer-readable storage medium" means a non-transitory storage medium. A computer-readable storage medium may be, for example, but not exclusively, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.More specific examples (a non-exhaustive list) of the computer-readable storage medium include: a portable computer diskette, a hard disk drive (HDD), a solid-state drive (SSD), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), an optical storage medium, a magnetic storage medium, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, device, or apparatus.

[0109] Modules, as used herein, generally include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific data types. In further aspects, memory generally stores the noted modules. The memory associated with a module may be a buffer or cache embedded in a processor, RAM, ROM, flash memory, or other suitable electronic storage medium. In still further aspects, a module as used herein is implemented as an application-specific integrated circuit (ASIC), a hardware component of a system on a chip (SoC), a programmable logic array (PLA), or other suitable hardware component embedding a defined set of configurations (e.g., instructions) for performing the specified functions.

[0110] Program code embodied on a computer-readable medium may be transmitted via any suitable medium, including, but not limited to, wireless, wired, fiber optic, cable, RF-based transmission, etc., or any suitable combination of the foregoing. The computer program code for performing operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java™, Smalltalk, C++, or similar, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.The program code may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer over any network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., over the Internet using an Internet service provider).

[0111] The terms "a" and "an," as used herein, are defined as one or more than one. The term "plural," as used herein, is defined as two or more than two. The term "another," as used herein, is defined as at least a second or more. The terms "including" and "having" as used herein are defined as "comprehensive" (i.e., open-ended language). The phrase "at least one of ... and ...," as used herein, refers to and includes all possible combinations of one or more of the listed items. For example, the phrase "at least one of A, B, and C" includes only A, only B, only C, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0112] The aspects contained herein may be embodied in other forms without departing from the spirit or essential characteristics. Accordingly, reference should be made to the following claims, rather than to the foregoing description, to indicate the scope of the present description. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 555,707

[0002]

Claims

[1] A system comprising a processor and a memory device communicatively coupled to the processor and storing machine-readable instructions that, when executed by the processor, cause the processor to Providing a vehicle testing facility that is equipped to operate physical and virtual vehicle components, Receiving a test plan for a set of vehicle components, Determining test system components and test system connections that contain any virtual components to implement the test plan on the vehicle test system, Creating the virtual components to execute the test plan and Implementing test system connections to enable testing of vehicle components. [2] The system of claim 1, wherein implementing the test bench connections includes duplicating any test bench signals. [3] The system of claim 1, wherein implementing the test bench connections includes generating simulated test bench signals. [4] The system of claim 1, wherein the instructions further cause the vehicle test system to be configured to support testing of a new component upon its connection to the vehicle test system. [5] The system of claim 4, wherein configuring the vehicle test facility to support testing of the new component includes implementing an HMI rule for translating HMI signals related to the new component. [6] The system of claim 1, wherein the instructions further cause to determine a first set of vehicle components to perform closed-loop testing and a second set of vehicle components to perform open-loop testing. [7] The system of claim 1, wherein the instructions further cause to determine a first set of vehicle components to perform closed-loop testing and a second set of vehicle components to perform simulated closed-loop testing. [8] Non-transitory computer-readable medium containing instructions that, when executed by one or more processors, cause the one or more processors to Providing a vehicle testing facility that is equipped to operate physical and virtual vehicle components, Receiving a test plan for a set of vehicle components, determining test system components and test system connections containing any virtual components to implement the test plan on the vehicle test system, Creating the virtual components to execute the test plan and implementing the test system components to enable testing of the vehicle components. [9] The non-transitory computer-readable medium of claim 8, wherein implementing the test bench connections includes duplicating any test bench signals. [10] The non-transitory computer-readable medium of claim 8, wherein implementing the test bench connections includes generating simulated test bench signals. [11] The non-transitory computer-readable medium of claim 8, wherein the instructions further cause the vehicle test system to be configured to support testing of a new component upon its connection to the vehicle test system. [12] The non-transitory computer-readable medium of claim 11, wherein configuring the vehicle test system to support testing of the new component includes implementing an HMI rule for translating HMI signals related to the new component. [13] The non-transitory computer-readable medium of claim 8, wherein the instructions further cause to determine a first set of vehicle components to perform closed-loop testing and a second set of vehicle components to perform open-loop testing. [14] Procedure with Providing a vehicle testing facility that is equipped to operate physical and virtual vehicle components, Receiving a test plan for a set of vehicle components, determining test system components and test system connections containing any virtual components to implement the test plan on the vehicle test system, Creating the virtual components to execute the test plan and implementing the test system connections to enable testing of the vehicle components. [15] The method of claim 14, wherein implementing the test bench connections includes duplicating any test bench signals. [16] The method of claim 14, wherein implementing the test bench connections includes generating simulated test bench signals. [17] The method of claim 14, further comprising configuring the vehicle test system to support testing of a new component upon its connection to the vehicle test system. [18] The method of claim 17, wherein configuring the vehicle test facility to support testing of the new component includes implementing an HMI rule for translating HMI signals related to the new component. [19] The method of claim 14, further comprising determining a first set of vehicle components for performing closed-loop testing and a second set of vehicle components for performing open-loop testing. [20] The method of claim 14, further comprising determining a first set of vehicle components for performing closed-loop testing and a second set of vehicle components for performing simulated closed-loop testing.

Images