Test bench of hardware-in-the-loop of plug-and-play associated with autonomous vehicle development

Abstract

To provide a system and a method associated with using a multi-modal base model, which are described in the present specification.SOLUTION: According to an embodiment, a method comprises: providing a test bench of a vehicle in which components of a physical vehicle and components of a virtual vehicle can be operated; receiving a test plan associated with a set of components of the vehicle; determining test bench components including any virtual components and test bench connection so as to implement the test plan in the test bench of the vehicle; generating virtual components to perform a test plan; and implementing the test bench connection to enable testing components of the vehicle.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The subject matter described herein generally relates to strategies for testing autonomous vehicles, which may include the need for multiple component testing.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 555,707, filed February 20, 2024, which is incorporated herein by reference in its entirety and made a part hereof. Any and all priority claims identified in the Application Data Sheet, or any amendments thereto, are hereby incorporated by reference under 37 CFR 1.57. [Background technology]

[0003] With regard to the development of autonomous vehicles, it is advantageous to be able to provide testing of autonomous functions on vehicle hardware before releasing those functions to the public. One technique used to validate vehicle hardware is to simulate the behavior of a virtual vehicle under autonomous control in a virtual environment. However, virtual environments are limited in terms of the information they can contain for testing, and important edge cases may be missed when testing of autonomous functions relies solely on the virtual representation. Additionally, the use of a virtual vehicle or environment may present an idealized representation of vehicle behavior, such that the real-world behavior of the vehicle under autonomous control is not fully tested. Summary of the Invention

[0004] In one embodiment, a system is disclosed that includes one or more processors and a memory communicatively coupled to the one or more processors. The memory stores a command module including instructions that, when executed by the one or more processors, cause the one or more processors to provide a vehicle test bench capable of operating physical and virtual vehicle components, receive a test plan for a set of vehicle components, determine test bench components and test bench connections, including any virtual components, to implement the test plan at the vehicle test bench, generate the virtual components to execute the test plan, and implement the test bench connections to enable testing of the vehicle components.

[0005] In one embodiment, a non-transitory computer-readable medium is disclosed that includes instructions that, when executed by one or more processors, cause the one or more processors to perform one or more functions, including instructions for providing a vehicle test bench capable of operating physical and virtual vehicle components, instructions for receiving a test plan for a set of vehicle components, instructions for determining test bench components and test bench connections, including any virtual components, to implement the test plan on the vehicle test bench, instructions for generating the virtual components to execute the test plan, and instructions for implementing the test bench connections to enable testing of the vehicle components.

[0006] In one embodiment, a method is disclosed that includes providing a vehicle test bench capable of operating physical and virtual vehicle components, receiving a test plan for a set of vehicle components, determining test bench components and test bench connections, including any virtual components, to implement the test plan at the vehicle test bench, generating the virtual components to execute the test plan, and implementing the test bench connections to enable testing of the vehicle components. [Brief explanation of the drawings]

[0007] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate various systems, methods, and other embodiments of the present disclosure. It will be understood that the boundaries of elements shown in the figures (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundaries. In some embodiments, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Additionally, elements may not be drawn to scale.

[0008] [Figure 1] FIG. 1 illustrates one embodiment of a vehicle in which the systems and methods disclosed herein may be implemented. [Figure 2] FIG. 2 illustrates one embodiment of an autonomous driving system associated with a multimodal foundation model. [Figure 3] FIG. 3 illustrates one embodiment of a cloud computing environment in which the systems and methods described herein may operate. [Figure 4] FIG. 4 shows an embodiment of a test bench for autonomous driving. [Figure 5] FIG. 5 illustrates one embodiment of a vehicle interface component. [Figure 6] FIG. 6 shows an embodiment of a vehicle environment simulator. [Figure 7A] FIG. 7A illustrates one embodiment of an autonomous driving software stack. [Figure 7B] FIG. 7B shows an example of a state transition diagram for three different operating states that may be implemented by the safety controller. [Figure 8] FIG. 8 illustrates one embodiment of the HMI hardware components. [Figure 9] FIG. 9 shows one embodiment of the left hand side component. [Figure 10] FIG. 10 shows one embodiment of the right hand component. [Figure 11] Figure 11 shows an example of a virtual representation of a vehicle in a test environment. [Figure 12] FIG. 12 illustrates an example method for testing an autonomous vehicle that may include both test drivers and safety-trained drivers. [Figure 13] FIG. 13 shows an example of how vehicle control is handled for an implementation having both a test driver and a safety-trained driver. [Figure 14] FIG. 14 illustrates an example of how multiple instances of production, prototype, and virtual components can be handled during testing. DETAILED DESCRIPTION OF THE INVENTION

[0009] Systems, methods, and other embodiments associated with a hardware-in-the-loop test bench with dual-cockpit driver support for testing autonomous functions are described herein. Regarding previous approaches, various methods, including hardware-in-the-loop (HIL), software-in-the-loop (SIL), model-in-the-loop (MIL), simulator-in-the-loop (SimIL), process-in-the-loop (PIL), and human-in-the-loop (HuIL), have been attempted to solve verification and validation problems related to autonomous functions. However, these various approaches, by themselves, have been insufficient to effectively present real-world scenarios for testing a driver under test (DUT), a safety-trained driver (TSD), and an autonomous system (Guardian) operating simultaneously within a shared vehicle environment. For example, in such an environment, various control inputs in a drive-by-wire vehicle environment may be blended (e.g., combined via weighting) to enable smooth and predictable transitions between the DUT, TSD, or Guardian. Additionally, these various approaches do not address the need for a testbed that is highly adaptable and capable of testing different vehicle components, in order to test various combinations of systems from different suppliers.

[0010] Test benches utilizing a dual cockpit design offer an advantage over previous test bench approaches in that they enable the test bench to monitor the DUT and correct any mistakes made by the DUT, the administrator, or both while the vehicle is being driven. In this way, as the autonomous function matures during development, the TSD can (i) make corrections that the administrator is insufficiently trained to make and (ii) also correct any mistakes in the administrator's training, both of which can then be iteratively incorporated into further training of the autonomous function.

[0011] Furthermore, it is important to be able to provide autonomous function testing at all stages of development for a vehicle. Accordingly, the test bench presented herein may test autonomous functions in real or virtual environments on virtual components (e.g., a proposed model of a braking system), prototype components (e.g., a prototype braking system), and production components (e.g., a production braking system), or combinations thereof.

[0012] Referring to FIG. 1 , an example of a vehicle 100 is shown. As used herein, a “vehicle” refers to any form of motorized transportation. In one or more implementations, the vehicle 100 is an automobile. While features related to automobiles are described herein, it will be understood that the embodiments are not limited to automobiles. In some implementations, the vehicle 100 may be in the form of any robotic device or motorized transportation, which may include, for example, sensors that perceive aspects of the surrounding environment and thus benefit from the functionality described herein associated with strategies for testing autonomous vehicles, which may include both a test driver and a safety-trained driver. As a further note, this disclosure generally refers to the vehicle 100 as a vehicle capable of traveling on a roadway with surrounding vehicles, which are intended to be construed as the vehicle 100 itself. That is, surrounding vehicles may include any vehicles that the vehicle 100 may encounter on the roadway.

[0013] Vehicle 100 also includes various elements. It will be understood that in various embodiments, vehicle 100 may not necessarily have all of the elements shown in FIG. 1 . Vehicle 100 may have any combination of the various elements shown in FIG. 1 . Furthermore, vehicle 100 may have additional elements to those shown in FIG. 1 . In some arrangements, vehicle 100 may be implemented without one or more of the elements shown in FIG. 1 . While various elements are shown as being located within vehicle 100 in FIG. 1 , it will be understood that one or more of the elements may be located external to vehicle 100. Furthermore, the elements shown may be physically separated by large distances. For example, as discussed, one or more components of the system of the present disclosure may be implemented within the vehicle, while additional components of the system are implemented in a cloud computing environment or other system remote from vehicle 100.

[0014] Some of the possible elements of vehicle 100 are shown in FIG. 1 and described in conjunction with subsequent figures. However, descriptions of many of the elements in FIG. 1 are provided following the description of FIGS. 2-12 for brevity of this description. Furthermore, it will be understood that, for ease of illustration and clarity, reference numerals are appropriately repeated among different figures to indicate corresponding or similar elements. Additionally, the description outlines numerous specific details to provide a thorough understanding of the embodiments described herein. However, those skilled in the art will appreciate that the embodiments described herein may be implemented using various combinations of such elements. In any case, vehicle 100 includes test bench system 170, which is implemented to perform the methods and other functions as disclosed herein. As described in more detail below, in various embodiments, test bench system 170 is implemented in part within vehicle 100 as a cloud-based service. For example, in one approach, functionality associated with at least one module of test bench system 170 is implemented within vehicle 100, while additional functionality is implemented within a cloud-based computing system.

[0015] Referring to FIG. 2 , one embodiment of the test bench system 170 of FIG. 1 is further illustrated. The test bench system 170 is illustrated as including the processor 110 of the vehicle 100 of FIG. 1 . Thus, the processor 110 may be part of the test bench system 170, the test bench system 170 may include a processor separate from the processor 110 of the vehicle 100, or the test bench system 170 may access the processor 110 through a data bus or another communication path. In one embodiment, the test bench system 170 includes a memory 210 that stores the detection module 220 and the command module 230. The memory 210 may be a random access memory (RAM), a read-only memory (ROM), a hard disk drive, a flash memory, or other suitable memory that stores the detection module 220 and the command module 230. The detection module 220 and the command module 230 are, for example, computer-readable instructions that, when executed by the processor 110, cause the processor 110 to perform the various functions disclosed herein.

[0016] 2 is generally an abstract form of test bench system 170 as it may be implemented between vehicle 100 and a cloud computing environment. Thus, test bench system 170 may be embodied at least in part within a cloud computing environment to perform the methods described herein.

[0017] 2 , detection module 220 generally includes instructions that function to control processor 110 to receive data input from one or more sensors of vehicle 100. In one embodiment, the input is observation of one or more objects in an environment proximate vehicle 100, other aspects of the surroundings, or both. In one embodiment, as provided herein, detection module 220 acquires sensor data 250 that includes at least camera imagery. In a further mechanism, detection module 220 acquires sensor data 250 from additional sensors, such as radar 123, LiDAR 124, and other sensors that may be suitable for identifying vehicles, vehicle locations, lane markers, crosswalks, traffic signs, vehicle parking areas, road surface types, curbs, vehicle barriers, etc.

[0018] Thus, in one embodiment, detection module 220 controls each sensor to provide sensor data 250. Further, although detection module 220 is described as controlling various sensors to provide sensor data 250, in one or more embodiments, detection module 220 may employ other techniques to acquire sensor data 250, either active or passive. For example, detection module 220 may passively discover sensor data 250 from electronic information streams provided by various sensors to additional components within vehicle 100. Further, detection module 220 may perform various techniques to fuse data from multiple sensors when providing sensor data 250, from sensor data acquired over a wireless communication link (e.g., v2v) from one or more of the surrounding vehicles, or a combination thereof. Thus, in one embodiment, sensor data 250 represents a combination of perceptions acquired from multiple sensors.

[0019] In addition to the location of surrounding vehicles, sensor data 250 may also include, for example, odometry information, GPS data, or other location data. Further, in one embodiment, detection module 220 controls sensors to acquire sensor data for an area encompassing 360 degrees around vehicle 100, which may then be stored in sensor data 250. In some embodiments, the area sensor data may be used to provide a comprehensive assessment of the environment surrounding vehicle 100. Of course, in alternative embodiments, detection module 220 may acquire sensor data for only the forward direction, for example, when vehicle 100 does not include additional sensors to include additional areas around the vehicle, or when additional areas are not scanned for other reasons (e.g., because they are unnecessary due to known current conditions).

[0020] Additionally, in one embodiment, test bench system 170 includes database 240. In one embodiment, database 240 is an electronic data structure stored in memory 210 or another data store that is comprised of routines that can be executed by processor 110 to analyze the stored data, present the stored data, organize the stored data, etc. Thus, in one embodiment, database 240 stores data used by detection module 220 and command module 230 in performing various functions. In one embodiment, database 240 includes sensor data 250 along with metadata that characterizes various aspects of sensor data 250, for example. For example, metadata may include location coordinates (e.g., longitude and latitude), relative map coordinates or tile identifiers, timestamps / datestamps from when the individual sensor data 250 was generated, etc.

[0021] In one embodiment, command module 230 generally includes instructions that function to control processor 110 or a collection of processors within cloud computing environment 300 as shown in FIG.

[0022] 3, vehicle 100 may be connected to network 305, which enables communication between vehicle 100 and cloud servers (e.g., cloud server 310), infrastructure devices (e.g., infrastructure device 340), other vehicles (e.g., vehicle 380), and any other systems connected to network 305. With respect to network 305, the network may use any form of communication or networking to exchange data, including, but not limited to, the Internet, Directed Short Range Communications (DSRC) services, LTE, 5G, millimeter wave (mmWave) communications, etc.

[0023] Cloud server 310 is shown as including a processor 315 that may be part of test bench system 170 via network 305 via communications unit 335. In one embodiment, cloud server 310 includes memory 320 that stores communications module 325. Memory 320 may be random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable memory that stores communications module 325. Communications module 325 may be, for example, computer-readable instructions that, when executed by processor 315, cause processor 315 to perform various functions disclosed herein. Additionally, in one embodiment, cloud server 310 includes database 330. In one embodiment, database 330 is an electronic data structure stored in memory 320 or another data store that is comprised of routines that may be executed by processor 315 to analyze stored data, present stored data, organize stored data, etc.

[0024] Infrastructure device 340 is shown as including a processor 345 that may be part of test bench system 170 via network 305 via communication unit 370. In one embodiment, infrastructure device 340 includes memory 350 that stores communication module 355. Memory 350 may be random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable memory that stores communication module 355. Communication module 355 may be, for example, computer-readable instructions that, when executed by processor 345, cause processor 345 to perform the various functions disclosed herein. Additionally, in one embodiment, infrastructure device 340 includes database 360. In one embodiment, database 360 ​​is an electronic data structure stored in memory 350 or another data store that is comprised of routines that may be executed by processor 345 to analyze the stored data, present the stored data, organize the stored data, etc.

[0025] Thus, in addition to information obtained from sensor data 250, test bench system 170 may obtain information from a cloud server (e.g., cloud server 310), infrastructure devices (e.g., infrastructure device 340), other vehicles (e.g., vehicle 380), and any other systems connected to network 305. For example, a cloud server (e.g., cloud server 310) may be used to perform the same tasks described herein with respect to command module 230.

[0026] FIG. 4 illustrates an example of an autonomous driving test bench 400 that may be implemented via vehicle 100 using test bench system 170. Autonomous driving test bench 400 may include vehicle interface components 410, a vehicle environment simulator 420, an autonomous driving software stack 430, HMI hardware components 440, left-hand side (LHS) components 450, and right-hand side (RHS) components 460. These components may be directly or indirectly connected to each other to facilitate electronic communication between the various components of autonomous driving test bench 400. For example, as shown in FIG. 4 , vehicle interface components 410 may be connected by a CAN bus to exchange messages with vehicle environment simulator 420, HMI hardware components 440, LHS components 450, and RHS components 460. In addition, vehicle interface components 410 may be connected by Ethernet to exchange messages with autonomous driving software stack 430. Ethernet may also be used to connect autonomous driving software stack 430 to HMI hardware components 440. Similarly, a CAN bus may also be used to connect both the LHS components 450 and the RHS components 460 to both the vehicle's environment simulator 420 and the HMI hardware components 440. In various embodiments, other forms of connection may be used instead of or in addition to that shown in Figure 4. For example, the CAN bus may be replaced by an Ethernet network, one or more gateways may be implemented to allow CAN bus messages to be transmitted over Ethernet, the wired connection may be replaced with a wireless or optical connection, other protocols besides Ethernet may be used, etc.

[0027] 5, an example of a vehicle interface component 500 is shown that may be used to provide the vehicle interface component 410 as described above. The vehicle interface component 500 may include a vehicle control interface 510 and a prototype control interface 520. The vehicle control interface 510 may be configured to operate a vehicle (e.g., vehicle 100) or a portion thereof (e.g., a production component). The prototype control interface 520 may be configured to operate one or more prototype hardware components that may be fully or only partially integrated within the vehicle (or portion thereof).

[0028] For example, vehicle control interface 510 may provide all vehicle functions for vehicle 100 except for the steering system, while prototype control interface 520 may provide vehicle functions for a prototype steering system that interacts with vehicle 100. In this manner, the prototype steering system provided by prototype control interface 520 may be configured to function as the steering system for vehicle 100.

[0029] In some embodiments, prototype control interface 520 may configure the prototype steering system as the steering system of vehicle 100 by instructing vehicle 100 to send to the prototype steering system all messages received by the default steering system of vehicle 100. Additionally, prototype control interface 520 may be configured to provide any functionality needed to enable operation of one or more prototype hardware components that the vehicle's control interface 510 may not be able to provide (e.g., operation of a cooling system for a prototype hardware component that the vehicle's control interface 510 was not designed to support).

[0030] With reference to FIG. 6 , an example of a vehicle environment simulator 600 is shown that may be used to provide the vehicle environment simulator 420 described above. The vehicle environment simulator 600 may include a vehicle communication agent 610 and a vehicle environment simulator 620. The vehicle communication agent 610 may operate to enable one or more vehicle-based communication protocols (e.g., vehicle CAN bus agent, Ethernet) to support the interconnection of hardware and software components within a test bench (e.g., autonomous driving test bench 400). For example, the vehicle communication agent 610 may provide vehicle-based communication functionality to support different vehicle types, such as hybrid vehicles, internal combustion engine vehicles, electric vehicles, or others, so that production components, prototype components, or simulation components may operate with each other. For example, the vehicle communication agent 610 may simultaneously mimic several CAN bus connections. The vehicle communication agent 610 may also translate messages (e.g., translate a CAN bus message from a component into an Ethernet message that can be received by another component, or vice versa) so that production, prototype, or simulation components can seamlessly communicate with each other even if they have incompatible communication protocols. The vehicle communication agent 610 may also replicate messages, such as where multiple instances of a production, prototype, or simulation component are being tested. For example, the vehicle communication agent 610 may replicate a brake control input from the LHS component 900 so that one or more brake vehicle components, one or more brake prototype components, one or more brake virtual components, or a combination thereof, are all operated by the same brake control input message. In some embodiments, when duplicating a message, the vehicle communication agent 610 may also function to coordinate the timing of the message replication, thereby allowing multiple instances of a production, prototype, or simulation component to operate in a synchronized manner (or alternatively, in an offset arrangement, if desired).

[0031] The vehicle environment simulator 620 may function to provide simulation and evaluation capabilities based on information received from any of the production components, prototype components, or simulation components. For example, as shown in FIG. 11 , information collected by the vehicle environment simulator 620 may be used to model a virtual representation of a vehicle in a test environment. The vehicle environment simulator 620 may also be capable of evaluating the performance of a virtual vehicle in a virtual environment, such as the effects of various vehicle controls in extreme driving scenarios (e.g., high-speed driving, loss of traffic lights, unstable communication, unexpected events). The vehicle environment simulator 620 may also compare its evaluations with data obtained from outside the test bench (e.g., from external sensors at a test track) and may enable adjustment of its evaluations based on the comparison.

[0032] With reference to FIG. 7 , an example of an autonomous driving software stack 700 that can be used to provide the autonomous driving software stack 430 as described above is shown. The autonomous driving software stack 700 can provide any semi-autonomous or autonomous functionality as described herein, including simulating scenarios and performing perception, planning, and control functions. The autonomous driving software stack 700 can also include a safety controller 710. The safety controller 710 can serve to implement different operating states, where each operating state can specify how a vehicle (e.g., vehicle 100) is controlled between the LHS component 900, the RHS component 1000, the autonomous driving software stack 700, other control input sources, or a combination thereof. For example, FIG. 7B shows a state transition diagram for three different operating states that can be implemented by the safety controller 710. In this example, the safety controller 710 provides three different operating states.

[0033] In operating state S1, which may be the default state, safety controller 710 may configure the test bench so that the vehicle operates solely under the control of LHS components 900 (e.g., as if the vehicle were a conventional vehicle).

[0034] In operating state S2, which may be a test state, the safety controller 710 may configure the test bench to operate the vehicle in sub-states S2a, S2b1, or S2b2.

[0035] In sub-state S2a, the vehicle may operate solely under the control of the RHS component 1000. In some embodiments, sub-state S2a may allow joint control by the LHS component 900 and the RHS component 1000, which may further include the requirement that the RHS component 1000 take precedence over the LHS component 900 with respect to vehicle control in the event of a difference between the two.

[0036] In substate S2b1, the vehicle may operate solely under the control of autonomous driving software stack 700. In some embodiments, substate S2b1 may allow LHS component 900, RHS component 1000, or both to provide control inputs to autonomous driving software stack 700, which may then be evaluated by autonomous driving software stack 700 in its determination of control inputs for the vehicle.

[0037] In sub-state S2b2, the vehicle may operate as described above for sub-state S2b1, except that if a supervisor interrupt mode (described below) is activated, the RHS component 1000 may then fully or partially override the autonomous driving software stack 700, the LHS component 900, or both. For example, a first supervisor interrupt mode activation may result in an operating state requiring that the RHS component 1000 may override the autonomous driving software stack 700, the LHS component 900, or both, until such time as the supervisor interrupt becomes inactive. As another example, a second supervisor interrupt mode activation may result in an operating state requiring that steering and throttle control adjustments made by the RHS component 1000 are implemented, but still allow the autonomous driving software stack 700, the LHS component 900, or both to affect braking control if no opposing input is received from the RHS component 1000.

[0038] In operational state SO, which may be an emergency state, the vehicle may be operated to safely shut down, for example, by slowing to a stop, pulling over to the side of the road, releasing the throttle, etc. State SO may be a state required by safety controller 710 to be reachable from any other state when predetermined conditions are met (e.g., a required system fails). In such an embodiment, operational states that prevent the vehicle from reaching the emergency state when the predetermined conditions are met may be rejected by safety controller 710 as invalid.

[0039] The safety controller 710 may determine the extent to which one or more components (e.g., the autonomous driving software stack 700, the LHS component 900, and the RHS component 1000) are allowed to operate the vehicle based on the operational state. Thus, the safety controller 710 may allow individual or joint control between the LHS component 900, the RHS component 1000, or the autonomous driving software stack 700 as determined by each operational state. If the operational state specifies joint control, the operational state may determine which control input takes priority (e.g., the autonomous driving software stack 700 always takes priority over the LHS component 900 and the RHS component 1000). In some embodiments, priority may be determined by weighting of the control inputs. For example, the operational state may specify that a control signal from the RHS component 1000 has twice the influence as a control signal from the LHS component 900. As another example, the operational state may utilize a weighting function applied to one or more control inputs of the same type to form a combined control of that type.

[0040] In some embodiments, the safety controller 710 may establish concurrency rules regarding the visibility of the control input devices. For example, a simple concurrency rule is that the steering wheel of the RHS component 1000 and the steering wheel of the LHS component 900 should be positioned to be identical in terms of steering control input (e.g., if the RHS component 1000 is generating a control input, it causes the LHS component 900 to adjust to reflect that control input, or vice versa) or steering feedback (e.g., feedback signals from the steering system in the vehicle 100). In this way, the steering wheels may appear to function identically depending on driving inputs provided by either driver, both drivers, the autonomous driving software stack 700, etc. In some embodiments, the positioning of the control input devices may be specified by the combined control inputs (e.g., created by a weighting function).

[0041] As another example, in an operating state in which the RHS component 1000 and the LHS component 900 may jointly operate the vehicle and the LHS component 900 may have priority over the RHS component 1000, concurrency rules may require that (i) the control input device of the RHS component 1000 mimics the control input device of the LHS component 900 when no driving input is made to the control input device of the RHS component 1000, and (ii) the control input device of the LHS component 900 always mimics the control input device of the RHS component 1000 when driving input is made to the control input device of the RHS component 1000. In this way, the control input device of the RHS component 1000 may reflect the actions of a first driver who is primarily operating the vehicle through the control input device of the LHS component 900, except when a second driver uses the control input device of the RHS component 1000 to take control from the first driver.

[0042] In some embodiments, the safety controller 710 may also establish transition rules to handle changes between operational states involving different concurrency rules. For example, if the safety controller 710 transitions from an operational state that allows control of the LHS component 900 to an operational state that only allows joint control by the LHS component 900 or the RHS component 1000, the transition rule may specify a ramp function that the safety controller 710 applies to the implementation of the concurrency rules for that operational state, which may provide a smoother transition appearance to the driver when a change in operational state occurs. In some embodiments, such transition rules may not only be aesthetically pleasing in their results, but may also provide safety benefits by limiting sudden changes that could surprise or injure the driver due to a sudden implementation of the concurrency rules. In some embodiments, the transition rules may only affect how the implementation of the concurrency rules affects the physical position of the control input device of the LHS component 900 or the RHS component 1000. For example, even if it is desirable to provide a smoother transition appearance as described above, an input to an operating state may nevertheless require immediate implementation of a concurrency rule other than one related to the physical location of the control device, such as when the input to a new operating state causes autonomous driving software stack 700 to perform an extreme maneuver that is important for safety purposes.

[0043] 8, an example of an HMI hardware component 800 is shown. The HMI hardware component 800 may include a vehicle notification system 810, an administrator interrupt component 820 (e.g., a pedal that can be actuated by the safety driver to take full control of the vehicle), or other HMI components not provided by the LHS component 900 or the RHS component 1000 as described below.

[0044] Vehicle notification system 810 may provide support for any audio, visual, tactile, or other type of alert or indicator that may be used to communicate with a vehicle operator. For example, vehicle notification system 810 may utilize visual indicators (e.g., colored LEDs), audio indicators (e.g., different tones, voice announcements), vibrations, etc. to indicate the health of system components or the overall system, the operating state of the vehicle (e.g., manual, autonomous active), activation of supervisor interrupt mode, which driver is in control of the vehicle (e.g., DUT, TSD, supervisor), emergency vehicle operation, etc.

[0045] In some embodiments, vehicle notification system 810 may utilize HMI rules to facilitate the conversion of HMI messages. For example, different displays, steering wheels, or other HMI components may be utilized within a test bench, each having a different HMI interface. Accordingly, vehicle notification system 810 may store and utilize HMI rules that enable HMIs to be implemented between the test bench and one or more HMI devices. In some embodiments, when a new component is added to a test bench, vehicle notification system 810 may detect a change in the available HMI interfaces, define one or more HMI rules based on that detection, and then implement the one or more HMI rules. Similarly, when a component is removed from a test bench, vehicle notification system 810 may detect the change in the HMI interfaces and discontinue use of any HMI rules that are no longer needed.

[0046] The supervisor interrupt component 820 may enable a set of input control devices (e.g., those used by the TSD via the RHS component 1000) to have partial or complete control of the vehicle, overriding any other driving input. For example, the supervisor interrupt component 820 may utilize a pedal, button, voice command, or other form of HMI interface to allow the driver to activate or deactivate a supervisor interrupt mode.

[0047] When supervisor interrupt mode is active, the safety controller 710 may implement an operating state in which one or more control signals take precedence over any other control signals. For example, if the TSD operating the RHS component 1000 activates supervisor mode, the steering, brakes, throttle, and any other control inputs of the RHS component 1000 may take precedence over any control inputs from the LHS component 900, the autonomous driving software stack 700, or other sources. In some embodiments, this priority may suppress any control signals other than those authorized by the operating state triggered by supervisor interrupt mode activation (e.g., RHS only, no LHS, and no autonomous driving software stack 700). In some embodiments, this suppression may be limited to only some controls (e.g., RHS steering and throttle only, but LHS brakes or autonomous driving software stack 700 control inputs may still activate the brake system).

[0048] In some embodiments, supervisor interrupt component 820 may provide multiple supervisor interrupt modes with different priorities. For example, LHS component 900 may have an LHS supervisor interrupt pedal that allows LHS component 900 to inhibit control inputs to autonomous driving software stack 700, while RHS component 1000 may have an RHS supervisor interrupt pedal that allows RHS component 1000 to inhibit control inputs to either LHS component 900 or autonomous driving software stack 700.

[0049] In some embodiments, the supervisor interrupt interface may be operated by a test bench user other than the DUT or TSD. For example, a track observer may activate the supervisor interrupt mode via a mobile application interface in communication with the test bench, which provides control inputs for the autonomous driving software stack 700 that override control inputs for the LHS component 900 or the RHS component 1000. This configuration may be useful when only one driver (e.g., the DUT) is operating the test bench vehicle and imposing the autonomous driving software stack 700 is desired to ensure safe operation of the vehicle.

[0050] With reference to FIG. 9 , an example of an LHS component 900 is shown. The LHS component 900 may comprise any component used to generate a control input based on the action of a left-hand side driver. For example, the LHS component 900 may utilize an LHS brake pedal 910 along with an LHS brake actuator 920 and an LHS brake ECU 930 to receive a brake input from the LHS driver and generate an LHS brake control input. The LHS component 900 may also utilize an LHS throttle pedal 940 along with an LHS throttle actuator 950 and an LHS throttle ECU 960 to receive a throttle input from the LHS driver and generate an LHS throttle control input. Similarly, the LHS component 900 may also utilize an LHS steering wheel 970 along with an LHS steering actuator 980 and an LHS steering ECU 990 to receive a steering input from the LHS driver and generate an LHS steering control input. In some embodiments, the LHS component 900 may include other features of an LHS device that generate a control input based on the action of the LHS driver. For example, LHS brake pedal 910, LHS brake actuator 920, and LHS brake ECU 930 may be omitted if the throttle component provides both throttle and braking functions (e.g., single pedal operation). As another example, an LHS device used to generate control inputs based on the LHS driver's vocal commands may be added to LHS component 900.

[0051] 10, an example of an RHS component 1000 is shown. The RHS component 1000 may be identical in configuration to the LHS component 900, except that the devices of the RHS component 1000 may be used to generate RHS control inputs based on actions of the RHS driver.

[0052] In some embodiments, the devices of the RHS component 1000 may differ from those of the LHS component 900. For example, the LHS component 900 may utilize a prototype steering wheel, while the RHS component 1000 may utilize a standard production steering wheel. As another example, the LHS component 900 may utilize devices used to generate LHS control inputs based on the LHS driver's voice commands, while the RHS component 1000 may utilize standard production control input devices (e.g., brake, throttle, steering wheel). Additionally, even when the RHS component 1000 uses the same devices as the LHS component 900, it should be understood that the layout of such components between the RHS component 1000 and the LHS component 900 may be the same, reversed, mirrored, or otherwise arranged as desired.

[0053] While examples provided herein may make reference to an LHS and a RHS, it should be understood that these references are merely exemplary. For example, the location of the LHS components and the LHS operator may be anywhere, such as within the vehicle (e.g., left-hand side of the vehicle, center of the vehicle, right-hand side of the vehicle, rear of the vehicle) or outside the vehicle (e.g., desk, dual cockpit). Similarly, the location of the RHS components and the RHS operator may be anywhere. Thus, any examples provided herein with respect to a left-hand drive vehicle should be understood to be applicable to other vehicle configurations (e.g., center drive, right-hand drive, remote drive).

[0054] In some embodiments, autonomous driving test bench 400 may support testing of any mechanism with one or more production components, one or more prototype components, one or more virtual components, or a combination thereof. For example, it may be desired to perform tests comparing prototype brake components to production brake components in use as well as virtual brake components (which may represent some ideal brake components, for example). To perform such tests, any control signals, feedback signals, sensor signals, or other data communicated within autonomous driving test bench 400 may be replicated. For example, brake control inputs from autonomous driving software stack 700, LHS component 900, RHS component 1000, or a combination thereof may be replicated so that each of the production brake components, prototype brake components, or virtual brake components receives the same set of brake control inputs.

[0055] In some cases, open-loop testing through replication of control inputs or other data may be sufficient to test all components. For example, production, prototype, or virtual door lock components may be tested to determine when failure (e.g., due to an inability to lock or unlock within specified parameters) may occur for each component at extreme temperatures, the testing utilizing replication of door lock control signals and temperature sensor data. In tests involving virtual components, the virtual components may include deterministic or probabilistic models of how the virtual components may degrade, fail, or otherwise behave as system test parameters change.

[0056] In some cases, closed-loop testing may be required to fully test a component. For example, the output signal of a brake component may affect overall vehicle behavior, resulting in changes to future brake control inputs. In such situations, the autonomous driving test bench 400 may perform the following techniques, among others, to test a set of production, prototype, or virtual components:

[0057] First, to the extent that the necessary resources for closed-loop testing are available within or to autonomous driving test bench 400, the test bench may attempt to create as many instances of closed-loop tests as possible. For example, even if autonomous driving test bench 400 has access to only one vehicle, that vehicle may have four sets of braking systems, allowing simultaneous closed-loop testing of up to four production, prototype, or virtual braking components. In such a situation, autonomous driving test bench 400 may be provided with a selection of possible closed-loop test options from which the user can choose (e.g., front brakes only, rear brakes only) to limit the closed-loop testing to the particular configuration of interest.

[0058] Second, to the extent permitted by the test bench configuration, the autonomous driving test bench 400 may attempt to schedule closed-loop tests for a series of subsets of a set of production, prototype, or virtual components according to time-based intervals. For example, a closed-loop test may proceed by testing two brake components simultaneously until completion. In some embodiments, a test sequence may be constructed using time-division multiplexing, for example, where a closed-loop test may be resolved before adjusting system test parameters to the new condition (e.g., testing all brake components at -20°F before reducing to -25°F).

[0059] Third, in some embodiments, autonomous driving test bench 400 may be instructed to perform closed-loop testing of a first subset of a set of production, prototype, or virtual components, and open-loop testing of a second subset of the set of production, prototype, or virtual components. This approach may be useful for instances where a component's open-loop test failure may preclude the need for further closed-loop testing.

[0060] Fourth, in some embodiments, autonomous driving test bench 400 may generate simulated feedback signals or other data for a set of production, prototype, or virtual components that cannot be closed-loop tested, such that a subset may nevertheless be subjected to simulated closed-loop testing. Simulated feedback signals may be implemented, for example, by simulating in a virtual environment how the outputs of the subset may affect vehicle operation (e.g., via vehicle environment simulator 620) to generate simulated feedback signals, simulated sensor data, or other simulated data. This approach may be useful for instances where failure of a simulated closed-loop test of a component may preclude the need for further closed-loop testing of the component. In some embodiments, simulated closed-loop testing only requires simulating physical resources that may not be fully available for closed-loop testing. It should also be understood that, in some embodiments, closed-loop testing may use virtual components (e.g., software prototypes), but unlike simulated closed-loop testing, there is no substitute for simulated components for production components, prototype components, virtual components, etc.

[0061] With respect to the test bench testing described above, it should be understood that production, prototype, or virtual components do not necessarily need to be located in the same vehicle, or even in the same city. For example, one practical aspect of autonomous driving test bench 400 is that it may enable testing of components at remote locations. For example, if a mechanic at a dealership wants to test whether a vehicle component is still reliable, the mechanic accesses autonomous driving test bench 400, connects the vehicle component of interest to the test bench, and then runs tests via the test bench to assess the component's reliability. This approach may be useful when the reliability of a component cannot be fully determined without real-world testing in a vehicle or vehicle subsystem. Similarly, a third-party supplier developing prototype components may access a test bench to test one or more prototype components. In some embodiments, a vehicle (e.g., vehicle 100) (or portion of a vehicle) present at a location may be used as part of autonomous driving test bench 400 to remotely test components.

[0062] Figure 12 shows a flowchart of a method 1200 associated with a strategy for testing an autonomous vehicle that may include both a test driver and a safety-trained driver. Method 1200 is described in terms of test bench system 170 of Figures 1 and 2. While method 1200 is described in conjunction with test bench system 170, it should be understood that method 1200 is not limited to implementation within test bench system 170, but is an example of a system in which method 1200 may be implemented.

[0063] In step 1210, command module 230 may initialize test bench system 170 to provide a test bench (e.g., autonomous driving test bench 400). In providing the test bench, command module 230 may configure one or more production components, one or more prototype components, one or more virtual components, or a combination thereof, to communicate with test bench system 170 and to be operable as vehicle components for tests that may be performed on test bench system 170. In some embodiments, command module 230 may provide plug-and-play functionality therefor, where a component added to test bench system 170 provides sufficient information to enable command module 230 to configure the component for interoperability with test bench system 170.

[0064] In step 1220, the command module 230 may receive instructions to perform testing on a subset of one or more production components, one or more prototype components, one or more virtual components, or any combination thereof. Upon receiving the instructions, the command module 230 may determine available test bench resources to perform the testing. For example, if closed-loop testing is required, the command module 230 may determine available resources to support the closed-loop testing. If insufficient resources are available for closed-loop testing, the command module 230 may provide alternative approaches, such as sequential closed-loop testing (e.g., performing closed-loop testing on one component after another until completion), closed-loop / open-loop testing (e.g., performing closed-loop testing on some components and open-loop testing on the remaining components), or closed-loop / simulated closed-loop testing (e.g., virtually adding resources via simulation to perform simulated closed-loop testing on components that cannot participate in closed-loop testing).

[0065] In step 1230, command module 230 may generate (and present to the testbench user) an adjustable test implementation plan that indicates what types of tests can be performed by testbench system 170. Command module 230 may then receive instructions that include modifications, selections, or other adjustments to the adjustable test implementation plan (e.g., selecting closed-loop / simulated closed-loop testing and then specifying which components should be tested using closed-loop vs. simulated closed-loop testing).

[0066] In step 1240, command module 230 may select one or more production components, one or more prototype components, one or more virtual components, or a combination thereof, to be tested according to the adjustable test implementation plan and configure the components for testing. To configure the components for testing, command module 230 may instruct test bench system 170 regarding mechanisms such as control signals, feedback signals, simulated signals, and sensor data that need to be communicated throughout test bench system 170. For example, command module 230 may instruct LHS component 900 to use a prototype throttle component for an impaired driver instead of a standard production throttle component. In some embodiments, command module 230 may generate simulation components, a simulation environment, or both, as needed, to facilitate simulated closed-loop testing. Additionally, instructions regarding mechanisms such as control signals, feedback signals, simulated signals, and sensor data may include decisions regarding routing, replicating, generating, or simulating information, or other adjustments necessary to enable the components to be tested.

[0067] At step 1250, the command module 230 may determine operational states to use in the adjustable test implementation plan. In some embodiments, the operational states may be generated automatically based on the components tested or used by the adjustable test implementation plan. In some examples, the operational states may also be generated based on the selection of a predefined template. The operational states may also be further refined by test bench user modifications to the operational states generated by the command module 230. In some embodiments, the command module 230 may generate operational states based on the number of drivers present or the availability of features in the autonomous driving software stack 700. For example, if no driver is detected at a location where the RHS component 1000 is to be operated, the operational states may be modified by removing states that would be used only if the RHS component 1000 were utilized. Similarly, if the autonomous driving software stack 700 lacks a feature to operate in a particular aspect of the test (e.g., because the TSD provides vehicle monitoring until that feature is fully developed), the operational states may be modified by removing states that would be used only if that feature were present. Determining the operating states by command module 230 may also include command module 230 determining concurrency rules, transition rules, weighting functions, etc. for each operating state. Command module 230 may also create or modify operating states to include criteria regarding changes in test or environmental data that may also result in a transition from one operating state to another. For example, certain operating states may be restricted based on geolocation data (e.g., used only on test tracks).

[0068] In step 1260, the command module 230 may perform tests that include following operational conditions. In following operational conditions, the command module 230 may source control signals, adjust the position of control devices, generate combined control signals, etc., based on one or more driver control inputs, the autonomous driving software stack 700, or other sources of vehicle control.

[0069] At step 1270, the command module 230 may receive an activation signal for an administrator interrupt mode. Upon receiving the administrator interrupt mode, the command module 230 may switch to a different operating state based on the activated administrator interrupt mode. While the administrator interrupt mode is active, the command module 230 may function to only obey certain control signals (e.g., those originating from the RHS component 1000) while ignoring others (e.g., from the autonomous driving software stack 700 or the LHS component 900). Furthermore, when the administrator interrupt mode is deactivated, the command module 230 may change to another operating state (e.g., the one before the administrator interrupt mode was activated).

[0070] 13 illustrates a flowchart of a method 1300 associated with processing vehicle control for an implementation having both a driver under test and a safety-trained driver. Method 1300 is described in terms of test bench system 170 of FIGS. 1 and 2. While method 1300 is described in conjunction with test bench system 170, it should be understood that method 1300 is not limited to implementation within test bench system 170, but is an example of a system in which method 1300 may be implemented.

[0071] In step 1310, command module 230 may provide a test bench capable of receiving a first set of inputs from a first set of devices operable by a first driver, a second set of inputs from a second set of devices operable by a second driver, and a third set of inputs from autonomous driving components. For example, vehicle 100 may be configured with controls for both the left-hand side and the right-hand side of the forward vehicle compartment. The left-hand side may be configured for a DUT. The right-hand side may be configured for a TSD including a supervisor pedal that activates a supervisor interrupt mode. Test bench system 170 of vehicle 100 may be configured to provide an autonomous driving test bench 400 in connection with operation of vehicle 100 by a DUT, a TSD, or an autonomous driving module.

[0072] In step 1320, command module 230 may generate operating states, each operating state determining how the first, second, and third sets of inputs can control the vehicle. For example, one operating state may be that the left-hand side of vehicle 100 can drive vehicle 100 without interference from the TSD or autonomous driving module, and another operating state may be that the left-hand side of vehicle 100 can drive vehicle 100 provided that the TSD or autonomous driving module takes priority.

[0073] In step 1330, the command module 230 may determine a current operating state for the vehicle. For example, based on the test being performed, parameters of the environment, behavior of the DUT, TSD, or autonomous driving module, or other factors, the command module 230 may determine a particular operating state in which the vehicle 100 is operating.

[0074] In step 1340, the command module 230 may configure vehicle control with the first set of inputs, the second set of inputs, and the third set of inputs based on the current operating state. For example, if the vehicle changes from a certain operating state, the command module 230 may adjust the capabilities of the DUT, the TSD, and the autonomous driving module to control the vehicle. Additionally, the command module 230 may use concurrency or transition rules when implementing the configuration of vehicle control.

[0075] 14 illustrates a flowchart of a method 1400 associated with processing multiple instances of production components, prototype components, and virtual components during testing. Method 1400 is described in terms of test bench system 170 of FIGS. 1 and 2. While method 1400 is described in conjunction with test bench system 170, it should be understood that method 1400 is not limited to implementation within test bench system 170, but is an example of a system in which method 1400 may be implemented.

[0076] In step 1410, command module 230 may provide a vehicle test bench capable of operating physical and virtual vehicle components. For example, test bench system 170 may initialize autonomous driving test bench 400 and provide plug-and-play functionality as described herein. For example, when a prototype component is added to the test bench, command module 230 may electronically receive an identifier from the prototype component that enables command module 230 to determine how to configure the test bench to interoperate with the prototype component.

[0077] In step 1420, the command module 230 may receive a test plan for a set of vehicle components. For example, the test plan may describe test procedures, components to be tested, test environment, test parameters, etc.

[0078] In step 1430, command module 230 may determine the test bench components and test bench connections, including any virtual components, to implement the test plan on the vehicle test bench. For example, the test plan may provide sufficient information about the components to be tested and also about closed-loop components whose use will be restricted during the test (e.g., only one component may use the closed-loop component at a time). In some embodiments, the type of test or components may provide command module 230 with sufficient information to determine which closed-loop components may be restricted. For example, if the type of test is for autonomous driving longitudinal-based control functions with different degrees of aggressiveness, command module 230 may determine that brake and throttle components should be restricted. Once restrictions are identified, command module 230 may then determine whether other components are available elsewhere (e.g., at a different location) or whether a virtual simulation can be used instead. If no substitutions are available, command module 230 may analyze the test plan and suggest modifications that allow the modified test plan to proceed. In some embodiments, the command module 230 may have settings that instruct the command module 230 on how to proceed with test modifications without approval from a testbench user.

[0079] In step 1440, the command module 230 may generate virtual components to implement the test plan. For example, if the command module 230 determines that the test plan requires a virtual steering system to control a virtual vehicle in a virtual environment, the command module 230 may construct that element for use in the test plan.

[0080] In step 1450, the command module 230 may implement test bench connections to enable testing of vehicle components. For example, as the command module 230 progresses through the test plan, components used in closed-loop testing or other types of testing may require modification. Accordingly, the command module 230 may adjust the test bench by changing the routing of test bed signals through the test bench. For example, if a portion of the testing has been completed on a production throttle ECU and needs to proceed to a prototype throttle ECU, the command module 230 may redirect the throttle control input through the prototype throttle ECU instead of the production throttle ECU.

[0081] 1 will now be described in greater detail as an exemplary environment in which the systems and methods disclosed herein may operate. In some examples, vehicle 100 is configured to selectively switch between various modes, such as an autonomous mode, one or more semi-autonomous operating modes, a manual mode, etc. Such switching may be implemented in any suitable manner now known or later developed. "Manual mode" means that all or most of the navigation / steering of the vehicle is performed according to inputs received from a user (e.g., a human driver). In one or more arrangements, vehicle 100 may be a conventional vehicle configured to operate exclusively in manual mode.

[0082] In one or more embodiments, vehicle 100 is an autonomous vehicle. As used herein, "autonomous vehicle" refers to a vehicle operating in an autonomous mode. "Autonomous mode" refers to the use of one or more computing systems to control vehicle 100 with minimal or no input from a human driver, e.g., to provide navigation / steering of vehicle 100 along a travel route. In one or more embodiments, vehicle 100 is either highly automated or fully automated. In one embodiment, vehicle 100 is configured with one or more semi-autonomous operating modes in which one or more computing systems perform a portion of the navigation / steering of the vehicle along a travel route, and a vehicle operator (i.e., a driver) provides input to the vehicle to perform a portion of the navigation / steering of vehicle 100 along a travel route.

[0083] Vehicle 100 may include one or more processors 110. In one or more arrangements, processor 110 may be the main processor of vehicle 100. For example, processor 110 may be an electronic control unit (ECU). Vehicle 100 may include one or more data stores 115 that store one or more types of data. Data store 115 may include volatile memory, non-volatile memory, or both. Examples of suitable data stores 115 include RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, magnetic disk, optical disk, hard drive, or any other suitable storage medium, or any combination thereof. Data store 115 may be a component of processor 110, or data store 115 may be operatively connected to processor 110 for use by processor 110. As used throughout this specification, the term "operably connected" includes direct or indirect connections, and can include connections without direct physical contact.

[0084] In one or more arrangements, the data store 115 may include map data 116. The map data 116 may include maps of one or more geographic areas. In some examples, the map data 116 may include information or data about roads, traffic control devices, road markings, structures, features, landmarks, or any combination thereof, within one or more geographic areas. The map data 116 may be in any suitable form. In some examples, the map data 116 may include aerial photographs of an area. In some examples, the map data 116 may include ground photographs of an area, including 360-degree ground photographs. The map data 116 may include measurements, dimensions, distances, information, or any combination thereof, for one or more items included in the map data 116. The map data 116 may also include measurements, dimensions, distances, information, or any combination thereof, for other items included in the map data 116. The map data 116 may include digital maps with information about road geometry. The map data 116 may be high quality, high definition, or both.

[0085] In one or more arrangements, the map data 116 may include one or more terrain maps 117. The terrain maps 117 may include information about the ground, terrain, roads, terrain surfaces, other features, or any combination thereof, of one or more geographic areas. The terrain maps 117 may include elevation data within one or more geographic areas. The terrain maps 117 may be high quality, high definition, or both. The terrain maps 117 may define one or more ground surfaces, which may include paved roads, unpaved roads, land, and other features that define a ground surface.

[0086] In one or more arrangements, the map data 116 may include one or more stationary obstacle maps 118. The stationary obstacle map 118 may include information about one or more stationary obstacles located within one or more geographic areas. A "stationary obstacle" is a physical object whose position does not change or does not substantially change over a period of time and whose size does not change or does not substantially change over a period of time. Examples of stationary obstacles include trees, buildings, curbs, fences, guardrails, centerlines, utility poles, statues, monuments, signs, benches, furniture, mailboxes, large rocks, and hills. A stationary obstacle may be an object that extends above ground level. One or more stationary obstacles included in the stationary obstacle map 118 may have location data, size data, dimension data, material data, other data, or any combination thereof associated therewith. The stationary obstacle map 118 may include measurements, dimensions, distances, information, or any combination thereof regarding one or more stationary obstacles. The static obstacle map 118 may be high quality, high definition, or both. The static obstacle map 118 may be updated to reflect changes in the mapped area.

[0087] Data store 115 may include sensor data 119. In this context, "sensor data" means any information related to sensors equipped on vehicle 100, including capabilities and other information related to those sensors. As described below, vehicle 100 may include sensor system 120. Sensor data 119 may relate to one or more sensors of sensor system 120. As an example, in one or more arrangements, sensor data 119 may include information related to one or more LIDAR sensors 124 of sensor system 120.

[0088] In some examples, at least a portion of the map data 116 or sensor data 119 may be located in a data store 115 located on-board the vehicle 100. Alternatively, or in addition, at least a portion of the map data 116 or sensor data 119 may be located in a data store 115 located remotely from the vehicle 100.

[0089] As described above, vehicle 100 may include sensor system 120. Sensor system 120 may include one or more sensors. A "sensor" refers to any device, component, or system that can detect or sense something. One or more sensors may be configured to sense, detect, or function in real time. As used herein, the term "real time" refers to a level of processing responsiveness that allows a user or system to sense a particular process or decision to be made quickly enough or that allows a processor to keep up with some external process.

[0090] In arrangements where sensor system 120 includes multiple sensors, the sensors may function independently of one another. Alternatively, two or more of the sensors may function in combination with one another. In such embodiments, the two or more sensors may form a sensor network. Sensor system 120, one or more sensors, or both may be operatively connected to processor 110 of vehicle 100, data store 115, another element (including any of the elements shown in FIG. 1), or any combination thereof. Sensor system 120 may acquire data regarding at least a portion of the external environment of vehicle 100 (e.g., nearby vehicles).

[0091] The sensor system 120 may include any suitable type of sensor. Various examples of different types of sensors are described herein. However, it will be understood that embodiments are not limited to the particular sensors described. The sensor system 120 may include one or more vehicle sensors 121. The vehicle sensors 121 may detect, determine, sense, or acquire information related to the vehicle 100 itself. In one or more mechanisms, the vehicle sensors 121 may be configured to detect, determine, or acquire information related to changes in the position and orientation of the vehicle 100, for example, based on inertial acceleration. In one or more mechanisms, the vehicle sensors 121 may include one or more accelerometers, one or more gyroscopes, an inertial measurement unit (IMU), a dead reckoning system, a global navigation satellite system (GNSS), a global positioning system (GPS), a navigation system 147, other suitable sensors, or any combination thereof. The vehicle sensors 121 may be configured to detect, determine, or acquire information related to one or more characteristics of the vehicle 100. In one or more arrangements, the vehicle sensors 121 may include a speedometer for determining the current speed of the vehicle 100 .

[0092] Alternatively or in addition, sensor system 120 may include one or more environmental sensors 122 configured to obtain, sense, or a combination of obtain driving environment data. "Driving environment data" includes data or information regarding the external environment or one or more portions thereof in which the autonomous vehicle is located. For example, environmental sensors 122 may be configured to detect, quantify, sense, or obtain, in any combination, obstacles, information / data about such obstacles, or a combination thereof, in at least a portion of the external environment of vehicle 100. The obstacles may comprise stationary objects, dynamic objects, or a combination thereof. Environmental sensors 122 may be configured to detect, measure, quantify, sense, or obtain, in any combination, other features in the external environment of vehicle 100, such as lane markers, signs, traffic lights, traffic signals, lanes, crosswalks, curbs near vehicle 100, off-road objects, etc.

[0093] Described herein are various examples of sensors for sensor system 120. Exemplary sensors may be part of one or more environmental sensors 122, one or more vehicle sensors 121, or both. However, it will be understood that embodiments are not limited to the particular sensors described.

[0094] By way of example, in one or more arrangements, sensor system 120 may include one or more radar sensors 123, one or more LIDAR sensors 124, one or more sonar sensors 125, one or more cameras 126, or any combination thereof. In one or more arrangements, camera 126 may be a high dynamic range (HDR) camera or an infrared (IR) camera.

[0095] Vehicle 100 may include input system 130. An "input system" includes any device, component, system, element, or mechanism, or group thereof, that allows information / data to be input into a machine. Input system 130 may receive input from a vehicle occupant (e.g., a driver or passenger). Vehicle 100 may include output system 135. An "output system" includes any device, component, or mechanism, or group thereof, that allows information / data to be presented to a vehicle occupant (e.g., a person, a vehicle passenger, etc.).

[0096] Vehicle 100 may include one or more vehicle systems 140. Various examples of vehicle systems 140 are shown in FIG. 1 . However, vehicle 100 may include more, fewer, or different vehicle systems. While certain vehicle systems are defined separately, it should be understood that each or any of the systems or portions thereof may be otherwise combined or separated within vehicle 100 via hardware, software, or a combination thereof. Vehicle 100 may include a propulsion system 141, a braking system 142, a steering system 143, a throttle system 144, a transmission system 145, a signaling system 146, a navigation system 147, other systems, or any combination thereof. Each of these systems may include one or more devices, components, or combinations thereof, now known or later developed.

[0097] Navigation system 147 may include one or more devices, applications, and / or combinations thereof, now known or later developed, configured to determine the geographic location of vehicle 100, determine a travel route for vehicle 100, or both. Navigation system 147 may include one or more mapping applications for determining a travel route for vehicle 100. Navigation system 147 may include a global positioning system, a local positioning system, a geolocation system, or any combination thereof.

[0098] Processor 110, test bench system 170, autonomous driving module 160, or any combination thereof, may be operatively connected to communicate with various aspects of vehicle systems 140 or individual components thereof. For example, returning to FIG. 1 , processor 110, autonomous driving module 160, or any combination thereof may communicate to send or receive information from various aspects of vehicle systems 140 to control the movement, speed, steering, heading, direction, etc. of vehicle 100. Processor 110, test bench system 170, autonomous driving module 160, or any combination thereof may control some or all of these vehicle systems 140 and, therefore, may be partially or fully autonomous.

[0099] Processor 110, test bench system 170, autonomous driving module 160, or any combination thereof may be operable to control at least one of the navigation and steering of vehicle 100 by controlling vehicle systems 140 or one or more of its components. For example, when operating in an autonomous mode, processor 110, test bench system 170, autonomous driving module 160, or any combination thereof may control the direction, speed, or both of vehicle 100. Processor 110, test bench system 170, autonomous driving module 160, or any combination thereof may cause vehicle 100 to accelerate (e.g., by increasing the supply of fuel provided to the engine), slow down (e.g., by decreasing the supply of fuel to the engine or by applying the brakes), change direction (e.g., by turning the front two wheels), or any combination thereof. As used herein, "cause" or "causing" means, either directly or indirectly, to make, force, compel, direct, command, instruct, or enable an event or action to occur, or any combination thereof, or to make, force, compel, direct, instruct, instruct, or at least enable an event or action to be in a state in which such event or action can occur, or any combination thereof.

[0100] Vehicle 100 may include one or more actuators 150. Actuator 150 may be any element or combination of elements operable to modify, adjust, change, or any combination thereof, related to vehicle system 140 or one or more of its components in response to receiving a signal or other input from processor 110, autonomous driving module 160, or a combination thereof. Any suitable actuator may be used. For example, actuator 150 may include a motor, a pneumatic actuator, a hydraulic piston, a relay, a solenoid, and a piezoelectric actuator, just to name a few possibilities.

[0101] Vehicle 100 may include one or more modules, at least some of which are described herein. The modules may be implemented as computer-readable program code that, when executed by processor 110, implements one or more of the various processes described herein. One or more of the modules may be components of processor 110, or one or more of the modules may be executed on or distributed among other processing systems to which processor 110 is operatively connected. The modules may include instructions (e.g., program logic) executable by processor 110. Alternatively, or in addition, data store 115 may include such instructions.

[0102] In one or more arrangements, one or more of the modules described herein may include artificial intelligence or computational intelligence elements, such as neural networks, fuzzy logic, or other machine learning algorithms. Further, in one or more arrangements, one or more of the modules may be distributed among multiple modules described herein. In one or more arrangements, two or more of the modules described herein may be combined into a single module.

[0103] Vehicle 100 may include one or more autonomous driving modules 160. Autonomous driving module 160 may be configured to receive data from sensor system 120 or from any other type of system capable of capturing information about vehicle 100, the vehicle's external environment, or a combination thereof. In one or more mechanisms, autonomous driving module 160 may use the data to generate one or more driving scene models. Autonomous driving module 160 may determine the position and speed of vehicle 100. Autonomous driving module 160 may determine the location of obstacles, obstacles, or other environmental features, including traffic signs, trees, shrubs, nearby vehicles, pedestrians, etc.

[0104] The autonomous driving module 160 may be configured to receive, determine, or combine location information regarding obstacles in the external environment of the vehicle 100 that may be used by the processor 110, one or more of the modules described herein, or any combination thereof, to estimate the position or orientation of the vehicle 100, the vehicle's position or orientation in global coordinates based on signals from multiple satellites or other geolocation systems, or any other data / signals that may be used to determine the position or orientation of the vehicle 100 relative to the vehicle's 100 environment that is used either in creating a map or in determining the position of the vehicle 100 relative to the map data.

[0105] Autonomous driving module 160 may be configured, independently or in combination with test bench system 170, to determine a travel path, a current autonomous driving maneuver for vehicle 100, a future autonomous driving maneuver, modifications to the current autonomous driving maneuver, etc. Such decisions by autonomous driving module 160 may be based on data acquired by sensor system 120, a driving scene model, data from any other suitable source, such as determinations from sensor data 250, or any combination thereof. Generally, autonomous driving module 160 may function to implement various levels of automation, including advanced driver assistance (ADAS) functions, semi-autonomous functions, and fully autonomous functions. A "driving maneuver" refers to one or more actions that affect the movement of the vehicle. Examples of driving maneuvers include accelerating, decelerating, braking, turning, moving vehicle 100 laterally, changing lanes of travel, merging into lanes of travel, and reversing, just to name a few possibilities. Autonomous driving module 160 may be configured to implement driving maneuvers. Autonomous driving module 160 may implement such autonomous driving maneuvers directly or indirectly. As used herein, "causing" or "causing" means, either directly or indirectly, to cause, command, direct, enable, or cause an event or action to occur, to command, direct, or at least enable an event or action to occur, or any combination thereof, or to cause, command, direct, or at least enable an event or action to become in a state in which such event or action can occur, or any combination thereof. Autonomous driving module 160 may be configured to perform various vehicle functions and send data to, receive data from, interact with, or control vehicle 100 or one or more of its systems (e.g., one or more of vehicle systems 140), whether individually or in combination.

[0106] Detailed embodiments are disclosed herein. However, it should be understood that the disclosed embodiments are intended merely as examples. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ the aspects of the present specification in substantially any suitable detailed configuration. Furthermore, the terms and phrases used herein are not intended to be limiting, but rather to provide an understandable description of possible implementations. While various embodiments are shown in FIGS. 1-14, the embodiments are not limited to the structures or applications shown.

[0107] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of code, comprising one or more executable instructions that implement the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved.

[0108] The above-described systems, components, or processes may be implemented in hardware or a combination of hardware and software, either centralized within one processing system or distributed with various elements spread across several interconnected processing systems. Any type of processing system or other apparatus configured to perform the methods described herein is suitable. A typical combination of hardware and software may be a processing system having computer-usable program code that, when loaded and executed, controls the processing system such that the processing system performs the methods described herein. The systems, components, or processes may also be embodied in a computer-readable storage, such as a computer program product or other data program storage device, tangibly embodying a program of instructions executable by the machine to perform the methods and processes described herein. These elements may also be embodied in an application product, which comprises all features enabling implementation of the methods described herein and, when loaded in a processing system, is capable of performing the methods.

[0109] Furthermore, the mechanisms described herein may take the form of a computer program product having computer-readable program code embodied in, e.g., stored on, one or more computer-readable media. Any combination of one or more computer-readable media may be utilized. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The phrase "computer-readable storage medium" refers to a non-transitory storage medium. A computer-readable storage medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include the following: portable computer diskettes, hard disk drives (HDDs), solid-state drives (SSDs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), portable compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this specification, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0110] Generally, a module as used herein includes a routine, program, object, component, data structure, etc. that performs a particular task or implements a particular data type. In a further aspect, a memory generally stores the noted modules. The memory associated with a module may be a buffer or cache integrated within a processor, RAM, ROM, flash memory, or another suitable electronic storage medium. In still further aspects, a module contemplated by the present disclosure is implemented as an application-specific integrated circuit (ASIC), as a hardware component of a system-on-chip (SoC), as a programmable logic array (PLA), or as another suitable hardware component incorporating a defined configuration set (e.g., instructions) to perform the functions of the present disclosure.

[0111] Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wired, fiber optic, cable, RF, or the like, or any suitable combination thereof. Computer program code for performing operations for aspects of the present mechanism may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java™, Smalltalk, C++, or the like, and traditional procedural programming languages ​​such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection to the external computer may be made (e.g., through the Internet using an Internet Service Provider).

[0112] The terms "a" and "an," as used herein, are defined as one or more than one. The term "plurality," as used herein, is defined as two or more than two. The term "another," as used herein, is defined as at least a second or more. The terms "including" and "having," as used herein, are defined as comprising (i.e., open language). The phrase "and at least one of," as used herein, refers to and includes any and all possible combinations of one or more of the associated listed items. As an example, the phrase "at least one of A, B, and C" includes A only, B only, C only, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0113] The aspects herein may be embodied in other forms without departing from the spirit or essential attributes thereof, and reference should accordingly be made to the following claims, rather than the foregoing specification, as indicating the scope of the present invention.

Claims

1. 1. A system comprising: a processor; a memory communicatively coupled to the processor and storing machine-readable instructions; the machine-readable instructions, when executed by the processor, cause the processor to: providing a vehicle test bench capable of operating physical and virtual vehicle components; receiving a test plan for a set of vehicle components; determining test bench components, including any virtual components, and test bench connections for implementing the test plan on a test bench for the vehicle; generating the virtual components to execute the test plan; and implementing the test bench connections to enable testing of components of the vehicle.

2. The system of claim 1 , wherein implementing the test bench connections comprises replicating any test bench signals.

3. The system of claim 1 , wherein implementing the test bench connections comprises generating simulated test bench signals.

4. The instruction: The system of claim 1 , further comprising configuring the vehicle test bench to support testing of the new component upon connection of the new component to the vehicle test bench.

5. 5. The system of claim 4, wherein configuring the vehicle test bench to support testing of the new component includes implementing HMI rules to transform HMI signals associated with the new component.

6. The instruction: The system of claim 1 , further comprising determining a first set of vehicle components to perform closed-loop testing on and a second set of vehicle components to perform open-loop testing on.

7. The instruction: The system of claim 1 , further comprising determining a first set of vehicle components to perform closed-loop testing on and a second set of vehicle components to perform simulated closed-loop testing on.

8. A non-transitory computer-readable medium containing instructions that, when executed by one or more processors, cause the one or more processors to: providing a vehicle test bench capable of operating physical and virtual vehicle components; receiving a test plan for a set of vehicle components; determining test bench components, including any virtual components, and test bench connections for implementing the test plan on a test bench for the vehicle; generating the virtual components to execute the test plan; A non-transitory computer-readable medium for implementing the test bench connection to enable testing of components of the vehicle.

9. 10. The non-transitory computer-readable medium of claim 8, wherein implementing the test bench connections includes replicating any test bench signals.

10. The non-transitory computer-readable medium of claim 8 , wherein implementing the test bench connections comprises generating simulated test bench signals.

11. The instruction:

10. The non-transitory computer-readable medium of claim 8, further comprising configuring the vehicle test bench to support testing of the new component upon connection of the new component to the vehicle test bench.

12. 12. The non-transitory computer-readable medium of claim 11, wherein configuring the vehicle test bench to support testing of the new component includes implementing HMI rules to transform HMI signals associated with the new component.

13. The instruction:

10. The non-transitory computer-readable medium of claim 8, further comprising determining a first set of vehicle components to perform closed-loop testing on and a second set of vehicle components to perform open-loop testing on.

14. Providing a vehicle test bench capable of operating physical and virtual vehicle components; receiving a test plan for a set of vehicle components; determining test bench components, including any virtual components, and test bench connections for implementing the test plan on a test bench for the vehicle; generating the virtual components to execute the test plan; implementing the test bench connections to enable testing of components of the vehicle; A method comprising:

15. 15. The method of claim 14, wherein implementing the test bench connections includes replicating any test bench signals.

16. 15. The method of claim 14, wherein implementing the test bench connections comprises generating simulated test bench signals.

17. The method of claim 14 , further comprising configuring the vehicle test bench to support testing of the new component upon connection of the new component to the vehicle test bench.

18. 20. The method of claim 17, wherein configuring the vehicle test bench to support testing of the new component includes implementing HMI rules to transform HMI signals associated with the new component.

19. The method of claim 14 , further comprising determining a first set of vehicle components to perform closed-loop testing on and a second set of vehicle components to perform open-loop testing on.

20. The method of claim 14 , further comprising determining a first set of vehicle components to perform closed-loop testing on and a second set of vehicle components to perform simulated closed-loop testing on.

Images