Sensor aggregation in autonomous vehicles

Abstract

To provide techniques for sensor aggregation in an autonomous vehicle.SOLUTION: A system and method for operating an autonomous vehicle via a safety gateway, the safety gateway aggregating input data streams from a variety of sensors via different communication protocols and providing an abstraction mechanism for issuing torque, brake, steering, and any other vehicle specific commands.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] An autonomous vehicle (e.g., a self-driving truck) includes sensors, devices, and systems that may function together to generate sensor data indicative of the vehicle's position, speed, operating characteristics, indicators of the autonomous vehicle's specific surroundings, and various parameter values ​​related to the vehicle's condition. These sensors and other devices may include cameras, vehicle diagnostic devices, lidar, temperature and pressure sensors, etc. Many of these sensors or devices use different data formats and transmission protocols. The autonomous vehicle may have one or more central computers that collect, store, and process this data. [Background technology]

[0002] Unfortunately, many of these sensors or devices communicate data over different protocols, including controller area network ("CAN") buses, different Ethernet protocols (including different connector types such as RJ-45, fiber optic, and coaxial), serial communications (such as RS-232), and the like. This can result in a difficult assortment of cables, wires, connectors, and other adapters to manage. These connections are susceptible to vibration and failure in an autonomous driving environment. Furthermore, they are difficult to manage and maintain. This type of connection approach also makes it difficult to implement redundant computing systems, as wiring and connection issues are exacerbated with multiple computing systems.

[0003] Therefore, there is a need for a system and method for aggregating sensors and other devices in autonomous vehicles. [Brief explanation of the drawings]

[0004] The features and advantages of the illustrative embodiments, and the manner in which they are achieved, will become more readily apparent by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0005] [Figure 1] FIG. 1 is an example block diagram of a control system that may be deployed in an autonomous vehicle, according to some embodiments.

[0006] [Figure 2] FIG. 1 is an exemplary block diagram of a control system, according to some embodiments.

[0007] [Figure 3] 1 is an exemplary depiction of communication between a gateway and a computing system, according to some embodiments.

[0008] [Figure 4] 1 is an exemplary depiction of a configuration personalization flow between a gateway and a computing system, according to some embodiments.

[0009] [Figure 5A] FIG. 1 is an exemplary block diagram of a redundant control system, according to some embodiments. [Figure 5B] FIG. 1 is an exemplary block diagram of a redundant control system, according to some embodiments.

[0010] [Figure 6] FIG. 2 is an example block diagram of a network architecture of a gateway, according to some embodiments.

[0011] [Figure 7A] 1 is an exemplary view of an autonomous vehicle, according to some embodiments. [Figure 7B] 1 is an exemplary view of an autonomous vehicle, according to some embodiments. [Figure 7C] 1 is an exemplary view of an autonomous vehicle, according to some embodiments.

[0012] Throughout the drawings and detailed description, unless otherwise stated, like drawing reference numbers will be understood to refer to like elements, features, and structures. The relative size and depiction of these elements may be exaggerated or adjusted for clarity, illustration, and / or convenience. DETAILED DESCRIPTION OF THE INVENTION

[0013] In the following description, specific details are set forth to provide a thorough understanding of various exemplary embodiments. It should be understood that various modifications to the embodiments will be readily apparent to those skilled in the art, and that one or more principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Moreover, in the following description, numerous details are set forth for purposes of explanation. However, those skilled in the art should understand that the embodiments may be practiced without these specific details. In other instances, well-known structures, methods, procedures, components, and circuits are not shown or described so as not to obscure the description with unnecessary detail. Thus, the present disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0014] For convenience and clarity of presentation, a number of terms are used herein. For example, the term "semi-truck" is used to refer to a vehicle in which the system of the exemplary embodiments may be used. The terms "semi-truck," "truck," "tractor," "vehicle," and "semi" may be used interchangeably herein. However, it will be understood that the scope of the present invention is not limited to use within semi-trucks.

[0015] Prior to describing the sensor aggregation systems and methods of the present invention, reference is first made to Figure 1, which illustrates an embodiment of a control system 100 that may be deployed on an autonomous or semi-autonomous vehicle, such as, but not limited to, a semi-truck (such as semi-truck 700 shown in Figures 7A-7C). Control system 100 is described to introduce and illustrate the types of sensors and other devices that may be in communication with a computer system 140 on an autonomous or semi-autonomous vehicle (such vehicles will hereinafter be referred to generally as "autonomous" vehicles for convenience).

[0016] Control system 100 may include sensors 110 that collect data and information to be provided to computer system 140 to perform operations, including, for example, control operations to control vehicle components via gateway 180. According to some embodiments, gateway 180 is configured to enable computer system 140 to control vehicle components from different manufacturers. Computer system 140 may be configured with one or more central processing units (CPUs) 142 to perform processing, including processing to implement features of embodiments of the present invention as described elsewhere herein, and to receive sensor data from sensors 110 for use in generating control signals to control one or more actuators or other controllers associated with systems of the vehicle in which control system 100 is deployed (e.g., actuators or controllers that enable control of throttle 184, steering system 186, brakes 188, and / or other devices and systems). In general, control system 100 may be configured to operate a vehicle (e.g., semi-truck 700) in an autonomous (or semi-autonomous) operating mode.

[0017] For example, the control system 100 may be operated to capture images from one or more cameras 112 mounted at various locations on the semi-truck 700 and perform processing (e.g., image processing) on ​​these captured images to identify objects proximate to or within the path of the semi-truck 700. In some aspects, one or more lidar 114 and radar 116 sensors may be disposed on the vehicle to sense or detect the presence and volume of objects proximate to or within the path of the semi-truck 700. Other sensors may also be disposed or mounted at various locations on the semi-truck 700 to capture other information, such as position data. For example, the sensors may include one or more satellite positioning sensors and / or an inertial navigation system, such as the GNSS / IMU 118. The Global Navigation Satellite System (GNSS) is a space-based system of satellites that provides location information (longitude, latitude, altitude) and time information in all weather conditions on or near the Earth to devices called GNSS receivers. GPS is the most widely used GNSS system in the world and may be used interchangeably herein. An inertial measurement unit ("IMU") is an inertial navigation system. Generally, an inertial navigation system ("INS") measures and integrates the orientation, position, velocity, and acceleration of a moving object. The INS integrates the measurement data, and the GNSS is used as a correction for integration errors in the INS orientation calculation. Any number of different types of GNSS / IMU 118 sensors may be used in conjunction with features of the present invention.

[0018] Data collected by each of the sensors 110 may be processed by the computer system 140 to generate control signals that may be used to control the operation of the semi-truck 700. For example, image and location information may be processed to identify or detect objects around or in the path of the semi-truck 700, and control signals may be transmitted via the controller 182 to adjust the engine 184, steering 186, and / or brakes 188, as needed, to safely operate the semi-truck 700 in an autonomous or semi-autonomous manner. While example sensors, actuators, and other vehicle systems and devices are shown in FIG. 1 , one skilled in the art will understand, after reading this disclosure, that other sensors, actuators, and systems may also be included in the system 100 consistent with the present disclosure. For example, in some embodiments, actuators may also be provided that provide a mechanism allowing control of the transmission of a vehicle (e.g., the semi-truck 700).

[0019] Control system 100 may include a computer system 140 (e.g., a computer server) configured to provide a computing environment in which one or more software, firmware, and control applications (e.g., items 160-182) may execute to perform at least some of the processes described herein. In some embodiments, computer system 140 includes components deployed on a vehicle (e.g., deployed in a system rack 740 located within a sleeper compartment 712 of the semi-truck, as shown in FIG. 7C). Computer system 140 may communicate with other computer systems (not shown) that may be local to semi-truck 700 and / or remote from the semi-truck (e.g., computer system 140 may communicate with one or more remote ground-based or cloud-based computer systems via a wireless communication network connection).

[0020] According to various embodiments described herein, computer system 140 may be implemented as a server. In some embodiments, computer system 140 may be configured using any of a number of computing systems, environments, and / or configurations, such as, but not limited to, a personal computer system, a cloud platform, a server computer system, a thin client, a thick client, a handheld or laptop device, a tablet, a smartphone, a database, a multiprocessor system, a microprocessor-based system, a set-top box, a programmable consumer electronics device, a network PC, a minicomputer system, a mainframe computer system, a distributed cloud computing environment, and the like, which may include any of the above systems or devices.

[0021] Different software applications or components may be executed by the computer system 140 and the control system 100. For example, as shown in active learning component 160, an application performing active learning machine processing may be provided to process images captured by one or more cameras 112 and information acquired by LIDAR 114. For example, image data may be processed using a deep learning segmentation model 162 to identify objects of interest (e.g., other vehicles, construction signs, etc.) in the captured images. In some aspects herein, deep learning segmentation may be used to identify lane points in LIDAR scans. As an example, the system may use an intensity-based voxel filter to identify lane points in LIDAR scans. The LIDAR data may be processed by a machine learning application 164 that draws or identifies bounding boxes on the image data to identify objects of interest located by the LIDAR sensor.

[0022] Information output from the machine learning applications may be provided as input to object fusion 168 and vision map fusion 170 software components, which may perform processing to predict the behavior of other road users, fuse local vehicle pose with global map geometry in real time, and enable on-the-fly map correction. Output from the machine learning applications may be supplemented with information from radar 116 and map localization 166 application data (as well as positioning data). In some aspects, these applications enable control system 100 to be less reliant on maps and better able to handle constantly changing road environments. Furthermore, by correcting any map errors on the fly, control system 100 may facilitate safer, more scalable, and more efficient operation compared to alternative map-centric approaches.

[0023] The information is provided to a predictive planning application 172, which provides input to components of trajectory planning 174, allowing a trajectory to be generated in real time by a trajectory generation system 176 based on interactions and predicted interactions between the semi-truck 700 and other associated vehicles in the truck's operating environment. In some embodiments, for example, the control system 100 generates a 60-second planning period and analyzes the relevant actors and available trajectories. The plan that best meets multiple criteria (including safety, comfort, and route preferences) may be selected, and any associated control inputs needed to implement the plan are provided to a controller 182 to control the movement of the semi-truck 700.

[0024] In some embodiments, these disclosed applications or components (as well as other components or flows described herein) may be implemented in hardware, a computer program executed by a processor, firmware, or a combination of the above, unless otherwise specified. In some cases, the computer program may be embodied on a computer-readable medium, such as a storage medium or storage device. For example, the computer program, code, or instructions may reside in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, a compact disk read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art.

[0025] A non-transitory storage medium may be coupled to a processor such that the processor may read information from and write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In alternative embodiments, the processor and the storage medium may reside as separate components. For example, FIG. 1 illustrates an exemplary computer system 140 that may represent or be integrated with any of the components disclosed below. As such, FIG. 1 is not intended to suggest any limitation regarding the scope of use or functionality of embodiments of the systems and methods disclosed herein. The computer system 140 may implement and / or perform any of the functionality disclosed herein.

[0026] Computer system 140 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system 140 may also be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including non-transitory memory storage devices.

[0027] 1 , computer system 140 is shown in the form of a general-purpose computing device. Components of computer system 140 may include, but are not limited to, one or more processors (e.g., CPU 142 and GPU 144), a communications interface 146, one or more input / output interfaces 148, and one or more storage devices 150. Although not shown, computer system 140 may also include a system bus that couples various system components, including system memory, to CPU 142. In some embodiments, input / output (I / O) interface 148 may also include a network interface. For example, in some embodiments, some or all of the components of control system 100 may communicate via a controller area network (CAN) bus or the like that interconnects various components within the vehicle in which control system 100 is deployed and associated.

[0028] In some embodiments, storage device 150 may include various types and forms of non-transitory computer-readable media. Such media may be any available media accessible by a computer system / server and may include both volatile and non-volatile media, removable and non-removable media. In one embodiment, system memory implements processes represented by flow diagrams in other figures herein. System memory may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. As another example, storage device 150 may read from and write to non-removable, non-volatile magnetic media (not shown, typically referred to as a “hard drive” or “solid-state drive”). Although not shown, storage device 150 may also include one or more removable, non-volatile disk drives, such as magnetic, tape, or optical disk drives. In such cases, each may be connected to a bus by one or more data media interfaces. Storage device 150 may include at least one program product having program modules, codes, and / or sets of instructions (e.g., at least one) configured to perform the functions of various embodiments of an application.

[0029] As described above, wiring and connections enable the sensors and other devices of FIG. 1 to exchange data with computer system 140, including a variety of different wiring types and communication protocols. For example, some of the vehicle systems (e.g., throttle 184, steering 186, brake 188) may communicate using a CAN bus, while other devices, such as sensors 112-118, may communicate via other protocols or communication methods (e.g., RS232 / 485, different Ethernet connector types, etc.). As a result, managing the assortment of cables, wires, and connections can be complex and difficult. Furthermore, some connection types may not tolerate the vibrations or difficult conditions experienced by moving vehicles (especially semi-trucks). As a result, wiring and connections are difficult to connect and maintain, making it difficult to implement any computer system redundancy. To further complicate matters, different vehicles may have different configurations of sensors or vehicle systems.

[0030] Referring now to FIG. 2, a control system 200 that ameliorates these and other problems is shown. As shown, control system 200 includes a computer system 140 similar to computer system 140 of FIG. 1. In the embodiment shown in FIG. 2, a safety gateway 280 is provided that serves as a communications gateway between various sensors (including GPS, lidar, radar, IMU, cameras, etc.) and computer system 140, as well as between other devices (e.g., track systems such as throttle, brakes, steering, etc.). Safety gateway 280 may communicate with computer system 140 using a single connection, e.g., an Ethernet uplink connection. This single connection with computer system 140 provides many advantages, as described further herein. While a single connection is used in some embodiments, one or more additional connections between safety gateway 280 and computer system 140 may also be used (e.g., for redundancy, etc.).

[0031] In the embodiment shown in FIG. 2, safety gateway 280 is configured to receive data from one or more sensors 110 and to send and receive data from vehicle systems 184-188. For example, safety gateway 280 receives input data from sensors 110 (message path (1)), provides that data to computer system 140 (message path (2)), receives commands from computer system 140 (message path (3)), receives vehicle state updates from vehicle systems 184-188 (message path (4)), and transmits vehicle system commands (message path (5)). In some embodiments, additional message paths may be provided for issuing sensor commands to sensors 110 (not shown in FIG. 2). Gateway 280 is configured with multiple connector ports (not shown in FIG. 2) for connecting with various sensors 110 and vehicle systems 184-188. Different configurations of ports on gateway 280 are further described below in conjunction with the description of FIGS. 5 and 6.

[0032] According to some embodiments, safety gateway 280 is implemented using a network processor and one or more network connectors. According to some embodiments, safety gateway 280 is configured with a 2.5G Ethernet connection (for connection to computer system 140), a port for connecting to a vehicle CAN bus (for communication with vehicle systems 184-188), an RS232 connector, one or more GPS connectors (using universal asynchronous receiver / transmitter (“UART”) serial communication), and other Ethernet ports (e.g., for receiving lidar sensor data, camera data, and other sensor data).

[0033] Because different vehicles may be configured with different sensors and vehicle systems, the operation of safety gateway 280 is configured for a particular vehicle and a particular computer system 140 (and, e.g., a particular configuration of the software operated by computer system 140) with a configuration file that defines parameters such as the human-readable purpose of each communication channel (e.g., "left-center rider," "brake unit CAN bus," etc.), transport parameters (e.g., UDP port, priority, etc.), data format of each channel (e.g., "CAN frame," "IMU packet," etc.), and physical interface details of each channel used on the gateway side (e.g., Linux device, baud rate, etc.).

[0034] According to some embodiments, safety gateway 280 abstracts various inputs and outputs from a specific vehicle's configuration, allowing a standard configuration of safety gateway 280 to be used with different vehicles having different sensor and system configurations. The configuration file for each safety gateway 280 defines the IP addresses and ports of the master and slave computers, as well as any number of logical channels between safety gateway 280 and computer system 140. Each logical channel is unidirectional (e.g., each of channels (1) through (5) depicted in FIG. 2 is unidirectional). An illustrative example of logical channels that may be implemented using features of the present invention is shown in FIG. 3, where multiple channels are shown between safety gateway 280 and computer system 140. Channels may include input channels (inputs to computer system 140) for transferring data associated with sensors and vehicle systems (e.g., "left medium-range lidar," "left short-range lidar," "brake unit update," "engine update," etc.). Channels may also include output channels (output from computer system 140 for routing by safety gateway 280) for transferring commands or other data (e.g., "brake unit commands," "throttle commands," "steering commands," etc.) that are delivered by safety gateway 280 to associated vehicle systems or devices. Channels may include one or more channels associated with a configuration of safety gateway 280 for use with a particular vehicle.

[0035] Embodiments allow a standard hardware configuration of safety gateway 280 to be used in different vehicle environments. According to some embodiments, this is achieved in part through the use of one or more configuration files that are provisioned when safety gateway 280 is used in a vehicle. According to some embodiments, each configuration file is provisioned using a process such as provisioning process 400 shown in FIG. 4. This provisioning process 400 is executed each time the vehicle is driven and ensures that safety gateway 280 is properly configured even if changes are made to the configuration or software of computer system 140 or the configuration or software of safety gateway 280. For example, embodiments ensure that safety gateway 280 is properly configured for operation in a particular vehicle, even in situations where a different software build process and team runs on the software of safety gateway 280 than runs on the software of computer system 140. Because each channel is associated with a specific data type, thereby enabling the generation of automated test data, embodiments allow for formal verification of the configuration of safety gateway 280 in a simplified and more repeatable manner.

[0036] 4, provisioning process 400 involves operating safety gateway 280 to determine whether safety gateway 280 is unprovisioned (e.g., safety gateway 280 is newly installed) or whether a configuration file exists ("saved" provisioning, such as when the gateway was previously provisioned for the vehicle). Because it is important that both computer 140 and safety gateway 280 have identical configurations, provisioning process 400 is performed even in situations where safety gateway 280 has previously been provisioned (e.g., to avoid situations where other aspects of computer 140 or the vehicle have changed since the configuration file was last provisioned). For example, provisioning may not be required if the vehicle is simply stopped while in motion.

[0037] If the determination is that safety gateway 280 is not provisioned or the configuration has been saved, the process continues with safety gateway 280 operating to request configuration content from computer system 140. Computer system 140 is configured to respond to such requests for configuration content by returning a configuration snapshot that includes information such as the vehicle model and a list of configuration files (and, in some embodiments, version information identifying the current version of each file). In some embodiments, safety gateway 280 may iteratively request each configuration file in the list. In other embodiments, configuration files may be provided in batches. In the embodiment depicted in FIG. 4, safety gateway 280 responds to the list provided by computer system 140 by iteratively looping through the list of configuration files and requesting each configuration file. In some embodiments, not all configuration files may require synchronization; for example, in some embodiments, safety gateway 280 may already have the correct version of a configuration file, in which case the file need not be requested.

[0038] In some embodiments, the configuration files may be provided by a remote server. For example, computer system 140 may request one or more configuration files from a server via network connection 190. The configuration file contents are then stored on computer system 140 and provided to safety gateway 280, so that both computer system 140 and safety gateway 280 share the same configuration details.

[0039] In some embodiments, the configuration file contains a list of channels and their configurations and is stored in a human-readable text file. For example, in some embodiments, the file is implemented using protocol buffers (also referred to as “protobuf,” which is an open-source, cross-platform data format used to serialize structured data and developed by Google®). Other formats (e.g., ROS messages, flatbuffers, XML, etc.) may be used as well. In embodiments using protobuf, a proto definition file (e.g., “.proto”) may describe the data structures, messages, and services for implementing the configuration of each sensor associated with the vehicle. In some cases, other data formats may be used to formally define the data in the configuration files herein. In some embodiments, each channel can be viewed as an input for one node or device and as an output for another side of the communication (e.g., as shown in FIG. 3 ). In some embodiments, the configuration file enumerates input and output channels from the master perspective (the device receiving the input). 3, safety gateway 280 acts as a slave when sending input to computer system 140 and listens for changes in output channels. The configuration file may include information identifying the IP address of the master (e.g., computer system 140) and information identifying the IP address of the slave (e.g., safety gateway 280). In some embodiments, the configuration file may further specify the address of a remote server that acts as a configuration server that maintains current configuration files for different vehicles.

[0040] In some embodiments, the configuration file may include a notion of a data format for a particular channel. For example, a data channel format may include information identifying the type of channel (e.g., "input," "output," etc., where input is information sent from a slave to a master and output is information sent from a master to a slave). For example, an input may be a channel containing information received by safety gateway 280 from a sensor, such as a camera, which is transmitted to computer system 140 along with configuration information. As another example, an output may be a channel containing information for controlling the operation of a vehicle system that is transmitted from computer system 140 to safety gateway 280 for use in controlling the vehicle system.

[0041] The configuration file may include human-readable information identifying each input and each output. For example, an input containing information from a camera may include a human-readable label identifying a particular camera on the vehicle (e.g., "front right camera"). Each input and output may further include information identifying the data port over which the channel is transmitted, as well as the format of the channel information. Many different types of formats may be used. Different types of formats may require different types of processing (e.g., some types of formats may require checking for dropped packets, checking for long payload fragmentation, etc.). In illustrative, but non-limiting examples, formats may include: (i) RAW_DATA (a format in which data is transmitted in raw format, such as when camera data is transmitted); (ii) ANY_PAYLOAD (a format in which any payload is wrapped in a packet with a common header containing the transmission time, fragmentation information, and an auto-increment counter to check for lost packets); or (iii) FRAGMENTED_DATA (a format for transmitting large payloads in which use of the ANY_PAYLOAD format results in extra serialization / deserialization calls on each fragment).

[0042] In some embodiments, each input or output further includes information identifying a maximum transmission unit (or "MTU"), which specifies the maximum size of a packet that may be transmitted. In some embodiments, a payload format may also be specified (in addition to the channel format information). For example, a channel over which data is transmitted using the ANY_PAYLOAD format may include information specifying that the payload format is: (i) CAN_MESSAGE (a format for transmitting CAN message data); (ii) CAN_BATCH (a format for transmitting a batch of CAN messages); (iii) POWER_STATE (a format for transmitting power state data associated with different vehicle sensors or devices); (iv) IMU_PACKET (a format for transmitting IMU update data); or (v) HEALTH_REPORT (detailed information about the health status of the safety gateway). Other specializations may be easily added as needed. The use of a payload format in addition to a channel format allows a receiver to appropriately process the received data. This significantly reduces the number and types of cabling and connections required to operate computer system 140.

[0043] The channel information may further include a data sample, which may be, for example, a link to a file that can be used to simulate a realistic data stream for testing purposes. The message body may also include information identifying other details of the message (e.g., information identifying the time interval at which the body is updated, or information further identifying the format of the message body, etc.).

[0044] In this manner, embodiments allow computer system 140 and safety gateway 280 to dynamically define (based on a configuration file) how each should communicate with one another. For example, based on the contents of the configuration file, computer system 140 and safety gateway 280 know which channels are available, where to find updated data, the update interval at which updates should be made available, and the format and content of the data on each channel. Furthermore, because the configuration file defines the format in which messages are transmitted, embodiments allow computer system 140 to communicate with safety gateway 280 using a unified communication channel (e.g., via an Ethernet connection, etc.) even if different sensors or vehicle systems communicate with safety gateway 280 using other communication protocols (e.g., serial interface, CAN, etc.). Computer system 140 may transmit commands to modify the operation of vehicle systems (e.g., steering control, braking system, etc.) to safety gateway 280 using the Ethernet connection, and safety gateway 280 can convert the commands to a CAN bus protocol (or similar) for transmission to the vehicle systems. Similar transformations can occur with sensor data and vehicle data received by safety gateway 280 .

[0045] Embodiments enable the use of a redundant configuration, such as that shown in FIG. 5. System 500 of FIG. 5A provides two dual-redundant data paths that supply data to computer system 140. Each data path includes a safety gateway 280a or 280b and a corresponding switch 192a or 192b. Data received from each sensor 110 is replicated using a set of switches 194a-194d and connectors 196. For example, one switch 194a may transmit sensor data to each of two switches 192a, 192b. A second switch 194d may transmit sensor data to each of two safety gateways 280a, 280b. By providing redundant safety gateways 280, embodiments substantially reduce or eliminate types of failures that can prevent an autonomous vehicle (such as semi-truck 700 of FIG. 7) from having the information needed to operate the vehicle. The two safety gateways 280 are each configured with the same configuration file and generate the same information needed by the computer system 140 to monitor and control the operation of the vehicle.

[0046] Additionally, in some embodiments, sensors around the vehicle may be striped or staggered across the switches 194 so that no single switch 194 carries similar sensor data. For example, switch 194a in FIG. 5A may receive sensor data from a camera located on the front left side of the vehicle, while a different switch 194d may receive sensor data from a camera located on the front right side of the vehicle. A similar configuration may be alternated or varied across the switches 194 for other types of sensor data (e.g., lidar, etc.). Such a configuration further reduces or eliminates types of failures that could hinder or inhibit operation of the autonomous vehicle.

[0047] The configuration depicted in FIG. 5A shows several Ethernet connections 196. For non-Ethernet signals (such as CAN data), each safety gateway 280a, 280b may be configured to receive data from an associated input source (e.g., for CAN bus data, each safety gateway 280a, 280b may be coupled to a CAN bus, ensuring that the computer system 140 has two sources of CAN bus data). Such a configuration is shown in FIG. 5B. The configurations of FIGS. 5A and 5B are used in a single system, and are shown separately for convenience and ease of illustration. In FIG. 5B, one or more sensors 110 that do not communicate via Ethernet are shown providing data to two safety gateways 280a, 280b. Each safety gateway 280 receives the same information, thereby providing redundancy for vehicle operation. If one of the safety gateways 280 becomes unavailable, the other continues to provide sensor data to the computer system 140, allowing the computer system 140 to continue to operate, monitor, and control the vehicle. Examples of sensors that may not communicate over Ethernet include, for example, radar devices, CAN bus devices, IMU devices (which generally communicate using RS232 / 422), and GPS devices. Although item 110 is labeled "sensor," the term is used more generally and may include any vehicle system or device that generates data that the computer system 140 uses to operate, monitor, and control.

[0048] 5A and 5B, a single computer system 140 is shown. In some embodiments, redundant computer systems 140 may also be provided. Additionally, in some embodiments, each of the safety gateways 280 may be configured to operate as a computer system 140. For example, if a computer system 140 is rendered inoperable or unavailable, the safety gateway 280 may take over control functions and safely operate the vehicle with limited or minimal risk maneuvers (e.g., to safely stop the vehicle or allow sufficient time for a human operator to resume control).

[0049] The safety gateway 280 may be configured in many different ways. For example, referring to Figure 6, a safety gateway 280 is shown that uses a vehicle network processor 198 in conjunction with several switches 194 (e.g., Ethernet switches, etc.) to communicate over multiple network connections 196 to receive and / or transmit data to one or more vehicle sensors or systems. The safety gateway 280 communicates with the remote computer system 140 over an uplink connection, thereby substantially eliminating the need for multiple cables associated with the computer system 140. Other configurations may also be provided, and the configuration depicted in Figure 6 is for illustrative purposes only.

[0050] 7A-7C are exemplary depictions of exterior views of a semi-truck 700 that may be associated with or used in accordance with exemplary embodiments. The semi-truck 700 is shown for illustrative purposes only. As such, those skilled in the art will understand, after reading this disclosure, that the embodiments may be used in conjunction with many different types of vehicles and are not limited to the type of vehicle shown in FIGS. 7A-7C. The exemplary semi-truck 700 shown in FIGS. 7A-7C is one style of truck configuration common in North America, including an engine 706 located forward of the cab 702, a steering axle 714, and two drive axles 716. A trailer (not shown) may be attached to the semi-truck 700 via a fifth wheel trailer coupling typically mounted on a frame 718 and located on the drive axles 716. As shown in FIGS. 7A and 7C, a sleeper compartment 712 may be located behind the cab 702. FIGS. 7A-7C further illustrate multiple sensors located in different locations on the semi-truck 700. For example, one or more sensors may be mounted on the roof of the cab 702 on a sensor rack 720. Sensors may also be mounted on the side mirrors 710 and elsewhere on the semi-truck. Sensors may also be mounted on the bumper 704, as well as on the side of the cab 702 and elsewhere. For example, in FIG. 7A , rear-facing radar 736 is shown mounted on the side of the cab 702. Embodiments may be used with other configurations of trucks and other vehicles (e.g., semi-trucks having cab-over or cab-forward configurations, etc.). In general, without limiting the embodiments of the present disclosure, features of the present invention may be used with desired results in vehicles that haul cargo over long distances, such as long-distance semi-truck routes.

[0051] FIG. 7B is a front view of semi-truck 700 showing numerous sensors and sensor locations. A sensor rack may mount and position several sensors on windshield 708, including long-range lidar 722, long-range camera 724, GPS antenna 734, and mid-range front-facing camera 726. Side mirrors 710 may provide mounting locations for rear-facing camera 728 and mid-range lidar 730. Front radar 732 may be mounted on bumper 704. Other sensors (both shown and not shown) may be mounted or located elsewhere on semi-truck 700. As such, the locations and mounting shown in FIGS. 7A-7C are for illustrative purposes only.

[0052] 7C, there is shown a partial view of semi-truck 700 showing the interior of cab 702 and some aspects of sleeper compartment 712. In some embodiments, a portion of control system 200 of FIG. 2 is located in a system rack 740 within sleeper compartment 712, allowing easy access to components of control system 200 for maintenance and operation.

[0053] As will be understood based on the foregoing specification, the above-described examples of the present disclosure may be implemented using computer programming or engineering techniques, including computer software, firmware, hardware, or any combination or subset thereof. Such resulting programs having computer-readable code may be embodied or provided in one or more non-transitory computer-readable media, thereby creating a computer program product, i.e., an article of manufacture, in accordance with the discussed examples of the present disclosure. For example, the non-transitory computer-readable medium may be, but is not limited to, a fixed drive, a diskette, an optical disk, a magnetic tape, flash memory, an external drive, semiconductor memory such as read-only memory (ROM), random access memory (RAM), and / or any other non-transitory transmission and / or reception medium, such as the Internet, cloud storage, the Internet of Things (IoT), or other communications networks or links. An article of manufacture containing the computer code may be created and / or used by executing the code directly from one medium, by copying the code from one medium to another, or by transmitting the code over a network.

[0054] A computer program (also referred to as a program, software, software application, "app," or code) may include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, cloud storage, Internet of Things, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. However, "machine-readable medium" and "computer-readable medium" do not include transitory signals. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and / or any other type of data to a programmable processor.

[0055] The above descriptions and illustrations of processes herein should not be construed as implying a fixed order for performing process steps. Rather, process steps may be performed in any order practicable, including simultaneous performance of at least some steps. While the present disclosure has been described in connection with specific examples, it should be understood that various changes, substitutions, and alterations apparent to those skilled in the art could be made to the disclosed embodiments without departing from the spirit and scope of the present disclosure, as set forth in the appended claims.

Claims

1. A plurality of sensors; a plurality of vehicle control systems; a computer system configured to control and monitor operation of an autonomous vehicle, the computer system including a first memory that stores a first configuration file; a safety gateway in direct communication with the computer system, the plurality of sensors, and a safety gateway in direct communication with the plurality of vehicle control systems, the safety gateway including a second memory that stores a second configuration file; and a processor, the processor including: determining a status of the second configuration file; requesting configuration content from the computer system based on the status, the configuration content defining one or more data structures associated with the plurality of sensors; updating the second configuration file with the configuration content such that the second configuration file has the same configuration content as the first configuration file; receiving sensor data from the plurality of sensors; reformatting the sensor data for transmission from the safety gateway to the computer system via a network interface based on the updated second configuration file; routing commands from the computer system to the vehicle control systems via the network interface; the updated second configuration file includes a list of unidirectional channels provided between the safety gateway and the computer system through the network interface, and further includes configuration information for each channel; The system, wherein the unidirectional channel includes an input channel for receiving the sensor data from the plurality of sensors, and further includes an output channel for providing the commands to the plurality of vehicle control systems.

2. The system of claim 1 , wherein the configuration content includes a sensor name, a data port, and a payload format for each of the plurality of sensors.

3. 2. The system of claim 1, wherein the safety gateway is configured to process messages from each of the plurality of sensors using information from the configuration content to normalize the messages into a standardized format for transmission to the computer system via the network interface.

4. The system of claim 1 , wherein the network interface is an Ethernet protocol network interface, and the sensor data received by the safety gateway includes sensor data received from at least one sensor via a serial interface.

5. 5. The system of claim 4, wherein the sensor data received from the at least one sensor via the serial interface is converted by the safety gateway into a message format compatible with an Ethernet protocol based on information from the configuration content.

6. 2. The system of claim 1, wherein the status of the second configuration file is one of: (i) a status where the second configuration file is not provisioned; and (ii) a status where the second configuration file is saved.

7. The system of claim 1 , wherein the processor is further configured to determine that the autonomous vehicle is beginning a new route.

8. 2. The system of claim 1, further comprising a second safety gateway, wherein the safety gateway and the second safety gateway are arranged in a redundant configuration to provide multiple redundant data paths for providing the sensor data to the computer system.

9. The system of claim 8 , wherein the safety gateway and the second safety gateway receive the same sensor data.

10. 1. A method performed by a safety gateway of an autonomous vehicle, comprising: determining a status of a first configuration file stored in a first memory of the safety gateway; requesting a second configuration file stored in a second memory of a computer system in direct communication with the safety gateway based on the status; updating the first configuration file to be the same as the second configuration file, the updated first configuration file defining a plurality of data structures associated with a plurality of sensors that communicate directly with the safety gateway; receiving sensor data from the plurality of sensors; reformatting the sensor data for transmission from the safety gateway to the computer system via a network interface based on the updated first configuration file; routing commands from the computer system to a plurality of vehicle control systems via the network interface, the vehicle control systems communicating directly with the safety gateway; the updated first configuration file includes a list of unidirectional channels provided between the safety gateway and the computer system through the network interface, and further includes configuration information for each channel; The method, wherein the unidirectional channel includes an input channel for receiving the sensor data from the plurality of sensors, and further includes an output channel for providing the commands to the plurality of vehicle control systems.

11. The method of claim 10 , wherein the safety gateway is arranged in a redundant configuration with at least a second safety gateway to provide multiple redundant data paths for providing the sensor data to the computer system.

12. The method of claim 11 , wherein the safety gateway and the second safety gateway receive the same sensor data.

13. The method of claim 10 , wherein the updated first configuration file identifies a sensor name, a data port, and a payload format for each of the plurality of sensors.

14. 11. The method of claim 10, wherein the sensor data is transmitted from the safety gateway to the computer system via an Ethernet protocol network interface, and the sensor data received by the safety gateway includes sensor data received from at least one sensor via a serial interface.

15. 15. The method of claim 14, wherein the sensor data received from the at least one sensor via the serial interface is converted by the safety gateway into a message format compatible with an Ethernet protocol based on information from the updated first configuration file.

16. 11. The method of claim 10, wherein the status of the first configuration file is one of: (i) a status where the first configuration file is not provisioned; and (ii) a status where the first configuration file is saved.

17. receiving vehicle commands from the computer system in a first format defined by the updated first configuration file; converting the vehicle commands to a second format defined by the updated first configuration file; The method of claim 10 , further comprising transmitting the vehicle command to a vehicle system to cause the vehicle system to modify its operation.

18. 18. The method of claim 17, wherein the first format is compatible with an Ethernet transmission protocol and the second format is compatible with a CAN bus protocol.

Images