Automated truck mounted attenuator

Abstract

Systems and methods for automated control of a vehicle are described. In one example, a method of automated vehicle operation includes tracking path markers relative to a centerline of a first vehicle and a centerline of a second vehicle. The path markers define a path. The method further includes determining centerline distance ratios comparing respective distances between the centerline of the first vehicle and each of the path markers at a defined location on the path. The method further includes causing the centerline of the second vehicle to be positioned at a defined lateral distance from one of the path markers at the defined location on the path based on the centerline distance ratios.

Description

Attorney Docket: 222204-2985 AUTOMATED TRUCK MOUNTED ATTENUATOR CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Application Serial No. 63 / 622,927, filed January 19, 2024, titled “AUTOMATED TRUCK MOUNTED ATTENUATOR,” the entire contents of which are hereby incorporated herein by reference. This application also claims the benefit of and priority to U.S. Provisional Application Serial No. 63 / 553,442, filed February 14, 2024, titled “AUTOMATED TRUCK MOUNTED ATTENUATOR,” the entire contents of which are hereby incorporated herein by reference. BACKGROUND

[0002] Truck-Mounted Attenuators (TMAs) are energy-absorbing devices added to heavy shadow vehicles to provide a mobile barrier that protects work crews from errant vehicles entering active work zones. In mobile and short-duration operations, drivers manually operate the TMA equipped truck (or simply, “TMA"), keeping pace with the work zone as needed to function as a mobile barrier protecting work crews. While the TMA is designed to absorb or redirect the energy from a colliding vehicle, there is still significant risk of injury to the TMA driver when struck. TMA crashes are a serious problem in many places. In one location, crashes increased each year from 2011 to 2014 despite a decrease in the number of active construction sites between 2013 and 2014. In one state, the 4-year rolling average number of crashes increased by 10% from 2018 to 2019.

[0003] Although various efforts have been made to improve TMA vehicle crashworthiness (e.g., by adding interior padding, harnesses, and supplemental head restraints), the most effective way to protect TMA drivers may be to remove them from the vehicle altogether. Recent advances in automated vehicle technologies—including advanced sensing, high-precision differential global positioning system (GPS), inertial sensing, advanced control algorithms, and machine learning have enabled the development of automated systems capable of controlling TMA’s. Furthermore, the relatively low operating speeds and platoon-like operating movements of TMA’s make a leader-follower automated control concept feasible for a variety of mobile and short-duration use cases without the cost or complexity of full autonomy.Attorney Docket: 222204-2985 SUMMARY

[0004] The present disclosure is directed to systems and methods for automated control of a Truck-Mounted Attenuator (TMA). An automated control system and method for TMAs are described in embodiments herein. The automated TMA (ATMA) system and method of the present disclosure use a leader-follower control concept in various examples. The embodiments implement machine-vision and vision-based navigation techniques to facilitate autonomous object detection and collision prevention, as well as automated lateral offsets, longitudinal offsets, and speed control of a TMA. Some embodiments implement at least one of a global positioning system (GPS) or real-time kinematic (RTK) approach as a primary source of automated lateral control of a TMA with a vision-based approach being implemented as a secondary, redundant, or backup source of automated lateral control of a TMA. The embodiments thereby allow for a driver to be removed from potential harm associated with driving an at-risk TMA.

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description or can be learned from the description or through practice of the embodiments. Other aspects and advantages of embodiments of the present disclosure will become better understood with reference to the appended claims and the accompanying drawings, all of which are incorporated in and constitute a part of this specification. The drawings illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related concepts of the present disclosure.

[0006] According to one example embodiment, a method of automated vehicle operation includes determining a first pixel distance of pixels in first image data that correspond to a lateral distance between a path marker partly defining a path and a centerline of a first vehicle at a defined location on the path. The first image data is captured by a first camera coupled to the first vehicle. The method further includes calculating a second pixel distance of pixels in second image data that correspond to the lateral distance. The second image data is captured by a second camera coupled to a second vehicle on the path. The second pixel distance of pixels is calculated based on the first pixel distance of pixels. The method further includes causing a centerline of the second vehicle to be positioned at the lateral distance from the path marker at the defined location on the path based on the second pixel distance of pixels in the second image data.Attorney Docket: 222204-2985 BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Many aspects of the present disclosure can be better understood with reference to the following figures. The components in the figures are not necessarily to scale, with emphasis instead being placed upon clearly illustrating the concepts of the disclosure. Moreover, repeated use of reference characters or numerals in the figures is intended to represent the same or analogous features, elements, or operations across different figures. Repeated description of such repeated reference characters or numerals is omitted for brevity.

[0008] FIG. 1 illustrates a diagram of an example environment according to various aspects and embodiments of the present disclosure.

[0009] FIG.2 illustrates a block diagram of an example computing environment according to various aspects and embodiments of the present disclosure.

[0010] FIG. 3 illustrates an example annotated visual representation according to various aspects and embodiments of the present disclosure.

[0011] FIG. 4 illustrates an example human-machine interface according to various aspects and embodiments of the present disclosure.

[0012] FIG. 5 illustrates a diagram of another example environment according to various aspects and embodiments of the present disclosure.

[0013] FIG. 6 illustrates a block diagram of an example automated lead vehicle operation system according to various aspects and embodiments of the present disclosure.

[0014] FIG. 7 illustrates a block diagram of an example automated follow vehicle operation system according to various aspects and embodiments of the present disclosure.

[0015] FIG.8 illustrates a block diagram of an example automated lead-follow vehicle remote operation system according to various aspects and embodiments of the present disclosure.

[0016] FIG. 9 illustrates a block diagram of an example automated vehicle safety system according to various aspects and embodiments of the present disclosure.

[0017] FIG. 10 illustrates an example visual representation according to various aspects and embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] As noted above, Truck Mounted Attenuators (TMAs) provide an added measure of safety to work zones by acting as a physical barrier between work crews and passing traffic or errant vehicles. However, even with energy-absorbing crash barriers and safety restraintAttorney Docket: 222204-2985 protections, the human driver of the TMA-equipped vehicle is put at risk of injury when that vehicle is struck by a car or heavy truck. This is particularly true in mobile and short-duration work zone operations when the driver remains in the vehicle during the work activity.

[0019] The emerging field of vehicle automation offers a potential solution that would allow the TMA to be operated without a human driver occupying the vehicle, which eliminates the risk of injury should the TMA vehicle be hit. Embodiments of the present disclosure are directed to systems and methods for automated control of a vehicle such as a TMA. The embodiments use a leader-follower control concept in various examples to allow for a driver to be removed from potential harm associated with driving an at-risk TMA.

[0020] The embodiments can be designed and implemented as a leader-follower automated TMA (ATMA) system and method. The embodiments include a multitude of functional capabilities and may be implemented under various operation design domain (ODD) conditions. Example ODD conditions include, but are not limited to, at least one of speeds of approximately 35 miles per hour (mph), a dependable global positioning system (GPS) signal or a GPS-denied environment, commanded following distances of approximately 50 feet (ft) to approximately 400 ft, commanded lateral offsets of approximately + / - 12 ft, clear weather, or night and day operations. The embodiments can be implemented in a closed course setting in some examples or on a public roadway in live work zones in other examples.

[0021] The embodiments can successfully monitor, detect, and respond to object intrusions in a safety zone (e.g., an area between the lead and follower vehicles) using a light detection and ranging (LiDAR) based system in some examples. Some embodiments can include a human- machine interface (HMI) system to allow an operator in a lead vehicle to control the lateral and longitudinal offsets of a follower vehicle, apply waypoint holds and releases, and provide situational awareness of an entire operation. The HMI system is intuitive and provides a means for the lead operator to safely drive and control the vehicle with minimal operator effort and attention. Some embodiments can include a remote operation feature that utilizes a joystick to control a following vehicle at a range of up to approximately 500 meters (m) in some examples.

[0022] The embodiments can operate in GPS-denied environments, including under large overpasses, in tunnels, and in urban canyons, as such environments are a part of typical work zone operations. Some embodiments implement a LiDAR-based solution to support operation in a GPS- denied environment. LiDAR provides "point cloud" data that can be processed for at least one of object detection on a desired path or route navigation. One embodiment includes an ATMA systemAttorney Docket: 222204-2985 that can implement a LiDAR-based simultaneous localization and mapping (SLAM) solution to support operation in a GPS-denied environment. The SLAM method uses point cloud returns (e.g., point cloud data) from LiDAR to map physical features near the roadway to provide landmarks that can be used by the ATMA system for self-localization and navigation. In one example, a LiDAR sensor intended for use in forward object detection applications rather than LiDAR-based mapping can be used. However, the accuracy and range of 16 beam LiDAR units used in a SLAM model in this example may be limited and ultimately the lateral tracking accuracy may suffer. One option to resolve this issue is to use a different class of LiDAR unit that would provide the resolution and range necessary for accurate roadside LiDAR mapping. However, higher performance LiDAR units can be cost prohibitive for a commercially viable product.

[0023] As a solution to the above-described problems associated with GPS-denied environments and LiDAR-based approaches, some embodiments herein include a camera-based perception system that can track lane lines and road edges. In one embodiment, a single LiDAR unit is used to support forward object detection. Use of such a LiDAR unit for mapping is omitted in this example. Higher performance cameras are included in some embodiments, as well as a lead vehicle technology package to support accurate camera-based mapping. The resulting performance achieved by and economics associated with the embodiments in many examples can support operations in GPS-denied environments at a lower per-unit cost relative to existing GPS and LiDAR-based systems. The embodiments also improve the ease of operation, stability of operation, and accuracy performance at speeds up to approximately 35 mph over long distances (e.g., miles) and durations (e.g., minutes, hours) relative to such existing systems.

[0024] Some embodiments also leverage state-of-the-art and relatively more capable component products such as improved sensing, computing, and communication equipment. Incorporating such component products in these embodiments allows for a reduced amount of total computers (e.g., from five to two in one example), improves the range, capability, and reliability of the wireless link, and achieves more accurate GPS signals at a much lower cost.

[0025] For context, FIG.1 illustrates a diagram of an example environment 100 according to various aspects and embodiments of the present disclosure. The environment 100 can facilitate automated vehicle operation according to various aspects and embodiments of the present disclosure. For instance, the environment 100 can facilitate automated operation of a vehicle such as a Truck Mounted Attenuator (TMA).Attorney Docket: 222204-2985

[0026] The environment 100 in this example includes a lead vehicle 110 and a follow vehicle 120 on a path 130 and in communication with one another by way of one or more networks 140 (or “networks 140”) and computing devices 112, 122 of the lead vehicle 110 and the follow vehicle 120, respectively. The path 130 includes one or more path markers 132a, 132b, 132c (or “path markers 132”) and one or more path edges 134a, 134b (or “path edges 134”). Each of the path markers 132 partly defines at least one of the path 130 or one or more lanes 136a, 136b (or “the lanes 136”) of the path 130.

[0027] The networks 140 can include, for instance, at least one of the Internet, an intranet, an extranet, a wide area network (WAN), a local area network (LAN), a wired network, a wireless network (e.g., cellular, WiFi®), a cable network, a satellite network, a controller area network (CAN), or another suitable network. The computing devices 112, 122 can communicate with one another using any suitable systems interconnect models and / or protocols such as, for instance, dedicated short-range communications (DSRC), Hypertext Transfer Protocol (HTTP), Simple Object Access Protocol (SOAP), representational state transfer (REST), Real-time Transport Protocol (RTP), Real-time Streaming Protocol (RTSP), Real-time Messaging Protocol (RTMP), User Datagram Protocol (UDP), Internet Protocol (IP), Transmission Control Protocol (TCP), and / or other protocols for communicating data over the networks 140, without limitation. Although not illustrated, the networks 140 can include connections to any number of other network hosts, such as website servers, file servers, networked computing resources, databases, data stores, or any other network or computing architectures.

[0028] The computing device 112 of the lead vehicle 110 can be configured in many examples to perform a method of automated vehicle operation to control the follow vehicle 120 by way of the computing device 122. To automate control of the follow vehicle 120, the computing device 112 can employ machine vision devices and methods to track a centerline of at least one of the path markers 132 or the path edges 134 relative to a centerline 1 of the lead vehicle 110 and a centerline 2 of the follow vehicle 120. The computing device 112 can then use machine vision data captured during such centerline tracking in many examples to determine localizing data or information for at least one of the lead vehicle 110 or the follow vehicle 120. For instance, the computing device 112 can determine localizing data or information indicative of a lateral position of at least one of the centerline 1 of the lead vehicle 110 or the centerline 2 of the follow vehicle 120 relative to at least one of the path markers 132 or the path edges 134. In another example, the computing device 112 can determine localizing data or information indicative of the lateralAttorney Docket: 222204-2985 positions of the centerline 1 of the lead vehicle 110 and the centerline 2 of the follow vehicle 120 relative to one another. The computing device 112 can use such localizing data or information in many examples to control a lateral position of the centerline2of the follow vehicle 120 relative to some reference point. For instance, the computing device 112 can use localizing data or information determined for the lead vehicle 110 to control a lateral position of the centerline2of the follow vehicle 120 relative to the centerline1of the lead vehicle 110. In another example, the computing device 112 can use localizing data or information determined for the lead vehicle 110 to control a lateral position of the centerline2of the follow vehicle 120 relative to the centerline of at least one of the path markers 132 or the path edges 134.

[0029] Although not illustrated in FIG. 1 for clarity, each of the lead vehicle 110 and the follow vehicle 120 includes or is coupled to one or more on-board perception sensors that can include, but are not limited to, a camera (e.g., optical, thermographic), a stereo camera, a microphone, radar, ultrasound or sonar, light detection and ranging (LiDAR), receiver(s) for one or more global navigation satellite systems (GNSS) such as, for instance, the Global Positioning System (GPS), odometry, an inertial measurement unit (e.g., accelerometer, gyroscope, magnetometer), temperature, precipitation, pressure, and other types of sensors. Additionally, each of the computing device 112 of the lead vehicle 110 and the computing device 122 of the follow vehicle 120 is coupled to such on-board perception sensor(s) to capture or measure data such as, for instance, image data, video data, audio data, radar data, ultrasonic data, point cloud data, LiDAR data, location data, position data, orientation data, operational or state data, any combination thereof, or other types of data in some cases. For instance, each of the computing device 112 of the lead vehicle 110 and the computing device 122 of the follow vehicle 120 can use such on-board perception sensors to capture image or video data indicative of a centerline of at least one of the path markers 132 or the path edges 134 relative to each of the centerline 1 of the lead vehicle 110 and the centerline 2 of the follow vehicle 120. Some embodiments herein can implement at least one of a GNSS or a GPS solution that uses a real-time kinematic (RTK) positioning correction method. The method may be implemented to correct common errors in most or all GNSS or GPS systems (e.g., GPS systems). Some embodiments herein can implement at least one of a GNSS or a GPS solution that uses an RTK positioning correction method to allow for centimeter precision down to a range of approximately 4 centimeters (cm) to approximately 10 cm.Attorney Docket: 222204-2985

[0030] The lead vehicle 110 and the follow vehicle 120 in the example shown each include or is coupled to a first camera and a second camera, respectively. The first camera can be configured and operable by the computing device 112 to capture first image data (e.g., first video or images) of at least one of the path 130, the path markers 132, the path edges 134, or the lanes 136. The second camera can be configured and operable by at least one of the computing device 112 or the computing device 122 to capture second image data (e.g., second video or images) of at least one of the path 130, the path markers 132, the path edges 134, or the lanes 136.

[0031] The computing device 112 in many examples can use the first image data captured by the first camera of the lead vehicle 110 and the second image data captured by the second camera of the follow vehicle 120 to concurrently track a centerline of at least one of the path markers 132 or the path edges 134 relative to the centerline 1 of the lead vehicle 110 and the centerline 2 of the follow vehicle 120. In one example, the computing device 112 may implement a machine or computer vision model such as, for instance, a You Only Look Once (YOLO) model to perform real-time or near-real-time object detection and concurrently track a centerline of at least one of the path markers 132 or the path edges 134 against the centerline 1 of the lead vehicle 110 and the centerline 2 of the follow vehicle 120. For instance, the computing device 112 can implement a YOLO version 8 (YOLOv8) model to perform real-time or near-real-time object detection and concurrently track a centerline of at least one of the path markers 132 or the path edges 134 against the centerline1of the lead vehicle 110 and the centerlineof the follow vehicle 120.

[0032] To automate control of the follow vehicle 120, the computing device 112 of the lead vehicle 110 can implement a machine vision navigation process in many examples to determine localizing data or information for the lead vehicle 110 relative to at least one of the path markers 132 or the path edges 134 at a defined location L on the path 130. For instance, the computing device 112 can implement a machine vision navigation process to determine centerline distance ratios comparing respective distances d1a,d3a, d4a, d5a between the centerline 1 of the lead vehicle 110 and each of the path markers 132 or each of the path edges 134 at a defined location L on the path 130. The distances d1a, d2a, d3a denote respective distances between the centerline 1 of the lead vehicle 110 and each of the path markers 132a, 132b, 132c. The distances d4a, d5a denote respective distances between the centerline 1 of the lead vehicle 110 and each of the path edges 134a, 134b.

[0033] To automate control of the follow vehicle 120, the computing device 112 can use the aforementioned localizing data or centerline distance ratios corresponding to the lead vehicle 110Attorney Docket: 222204-2985 to cause the centerline 2 of the follow vehicle 120 to be at least momentarily positioned at or maintain a defined position on the path 130. For example, using the localizing data or centerline distance ratios of the lead vehicle 110, the computing device 112 can cause the centerlineof the follow vehicle 120 to be at least momentarily positioned at or maintain a defined lateral distance from at least one of the path markers 132 or the path edges 134 at the defined location L on the path 130 or another location on the path 130. For instance, using the localizing data or centerline distance ratios of the lead vehicle 110, the computing device 112 can cause the centerline2of the follow vehicle 120 to be at least momentarily positioned at or maintain any of distances d1b, d2b, d3b, d4b, d5b at the defined location L on the path 130. The distances d1b, d2b, d3b denote respective distances between the centerline 2 of the follow vehicle 120 and each of the path markers 132a, 132b, 132c. The distances d4b, d5b denote respective distances between the centerline 2 of the follow vehicle 120 and each of the path edges 134a, 134b.

[0034] As another example, the computing device 112 can use the aforementioned localizing data or centerline distance ratios corresponding to the lead vehicle 110 to cause the centerline 2 of the follow vehicle 120 to be at least momentarily positioned at or maintain a defined distance from the centerline 1 of the lead vehicle 110 on the path 130. For instance, using the localizing data or centerline distance ratios of the lead vehicle 110, the computing device 112 can cause the centerline2of the follow vehicle 120 to be at least momentarily positioned at or maintain a distance d6from the centerline1of the lead vehicle 110 on the path 130 or another location on the path 130. The distance d6denotes a lateral or offset distance between the centerline2of the follow vehicle 120 and the centerline1of the lead vehicle 110. The distance d6can range from a value of zero to a value of approximately 12 feet (ft) or greater in some cases. The distance d6can be equal to zero in one example such that the computing device 112 causes the centerline2of the follow vehicle 120 to be at least momentarily positioned at or maintain the same position as the centerline 1 of the lead vehicle 110 at the defined location L on the path 130 or another location on the path 130. In the example shown, the distance d6 has a value greater than zero and less than approximately 12 ft.

[0035] The computing device 112 can also employ the aforementioned machine vision devices, model, and navigation process to cause the follow vehicle 120 to be at least momentarily positioned at or maintain a defined longitudinal distance from the lead vehicle 110 on the path 130. For instance, the computing device 112 can cause the follow vehicle 120 to be at least momentarily positioned at or maintain a distance d7from the lead vehicle 110. The distance d7Attorney Docket: 222204-2985 denotes at least one of a defined linear distance or a defined longitudinal distance between a back portion of the lead vehicle 110 and a front portion of the follow vehicle 120. The distance d7can range from a value of zero to a value of approximately 400 ft or greater in some cases.

[0036] The computing device 112 can further employ the aforementioned machine vision devices, model, and navigation process to cause the follow vehicle 120 to at least momentarily travel at or maintain a defined speed on the path 130. The defined speed may be relative to at least one of the path 130, one or more of the path markers 132, one or more of the path edges 134, or the lead vehicle 110. The defined speed may range from a value of zero to a value of approximately 45 miles per hour (mph) or greater in some cases. For instance, the defined speed may range from a value of zero to a value of approximately 45 mph relative to the path 130.

[0037] To automate control of the follow vehicle 120, the computing device 112 can use the aforementioned first image data (e.g., video, images) captured by the first camera of the lead vehicle 110 to determine a lateral distance between a centerline of at least one of the path markers 132 or the path edges 134 and the centerline 1 of the lead vehicle 110 at the defined location L on the path 130. For instance, the computing device 112 can use the first image data to determine a first pixel distance of pixels along a linear segment, perpendicular to centerline 1, of the first image data that corresponds to the distance d1abetween a centerline of the path marker 132a and the centerline1of the lead vehicle 110 at the defined location L on the path 130. The computing device 112 can determine the first pixel distance of pixels in one example based on the size, number, and arrangement of pixels positioned along such a linear segment of the first image data between the centerline of the path marker 132a and the centerline1of the lead vehicle 110. For instance, the computing device 112 can determine the first pixel distance of pixels by determining the size (e.g., diameter, width) and number of pixels arranged along such a linear segment of the first image data between the centerline of the path marker 132a and the centerline 1 of the lead vehicle 110 at the defined location L on the path 130.

[0038] The computing device 112 can further use the first image data to determine a second lateral distance between a centerline of another path marker 132 or path edge or guard rail 134 and the centerline 1 of the lead vehicle 110 at the defined location L on the path 130. For instance, the computing device 112 can use the first image data to determine a second pixel distance of pixels along a linear segment of the first image data that corresponds to the distance d2a between a centerline of the path marker 132b and the centerline 1 of the lead vehicle 110 at the defined location L on the path 130. The computing device 112 can then use the first pixel distance of pixelsAttorney Docket: 222204-2985 corresponding to the distance d1a and the second pixel distance of pixels corresponding to the distance d2ato calculate a centerline distance ratio corresponding to the lead vehicle 110. The centerline distance ratio of the lead vehicle 110 can be indicative of its absolute position on the path 130 relative to at least one of the path markers 132a, 132b or the lane 136a in this example. The centerline distance ratio includes and compares the first pixel distance of pixels corresponding to the distance d1aand the second pixel distance of pixels corresponding to the distance d2a. Different lateral and longitudinal distances may be calculated in various examples with or without the use of GPS systems or methodologies. For example, one embodiment uses a combination of radar and vehicle odometry to determine vehicle distance traveled (e.g., by the follow vehicle 120) and proximity to a preceding vehicle (e.g., proximity of the follow vehicle 120 to the lead vehicle 110).

[0039] To automate control of the follow vehicle 120, the computing device 112 or 122 can use the aforementioned second image data (e.g., video, images) captured by the second camera of the follow vehicle 120 to determine a lateral distance between a centerline of at least one of the path markers 132 or the path edges 134 and the follow vehicle 120 at the defined location L on the path 130. For instance, the computing device 112 or 122 can use at least one of the first pixel distance of pixels corresponding to the distance d1a, the second pixel distance of pixels corresponding to the distance d2a, or the centerline distance ratio corresponding to the lead vehicle 110 to calculate a third pixel distance of pixels along a linear segment of the second image data that corresponds to the distance d1bbetween the centerline of the path marker 132a and the centerline2of the follow vehicle 120 at the defined location L on the path 130. The computing device 112 or 122 can also use at least one of the first pixel distance of pixels corresponding to the distance d1a, the second pixel distance of pixels corresponding to the distance d2a, or the centerline distance ratio corresponding to the lead vehicle 110 to calculate a fourth pixel distance of pixels along a linear segment of the second image data that corresponds to the distance d2b between the centerline of the path marker 132a and the centerline 2 of the follow vehicle 120 at the defined location L on the path 130.

[0040] In one example, the computing device 112 can use at least one of the first pixel distance of pixels corresponding to the distance d1a, the second pixel distance of pixels corresponding to the distance d2a, or the centerline distance ratio corresponding to the lead vehicle 110 to calculate at least one of the third pixel distance of pixels corresponding to the distance d1b or the fourth pixel distance of pixels corresponding to the distance d2bbased on one or more attributes of any or allAttorney Docket: 222204-2985 of the lead vehicle 110, the follow vehicle 120, the first camera coupled to the lead vehicle 110, or the second camera coupled to the follow vehicle 120. For instance, the computing device 112 can use at least one of the first pixel distance, the second pixel distance, or the centerline distance ratio to calculate the third pixel distance and / or the fourth pixel distance based on one or more attributes including, but not limited to, vehicle specifications (e.g., height, width, length) of the lead vehicle 110 and / or the follow vehicle 120, camera specifications of the first camera and / or the second camera, camera pose of the first camera and / or the second camera, and camera location of the first camera on the lead vehicle 110 or the second camera on the follow vehicle 120.

[0041] In one example, the computing device 112 can determine the third pixel distance of pixels corresponding to the distance d1b based on the size, number, and arrangement of pixels positioned along the aforementioned linear segment of the second image data between the centerline of the path marker 132a and the centerline 2 of the follow vehicle 120. For instance, the computing device 112 can determine the third pixel distance of pixels by determining the size (e.g., diameter, width) and number of pixels arranged along such a linear segment of the second image data between the centerline of the path marker 132a and the centerline 2 of the follow vehicle 120 at the defined location L on the path 130. Similarly, the computing device 112 can determine the fourth pixel distance of pixels corresponding to the distance d2bbased on the size, number, and arrangement of pixels positioned along the aforementioned linear segment of the second image data between the centerline of the path marker 132b and the centerline2of the follow vehicle 120. For instance, the computing device 112 can determine the fourth pixel distance of pixels by determining the size (e.g., diameter, width) and number of pixels arranged along such a linear segment of the second image data between the centerline of the path marker 132b and the centerline2of the follow vehicle 120 at the defined location L on the path 130.

[0042] To automate control of the follow vehicle 120, based on determining at least one of the third pixel distance corresponding to the distance d1b or the fourth pixel distance corresponding to the distance d2b, the computing device 112 can use the third pixel distance and / or the fourth pixel distance to cause (e.g., via the computing device 122) the centerline 2 of the follow vehicle 120 to be positioned at the distance d1b and the distance d2b at the defined location L on the path 130. Additionally, the computing device 112 can cause (e.g., via the computing device 122) the speed of the follow vehicle 120 to be adjusted or maintained such that at the defined location L on the path 130 the centerline 2 of the follow vehicle 120 is positioned at the distance d1b from the path marker 132a based on the third pixel distance and at the distance d2bfrom the path markerAttorney Docket: 222204-2985 132b based on the fourth pixel distance. In some examples, the third pixel distance corresponding to the distance d1bmay be equal to the first pixel distance corresponding to the distance d1a, and the fourth pixel distance corresponding to the distance d2bmay be equal to the second pixel distance corresponding to the distance d2a. Thus, in these examples the distance d1bis equal to the distance d1aand the distance d2bis equal to the distance d2a. In other examples, the third pixel distance corresponding to the distance d1bmay be different from the first pixel distance corresponding to the distance d1aand the fourth pixel distance corresponding to the distance d2bmay be different from the second pixel distance corresponding to the distance d2a. Thus, in these examples the distance d1b is thus different from the distance d1a and the distance d2b is different from the distance d2a.

[0043] The computing device 112 can provide (e.g., via the networks 140) the computing device 122 with at least one of the first pixel distance of pixels corresponding to the distance d1a, the second pixel distance of pixels corresponding to the distance d2a, or the centerline distance ratio corresponding to the lead vehicle 110. The computing device 122 can use at least one of the first pixel distance, the second pixel distance, or the centerline distance ratio to calculate the desired third pixel distance corresponding to the distance d1b and / or the fourth pixel distance corresponding to the distance d2bin some cases. For instance, the computing device 122 can use the first pixel distance, the second pixel distance, and / or the centerline distance ratio, as well as the aforementioned second image data captured by the second camera of the follow vehicle 120, to calculate the third pixel distance corresponding to the distance d1band / or the fourth pixel distance corresponding to the distance d2bin the same manner as described above (e.g., in the same manner as the computing device 112 calculates such distances). In some examples, based on determining at least one of the third pixel distance corresponding to the distance d1bor the fourth pixel distance corresponding to the distance d2b, the computing device 122 can then use the third pixel distance and / or the fourth pixel distance to cause the centerline 2 of the follow vehicle 120 to be positioned at the distance d1b and the distance d2b at the defined location L on the path 130. For instance, the computing device 122 can use the third pixel distance and / or the fourth pixel distance to cause the centerline 2 of the follow vehicle 120 to be positioned at the distance d1b and the distance d2b at the defined location L on the path 130 in the same manner as described above (e.g., in the same manner as the computing device 112 can cause the centerline 2 of the follow vehicle 120 to be positioned at such distances). In one example, the computing device 112 or 122 can manipulate the follow vehicle 120 to make the distance d2band the distance d1bAttorney Docket: 222204-2985 equivalent to the distance d2a and the distance d1a, respectively, at position L once the follow vehicle 120 has traveled the length of the distance d7.

[0044] To further facilitate automated operation of the follow vehicle 120 in some examples, at least one of the computing devices 112, 122 can generate an annotated visual representation of the first image data captured by the first camera of the lead vehicle 110. For instance, the annotated visual representation can include various annotations overlaid (e.g., superimposed) on at least one of the first image data or the second image data. The annotations can denote, for example, any or all of a centerline of at least one of the path markers 132, a centerline of at least one of the path edges 134, the centerline 1 of the lead vehicle 110, the centerline 2 of the follow vehicle 120, or a horizontal reference line. Based on determining the first, second, third, and fourth pixel distances respectively corresponding to the distances d1a, d2a, d1b, d2b, the computing device 112 can generate the annotated visual representation such that it includes annotations denoting such distances. The computing device 112 can also generate the annotated visual representation such that it includes additional annotations denoting additional pixel distances of pixels in the first image data or the second image data that correspond to additional lateral distances between the centerline 1 of the lead vehicle 110 or the centerline 2 of the follow vehicle 120 and a centerline of at least one of the path markers 132 or the path edges 134. For instance, the computing device 112 can generate the annotated visual representation such that it includes additional annotations denoting additional pixel distances of pixels in the first image data or the second image data that correspond to any or all of the distances d3a, d4a, d5a, d3b, d4b, d5bbetween the centerline1of the lead vehicle 110 or the follow vehicle 120 and a centerline of at least one of the path markers 132 or the path edges 134.

[0045] The computing device 112 can generate an annotated visual representation of the first image data in one example such that it includes a first centerline distance ratio line overlaid on the first image data between a centerline of the path marker 132a and the centerline 1 of the lead vehicle 110 at the defined location L on the path 130. The first centerline distance ratio line in this example denotes the first pixel distance of pixels in the first image data that corresponds to the distance d1a between the centerline of the path marker 132a and the centerline 1 of the lead vehicle 110. The computing device 112 can generate the annotated visual representation such that it also includes a second centerline distance ratio line overlaid on the first image data between a centerline of the path marker 132b and the centerline 1 of the lead vehicle 110 at the defined location L on the path 130. The second centerline distance ratio line in this example denotes the second pixel distance of pixels in the first image data corresponding to the distance d2a. Together, the firstAttorney Docket: 222204-2985 centerline distance ratio line and the second centerline distance ratio line in this example visually represent the centerline distance ratio corresponding to the lead vehicle 110 at the defined location L on the path 130.

[0046] To further facilitate automated operation of the follow vehicle 120 in some examples, the computing device 112 can generate a human-machine interface (HMI) having interactive operational inputs corresponding to vehicular operations associated with at least one of the lead vehicle 110 or the follow vehicle 120. For example, the computing device 112 can generate an HMI having interactive operational inputs that can include, but are not limited to, at least one of a speed indicator for the lead vehicle 110, a speed indicator for the follow vehicle 120, a lateral setting input and indicator for the follow vehicle 120, a longitudinal setting input and indicator for the follow vehicle 120, a hold input for the lead vehicle 110, a hold input for the follow vehicle 120, a lead hold input for stopping the follow vehicle 120 at a position where the lead vehicle 110 was when the lead hold input was engaged, and a stop input for stopping the follow vehicle 120. In these examples, the computing device 112 can cause the follow vehicle 120 to perform one or more vehicular operations based on input received by way of the interactive operational inputs of such an HMI.

[0047] The computing devices 112, 122 and the aforementioned on-board perception sensors included with or coupled to the lead vehicle 110 and the follow vehicle 120 can be configured with capabilities to support remote operator control (e.g., remote operator override control). For example, the computing devices 112, 122 and on-board perception sensors of the lead vehicle 110 and the follow vehicle 120 can include or be configured to implement automated lead-follow vehicle remote operation system 800 described herein and illustrated in FIG.8 to facilitate remote operation of the follow vehicle 120. The automated lead-follow vehicle remote operation system 800 (or “remote operation system 800”) can facilitate remote operation of the follow vehicle 120 in one example by exposing drive-by-wire (DBW) controls to a remote controller using a cellular vehicle-to-everything (C-V2X) communication link and a 5G / 4G-LTE link. The remote operation system 800 can implement a watchdog timer in some cases to keep a low latency and safe communication link between the ATMA components of the lead vehicle 110 and those of the follow vehicle 120, which can include a robotic operating system (ROS) software platform in many examples.

[0048] In some embodiments, at least one of the computing devices 112, 122 can be configured to generate and render a data telemetry packet on a display device such as a monitor orAttorney Docket: 222204-2985 a digital screen. The data telemetry packet can be embodied as a visual display such as a virtual dashboard or an interactive display that can include kinematics data, automation status, latency data, and error codes in some examples. In some embodiments, at least one of the computing devices 112, 122 can be configured to employ a web-based interface to show telemetry data and also video views to allow an operator to understand how to handle object avoidance or path re- routing. These embodiments use WebRTC for video broadcasting, message queuing telemetry transport (MQTT) for data exchange between the vehicle platforms and XVIZ Server for the web data and video presentation with low latency performance.

[0049] In other embodiments, the computing devices 112, 122 and the aforementioned on- board perception sensors of the lead vehicle 110 and the follow vehicle 120 can also include or be configured to implement a separate channel for DBW commands which provides the ability to independently control steering, throttle, and brake with some applied thresholds for safety purposes. This data channel monitors timestamping and disables automation when a packet / timestamp is missed for more than 100 milliseconds (ms) in some examples. For instance, the computing devices 112, 122 and on-board perception sensors of the lead vehicle 110 and the follow vehicle 120 can include or implement automated vehicle safety system 900 described herein with reference to FIG. 9 to monitor the health and performance of onboard computing hardware, sensors, and communications equipment of the follow vehicle 120, and the lead vehicle 110 in some cases.

[0050] FIG. 2 illustrates a block diagram of an example computing environment 200 according to various aspects and embodiments of the present disclosure. The computing environment 200 can include or be coupled to a computing device 202. With reference to FIGS.1 and 2, the computing environment 200 can be embodied as or used, at least in part, to implement each of the lead vehicle 110 and the follow vehicle 120, and the computing device 202 can be embodied as or used, at least in part, to implement each of the computing device 112 and the computing device 122.

[0051] The computing device 202 can include at least one processing system, for example, having one or more processors 204 (or “processor 204”) and at least one memory 206 (or “memory 206”), both of which can be coupled to a local interface 208. The memory 206 can include a data store 210, an automated vehicle operation module 212, a communications stack 214, and a control module 216 in the example shown. The computing device 202 can also be coupled by way of the local interface 208 to one or more perception sensors 218 (or “perception sensors 218”), one orAttorney Docket: 222204-2985 more control systems 220 (or “control systems 220”), or a combination thereof. In some cases, the computing environment 200, the computing device 202, or both may also include other components that are not illustrated in FIG.2.

[0052] The lead vehicle 110 and the follow vehicle 120 can each include or be coupled to the on-board perception sensors listed above and various on-board vehicle control systems (e.g., powertrain control system, brake and suspension control system, steering control system). The perception sensors 218 can be embodied as such on-board perception sensors, and the control systems 220 can be embodied as such on-board vehicle control systems. In this example, the computing environment 200 and the computing device 202 (and thus, the computing devices 112, 122 of the lead vehicle 110 and the follow vehicle 120, respectively) can be coupled to the perception sensors 218 and the control systems 220.

[0053] The processor 204 can include any processing device (e.g., a processor core, a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a controller, or a microcontroller) and can include one or multiple processors that can be operatively connected. In some examples, the processor 204 can include one or more complex instruction set computing (CISC) microprocessors, one or more reduced instruction set computing (RISC) microprocessors, one or more very long instruction word (VLIW) microprocessors, or one or more processors that are configured to implement other instruction sets.

[0054] The memory 206 can be embodied as one or more memory devices and store data and software or executable-code components executable by the processor 204. For example, the memory 206 can store executable-code components associated with the automated vehicle operation module 212, the communications stack 214, and the control module 216 for execution by the processor 204. The memory 206 can also store data such as the data described below that can be stored in the data store 210, among other data. For example, the memory 206 can store computer-readable instructions for implementing various algorithms and machine-vision or computer-vision models (e.g., YOLOv8) used by the automated vehicle operation module 212, the communications stack 214, and the control module 216. The memory 206 can further store such algorithms and models, as well as any input data used to implement such algorithms and models, or output data generated by such algorithms and models as described in embodiments herein. The memory 206 can also store captured vehicle sensory data, vehicle perception information, vehicle localizing data or information, and vehicle centerline distance ratios generated by andAttorney Docket: 222204-2985 communicated between the lead vehicle 110 and the follow vehicle 120 as described in embodiments herein.

[0055] The memory 206 can store other executable-code components for execution by the processor 204. For example, an operating system can be stored in the memory 206 for execution by the processor 204. Where any component discussed herein is implemented in the form of software, any one of a number of programming languages can be employed such as, for example, C, C++, C#, Objective C, JAVA®, JAVASCRIPT®, Perl, PHP, VISUAL BASIC®, PYTHON®, RUBY, FLASH®, or other programming languages.

[0056] As discussed above, the memory 206 can store software for execution by the processor 204. In this respect, the terms “executable” or “for execution” refer to software forms that can ultimately be run or executed by the processor 204, whether in source, object, machine, or other form. Examples of executable programs include a compiled program that can be translated into a machine code format and loaded into a random-access portion of the memory 206 and executed by the processor 204, source code that can be expressed in an object code format and loaded into a random-access portion of the memory 206 and executed by the processor 204, source code that can be interpreted by another executable program to generate instructions in a random-access portion of the memory 206 and executed by the processor 204, or other executable programs or code.

[0057] The local interface 208 can be embodied as a data bus with an accompanying address / control bus or other addressing, control, and / or command lines. In part, the local interface 208 can be embodied as an in-vehicle interface such as, for instance, an on-board diagnostics (OBD) bus, a controller area network (CAN) bus, a Local Interconnect Network (LIN) bus, a Media Oriented Systems Transport (MOST) bus, ethernet, or another network interface.

[0058] The data store 210 can include data for the computing device 202 such as, for instance, one or more unique identifiers for the computing device 202, digital certificates, encryption keys, session keys and session parameters for communications, and other data for reference and processing. The data store 210 can also store computer-readable instructions for execution by the computing device 202 (e.g., via the processor 204), including instructions for the automated vehicle operation module 212, the communications stack 214, and the control module 216. For example, the data store 210 can store computer-readable instructions for implementing various algorithms and machine-vision or computer-vision models (e.g., YOLOv8) used by the automated vehicle operation module 212, the communications stack 214, and the control module 216. TheAttorney Docket: 222204-2985 data store 210 can further store such algorithms and models, as well as any input data used to implement such algorithms and models, or output data generated by such algorithms and models as described in embodiments herein. The data store 210 can also store captured vehicle sensory data, vehicle perception information, vehicle localizing data or information, and vehicle centerline distance ratios generated by and communicated between the lead vehicle 110 and the follow vehicle 120 as described in embodiments herein.

[0059] The automated vehicle operation module 212 can be embodied as one or more software applications or services executing on the computing device 202. For example, the automated vehicle operation module 212 can be embodied as one or more software applications or services that can be implemented (e.g., executed, run) by the processor 204 to perform automated vehicle operation as described in embodiments herein. In one example, the automated vehicle operation module 212 can be implemented by the computing device 202 (and thus, the computing devices 112, 122) to automate the operation of the follow vehicle 120 by the lead vehicle 110 (e.g., via the computing device 112) as described herein with reference to FIG. 1. In another example, the automated vehicle operation module 212 can be implemented by the computing device 202 (and thus, the computing device 112, 122) to generate an annotated visual representation of image data captured by at least one of a first camera of the lead vehicle 110 or a second camera of the follow vehicle 120 as described herein with reference to FIG. 3. In yet another example, the automated vehicle operation module 212 can be implemented by the computing device 202 (and thus, the computing devices 112, 122) to generate a human-machine interface having interactive operational inputs corresponding to vehicular operations associated with at least one of the lead vehicle 110 or the follow vehicle 120 as described herein with reference to FIG.4.

[0060] The communications stack 214 can include software and hardware layers to implement data communications such as, for instance, Bluetooth®, Bluetooth® Low Energy (BLE), WiFi®, cellular data communications interfaces, DSRC, C-V2X communications interfaces, or a combination thereof. Thus, the communications stack 214 can be relied upon by the computing device 202 to establish DSRC, C-V2X, cellular, Bluetooth®, WiFi®, and other communications channels with the networks 140, and further allow such communications channels to be established between the lead vehicle 110 and the follow vehicle 120. The communications stack 214 can include the software and hardware to implement Bluetooth®, BLE, DSRC, C-V2X, and related networking interfaces, which provide for a variety of different network configurations and flexibleAttorney Docket: 222204-2985 networking protocols for short-range, low-power wireless communications. The communications stack 214 can also include the software and hardware to implement WiFi® communication, DSRC / C-V2X communication, and cellular communication, which also offers a variety of different network configurations and flexible networking protocols for mid-range, long-range, wireless, and cellular communications. The communications stack 214 can also incorporate the software and hardware to implement other communications interfaces, such as X10®, ZigBee®, Z-Wave®, and others. The communications stack 214 can be configured to communicate captured vehicle sensory data, vehicle perception information, vehicle localizing data or information, and vehicle centerline distance ratios between the lead vehicle 110 and the follow vehicle 120 as described in embodiments herein.

[0061] The control module 216 can be embodied as one or more software applications or services executing on the computing device 202. The control module 216 can be embodied as one or more software applications or services that can be implemented (e.g., executed, run) by the processor 204 to cause any of the control systems 220 to perform one or more operations to facilitate automated driving of the follow vehicle 120 by the lead vehicle 110 (e.g., via the computing device 112). For example, the control module 216 can be implemented to cause any of the control systems 220 to perform one or more driving operations that can include, but are not limited to, accelerating, braking, or steering, among others. In at least one example, the control module 216 can be implemented to cause any of the control systems 220 to perform such driving operation(s) based at least in part on captured vehicle sensory data, vehicle perception information, vehicle localizing data or information, and vehicle centerline distance ratios generated by and communicated between the lead vehicle 110 and the follow vehicle 120 as described in embodiments herein.

[0062] The perception sensors 218 can be embodied as one or more on-board perception sensors that can be included in or coupled (e.g., communicatively, operatively) to and used by each of the lead vehicle 110 and the follow vehicle 120 to capture or measure sensor data (e.g., observational data, image data, video data). The sensor data can be indicative of an environment surrounding the lead vehicle 110 and the follow vehicle 120 (e.g., the environment 100, the path 130, the path markers 132, the path edges 134) and can be used by the lead vehicle 110 (e.g., the computing device 112) to automate operation of the follow vehicle 120 as described in embodiments herein. The perception sensors 218 can include, but are not limited to, a camera (e.g., optical, thermographic), a stereo camera, a microphone, radar, ultrasound or sonar, LiDAR,Attorney Docket: 222204-2985 receivers for one or more GNSS such as, for instance, GPS, odometry, an inertial measurement unit (e.g., accelerometer, gyroscope, magnetometer), temperature, precipitation, pressure, and other types of sensors.

[0063] The control systems 220 can be embodied as one or more on-board vehicle control systems that can be included in or coupled (e.g., communicatively, operatively) to and used by each of the lead vehicle 110 and the follow vehicle 120 to facilitate automated driving of the follow vehicle 120 by the lead vehicle 110. The control systems 220 can be embodied as one or more on- board vehicle control systems that can include, but are not limited to, a powertrain control system (e.g., for the motor, inverter, battery), a chassis control system (e.g., for the brakes, suspension), a steering control system, a vehicle lighting and signal control system (e.g., for the internal and external lights, blinkers, horn), or another control system.

[0064] The control systems 220 can be configured to perform various driving operations (e.g., accelerating, braking, steering) based on receipt of direction (e.g., instructions) from the control module 216 as described above. For example, the control systems 220 can be configured to perform such driving operation(s) based on receipt of instructions generated by the control module 216 in response to captured vehicle sensory data, vehicle perception information, vehicle localizing data or centerline distance ratios generated by and communicated between the lead vehicle 110 and the follow vehicle 120, vehicle speed settings, and vehicle lateral and / or longitudinal or linear offset settings as described in embodiments herein.

[0065] FIG.3 illustrates an example annotated visual representation 300 according to various aspects and embodiments of the present disclosure. The computing device 112 (e.g., via the automated vehicle operation module 212) can generate the annotated visual representation 300 in the example shown using image data 310 captured by one or more of the perception sensors 218 (e.g., a camera) included in or coupled to the lead vehicle 110. The computing device 112 can generate the annotated visual representation 300 such that it includes various annotations overlaid (e.g., superimposed) on the image data 310 as shown in FIG. 3. For instance, the annotations include annotations 320a, 320b that respectively denote an upper and lower boundary of a region in the image data 310 where the computing device 112 computes lane markings and object identification. Defining such a region reduces the computational load requirement and speeds up processing. The annotations in this example further include an annotation 321 denoting the centerline 1 of the lead vehicle 110, an annotation 322a denoting a centerline of the path marker 132a, an annotation 322b denoting a centerline of the path marker 132b, an annotation 322cAttorney Docket: 222204-2985 denoting a centerline of the path marker 132c, an annotation 324a denoting a centerline of one path edge or guard rail 134a, and an annotation 324b denoting a centerline of another path edge or guard rail 134b.

[0066] The annotations in the example shown further include a first centerline distance ratio line 330a overlaid on the image data 310 between the annotations 321, 322a at the defined location L on the path 130. The first centerline distance ratio line 330a in this example denotes a first pixel distance of pixels in the image data 310 that corresponds to the distance d1a(FIG. 1) between the centerline of the path marker 132a and the centerline1of the lead vehicle 110. The computing device 112 can generate the annotated visual representation 300 such that it also includes a second centerline distance ratio line 330b overlaid on the image data 310 between the annotations 321, 322b at the defined location L on the path 130. The second centerline distance ratio line 330b in this example denotes a second pixel distance of pixels in the image data 310 that corresponds to the distance d2a (FIG. 1). Together, the first centerline distance ratio line 330a and the second centerline distance ratio line 330b in this example visually represent a centerline distance ratio corresponding to the lead vehicle 110 at the defined location L on the path 130 relative to the lane 136a.

[0067] The annotations in the example shown further include a third centerline distance ratio line 330c overlaid on the image data 310 between the annotations 321, 322c. The third centerline distance ratio line 330c in this example denotes a third pixel distance of pixels in the image data 310 that corresponds to the distance d3a(FIG. 1) between the centerline of the path marker 132c and the centerline1of the lead vehicle 110. The computing device 112 can generate the annotated visual representation 300 such that it also includes a fourth centerline distance ratio line 330d overlaid on the image data 310 between the annotations 321, 324a at the defined location L on the path 130. The fourth centerline distance ratio line 330d in this example denotes a fourth pixel distance of pixels in the image data 310 that corresponds to the distance d4a (FIG. 1). The computing device 112 can generate the annotated visual representation 300 such that it further includes a fifth centerline distance ratio line 330e overlaid on the image data 310 between the annotations 321, 324b at the defined location L on the path 130. The fifth centerline distance ratio line 330e in this example denotes a fifth pixel distance of pixels in the image data 310 that corresponds to a lateral distance between the centerline 1 of the lead vehicle 110 and the path edge or guard rail 134b.Attorney Docket: 222204-2985

[0068] The computing device 112 can generate the annotated visual representation 300 and the annotations thereof (e.g., the annotations 320a, 320b, 321, 322a, 322b, 322c, 324a, 324b and the centerline distance ratio lines 330a, 330b, 330c, 330d, 330e) in real-time or near-real-time in some examples. In other examples, the computing device 112 can generate the annotated visual representation 300 and the annotations thereof (e.g., the annotations 320a, 320b, 321, 322a, 322b, 322c, 324a, 324b and the centerline distance ratio lines 330a, 330b, 330c, 330d, 330e) periodically based on some defined tithe perception sensors 218me interval (e.g., every half second, every second, every two seconds).

[0069] FIG.4 illustrates an example human-machine interface 400 (or “HMI 400”) according to various aspects and embodiments of the present disclosure. To further facilitate automated operation of the follow vehicle 120, in some examples the computing device 112 can generate the HMI 400 such that it includes interactive operational inputs corresponding to vehicular operations associated with at least one of the lead vehicle 110 or the follow vehicle 120. In the example illustrated in FIG. 4, the computing device 112 can generate the HMI 400 such that it includes at least one of a visual representation 410 of the lead vehicle 110, a speed indicator 412 for the lead vehicle 110, a visual representation 420 of the follow vehicle 120, a speed indicator 422 for the follow vehicle 120, a lateral setting input and indicator 430 for the follow vehicle 120, a longitudinal setting input and indicator 440 for the follow vehicle 120, a lead hold input 450 for stopping the follow vehicle 120 at a location where the lead vehicle 110 was positioned when the lead hold input 450 was engaged, a following hold input 460 for the follow vehicle 120, and a stop input 470 for the follow vehicle 120. The computing device 112 can cause the follow vehicle 120 to perform one or more vehicular operations in some examples based on input received by way of the above-described interactive operational inputs of the HMI 400.

[0070] FIG.5 illustrates a diagram of another example environment 500 according to various aspects and embodiments of the present disclosure. The environment 500 can facilitate automated vehicle operation according to various aspects and embodiments of the present disclosure. For instance, the environment 500 can facilitate automated operation of multiple vehicles such as multiple Truck Mounted Attenuators (TMA).

[0071] The environment 500 in the example shown includes the lead vehicle 110, a follow vehicle 520 positioned behind the lead vehicle 110, and a second follow vehicle 520i positioned behind the follow vehicle 520 on the path 130. The lead vehicle 110 includes the computing device 112, the follow vehicle 520 includes a computing device 522, and the follow vehicle 520iincludesAttorney Docket: 222204-2985 a computing device 522i. Each of the computing devices 522, 522i can include the same or similar components, attributes, and functionality as that of the computing device 122 and the computing device 202 described herein with reference to FIGS.1 and 2. The lead vehicle 110 and the follow vehicles 520, 520iare in communication with one another in this example by way of the networks 140 and computing devices 112, 522, 522i. The follow vehicle 520iis represented in FIG. 5 to illustrate one embodiment of the present invention where two follow vehicles 520, 520ican be automated or remotely controlled in some cases to follow the lead vehicle 110. However, this illustration is not limiting and is meant to show that any number of follow vehicles 520, 520imay follow behind the lead vehicle 110 or a preceding vehicle in a platoon of follow vehicles. For instance, the follow vehicle 520 may be operated as such a preceding vehicle in a platoon of follow vehicles with the follow vehicle 520i being operated as a succeeding or subsequent vehicle relative to the follow vehicle 520. The path 130 includes the path markers 132a, 132b, 132c and the path edges 134a, 134b. Each of the path markers 132 partly defines at least one of the path 130 or one or more of the lanes 136a, 136b of the path 130.

[0072] As an example, the computing device 112 can employ the machine vision devices, model, and navigation process described herein using computed centerline distance ratios of the lead vehicle 110 to cause a centerline2iof any number of the follow vehicles 520ito be at least momentarily positioned at or maintain a defined distance from the centerline1of the lead vehicle 110 or the centerline2of a preceding vehicle in the platoon on the path 130 such as the follow vehicle 520. For instance, the computing device 112 can use the centerline distance ratios of the lead vehicle 110 to command navigation of any of the following vehicles 520, 520iby way of the computing devices 522, 522i, respectively, to cause any of the centerlines2,2iof the follow vehicles 520, 520ito be positioned at or maintain a distance d6ifrom the centerline1of the lead vehicle 110 at a defined location L on the path 130 or another location on the path 130. The distance d6i denotes a lateral or offset distance between the centerline 2i of the follow vehicle 520i and the centerline 1 of the lead vehicle 110. In another example, the computing device 112 can use the centerline distance ratios of the lead vehicle 110 to command navigation of any of the following vehicles 520i by way of the computing devices 522i to cause any of the centerlines 2i of the follow vehicles 520i to be positioned at or maintain a distance d6i-1 from the centerline 2 of the follow vehicle 520 at the defined location L or another location on the path 130. The distance d6i-1 denotes a lateral or offset distance between the centerline 2i of the follow vehicle 520i and the centerline 2 of the follow vehicle 520.Attorney Docket: 222204-2985

[0073] The distances d6i and d6i-1 can each range from a value of zero to a value of approximately 12 ft in many examples. For instance, the distance d6ican be equal to zero such that the computing device 112 by way of computing device 522ican cause the centerlineof the follow vehicle 520ito be positioned at or maintain the same position as the centerline1of the lead vehicle 110 at the defined location L or another location on the path 130. In another example, the distance d6i-1can be equal to zero such that the computing device 112 by way of computing device 522ican cause the centerline2iof the follow vehicle 520ito be positioned at or maintain the same position as the centerline2of the follow vehicle 520 at the defined location L or another location on the path 130. In the example shown, the distance d6i has an absolute value greater than zero and less than approximately 12 ft.

[0074] In another example, the computing device 112 can employ the machine vision devices, model, and navigation process described herein to cause the follow vehicle 520i to be at least momentarily positioned at or maintain a defined longitudinal distance from the lead vehicle 110 or the follow vehicle 520 on the path 130. For instance, the computing device 112 by way of the computing device 520i can cause the follow vehicle 520i to be positioned at or maintain a distance d7i from the lead vehicle 110. The distance d7i denotes at least one of a defined linear distance or a defined longitudinal distance between a back portion of the lead vehicle 110 and a front portion of the follow vehicle 520i. In another example, the computing device 112 by way of the computing device 520ican cause the follow vehicle 520ito be positioned at or maintain a distance d7i-1from the follow vehicle 520. The distance d7i-1denotes at least one of a defined linear distance or a defined longitudinal distance between a back portion of the follow vehicle 520 and a front portion of the follow vehicle 520i. The distances d7iand d7i-1can each range from a value of approximately 50 ft to a value of approximately 400 ft in many examples.

[0075] The embodiments of the present disclosure can also be designed and implemented as an ATMA remote operator automated system and method. For instance, the ATMA remote operator automated system and method described herein with reference to FIG.8 can provide the ability to a remote operator to control an ATMA truck in very specific scenarios when automation holds the system due to an object within the navigation path or when it requires assist to navigate through a challenging path. Some ATMA remote operator automated embodiments include a data telemetry packet which provides kinematics, automation status, latency, and error codes to a remote operator interface. Other ATMA remote operator automated embodiments include a separate channel for DBW commands which provides the ability to independently control steering,Attorney Docket: 222204-2985 throttle, and brake with some applied thresholds for safety purposes. This data channel monitors timestamping and disables automation when a packet / timestamp is missed for more than 100 milliseconds (ms) in some examples. Some ATMA remote operator automated embodiments further include a web-based interface to show telemetry data and also video views to allow an operator to understand how to handle object avoidance or path re-routing. These embodiments use WebRTC for video broadcasting, message queuing telemetry transport (MQTT) for data exchange between the vehicle platforms and the XVIZ Server for the web data and video presentation with low latency performance.

[0076] FIG.6 illustrates a block diagram of an example the automated lead vehicle operation system 600 (or “lead vehicle system 600”) according to various aspects and embodiments of the present disclosure. The lead vehicle system 600 can be implemented in the environment 100 (FIG. 1) or the environment 500 (FIG. 5) to facilitate automated operation of a vehicle such as a TMA. For instance, the lead vehicle system 600 can be included in the lead vehicle 110 and implemented by the computing device 112 to facilitate automated operation of at least one of the follow vehicle 120 or the follow vehicle 520i as described herein with reference to FIGS.1, 2, and 5.

[0077] The lead vehicle system 600 includes a lead vehicle operation architecture 602 and several on-board perception sensors and antennas, among other components. The lead vehicle operation architecture 602 in the example shown includes a wireless link 610, a wireless cellular vehicle-to-everything (C-V2X) on-board unit (OBU) wireless link 612 (or “C-V2X OBU wireless link 612”), a graphics processing unit (GPU) platform 614 (or “GPU platform 614”), a navigation system 616, a real-time GPS system 618, an ethernet switch 620, a universal serial bus (USB) switch 622 (or “USB switch 622”), and a vehicle power supply 630. The wireless link 610 and the C-V2X OBU wireless link 612 are coupled to the ethernet switch 620, the navigation system 616 and the real-time GPS system 618 are coupled to the USB switch 622, and the GPU platform 614 is coupled to both the ethernet switch 620 and the USB switch 622. The vehicle power supply 630 is coupled to the C-V2X OBU wireless link 612 and the GPU platform 614.

[0078] The on-board perception sensors and antennas of the lead vehicle system 600 in this example include a C-V2X-GPS antenna 640, a camera 642 (e.g., a high-definition camera, a stereo camera), GNSS antennas 644, and a GNSS + L-band antenna 646. The C-V2X-GPS antenna 640 is coupled to the C-V2X OBU wireless link 612, the camera 642 is coupled to the GPU platform 614, the GNSS antennas 644 are coupled to the navigation system 616, and the GNSS + L-band antenna 646 is coupled to the real-time GPS system 618.Attorney Docket: 222204-2985

[0079] FIG. 7 illustrates a block diagram of an example automated follow vehicle operation system 700 (or “follow vehicle system 700”) according to various aspects and embodiments of the present disclosure. The follow vehicle system 700 can be implemented in the environment 100 (FIG. 1) or the environment 500 (FIG. 5) to facilitate automated operation of a vehicle such as a TMA. For instance, the follow vehicle system 700 can be included in the follow vehicle 120 and implemented by the computing device 112 to facilitate automated operation of at least one of the follow vehicle 120 or the follow vehicle 520ias described herein with reference to FIGS.1, 2, and 5.

[0080] The follow vehicle system 700 includes a follow vehicle operation architecture 702 and several on-board perception sensors and antennas, among other components. The follow vehicle operation architecture 702 in the example shown includes a wireless link 710, a wireless cellular vehicle-to-everything (C-V2X) on-board unit (OBU) wireless link 712 (or “C-V2X OBU wireless link 712”), a graphics processing unit (GPU) platform 714 (or “GPU platform 714”), a navigation system 716, a real-time GPS system 718, an ethernet switch 720, a universal serial bus (USB) switch 722 (or “USB switch 722”), a vehicle power supply 730, a light detection and ranging (LiDAR) object detection system 740, a controller area network (CAN) controller 750, a drive-by-wire (DBW) steering control system 760, a DBW throttle control system 762, a DBW brake control system 764, a vehicle CAN 766, and a wireless e-stop system 768. The wireless link 710, the C-V2X OBU wireless link 712, and the LiDAR object detection system 740 are coupled to the ethernet switch 720 in the example shown. Additionally, the navigation system 716, the real- time GPS system 718, the CAN controller 750, and the DBW steering control system 760 are coupled to the USB switch 722 in this example, and the GPU platform 714 is coupled to both the ethernet switch 720 and the USB switch 722. Also, the vehicle power supply 730 is coupled to the C-V2X OBU wireless link 712 and the GPU platform 714 in this example, and the CAN controller 750 is coupled to the DBW throttle control system 762, the DBW brake control system 764, the vehicle CAN 766, and the wireless e-stop system 768, which are all coupled to one another.

[0081] The on-board perception sensors and antennas of the follow vehicle system 700 in this example include a C-V2X-GPS antenna 740, a camera 742 (e.g., a high-definition camera, a stereo camera), GNSS antennas 744, and a GNSS + L-band antenna 746. The C-V2X-GPS antenna 740 is coupled to the C-V2X OBU wireless link 712, the camera 742 is coupled to the GPU platform 714, the GNSS antennas 744 are coupled to the navigation system 716, and the GNSS + L-band antenna 746 is coupled to the real-time GPS system 718.Attorney Docket: 222204-2985

[0082] FIG.8 illustrates a block diagram of an example automated lead-follow vehicle remote operation system 800 (or “remote operation system 800”) according to various aspects and embodiments of the present disclosure. The remote operation system 800 can be implemented in the environment 100 (FIG.1) or the environment 500 (FIG.5) to facilitate remote operation of one or more automated vehicles such as one or more automated TMAs. For instance, various components of the remote operation system 800 described below can be included in the lead vehicle 110 and at least one of the follow vehicles 120, 520, 520idescribed herein with reference to FIGS.1, 2, and 5. Such components of the remote operation system 800 can be implemented by the computing devices 112, 122, 522, 522i to facilitate remote operation of an automated vehicle such as any of the follow vehicles 120, 520, 520i.

[0083] The remote operation system 800 can be embodied and implemented as an ATMA remote operator automated system and method in many examples. For instance, the remote operation system 800 can provide the ability to a remote operator to control an ATMA truck such as any of the follow vehicles 120, 520, 520i in very specific scenarios when the computing device 112 (e.g., via the automated vehicle operation module 212) stops or holds the follow vehicle due to an object within the navigation path or when the follow vehicle requires assistance to navigate through a challenging path. The remote operation system 800 in the example shown includes a lead vehicle remote operation architecture 802a, a follow vehicle operation architecture 802b, and C-V2X-GPS antennas 840a, 840b, among other components.

[0084] The lead vehicle remote operation architecture 802a in the example shown includes a C-V2X OBU wireless link 812a, an ethernet switch 820a, a cellular router 870a, a special-purpose computing device 880a (e.g., a special-purpose GPU, CPU, and system on a chip (SoS) device or hardware), a gamepad input device 882, and a display device 884. The C-V2X OBU wireless link 812a is coupled to the C-V2X-GPS antenna 840a and further coupled (e.g., via an ethernet connection) to the ethernet switch 820a in the example shown. The ethernet switch 820a is coupled (e.g., via an ethernet connection) to the cellular router 870a and further coupled to the special- purpose computing device 880a in this example. Also, the cellular router 870a is coupled to an alternating current (AC) power supply, and the special-purpose computing device 880a is coupled (e.g., via a USB connection) to the gamepad input device 882 and further coupled (e.g., via a high- definition multimedia interface (HDMI) connection) to the display device 884 (e.g., a digital screen or monitor).Attorney Docket: 222204-2985

[0085] The follow vehicle operation architecture 802b in the example shown includes a C- V2X OBU wireless link 812b, an ethernet switch 820b, a camera 842 (e.g., a FlexDAS forward color camera), a cellular router 870b, and a special-purpose computing device 880b (e.g., a special- purpose GPU, CPU, and system on a chip (SoS) device or hardware). The C-V2X OBU wireless link 812b is coupled to the C-V2X-GPS antenna 840b and further coupled (e.g., via an ethernet connection) to the ethernet switch 820b in the example shown. The special-purpose computing device 880b is coupled (e.g., via an ethernet connection) to the cellular router 870b and further coupled (e.g., via a USB connection) to the camera 842 in this example. Also, the C-V2X OBU wireless link 812b is coupled to a direct current (DC) power supply, and the cellular router 870a and the special-purpose computing device 880b are each coupled to an alternating current (AC) power supply.

[0086] The cellular router 870a of the lead vehicle remote operation architecture 802a and the cellular router 870b of the follow vehicle operation architecture 802b are coupled to one another in this example by way of a 5G / 4G LTE Link or VPN, although another connection may be used in some cases. Additionally, the C-V2X-GPS antenna 840a and the C-V2X-GPS antenna 840b are coupled to one another in this example by way of a C-V2X IEEE802.11p secured WSM connection to facilitate communication of various SAE J2735 data, gamepad data, and video feed data between the lead vehicle remote operation architecture 802a and the follow vehicle operation architecture 802b.

[0087] The remote operation system 800 can be configured with capabilities to support remote operator control (e.g., remote operator override control) by exposing drive-by-wire (DBW) controls to a remote controller using two communication links. In the example shown, the communication links include the C-V2X link between the C-V2X-GPS antennas 840a, 840b with a maximum of 400 meters range and the 5G / 4G-LTE link between the cellular routers 870a, 870b with range longer than 400 m. The remote operation system 800 can implement a watchdog timer to keep a low latency and safe communication link between the lead vehicle remote operation architecture 802a and the follow vehicle operation architecture 802b, which can include a robotic operating system (ROS) software platform in many examples.

[0088] Some embodiments of the remote operation system 800 can further include a data telemetry packet that can provide kinematics, automation status, latency, and error codes to a remote operator interface such as the display device 884. For example, the remote operation system 800 can be configured to generate visual representation 1000 illustrated in and described withAttorney Docket: 222204-2985 reference to FIG.10. For instance, the remote operation system 800 can be configured to generate the visual representation 1000 on the display device 884 of the lead vehicle remote operation architecture 802a. Some embodiments of the remote operation system 800 can also include a web- based interface to show telemetry data and also video views to allow an operator to understand how to handle object avoidance or path re-routing. These embodiments use WebRTC for video broadcasting, message queuing telemetry transport (MQTT) for data exchange between the vehicle platforms and XVIZ Server for the web data and video presentation with low latency performance.

[0089] Other embodiments of the remote operation system 800 can also include a separate channel for DBW commands which provides the ability to independently control steering, throttle, and brake with some applied thresholds for safety purposes. This data channel monitors timestamping and disables automation when a packet / timestamp is missed for more than 100 milliseconds (ms) in some examples. For example, the remote operation system 800 can include or implement automated vehicle safety system 900 described herein with reference to FIG. 9 to monitor the health and performance of onboard computing hardware, sensors, and communications equipment of at least one of the follow vehicles 120, 520, 520i, and the lead vehicle 110 in some cases.

[0090] FIG.9 illustrates a block diagram of an example automated vehicle safety system 900 (or “safety system 900”) according to various aspects and embodiments of the present disclosure. The safety system 900 can be embodied and implemented as a redundant and fail-safe hardware- based watchdog system that can be implemented in the environment 100 to monitor all critical components of the lead vehicle 110, the follow vehicle 120, the computing devices 112, 122, on- board perception sensors of the lead vehicle 110 and the follow vehicle 120, and the networks 140. The safety system 900 can be embodied and implemented in the environment 100 as a fallback programmable logic controller (PLC) system configured to monitor the health and performance of onboard computing hardware, sensors, and communications equipment of the follow vehicle 120 in many examples, and the lead vehicle 110 in some cases.

[0091] The safety system 900 in the example shown includes a PLC safety device 910, an ROS system 940, a data acquisition device or hardware 960, a DBW safety relay 970, a brake air pressure regulator 980, and a throttle electronic control unit (ECU) input 990. The PLC safety device 910 in this example includes a socket engine 916 that receives ROS system heartbeat signals 942 and ROS automation heartbeat signals and DBW commands 944 from the ROS system 940. The PLC safety device 910 also receives analog input signals 918 from the DAQ device orAttorney Docket: 222204-2985 hardware 960, which generates the analog input signals 918 based at least in part on receiving DBW brake control signals 946 and DBW throttle control signals 948 from the ROS system 940. The PLC safety device 910 further receives power monitoring digital input signals 912 and e-stop monitoring digital input signals 914. The PLC safety device 910 routes the analog input signals 918 to a PLC logic engine 920 included in the PLC safety device 910 in this example. The PLC safety device 910 also includes an inertial measurement unit (IMU) 928 that it can use as a secondary IMU to validate requests from the ROS system 940 or the DAQ device or hardware 960. Based on analyzing signals from the socket engine 916 and the analog input signals 918, the PLC logic engine 920 can generate and route analog output signals 922 to the DBW safety relay 970. The PLC logic engine 920 can also generate digital output relay control signals 924 and digital output relay control signals 926. In case of an error during operation, the DBW safety relay 970 can be operable to pass specific PLC signals (e.g., the analog output signals 922) to the brake air pressure regulator 980 and the throttle ECU input 990 to apply full braking and zero throttle, respectively, to avoid any additional errors.

[0092] FIG.9 illustrates an example high level integration of the PLC safety device 910 and the ROS system 940, as well as the signals that can be controlled by each engine while the PLC safety device 910 takes the primary priority to provide a specific DBW command in case of an error, misbehavior, sensor error or total software failure. During operation, the ROS system 940 can control the DBW brake control signals 946 and the DBW throttle control signals 948 using the DAQ device or hardware 960 (e.g., a LabJack T7) which can generate certain voltages to bring the follow vehicle 120, 520, or 520ito specific accelerations or speeds. These signals can be monitored by the PLC safety device 910 which uses the IMU 928 as a secondary IMU to validate software requests based on acceleration values and determine if the follow vehicle 120, 520, or 520iis responding properly to those requests. The DBW safety relay 970 can be operable to allow the ROS signals (e.g., the DBW brake control signals 946 and the DBW throttle control signals 948) received from the DAQ device or hardware 960 to pass to the brake air pressure regulator 980 and the throttle ECU throttle input 990 during normal operation. In case of an error, the DBW safety relay 970 can be operable to pass specific PLC signals to the brake air pressure regulator 980 and the throttle ECU input 990 to apply full braking and zero throttle, respectively, as a way to avoid any additional misbehavior from the DBW system.

[0093] The PLC-based safety system 900 can reliably detect and communicate errors to a lead vehicle human machine interface (HMI) and, at minimum, safely arrest the follow vehicle 120,Attorney Docket: 222204-2985 520, or 520i should a critical error be detected. The safety system 900 can bring the follow vehicle 120, 520, or 520ito a stop without communicating through software (e.g., the automated vehicle operation module 212) on the computing device 112 and it can do so in the case of power failure as well.

[0094] FIG.10 illustrates an example visual representation 1000 according to various aspects and embodiments of the present disclosure. The computing device 112 (e.g., via the automated vehicle operation module 212) can generate the visual representation 1000 such that it includes a multi-view visual representation 1300, a virtual dashboard 1350, and a human-machine-interface 1400 (or “HMI 1400”) as illustrated in FIG.10.

[0095] The computing device 112 can generate the multi-view visual representation 1300 portion of the visual representation 1000 in the example shown using image data 1310 captured by one or more of the perception sensors 218 (e.g., a camera) included in or coupled to the lead vehicle 110. The computing device 112 can generate the multi-view visual representation 1300 in some examples such that it includes the virtual dashboard 1350 and one or more camera or video views such as a forward view 1312 and a rear view 1314 as shown in FIG. 10. The virtual dashboard 1350 can include displays of critical truck gauges such as speed, fuel, lights, revolutions per minute (RPM), gear position, and other data.

[0096] The computing device 112 can generate the HMI 1400 portion of the visual representation 1000 such that it includes interactive operational inputs corresponding to vehicular operations associated with at least one of the lead vehicle 110 or any of the follow vehicles 120, 520, 520i. The computing device 112 can generate the HMI 1400 in the example shown such that it includes at least one of a visual representation 1410 of the lead vehicle 110, a speed indicator 1412 for the lead vehicle 110, a visual representation 1420 of the follow vehicle 120, a speed indicator 1422 for the follow vehicle 120, a lateral setting input and indicator 1430 for the follow vehicle 120, a longitudinal setting input and indicator 1440 for the follow vehicle 120, a lead hold input 1450 for stopping the follow vehicle 120 at a location where the lead vehicle 110 was positioned when the lead hold input 1450 was engaged, a following hold input 1460 for the follow vehicle 120, and a stop input 1470 for the follow vehicle 120. The computing device 112 can cause the follow vehicle 120 to perform one or more vehicular operations in some examples based on input received by way of the above-described interactive operational inputs of the HMI 1400.

[0097] The visual representation 1000 including the multi-view visual representation 1300, the virtual dashboard 1350, and the HMI 1400 allows an operator of the lead vehicle 110 to haveAttorney Docket: 222204-2985 a better sense of the surroundings while monitoring the ATMA functionality and determine any potential scenario when a hold or stop event should be triggered or automation should be immediately disengaged.

[0098] Referring now to FIG. 2, an executable program can be stored in any portion or component of the memory 206 including, for example, a random-access memory (RAM), read- only memory (ROM), magnetic or other hard disk drive, solid-state, semiconductor, Universal Serial Bus (USB) flash drive, memory card, optical disc (e.g., compact disc (CD) or digital versatile disc (DVD)), floppy disk, magnetic tape, or other types of memory devices.

[0099] In various embodiments, the memory 206 can include both volatile and nonvolatile memory and data storage components. Volatile components are those that do not retain data values upon loss of power. Nonvolatile components are those that retain data upon a loss of power. Thus, the memory 206 can include, for example, a RAM, ROM, magnetic or other hard disk drive, solid- state, semiconductor, or similar drive, USB flash drive, memory card accessed via a memory card reader, floppy disk accessed via an associated floppy disk drive, optical disc accessed via an optical disc drive, magnetic tape accessed via an appropriate tape drive, and / or other memory component, or any combination thereof. In addition, the RAM can include, for example, a static random-access memory (SRAM), dynamic random-access memory (DRAM), or magnetoresistive random-access memory (MRAM), and / or other similar memory device. The ROM can include, for example, a programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other similar memory device.

[0100] As discussed above, the automated vehicle operation module 212, the communications stack 214, and the control module 216 can each be embodied, at least in part, by software or executable-code components for execution by general purpose hardware. Alternatively, the same can be embodied in dedicated hardware or a combination of software, general, specific, and / or dedicated purpose hardware. If embodied in such hardware, each can be implemented as a circuit or state machine, for example, that employs any one of or a combination of a number of technologies. These technologies can include, but are not limited to, discrete logic circuits having logic gates for implementing various logic functions upon an application of one or more data signals, application specific integrated circuits (ASICs) having appropriate logic gates, field- programmable gate arrays (FPGAs), or other components.Attorney Docket: 222204-2985

[0101] Any flowchart or process diagram shown in any figure of the present disclosure is representative of certain processes, functionality, and operations of the embodiments discussed herein. Each block can represent one or a combination of steps or executions in a process. Alternatively, or additionally, each block can represent a module, segment, or portion of code that includes program instructions to implement the specified logical function(s). The program instructions can be embodied in the form of source code that includes human-readable statements written in a programming language or machine code that includes numerical instructions recognizable by a suitable execution system such as the processor 204. The machine code can be converted from the source code. Further, each block can represent, or be connected with, a circuit or a number of interconnected circuits to implement a certain logical function or process step.

[0102] Although a flowchart or process diagram illustrated in a figure of the present disclosure may include a specific order, it is understood that the order can differ from that which is depicted. For example, an order of execution of two or more blocks can be scrambled relative to the order shown. Also, two or more blocks shown in succession can be executed concurrently or with partial concurrence. Further, in some embodiments, one or more of the blocks can be skipped or omitted. In addition, any number of counters, state variables, warning semaphores, or messages might be added to the logical flow described herein, for purposes of enhanced utility, accounting, performance measurement, or providing troubleshooting aids. Such variations, as understood for implementing the process consistent with the concepts described herein, are within the scope of the embodiments.

[0103] Also, any logic or application described herein, including the automated vehicle operation module 212, the communications stack 214, and the control module 216 can be embodied, at least in part, by software or executable-code components, can be embodied or stored in any tangible or non-transitory computer-readable medium or device for execution by an instruction execution system such as a general-purpose processor. In this sense, the logic can be embodied as, for example, software or executable-code components that can be fetched from the computer-readable medium and executed by the instruction execution system. Thus, the instruction execution system can be directed by execution of the instructions to perform certain processes described herein and / or illustrated in a flowchart or process diagram of a figure of the present disclosure. In the context of the present disclosure, a non-transitory computer-readable medium can be any tangible medium that can contain, store, or maintain any logic, application,Attorney Docket: 222204-2985 software, or executable-code component described herein for use by or in connection with an instruction execution system.

[0104] The computer-readable medium can include any physical media such as, for example, magnetic, optical, or semiconductor media. More specific examples of suitable computer-readable media include, but are not limited to, magnetic tapes, magnetic floppy diskettes, magnetic hard drives, memory cards, solid-state drives, USB flash drives, or optical discs. Also, the computer- readable medium can include a RAM including, for example, an SRAM, DRAM, or MRAM. In addition, the computer-readable medium can include a ROM, a PROM, an EPROM, an EEPROM, or other similar memory device.

[0105] Disjunctive language, such as the phrase “at least one of X, Y, or Z,” unless specifically stated otherwise, is to be understood with the context as used in general to present that an item, term, or the like, can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is not generally intended to, and should not, imply that certain embodiments require at least one of X, at least one of Y, or at least one of Z to be each present. As referenced herein in the context of quantity, the terms “a” or “an” are intended to mean “at least one” and are not intended to imply “one and only one.”

[0106] As referred to herein, the terms “includes” and “including” are intended to be inclusive in a manner similar to the term “comprising.” As referenced herein, the terms “or” and “and / or” are generally intended to be inclusive, that is (i.e.), “A or B” or “A and / or B” are each intended to mean “A or B or both.” As referred to herein, the terms “first,” “second,” “third,” and so on, can be used interchangeably to distinguish one component or entity from another and are not intended to signify location, functionality, or importance of the individual components or entities. As referenced herein, the terms “couple,” “couples,” “coupled,” and / or “coupling” refer to chemical coupling (e.g., chemical bonding), communicative coupling, electrical and / or electromagnetic coupling (e.g., capacitive coupling, inductive coupling, direct and / or connected coupling), mechanical coupling, operative coupling, optical coupling, and / or physical coupling.

[0107] It should be emphasized that the above-described embodiments of the present disclosure are merely possible examples of implementations set forth for a clear understanding of the principles of the disclosure. Many variations and modifications can be made to the above- described embodiment(s) without departing substantially from the spirit and principles of the disclosure. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.

Claims

Attorney Docket: 222204-2985 CLAIMS Therefore, at least the following is claimed:

1. A method of automated vehicle operation, the method comprising: determining, by at least one computing device, a first pixel distance of pixels in first image data that correspond to a lateral distance between a path marker partly defining a path and a centerline of a first vehicle at a defined location on the path, the first image data being captured by a first camera coupled to the first vehicle; calculating, by the at least one computing device, a second pixel distance of pixels in second image data that correspond to the lateral distance, the second image data being captured by a second camera coupled to a second vehicle on the path, the second pixel distance of pixels being calculated based on the first pixel distance of pixels; and causing, by the at least one computing device, a centerline of the second vehicle to be positioned at the lateral distance from the path marker at the defined location on the path based on the second pixel distance of pixels in the second image data.

2. The method of claim 1, further comprising: determining, by the at least one computing device, a third pixel distance of pixels in the first image data that correspond to a second lateral distance between the centerline of the first vehicle at the defined location on the path and a second path marker partly defining the path; and calculating, by the at least one computing device, a fourth pixel distance of pixels in the second image data that correspond to the second lateral distance, the fourth pixel distance being calculated based on the third pixel distance of pixels in the first image data.

3. The method of claim 1, further comprising: determining, by the at least one computing device, a third pixel distance of pixels in the first image data that correspond to a second lateral distance between the centerline of the first vehicle at the defined location on the path and a second path marker partly defining the path; calculating, by the at least one computing device, a centerline distance ratio corresponding to the first vehicle, the centerline distance ratio comprising the first pixel distance of pixels and the third pixel distance of pixels.Attorney Docket: 222204-2985 4. The method of claim 3, further comprising: calculating, by the at least one computing device, a fourth pixel distance of pixels in the second image data that correspond to the second lateral distance, at least one of the second pixel distance or the fourth pixel distance being calculated using the centerline distance ratio corresponding to the first vehicle.

5. The method of claim 3, further comprising: calculating, by the at least one computing device, a fourth pixel distance of pixels in the second image data that correspond to the second lateral distance, at least one of the second pixel distance or the fourth pixel distance being calculated using the centerline distance ratio corresponding to the first vehicle and being based on at least one of: vehicle specifications of at least one of the first vehicle or the second vehicle, camera specifications of at least one of the first camera or the second camera, camera pose of at least one of the first camera or the second camera, or camera location of at least one of the first camera on the first vehicle or the second camera on the second vehicle.

6. The method of claim 3, further comprising: generating, by the at least one computing device, annotated visual representation of the first image data, the annotated visual representation comprising annotations of at least one of a centerline of the path marker, a centerline of the second path marker, the centerline of the first vehicle, a horizontal reference line, the centerline distance ratio, the first pixel distance of pixels in the first image data, the third pixel distance of pixels in the first image data, or additional pixel distances of pixels in the first image data that correspond to additional lateral distances between the centerline of the first vehicle and centerlines of additional path markers.

7. The method of claim 3, further comprising: generating, by the at least one computing device, a first centerline distance ratio line on a visual representation of the first image data between the path marker and the centerline of the first vehicle at the defined location on the path, the first centerline distance ratio line corresponding to the first pixel distance of pixels in the first image data and the lateral distance; andAttorney Docket: 222204-2985 generating, by the at least one computing device, a second centerline distance ratio line on the visual representation of the first image data between the second path marker and the centerline of the first vehicle at the defined location on the path, the second centerline distance ratio line corresponding to the third pixel distance of pixels in the first image data and the second lateral distance, wherein the first centerline distance ratio line and the second centerline distance ratio line visually represent the first pixel distance of pixels and the second pixel distance of pixels, respectively, and the centerline distance ratio, collectively.

8. The method of claim 1, further comprising: causing, by the at least one computing device, the second vehicle to travel at a defined speed on the path.

9. The method of claim 8, wherein the defined speed ranges up to 45 miles per hour.

10. The method of claim 1, further comprising: causing, by the at least one computing device, the second vehicle to be positioned at and maintain a defined distance from the first vehicle on the path, the defined distance being at least one of a defined lateral distance between the centerline of the first vehicle and the centerline of the second vehicle at a future point in space or a defined longitudinal distance between a back portion of the first vehicle and a front portion of the second vehicle.

11. The method of claim 10, wherein the defined lateral distance ranges up to 12 feet and the defined longitudinal distance ranges up to 400 feet.

12. The method of claim 1, further comprising: generating, by the at least one computing device, a human-machine interface comprising interactive operational inputs corresponding to vehicular operations associated with at least one of the first vehicle or the second vehicle; and causing, by the at least one computing device, the second vehicle to perform one or more of the vehicular operations based on input received by way of the interactive operational inputs of the human-machine interface and manual operation of the first vehicle.Attorney Docket: 222204-2985 13. The method of claim 12, wherein the interactive operational inputs comprise at least one of a lateral setting input for the second vehicle, a longitudinal setting input for the second vehicle, a hold input for the first vehicle, a hold input for the second vehicle, or a stop input for one or more or the first vehicle or the second vehicle.

14. The method of claim 1, further comprising: implementing, by the at least one computing device, a machine vision process to concurrently track a centerline of the path marker relative to the centerline of the first vehicle and the centerline of the second vehicle using the first image data and the second image data, respectively.

15. A method of automated vehicle operation, the method comprising: determining, by the at least one computing device, a first pixel distance of pixels in first image data that correspond to a first lateral distance between a path marker partly defining a path and a centerline of a first vehicle at a defined location on the path, the first image data being captured by a first camera coupled to the first vehicle; calculating, by the at least one computing device, a second pixel distance of pixels in second image data that correspond to a second lateral distance between the path marker and a centerline of a second vehicle at the defined location on the path, the second image data being captured by a second camera coupled to the second vehicle, the second pixel distance of pixels being calculated based on the first pixel distance of pixels; and causing, by the at least one computing device, the centerline of the second vehicle to be positioned at the second lateral distance from the path marker at the defined location on the path based on the second pixel distance of pixels in the second image data, the second lateral distance being different from the first lateral distance.

16. The method of claim 15, further comprising: determining, by the at least one computing device, a third pixel distance of pixels in the first image data that correspond to a third lateral distance between the centerline of the first vehicle at the defined location on the path and a second path marker partly defining the path; andAttorney Docket: 222204-2985 determining, by the at least one computing device, a fourth pixel distance of pixels in the second image data that correspond to a fourth lateral distance between the centerline of the second vehicle at the defined location on the path and the second path marker, the fourth pixel distance of pixels being calculated based on the third pixel distance of pixels, and the fourth lateral distance being different from the third lateral distance.

17. The method of claim 15, further comprising: determining, by the at least one computing device, a third pixel distance of pixels in the first image data that correspond to a third lateral distance between the centerline of the first vehicle at the defined location on the path and a second path marker partly defining the path; calculating, by the at least one computing device, a centerline distance ratio corresponding to the first vehicle, the centerline distance ratio comprising the first pixel distance and the third pixel distance.

18. The method of claim 17, further comprising: calculating, by the at least one computing device, a fourth pixel distance of pixels in the second image data that correspond to a fourth lateral distance between the centerline of the second vehicle at the defined location on the path and the second path marker, the fourth pixel distance being calculated using the centerline distance ratio corresponding to the first vehicle, and the fourth lateral distance being different from the third lateral distance.

19. The method of claim 17, further comprising: calculating, by the at least one computing device, a fourth pixel distance of pixels in the second image data that correspond to a fourth lateral distance between the centerline of the second vehicle at the defined location on the path and the second path marker, the fourth lateral distance being different from the third lateral distance, and the fourth pixel distance being calculated using the centerline distance ratio corresponding to the first vehicle and being based on at least one of: vehicle specifications of at least one of the first vehicle or the second vehicle, camera specifications of at least one of the first camera or the second camera, camera pose of at least one of the first camera or the second camera, or camera location of at least one of the first camera on the first vehicle or the second camera on the second vehicle.Attorney Docket: 222204-2985 20. A method of automated vehicle operation, the method comprising: tracking, by at least one computing device, path markers relative to a centerline of a first vehicle and a centerline of a second vehicle, the path markers defining a path; determining, by the at least one computing device, centerline distance ratios comparing respective distances between the centerline of the first vehicle and each of the path markers at a defined location on the path; and causing, by the at least one computing device, the centerline of the second vehicle to be positioned at a defined lateral distance from one of the path markers at the defined location on the path based on the centerline distance ratios.

21. A method of automated vehicle operation, the method comprising: determining, by at least one computing device, a first pixel distance of pixels in first image data that correspond to a first lateral distance between a path marker partly defining a path and a centerline of a first vehicle at a defined location on the path, the first image data being captured by a first camera coupled to the first vehicle; determining, by at least one computing device, a second pixel distance of pixels in second image data that correspond to a second lateral distance between a path marker partly defining a path and a centerline of a second vehicle at a defined location on the path, the second image data being captured by a second camera coupled to the second vehicle; calculating, by the at least one computing device, a third pixel distance of pixels in a third image data that correspond to a third lateral distance, the third image data being captured by a third camera coupled to a third vehicle on the path, the third pixel distance of pixels being calculated based on the first or second pixel distance of pixels; and causing, by the at least one computing device, a centerline of the third vehicle to be positioned at the first or second lateral distance from the path marker at the defined location on the path based on the third pixel distance of pixels in the third image data.

22. The method of claim 21, further comprising: calculating, by the at least one computing device, a fourth pixel distance of pixels in a fourth image data that correspond to a fourth lateral distance, the fourth image data being capturedAttorney Docket: 222204-2985 by a fourth camera coupled to a fourth vehicle on the path, the fourth pixel distance of pixels being calculated based on the first, second, or third pixel distance of pixels; and causing, by the at least one computing device, a centerline of the fourth vehicle to be positioned at the first, second, or third lateral distance from the path marker at the defined location on the path based on the fourth pixel distance of pixels in the fourth image data.

23. A method of automated vehicle operation, the method comprising: determining, by at least one computing device, localizing information of a first vehicle based on image data captured by a camera coupled to the first vehicle; and controlling, by the at least one computing device, a lateral position of a second vehicle relative to a centerline of the first vehicle based on the localizing information of the first vehicle.

Images