Systems and methods for distributed visualization generation

Abstract

The present invention relates to a visualization generation system, comprising: a memory configured to store data captured by at least first sensors of a vehicle; and a processor, wherein the processor is configured to receive a request to generate a visualization of the vehicle's operation, determine a number of worker nodes needed to generate the visualization, prepare the number of worker nodes, provide instructions to each worker node to cause each worker node to generate a portion of the visualization, reassemble the portions of the visualization into a final visualization, and deliver the final visualization to at least a first user.

Description

[Technical Field]

[0001] Autonomous or semi-autonomous vehicles rely on substantial amounts of sensor data and other vehicle data to understand the state of the road, the vehicle's condition, and the environment around the vehicle. After such a vehicle completes a trip, engineers, analysts, and other users often need to replay a visualization of a portion of the trip. These visualizations often include several different types of data from sensors, imaging devices, vehicle systems, etc., and it is difficult to generate a video that includes all of this data. Summary of the Invention [Means for solving the problem]

[0002] Systems, methods, and computer program code are provided for distributed visualization generation of data associated with portions of a journey of an autonomous or semi-autonomous vehicle. Some embodiments include a memory configured to store data associated with vehicle operation, including data captured by at least a first sensor of the vehicle, and a processor, the processor configured to: receive a request to generate a visualization of the vehicle operation, the request including a start timestamp and an end timestamp and further including information identifying a user interface configuration; determine a number of worker nodes needed to generate the visualization; provision the number of worker nodes, provide instructions to each worker node to cause each worker node to generate a portion of the visualization; reassemble the portions of the visualization into a final visualization; and deliver the visualization to at least a first user. [Brief explanation of the drawings]

[0003] The features and advantages of the illustrative embodiments, and the manner in which they are achieved, will become more readily apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0004] [Figure 1]FIG. 1 illustrates an example of a communication environment in which a semi-truck may operate, according to some embodiments. [Figure 2] FIG. 1 illustrates a system according to some embodiments. [Figure 3] FIG. 1 is a flow diagram illustrating a process according to some embodiments. [Figure 4] FIG. 10 is a flow diagram illustrating a further process according to some embodiments. [Figure 5A] 10A-10C illustrate user interfaces that can be used to generate distributed visualizations according to some embodiments. [Figure 5B] 10A-10C illustrate user interfaces that can be used to generate distributed visualizations according to some embodiments. [Figure 6] 7A-7C illustrate a control system that may be deployed on a vehicle such as the semi-truck depicted in FIGS. 7A-7C, according to an exemplary embodiment. [Figure 7A] FIG. 1 illustrates an exterior view of a semi-truck that may be used in accordance with an exemplary embodiment. [Figure 7B] FIG. 1 illustrates an exterior view of a semi-truck that may be used in accordance with an exemplary embodiment. [Figure 7C] FIG. 1 illustrates an exterior view of a semi-truck that may be used in accordance with an exemplary embodiment.

[0005] Throughout the drawings and detailed description, the same drawing reference numbers should be understood to refer to the same elements, features, and structures unless otherwise stated. The relative size and depiction of these elements may be exaggerated or adjusted for clarity, illustration, and / or convenience. DETAILED DESCRIPTION OF THE INVENTION

[0006] In the following description, specific details are set forth to provide a thorough understanding of various exemplary embodiments. It should be understood that various modifications to the embodiments will be readily apparent to those skilled in the art, and that the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Moreover, in the following description, numerous details are set forth for purposes of explanation. However, those skilled in the art should understand that the embodiments may be practiced without the use of these specific details. In other instances, well-known structures and processes are not shown or described so as not to obscure the description with unnecessary detail. Thus, the present disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0007] For convenience and ease of description, certain terminology is used herein. For example, the term "semi-truck" is used to refer to a vehicle in which the system of the exemplary embodiments may be used. The terms "semi-truck," "truck," "tractor," "vehicle," and "semi" may be used interchangeably herein. Additionally, as will be apparent to those skilled in the art upon reading this disclosure, embodiments of the present invention may be used in conjunction with other types of vehicles. In general, embodiments may be used with desirable results in conjunction with any vehicle that tows a trailer or carries cargo over long distances.

[0008] Features of some embodiments will be described first by referring to FIG. 1 , which depicts features of a system according to some embodiments. As shown in FIG. 1 , the system 100 may include one or more semi-trucks 102a-n that communicate with a remote monitoring system 140 via one or more networks 110, 120, 130. One or more user devices 150 may also communicate with the remote monitoring system 140 (and, in some embodiments, with one or more semi-trucks 102) via one or more networks 110, 120, 130. One or more of the user devices 150 may also communicate with a visualization system 160 to interact with data from the semi-trucks 102 after the completion of a trip or route. According to some embodiments, the semi-trucks 102 are autonomous or semi-autonomous vehicles that capture and generate a large amount of data as the semi-trucks 102 operate during a trip or route. For example, an autonomous or semi-autonomous vehicle may generate, capture, or otherwise produce vehicle diagnostic or operational data, image data (including video, camera images, and lidar data), position and navigation data, trajectory and position data, object detection data, etc.

[0009] In some embodiments, some or all of this data is transmitted or uploaded to a data store (not shown in FIG. 1 ) for use in performing analysis of the data after the semi-truck 102 completes the trip (data collected for a particular trip of a particular semi-truck 102 is generally referred to herein as “trip data”). According to some embodiments, a user operating a user device 150 can interact with the trip data via a visualization system 160. In some embodiments, the visualization system 160 can be implemented as a web server that allows a user to view trip data associated with one or more vehicles via a web browser interface, such as the interface shown and described below in conjunction with FIGS. 5A and 5B . According to some embodiments, a user operating a user device 150 that interacts with the visualization system 160 can view or interact with short segments of the trip (e.g., a 20-second segment of trip data). In some situations, it is desirable to create a view of a longer trip (e.g., a selected time of the trip). Embodiments enable the generation of longer segments and automatically produce a distributed visualization that includes several short segments from a variety of data sources.

[0010] Features of such automatic generation of distributed visualizations are described with reference to FIG. 2 (depicting a system for automatic creation of distributed visualizations according to some embodiments that may be used in conjunction with a system such as the system of FIG. 1), FIGS. 3-4 (depicting a method of operating the system of FIG. 2), FIGS. 5A-5B (depicting user interfaces for initiating the automated process of the present invention that may be presented to a user operating a user device such as user device 150 of FIG. 1), FIG. 6 (depicting control systems and sensors that collect data associated with a traveling semi-truck 102 that may be visualized and converted into a distributed visualization according to the present invention), and FIGS. 7A-7C (depicting selected components of a semi-truck 102 according to some embodiments).

[0011] To introduce features of some embodiments, an illustrative example is provided here. In the illustrative example, several semi-trucks are in operation, each traveling. During the trip, each semi-truck generates a large amount of data (including vehicle diagnostic data, image data, sensor data, etc.). At the end of the trip, the data is uploaded to a data store for use in evaluating or analyzing the performance of the traveling vehicle and its systems. Some users may interact with short segments of data to analyze specific moments in time of a selected trip of a selected semi-truck. These analyses may be performed by the user operating a web browser and visualization service on the user device. If the user wants to create a visualization of a longer period (e.g., a trip of several minutes or hours), the user may request that the distributed visualization service be invoked to automatically generate the visualization. According to some embodiments, the user need only identify the desired start and end timestamps of the selected vehicle's trip, and the distributed visualization service will generate the visualization. In some embodiments, the user may also specify a desired service level at which the visualization is to be created (e.g., by specifying when the visualization needs to be completed). The distributed visualization service uses these inputs to determine the amount of resources needed to produce the requested visualization within specified time constraints, and automatically provisions, deploys, and manages the resources to generate the visualization. In this way, embodiments enable efficient production of visualizations. Once the visualization is produced, it is made available to users and the resources used to produce the visualization are deactivated, thereby allowing the distributed visualization service to scale up and down as needed to save costs and resources.

[0012] Referring now to FIG. 2 , a system diagram depicting a distributed visualization generation system 200 is shown, according to some embodiments. As shown, the system 200 includes components or modules that interact to enable the generation of distributed visualizations of data from vehicle systems 220 from one or more semi-trucks 102 (not shown in FIG. 2 ). In some embodiments, the data from the vehicle systems 220 is provided to a data store 240 for storage and later retrieval for viewing by users operating user devices 210 via a visualization service 250. The data store 240 may be, for example, a data warehouse or a data lake. The data store 240 may include different data storage components. For example, some of the data from the vehicle systems 220 may be stored in one or more relational, time-series, or noSQL databases, while image data may be stored in an object data store. Additionally, visualizations produced by distributed visualization service 260 may be stored in object data storage data store 240 (e.g., the Simple Storage Service "S3" object store offered by Amazon Web Services or other similar data stores).

[0013] In some embodiments, data from vehicle systems 220 may be uploaded to data store 240 in one or more batches after a trip is completed. In other embodiments, some or all of the data may be streamed or otherwise uploaded to data store 240 while a trip is being performed. For brevity and ease of explanation, the embodiments will be described assuming that all of the relevant data from vehicle systems 220 is uploaded to data store 240 in batches after a trip is completed by a semi-truck 102. Furthermore, data from each semi-truck 102 is associated with information identifying the particular semi-truck 102 (e.g., using a vehicle identifier) ​​and the particular trip (e.g., using a trip identifier).

[0014] Generally, the vehicle system 220 includes components associated with a telemetry system 222, sensors 224, and other vehicle systems 226. The vehicle system 220 may also include several other components, including those shown and described below in conjunction with FIG. 6. For purposes of illustrating components associated with generating a distributed visualization according to some embodiments, only selected components are shown in FIG. 2. For example, while FIG. 2 shows only a single vehicle system 220, in a practical application, multiple vehicle systems 220 would provide data to the data store 240 for use in the system of the present invention. In a practical application, multiple vehicle systems 220 and user devices 210, 212 would be provided.

[0015] Telemetry system 222 may include systems or components that produce, modify, or otherwise generate telemetry data associated with the location, orientation, and movement of semi-truck 102 while traveling. Sensors 224 may include systems or components that produce, modify, or otherwise generate data about semi-truck 102, objects around or on semi-truck 102 as the vehicle travels. For example, sensor 224 data may include images obtained from lidar, cameras, or other sensors mounted in or on semi-truck 102. Other vehicle systems 226 may produce or otherwise generate data associated with the operation of semi-truck 102 while traveling (e.g., engine operating data, weight, vehicle operating modes, diagnostic data, etc.). Some or all of this data from vehicle systems 220 may be stored with a timestamp so that a view of the vehicle's current state and operation at any point during travel can be obtained.

[0016] The system 200 includes one or more user devices 210, 212 that can be operated by a user to interact with data from the data store 240. For example, some user devices 210 may interact with data from the data store 240 through one or more visualization services 250. In such interactions, the user may be presented with a user interface, such as the user interfaces of FIGS. 5A and 5B, which allow the user to select a desired vehicle and a desired trip to view. Once the user selects a vehicle and trip, the user may configure one or more tiles of the user interface to display different types of data associated with the vehicle and trip. The user may then select a timestamp (or other location) during the trip to view data associated with a portion of the selected trip. Due to web browser rendering limitations and large data volume sizes, users may be limited to viewing short (e.g., 20-second) slices of data. According to some embodiments, the user may further interact with user device 210 to specify a longer time period and cause distributed visualization service 260 to generate a visualization of that longer time period (and, e.g., automatically create a video of that longer time period for later viewing). The visualization produced by distributed visualization service 260 may be stored in a location in data store 240, allowing other users (e.g., users operating user device 212) to view the video. In this manner, embodiments enable the automated creation of distributed visualizations containing large amounts of data without requiring users to manually stitch screen recordings of the data.

[0017] In some embodiments, the distributed visualization service 260 is invoked on demand (e.g., when a user operating a user device 210 requests the creation of a distributed visualization of a selected period of a trip for a particular vehicle). According to some embodiments, the distributed visualization service 260 uses one or more worker nodes 264 that are created on demand (e.g., when a user operating a user device 210 requests the creation of a distributed visualization). The master node 262 may be used to coordinate the visualization tasks performed by each of the worker nodes 264. For example, if a user operating a user device 210 requests that a two-minute visualization of a portion of a trip by a vehicle be generated, the master node 262 may divide the job into six 20-second tasks and distribute those tasks among one or more worker nodes 264. For example, the master node 262 may divide the job into individual tasks based on timestamps (a first task may be to generate a visualization from a start timestamp up to 20 seconds, a second task may be to generate a visualization from 20 seconds to 40 seconds, etc.). In some embodiments, the number of worker nodes 264 prepared to handle a job may depend on the speed at which the user requires the visualization to be returned (the "service level" or "SLA"). As a simple example, if the SLA is that the user requires the visualization to be completed in under one minute, the master node 262 may distribute the job among six worker nodes 264 (assuming each worker node requires at least 20 seconds to complete its assigned task).

[0018] The master node 262 (or other application software associated with the distributed visualization service 260) can calculate the number of tasks and the amount of computing resources needed to handle the tasks within the SLA and ensure that an appropriate number of worker nodes 264 are available to complete the job within the SLA. In some embodiments, the master node 262 is configured to coordinate all tasks performed in a distributed manner. Each worker node 264 is provided with data with instructions regarding the particular task to be performed. For example, a first worker node 264a may be instructed to create a visualization of a portion of a particular vehicle's run starting at a particular timestamp and ending at a particular timestamp (e.g., 20 seconds later). The worker node 264a is provided with start and end timestamps and information identifying the particular run and vehicle. The worker node 264a is comprised of a data server 266a, a tile server 270a, and a web API 268a. These components are configured to retrieve particular items of data from the data store 240 associated with the timestamp, run, and vehicle.

[0019] For example, data server 266a retrieves vehicle and telemetry data from a specified vehicle, trip, and timestamp. Tile server 270a retrieves vehicle location and map data from a specified vehicle, trip, and timestamp. Web API 268a retrieves images and other data for virtual display in a headless browser associated with worker node 268a. Together, the data from data server 266a, tile server 270a, and web API 268a is virtually displayed or played in the headless web browser for the duration of the task (e.g., from the task start timestamp to the task end timestamp) to render a visualization for that task. Upon completion of the task, worker node 264a generates a visualization file (e.g., in a format such as .mp4).

[0020] In some embodiments, one or more worker nodes 264 are responsible for compiling a final visualization that includes some or all of the visualizations generated during a job. For example, in some embodiments, worker node 264a may be operated to compile a visualization that includes all of the tasks performed by that worker node 264a during a job and then provide that composite visualization to another worker node 264 that may be tasked with adding its visualizations to the composite visualization. In this manner, a sequence of composite visualizations may be stitched together to form a final visualization file from all of the worker nodes 264. In some embodiments, master node 262 (or other code associated with distributed visualization service 260) may be tasked with compiling the final visualization file. According to some embodiments, master node 262 and worker nodes 264 may be Apache Spark nodes. Master node 262 and worker nodes 264 may be implemented, for example, as Java applications. In some embodiments, the master node 262 is configured to receive requests (e.g., jobs) from the visualization service 250 (as described below in conjunction with FIG. 4). The master node 262 is configured to break the job down into smaller chunks (tasks) and distribute the work to various worker nodes 264. In some embodiments, once a worker 264 completes its portion of the job, it returns the results to the master 262.

[0021] In some embodiments, once all of the worker nodes 264 have returned their results, the master node 262 compiles all of the worker results and stores the final resulting visualization file in the data store 240 (e.g., in an object store). In some embodiments, the resulting visualization file may be stored in some other accessible location. In some embodiments, the master node 262 may be further responsible for ensuring that a notification is sent to the user who originally requested the visualization (and / or any other users registered to receive the visualization file). In some embodiments, the notification may include a link to access or retrieve the visualization file. The master node 262 may be further responsible for naming the visualization file according to a file naming convention (e.g., indicating the run, vehicle, and start and end timestamps). As discussed above, instead of the master node 262 performing such final processing, the worker node 264 may be responsible for generating the final visualization file, naming the file, storing the file, and notifying the user.

[0022] FIG. 3 illustrates a process 300 for generating a distributed visualization according to some embodiments of the present invention. Process 300 may be implemented using a distributed visualization system, such as system 200 of FIG. 2, for example. The flowcharts described herein do not imply a fixed order of steps, and embodiments of the present invention may be practiced in any order that is practicable. Note that any of the methods or processes described herein may be implemented by hardware, software, or any combination thereof. For example, a computer-readable storage medium may store instructions that, when executed by a machine or processor, result in performance according to any of the embodiments described herein.

[0023] Process 300 begins at 302, when distributed visualization service 260 receives a visualization request that includes information identifying a selected segment of a trip by a particular vehicle. This information may be received from a user interaction with user device 210 (e.g., as shown in the user interface of FIG. 5). For example, the user may interact with the user interface to select a vehicle and trip and place one or more tiles or frames on the user interface to display data from the vehicle and trip in a desired manner. The user may also interact with the user interface to select a desired start time and end time for the visualization.

[0024] Processing continues at 304, where the distributed visualization service 260 determines the resources needed to generate the visualization and any constraints. Further details of the processing are described below in conjunction with FIG. 4. Generally, processing at 304 includes determining the number of worker nodes 264 needed to generate the visualization within the required time.

[0025] Processing continues at 306, where the distributed visualization service 260 spawns or deploys the number of worker nodes 264 needed to generate the visualization. At 306, processing includes generating instructions for each worker node 264 so that each worker node 264 can begin processing (e.g., including start and end timestamps for when each task is to be performed). Processing continues at 308, where the distributed visualization service 260 performs processing to reassemble the packets or segments created by each worker node 264 into a final visualization. As discussed above, in some embodiments, the master node 262 is responsible for reassembling the segments created by each worker node 264. In some embodiments, the worker nodes 264 are tasked with assembling the segments into a final visualization.

[0026] Processing continues at 310, where distributed visualization service 260 delivers the final visualization to a storage location (e.g., data store 240 or other location) and notifies the requesting user (or any other users designated by the requesting user) of the location so that they can view the visualization.

[0027] Referring now to Figure 4, a process 400 is shown that may be implemented in conjunction with process 300 of Figure 3. For example, process 400 includes steps that may be implemented by components of system 200 of Figure 2 in response to a user request to generate a distributed visualization. For example, process 400 may include processing by distributed visualization service 260 upon receiving a user request to generate a distributed visualization of a portion of a trip involving a particular vehicle. Process 400 begins at 402, where distributed visualization service 260 calculates the total length of the requested portion of the trip. In some embodiments, the total length is calculated in seconds by subtracting the start timestamp from the end timestamp of the requested segment.

[0028] Processing continues at 404, where service 260 calculates the required rendering resources based on the selected segment and any other constraints. For example, if the user requesting the visualization specified a desired SLA, processing at 404 may include calculating the resources required to render the requested visualization within the SLA period.

[0029] Processing continues at 406, with the service 260 generating a timestamp for each portion or segment of the desired visualization. For example, in some embodiments, the requested view is divided into several 20-second segments for assignment to worker nodes. Processing continues at 408, with the service 260 assigning segments (or tasks) to one or more worker nodes 264. The assignment of segments or tasks is performed based on the number of rendering resources determined to be needed (in step 404) and the number of segments determined in 406. Each worker 264 may be assigned to two or more tasks or segments. The assignment of segments to workers at 408 may include processing by the master node 262 to cause the deployment of the desired number of worker nodes 264 and the communication of task instructions to those worker nodes 264 (including, for example, instructions regarding the timestamp of each job).

[0030] FIG. 5A illustrates an interactive user display 500, according to some embodiments. The display 500 may be viewed by a user operating the user device 210 to interact with data associated with a trip performed by the semi-truck 102. For example, the display 500 may be displayed on a display device of the user device 210 or on a display device associated with the visualization service 250. Generally, the display 500 may be accessed by any authenticated user via an http or https connection. In the exemplary embodiment depicted in FIG. 5, the display includes several regions or frames that display different data items. For example, as shown, the display 500 includes three frames 502, 504, and 506 that are selected by a user to display different items of data associated with a particular vehicle on a selected trip. In the exemplary interface, frame 502 depicts data from a camera, frame 504 depicts a feed of diagnostic data, and frame 506 depicts a 3D view. According to some embodiments, a user interacting with display 500 may select different data sources from a selection of available modules 510 to display in different frames on display 500. The user can resize, move, and reassemble each of the frames to create the desired display. The data and information displayed in each of the frames is synchronized (e.g., data in camera frame 502 is synchronized by time with data displayed in diagnostic frame 504 and data displayed in 3D frame 506).

[0031] In some embodiments, the different display areas can be resized and repositioned by the user by dragging each area to a different location within display 500. Some or all of the display areas can be configured to display data associated with the vehicle and the trip.

[0032] A user interacting with the display 500 may interact with configuration options to select different data sources (e.g., different camera views, etc.). While several different types of data sources may feed the display area, some examples include a fused map view displaying the vehicle's heading, speed, operating mode, lane position, one or more status alerts indicating a status change (e.g., an alert when a lane change is being initiated or when a lane change is completed, etc.), monitoring alerts (e.g., diagnostic or hardware alerts that should be reviewed, etc.), a map view showing the vehicle's route and location, etc. Because a vehicle may have multiple sensors (e.g., multiple cameras with different orientations, etc.), the user may further select a particular view from a particular sensor (e.g., the left front camera or the right rear lidar, etc.).

[0033] In this way, a user interacting with visualization service 250 can configure a user interface to display data in a desired manner (e.g., a user interested in viewing data from a forward-facing camera on a vehicle may display that data in conjunction with diagnostic data or ego status data to analyze different aspects of the ride).

[0034] Once the user configures the interface in a desired manner, the user may request that a distributed visualization of the interface and vehicle data be generated for a selected time period. Such a request may be submitted using an interface such as interface 520 shown in FIG. 5B . For example, the user may configure one or more parameters (e.g., by interacting with a modal or pop-up window presented on user interface 500 of FIG. 5A ) to select a desired time frame (specifying start and end timestamps), a required SLA (identifying how quickly the visualization is needed), and other options. Other options, such as selecting which users should be notified after the visualization is generated, may also be configured. According to some embodiments, once a user requests generation of a visualization, details about the request, as well as details about the configuration and layout of user interface 500, are sent to distributed visualization service 260 (shown in FIG. 2 ). Information about the configuration and layout of user interface 500 may be sent in a format such as JavaScript Object Notation (JSON), a markup language (such as “yet another markup language” or YAML), or the like. When the distributed visualization service 260 assigns segments to workers (e.g., in step 408 of process 400), the distributed visualization service 260 also provides configuration and layout details of the user interface 500 so that each worker node 264 can generate its assigned segment in the requested layout. For example, each worker node 264 may use a headless browser to interact with data from the data store, and the headless browser is configured to replicate the user interface layout in the request.

[0035] For example, in embodiments in which a user interface layout is communicated using JSON objects, the JSON objects may specify the version of the layout, a definition of one or more tabs to be displayed, a title for each tab, an identifier for each tab, an indicator of which tab is the active tab, the layout of each tab (e.g., including information identifying the size and position of each frame displayed in the tab and information identifying the module providing the data for each frame), and metadata associated with the layout (e.g., version number, modification date, creation date, etc.). The source of the data for each frame may be a particular channel or feed of data, or some other source of data. In this manner, embodiments enable a user interacting with user device 210 to visually create a desired user interface layout using desired data from the vehicle and trip and instruct the visualization service to generate a visualization using that particular layout and data. Furthermore, the visualization service can enlist the assistance of several worker nodes to generate segments of that visualization so that the visualization can be generated within a desired SLA.

[0036] Figure 6 illustrates a control system 600 that may be deployed on a vehicle, such as the semi-truck 700 depicted in Figures 7A-7C, according to an example embodiment. With reference to Figure 6, control system 600 may include several sensors 610 that collect data and information that are provided to a central computer system 640 to perform operations, including, for example, control operations to control vehicle components via a gateway 680. According to some embodiments, gateway 680 is configured to allow central computer system 640 to control several different components from different manufacturers.

[0037] Central computer system 640 may be configured, along with one or more central processing units (CPUs) 642, to perform processing for implementing features of embodiments of the present invention described elsewhere herein, as well as to receive sensor data from sensors 610 for use in generating control signals for controlling one or more actuators or other controllers associated with the vehicle's systems (including, for example, actuators or controllers that enable control of throttle 684, steering system 686, brakes 688, etc.) Generally, control system 600 may be configured to operate semi-truck 700 in an autonomous (or semi-autonomous) operating mode.

[0038] For example, the control system 600 may be operated to capture images from one or more cameras 612 mounted at various locations on the semi-truck 700 and perform processing (such as image processing) on ​​the images to identify objects proximate to or in the path of the semi-truck 700. Additionally, one or more lidar 614 and radar 616 sensors may be positioned to sense or detect the presence and volume of objects proximate to or in the path of the semi-truck 700.

[0039] Other sensors may also be positioned or mounted at various locations on the semi-truck 700 to capture other information, such as position data. For example, the sensors may include one or more satellite positioning sensors, such as a GNSS / IMU 618, and / or an inertial navigation system. A global navigation satellite system (GNSS) is a space-based satellite system that provides location information (longitude, latitude, altitude) and time information anywhere on or near the Earth, in all weather conditions, to devices called GNSS receivers. GPS is the most widely used GNSS system in the world. An inertial measurement unit ("IMU") is an inertial navigation system. Generally, an inertial navigation system ("INS") measures and integrates the orientation, position, velocity, and acceleration of a moving object. The INS integrates the measured data, and the GNSS is used as a correction for integration errors in the INS orientation calculation. Any number of different types of GNSS / IMU 618 sensors may be used in conjunction with the features of the present invention. Data collected by each of these sensors may be processed by computer system 640 to generate control signals that control the operation of semi-truck 700. Image and location information may be processed to identify or detect objects around or within the path of semi-truck 700, and control signals may be issued to adjust throttle 684, steering 686, or brakes 688 as needed to safely operate semi-truck 700. Computer system 640 may include computer code operative to implement a process, such as process 300 of FIG. 3, to transmit data from semi-truck 700 to a remote monitoring system using a low-bandwidth protocol. Computer system 640 may also cause control information to be received (and acted upon) from the remote monitoring system, such as via inbound command node 214. While illustrative example sensors and actuators or vehicle systems are shown in FIG. 6, those skilled in the art will understand, upon reading this disclosure, that other sensors, actuators, or systems may also be used.

[0040] Control system 600 may include a computer system 640 (e.g., a computer server) configured to provide a computing environment in which one or more software or control applications (e.g., items 660-682) may be executed to implement the processes described herein. In some embodiments, computer system 640 includes components deployed on semi-truck 700 (e.g., they may be deployed in a system rack 740 positioned within berth compartment 712, as shown in FIG. 7C). Computer system 640 may communicate with other computer systems (not shown in FIG. 6, but shown as items 260, 270, and 290 in FIG. 2) that may be remote from semi-truck 700 (e.g., the computer systems may communicate via a network connection).

[0041] According to various embodiments described herein, computer system 640 may be implemented as a server. In some embodiments, computer system 640 may be configured using any of several well-known computing systems, environments, and / or configurations, such as, but not limited to, a personal computer system, a cloud platform, a server computer system, a thin client, a thick client, a handheld or laptop device, a tablet, a smartphone, a database, a multiprocessor system, a microprocessor-based system, a set-top box, a programmable consumer electronics product, a network PC, a minicomputer system, a mainframe computer system, a distributed cloud computing environment, or the like, and may include any of the above systems or devices.

[0042] Several different software applications or components may be executed by computer system 640 and control system 600. For example, as shown, an application (active learning component 660) may be provided that performs active learning machine processing to process images captured by one or more cameras 612 and information obtained by lidar 614. For example, image data may be processed using a deep learning segmentation model 662 to identify objects of interest within those images (e.g., other vehicles, construction signs, etc.). Here, deep learning segmentation may be used to identify lane points within a lidar scan. As an example, the system may use an intensity-based voxel filter to identify lane points within a lidar scan.

[0043] The lidar data may be processed by machine learning application 664 to draw or identify bounding boxes on the image data to identify objects of interest located by the lidar sensor. Information output from the machine learning application may be provided as input to object fusion 668 and vision map fusion 670 software components, which may perform processing to predict the behavior of other road users, fuse local vehicle pose with global map geometry in real time, and enable on-the-fly map correction. For example, data from object fusion 668 may be used as a source of tracked object data that may be transmitted from semi-truck 700 to one or more remote monitoring systems using the low-bandwidth techniques of the present invention.

[0044] Output from the machine learning application may be supplemented with information (as well as positioning data) from radar 616 and map localization 666 application data. These applications allow control system 600 to be less map-dependent and more capable of handling constantly changing road environments. Furthermore, by correcting any map errors on the fly, control system 600 can facilitate safer, more scalable, and more efficient operation compared to alternative map-centric approaches. Information is provided to prediction and planning application 672, which provides input to trajectory planning 674 component, which enables real-time generation of trajectory 676 based on interactions and predicted interactions between semi-truck 700 and other associated vehicles in the environment. The generated trajectory 676 may be the source of a calculated trajectory for the vehicle that can be transmitted to one or more remote monitoring systems using the low-bandwidth features of the present invention. In some embodiments, for example, control system 600 generates a 60-second planning period and analyzes the relevant actors and available trajectories. The plan that best meets multiple criteria (including safety, comfort, and route preference) is selected, and any associated control inputs required to implement the plan are provided to the controller 682 to control the movement of the semi-truck 700.

[0045] These applications or components (as well as other components or flows described herein) may be implemented in hardware, a computer program executed by a processor, firmware, or a combination of the above. The computer program may be embodied on a computer-readable medium, such as a storage medium or device. For example, the computer program may reside in random access memory ("RAM"), flash memory, read-only memory ("ROM"), erasable programmable read-only memory ("EPROM"), electrically erasable programmable read-only memory ("EEPROM"), registers, a hard disk, a removable disk, a compact disk read-only memory ("CD-ROM"), or any other form of storage medium known in the art.

[0046] The storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit ("ASIC"). Alternatively, the processor and the storage medium may reside as separate components. For example, FIG. 6 illustrates an exemplary computer system 640 that may represent or be integrated with any of the components described above. FIG. 6 is not intended to suggest any limitation regarding the scope of use or functionality of the embodiments of the present application described herein. The computer system 640 may implement and / or perform any of the functions described above.

[0047] Computer system 640 may be described in the general context of computer system-executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system 640 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0048] 6, computer system 640 is shown in the form of a general-purpose computing device. Components of computer system 640 may include, but are not limited to, one or more processors (e.g., CPU 642 and GPU 644), a communications interface 646, one or more input / output interfaces 648, and one or more storage devices 650. Although not shown, computer system 640 may also include a system bus that couples various system components, including system memory, to CPU 642. In some embodiments, input / output interface 648 may also include a network interface. For example, in some embodiments, some or all of the components of control system 600 may communicate via a controller area network ("CAN") bus or the like.

[0049] The storage device 650 may include various types and forms of computer-readable media. Such media may be any available media accessible by the computer system / server and may include both volatile and nonvolatile media, removable and non-removable media. The storage device 650 may include a storage component such as the storage device 204 of FIG. 2. In one embodiment, the system memory implements the flow diagrams of other figures. The system memory may include computer-readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. As another example, the storage device 650 may read from and write to non-removable, non-volatile magnetic media (not shown, typically referred to as a "hard drive"). Although not shown, the storage device 650 may include one or more removable, non-volatile disk drives, such as magnetic, tape, or optical disk drives. In such an example, each may be connected to the bus by one or more data media interfaces. The storage device 650 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the application.

[0050] 7A-7C are diagrams illustrating the exterior of a semi-truck 700 that may be used in accordance with an exemplary embodiment. With reference to FIGS. 7A-7C, the semi-truck 700 is shown for illustrative purposes only; those skilled in the art will understand, upon reading this disclosure, that the embodiments may be used in conjunction with several different types of vehicles. The exemplary semi-truck 700 shown in FIGS. 7A-7C is configured in a typical North American style with an engine 706 forward of the cab 702, a steering axle 714, and two drive axles 716. A trailer (not shown) is attached to the semi-truck 700 via a fifth-wheel trailer coupling mounted on a frame 718 positioned above the drive axles 716. A sleeping compartment 712 is positioned behind the cab 702. Several sensors are positioned in different locations on the semi-truck 700. For example, sensors may be mounted on a sensor rack 720 on the roof of the cab 702. Sensors may also be mounted on the side mirrors 710, as well as other locations. As discussed, sensors may be mounted on the bumper 704, as well as on the side of the cab 702, or elsewhere. For example, rear-facing radar 736 is shown mounted on the side of the cab 702 in FIG. 7A . Embodiments may be used in other configurations of trucks or other vehicles (e.g., semi-trucks having cab-over or cab-forward configurations, etc.). For example, embodiments may be used in conjunction with other types of vehicles towing trailers to enable improved information regarding the orientation of the trailer. Generally, and without limiting embodiments of the present invention, features of the present invention may be used with desirable results in vehicles that haul cargo over long distances, such as long-distance semi-truck routes.

[0051] FIG. 7B is a front view of a semi-truck 700 illustrating several sensors and sensor locations. A sensor rack 720 may secure and position several sensors, including a long-range lidar 722, a long-range camera 724, a GPS antenna 734, and a medium-range forward-facing camera 726. Side mirrors 710 may provide mounting locations for a rear-facing camera 728 and a medium-range lidar 730. A front radar 732 may be mounted on the bumper 704. Those skilled in the art will understand that the locations, sensor types, and mounting depicted in FIGS. 7A-7C are for illustrative purposes only; sensors may be mounted or installed in other locations, and the types of sensors in various locations are not limited to the illustrative embodiment therein. Referring now to FIG. 7C, a partial view of the semi-truck 700 is shown showing the interior of the cab 702 and sleeper compartment 712. In some embodiments, portions of the control system 600 of FIG. 6 are deployed in a system rack 740 within the berth compartment 712, allowing easy access to the components of the control system 600 for maintenance and operation.

[0052] As will be understood based on the foregoing specification, the above-described examples of the present disclosure may be implemented using computer programming or engineering techniques, including computer software, firmware, hardware, or any combination or subset thereof. Any such resulting program having computer-readable code may be embodied or provided in one or more non-transitory computer-readable media, thereby creating a computer program product, i.e., an article of manufacture, according to the discussed examples of the present disclosure. For example, the non-transitory computer-readable medium may be, but is not limited to, a fixed drive, a diskette, an optical disk, a magnetic tape, flash memory, an external drive, semiconductor memory such as read-only memory (ROM), random access memory (RAM), and / or any other non-transitory transmission and / or reception medium, such as the Internet, cloud storage, the Internet of Things (IoT), or other communications network or link. An article of manufacture including the computer code may be created and / or used by executing the code directly from one medium, by copying the code from one medium to another, or by transmitting the code over a network.

[0053] A computer program (also referred to as a program, software, software application, "app," or code) may include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, cloud storage, Internet of Things, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. However, "machine-readable medium" and "computer-readable medium" do not include transitory signals. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and / or any other type of data to a programmable processor.

[0054] The above descriptions and illustrations of processes herein should not be construed as implying a fixed order for performing process steps. Rather, process steps may be performed in any order that is practicable, including the simultaneous performance of at least some steps. While the present disclosure has been described in conjunction with specific examples, it should be understood that various changes, substitutions, and alterations apparent to those skilled in the art may be made to the disclosed embodiments without departing from the spirit and scope of the present disclosure as set forth in the appended claims.

Claims

1. 1. A visualization generation system, comprising: a memory configured to store data captured by at least a first sensor of the vehicle; a processor, the processor comprising: receiving a request for generating a visualization of the vehicle's operation, the request including a start timestamp and an end timestamp, and further including information identifying a user interface configuration; determining the number of worker nodes required to generate the visualization; providing the number of worker nodes and providing instructions to each worker node to cause each worker node to generate a portion of the visualization; reassembling the portions of the visualization into a final visualization; and a visualization generation system configured to deliver the visualization to at least a first user based at least in part on the user interface configuration.

2. The processor:

2. The visualization generation system of claim 1, further configured to receive information specifying a desired service level, and wherein determining a number of worker nodes includes determining a number of worker nodes needed to generate the visualization within the desired service level.

3. 2. The visualization generation system of claim 1, wherein the information identifying a user interface configuration includes information identifying at least a first frame, the at least first frame displaying a selected type of data associated with the operation of the vehicle.

4. 4. The visualization generation system of claim 3, wherein the information identifying a user interface configuration includes information identifying at least a second frame, the at least second frame displaying a second selected type of data associated with the operation of the vehicle.

5. The visualization generation system of claim 1 , wherein each worker node is provided with one or more of a headless browser application, a data server, a web API, and a tile server.

6. The visualization generation system of claim 5 , wherein the instructions to each worker node include at least a start timestamp of a portion of the visualization and an end timestamp of the portion of the visualization.

7. The visualization generation system of claim 1 , wherein the information identifying a user interface configuration is provided in a data storage object.

8. 8. The visualization generation system of claim 7, wherein the data storage object includes information identifying a title, a layout including the placement and position of one or more frames, and a source of data for display in each of the one or more frames.

9. 9. The visualization generation system of claim 8, wherein each worker node uses the data storage object to render a user interface in a headless browser.

10. 1. A method for generating a visualization of a selected segment of a vehicle's journey, comprising: receiving a request to generate a visualization of a selected segment of the journey by the vehicle, the request including a start timestamp and an end timestamp and further including information identifying a user interface configuration; determining the number of worker nodes required to generate the visualization; providing the number of worker nodes and providing instructions to each worker node to cause each worker node to generate a portion of the visualization; reassembling the portions of the visualization into a final visualization; and and providing the visualization to at least a first user based at least in part on the user interface configuration.

11. The method of claim 10 further comprising receiving trip and vehicle selection information.

12. 12. The method of claim 11, wherein the information selecting a trip and a vehicle and the request to generate a visualization is received from a user operating a user device displaying a desired user interface configuration, and the information identifying the user interface configuration is provided in a data storage object.

13. The processor 11. The method of claim 10, further configured to receive information specifying a desired service level, and wherein determining a number of worker nodes comprises determining a number of worker nodes needed to generate the visualization within the desired service level.

14. The method of claim 10 , wherein the information identifying a user interface configuration includes information identifying at least a first frame, the at least first frame displaying a selected type of data associated with operation of the vehicle.

15. 15. The method of claim 14, wherein the information identifying a user interface configuration includes information identifying at least a second frame, the at least second frame displaying a second selected type of data associated with the operation of the vehicle.

16. The method of claim 10 , wherein each worker node is provided with a headless browser application, a data server, a web API, and a tile server.

17. The method of claim 16 , wherein the instructions to each worker node include at least a start timestamp of the portion of the visualization and an end timestamp of the portion of the visualization.

18. A non-transitory machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to: receiving a request to generate a visualization of a selected segment of a vehicle's journey, the request including a start timestamp and an end timestamp and further including information identifying a user interface configuration; determining the number of worker nodes required to generate the visualization; providing the number of worker nodes and providing instructions to each worker node to cause each worker node to generate a portion of the visualization; reassembling the portions of the visualization into a final visualization; and and providing the visualization to at least a first user based at least in part on the user interface configuration.

19. 20. The medium of claim 18, wherein the operations further include receiving information specifying a desired service level, and wherein determining a number of worker nodes includes determining a number of worker nodes needed to generate the visualization within the desired service level.

20. 20. The medium of claim 18, wherein the information identifying a user interface configuration includes information identifying at least a first frame, the at least first frame displaying a selected type of data associated with the operation of the vehicle.

Images