Systems and methods for identifying subsets of data in a database

Abstract

Systems and methods for identifying a subset of data in a dataset are presented herein. In an example, a computing device may receive data related to the operation of at least one vehicle, determine a plurality of events that occurred therebetween based on the operation of the at least one vehicle, receive data related to a request specifying one or more event types, and determine a set of events from among the plurality of events based on the one or more event types specified by the request. In an example, the computing device may generate event data related to the operation of the at least one vehicle in an environment at time points corresponding to each event in the set of events, and transmit the event data to a device associated with the request.

Description

[Technical Field]

[0001] [Cross-reference to related patent applications] This application claims priority to U.S. Provisional Application No. 63 / 480,912, filed January 20, 2023, which is incorporated herein by reference in its entirety for all purposes.

[0002] The present disclosure generally relates to identifying a subset of data in a dataset that is responsive to a request. [Background technology]

[0003] Recent technological advances have made it possible to automate many driving functions traditionally performed by humans. For example, many vehicles now include systems that enable operation at Society of Automotive Engineers (SAE) Levels 2 through 4. These systems can support partial automation (Level 2) for functions such as automatic emergency braking, adaptive cruise control, and lane-keep assist; conditional automation (Level 3) for functions such as full control of the vehicle while certain conditions are met; and high automation (Level 4) for functions such as full control of the vehicle without driver supervision when operating within a specific operational design domain. However, as the capabilities of these systems increase, so does the need for data collection and acquisition to support the underlying development efforts.

[0004] Traditional techniques for collecting data to support these development efforts involve driving a vehicle over tens to thousands of miles and storing the data generated by sensors installed on the vehicle during operation. This data can represent thousands of hours of vehicle operation, much of it problem-free. As the amount of data collected increases, it can become difficult to quickly and effectively isolate portions of the data relevant to specific development tasks. Furthermore, when developers collect data on different computing devices during development, portions of the data are unnecessarily duplicated, thus unnecessarily consuming computing resources. Summary of the Invention

[0005] To facilitate parsing and analysis of data collected during vehicle operation, systems and methods are provided herein for identifying subsets of data within a data set.

[0006]

[0009] In an embodiment, at least one processor is programmed to: receive data related to operation of at least one vehicle in an environment, the operation of the at least one vehicle occurring during at least one time period; determine a plurality of events that occurred during the at least one time period based on the operation of the at least one vehicle in the environment, each event having an event type; receive data related to a request from a device specifying one or more event types; determine a set of events from among the plurality of events based on the one or more event types specified by the request, wherein each event from the set of events occurs at a time point associated with the at least one time period; generate event data related to the operation of the at least one vehicle in the environment at a time point corresponding to each event of the set of events; and transmit, to a computing device associated with the request, the event data configured to cause a display associated with the computing device to generate a user interface representing the operation of the at least one vehicle.

[0007] The request may specify at least one event type corresponding to a change in a state of the at least one vehicle. When determining the plurality of events, the at least one processor may be programmed to determine a change in a state of the at least one vehicle during at least one time period. Additionally or alternatively, when determining the plurality of events, the at least one processor may be programmed to determine the presence of one or more scenarios, agents, or objects in the environment at a point in time during the at least one time period.

[0008] When generating event data related to the operation of at least one vehicle in the environment, the at least one processor may be programmed to generate the event data based on a set of events and data generated during the operation of the at least one vehicle at a time corresponding to the occurrence of each event in the set of events.

[0009] The at least one processor may be further programmed to: determine a trigger configured to cause a device installed in the at least one vehicle to store data related to operation of the at least one vehicle in the environment based on the request, and provide data related to the trigger to the at least one vehicle to cause the trigger to be implemented by the device installed in the at least one vehicle. The at least one processor may be further programmed to receive data related to operation of the at least one vehicle in the environment based on providing data related to the trigger to the at least one vehicle.

[0010] The at least one processor may be further programmed to: determine, based on the request, a trigger configured to cause a device installed in the at least one vehicle to store data related to operation of the at least one vehicle in the environment; and determine, based on applying the trigger to the data related to operation of the at least one vehicle in the environment, whether the trigger satisfies a probability threshold. The at least one processor may be further programmed to provide data associated with the trigger to the at least one vehicle based on determining that the trigger satisfies the probability threshold, to cause implementation of the trigger by the device installed in the at least one vehicle. The at least one processor may be further programmed to update the trigger based on determining that the trigger does not satisfy the probability threshold, and to provide data associated with the trigger to the at least one vehicle based on updating the trigger, to cause implementation of the trigger by the device installed in the at least one vehicle.

[0011] In one embodiment, a computer-implemented method includes receiving, by at least one processor, data related to operation of at least one vehicle in an environment, the operation of the at least one vehicle occurring during at least one time period; determining, by the at least one processor, a plurality of events that occurred during the at least one time period based on the operation of the at least one vehicle in the environment, each event having an event type; receiving, by the at least one processor, data related to a request from a device that specifies one or more event types; determining, by the at least one processor, a set of events from among the plurality of events based on the one or more event types specified by the request, wherein each event from the set of events occurs at a time point related to the at least one time period; generating, by the at least one processor, event data related to the operation of the at least one vehicle in the environment at a time point corresponding to each event of the set of events; and transmitting, by the at least one processor, the event data to a computing device associated with the request configured to cause a display associated with the computing device to generate a user interface representing the operation of the at least one vehicle.

[0012] In another embodiment, a non-transitory machine-readable medium has computer-executable instructions stored thereon, which when executed by one or more processors cause the one or more processors to perform operations including the following steps: receiving data associated with operation of at least one vehicle in an environment, the operation occurring during at least one time period; determining a plurality of events that occurred during the at least one time period based on the operation of the at least one vehicle in the environment, each event having an event type; receiving data associated with a request from a device specifying one or more event types; determining a set of events from among the plurality of events based on the one or more event types specified by the request, wherein each event from the set of events occurs at a time point associated with the at least one time period; generating event data associated with the operation of the at least one vehicle in the environment at a time point corresponding to each event in the set of events; and transmitting to a computing device associated with the request the event data configured to cause a display associated with the computing device to generate a user interface representing the operation of the at least one vehicle.

[0013] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are intended to provide further explanation of the embodiments described herein. [Brief explanation of the drawings]

[0014] Non-limiting embodiments of the present disclosure are described by way of example with reference to the accompanying drawings, which are schematic and are not intended to be drawn to scale, and unless indicated as representing background art, the figures represent aspects of the present disclosure.

[0015] [Figure 1A] 1 illustrates components of a system for collecting and analyzing data generated during vehicle operation, according to one embodiment.

[0016] [Figure 1B] 1 illustrates various sensors associated with an ego aircraft (ego), according to one embodiment.

[0017] [Figure 1C] 1 illustrates components of a vehicle according to one embodiment.

[0018] [Figure 2] FIG. 1 is a flow diagram of a method for collecting and analyzing data generated during vehicle operation, according to one embodiment.

[0019] [Figure 3A] FIG. 1 is a diagram of an implementation of a process for collecting and analyzing data generated during vehicle operation, according to one embodiment. [Figure 3B] FIG. 1 is a diagram of an implementation of a process for collecting and analyzing data generated during vehicle operation, according to one embodiment.

[0020] [Figure 4A] FIG. 1 is a diagram of an implementation of a process for identifying a subset of data in a dataset, according to one embodiment. [Figure 4B] FIG. 1 is a diagram of an implementation of a process for identifying a subset of data in a dataset, according to one embodiment. [Figure 4C] FIG. 1 is a diagram of an implementation of a process for identifying a subset of data in a dataset, according to one embodiment. [Figure 4D] FIG. 1 is a diagram of an implementation of a process for identifying a subset of data in a dataset, according to one embodiment. [Figure 4E] FIG. 1 is a diagram of an implementation of a process for identifying a subset of data in a dataset, according to one embodiment.

[0021] [Figure 5A]1 illustrates an exemplary user interface for identifying a subset of data in a dataset, according to one embodiment. [Figure 5B] 1 illustrates an exemplary user interface for identifying a subset of data in a dataset, according to one embodiment. [Figure 5C] 1 illustrates an exemplary user interface for identifying a subset of data in a dataset, according to one embodiment. [Figure 5D] 1 illustrates an exemplary user interface for identifying a subset of data in a dataset, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Reference will now be made to the exemplary embodiments illustrated in the drawings, and specific language will be used herein to describe the same. It will nevertheless be understood that no limitation of the scope of the claims or the present disclosure is intended thereby. Alterations and further modifications of the features shown herein, and further applications of the principles of the subject matter shown herein that would occur to one skilled in the art in possession of this disclosure, should be considered within the scope of the subject matter disclosed herein. Other embodiments may be used and / or other changes may be made without departing from the spirit or scope of the disclosure. The exemplary embodiments described in the detailed description are not meant to limit the presented subject matter.

[0023] Conventional techniques and technical solutions available today do not include systems, tools, or platforms that enable the efficient collection and analysis of data generated during vehicle operation. To address this need, techniques are disclosed herein that involve the careful collection and analysis of data generated during vehicle operation. More specifically, certain techniques described herein include receiving, by at least one processor, data related to operation of at least one vehicle in an environment, the operation of the at least one vehicle occurring during at least one time period; determining, by the at least one processor, a plurality of events that occurred during the at least one time period based on the operation of the at least one vehicle in the environment, each event having an event type; receiving, by the at least one processor, data related to a request specifying one or more event types from a device; determining, by the at least one processor, a set of events from among the plurality of events based on the one or more event types specified by the request, wherein each event from the set of events occurs at a time point related to the at least one time period; generating, by the at least one processor, event data related to the operation of the at least one vehicle in the environment at a time point corresponding to each event in the set of events; and providing, by the at least one processor, the event data to the device.

[0024] By implementing the techniques described herein, systems (e.g., computing devices, servers, and / or the like onboard vehicles (sometimes referred to as “ego machines”)) can interact to quickly and efficiently request and provide data related to the operation of at least one vehicle. For example, if a developer needs additional data to train a model (e.g., a machine learning model used for perception) to improve the model's ability to classify a particular type of object encountered by a vehicle under certain circumstances (e.g., a traffic cone in proximity to a vehicle traveling over 60 mph), the developer may send a request to a computing device specifying the type and circumstances of such object. A different computing device (e.g., a server) can receive the request and match the request with events that have occurred in a database (e.g., an event in which a traffic cone was identified in association with a vehicle traveling over 60 mph). Processing such requests through a dedicated computing device (e.g., a server that collects, analyzes, and manages data related to the operation of at least one vehicle) eliminates the need for duplicated effort between computing devices when reviewing data related to the operation of at least one vehicle. Then, as new events are defined, subsequent requests related to such requests can be processed while minimizing the use of computing resources. This reduces the need for extra processing used to identify data corresponding to the event responsive to the request. Furthermore, if the computing device sending the request has fewer computing resources compared to the computing resources receiving the request, the processing of the data related to the operation of the at least one vehicle can be performed more quickly than if it were performed solely by the computing device requesting such data.

[0025] In some embodiments, one or more aspects of the present disclosure relate to configuring and managing data provided by vehicles to a network service. As an illustrative example, aspects of the present application address managing data received from multiple vehicles. Illustratively, the vehicle data may include data related to the operation of at least one vehicle, including, but not limited to, vehicle operation information, vehicle parameters, sensor configuration information, sensor values, environmental information, etc. The data may be transmitted to a network service provider. The data may be comprised of various data types or formats. The data is managed by the network service provider, and third parties may access the vehicle data by communicating with the network service provider. Third parties may include, for example, but are not limited to, vehicle manager(s), vehicle manufacturer(s), vehicle service provider(s), vehicle owner(s), etc. Thus, the network service provider may manage the vehicle data to provide access to third parties.

[0026] According to example embodiments, one or more aspects of the present disclosure relate to the organization and accessibility of data related to the operation of at least one vehicle that is provided to one or more third parties. A network service provider can facilitate management of vehicle data to provide specific vehicle data requested by a third party. For example, the network service provider can be configured to process vehicle data received from multiple data sources and store it in its database (sometimes referred to as a data store). In this example, a third party can request access to specific data, and the network service provider can scan the data store and provide the requested data to the third party.

[0027] In general, conventional approaches for providing data to third parties present significant technical challenges for the third parties and the network service provider. More specifically, in response to receiving a specific request from a third party, the network service provider scans its data store and provides a list of relevant data. The third party then needs to download all of the data and further process the downloaded data to identify the relevant portions of the data responsive to the request. In this regard, even if the amount of data is large, the third party needs to download the entire data.

[0028] Furthermore, the network service provider may provide the data as a data bundle, and thus the third party must download the data bundle and further process the data to find the specific data requested. For example, if the data consists of multiple video clips and the third party's request corresponds to a scenario in which a vehicle enters a tunnel, the network service provider may provide any video clips that include the tunnel. In this example, even if each video clip includes a few frames showing the tunnel entrance, the third party must download the entire video clip and identify the frames that show the tunnel entrance. Furthermore, each video clip may include multiple sensor signals (e.g., vehicle attributes, driving environment, driving parameters, etc.), and the third party must process multiple sensor signals even if only one or two sensor signals are requested by the third party.

[0029] Additionally, third parties may have limitations on requesting data. For example, if a third party requests data related to one or more events, the third party must download all of the data and process the downloaded data. In some aspects, the third party may not be able to download the data because all of the data is too large. Instead, the third party may monitor the received data to identify received data that includes events.

[0030] To address at least some of the above inefficiencies, network service providers can facilitate data management and provide data to third parties more efficiently. The network service provider can process received data related to the operation of at least one vehicle and store the data in a specific data format, such as one or more arrays of sensor signals representing vehicle attributes, driving environment, operating parameters, etc. In one example, each video clip included in the data can be stored as an array, such as an array of reference time and vehicle speed, an array of reference time and vehicle power consumption, etc. These arrays can be stored as indexed metadata. Advantageously, the index can be used when scanning for data related to a user's request. For example, in response to receiving a request specifying sensor signals (e.g., sensor signal parameters) of interest to the user, the network service provider can scan the metadata by utilizing the index. In this example, only data related to the index relevant to the user's request can be provided to the user. Thus, the user can access data with an index relevant to the user's query without downloading the entire video clip including the vehicle data requested by the user. For example, a video clip composed of video data can be stored by indexing each video data. In this example, the network service provider can search for an index for each vehicle data within the video clip. In that case, the network service provider may provide access or data relevant only to the user's request, so the user does not need to download the video clip to further search for relevant video data. Instead, the network service provider provides only the vehicle data relevant to the user's request.

[0031] The network service provider may also provide an input interface through which a third party can request vehicle data using the input interface. In some embodiments, the third party may provide input by using logic programming, such as Python code. In these embodiments, the third party may utilize commercially available libraries, such as NumPy or SciPy. For example, if a third party needs to request vehicle data having vehicles operating above a certain speed for more than five hours, the third party may write a query using Python code containing the request. In some embodiments, the query may be written in a customizable format, such as by including any query related to vehicle operating parameters, environment, driver behavior, etc. The network service provider may then scan its metadata and provide a list of sensor signals containing the request. In some embodiments, the network service provider may receive the input as one or more triggers (trigger events). In these embodiments, the network service provider may automatically determine whether the received data is associated with a trigger. In other embodiments, the network service provider may scan its metadata to determine whether any of the stored vehicle data is associated with a trigger.

[0032] The network service provider may also manage data related to the operation of at least one vehicle by grouping data (e.g., portions of data) based on sensor signals. In some embodiments, data received from multiple vehicles may be stored with multiple arrays (each array representing a set of sensor signals). In these embodiments, the arrays may be grouped based on the sensor signals they represent. For example, data received from vehicle 1, vehicle 2, and vehicle 3 may be processed as arrays such as an array of vehicle speed vs. time and an array of vehicle jerk events vs. time. In this example, the vehicle speed vs. time arrays associated with vehicles 1, 2, and 3 may be grouped. Thus, if a third party requests data related to the operation of at least one vehicle while operating at a high speed, the network service provider may scan only the data whose speed meets the request. The network service provider may also manage its computing resources. For example, the network service provider may monitor and dynamically adjust computing resources.

[0033] 1A, a non-limiting example of system components capable of implementing the methods and systems described herein is shown. For example, an analytics server may collect and analyze data generated during vehicle operation. The analytics server can then provide event data (as described herein) to a device based on receiving a request for the event data, the request specifying one or more event types.

[0034] 1A , exemplary environment 100 may include analytic server 110a, system database 110b, administrator computing device 120, ego machines 140a-140c (collectively referred to as “ego machines 140” and sometimes individually referred to as “ego machines 140”), ego computing devices 141a-141c (collectively referred to as “ego computing devices 141” and sometimes individually referred to as “ego computing devices 141”), and server 160. Environment 100 is not limited to the components described herein and may include additional or other components not shown for purposes of brevity, which should be considered within the scope of the embodiments described herein.

[0035] Components referred to herein may interconnect (e.g., establish connections to communicate) via a network 130. Examples of network 130 may include, but are not limited to, private or public LANs, WLANs, MANs, WANs, and the Internet. Network 130 may include wired and / or wireless connections that facilitate communication according to one or more standards and / or via one or more transmission media.

[0036] Communications over network 130 may be performed according to various communication protocols, such as Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), and IEEE communications protocols. In one example, network 130 may include wireless communications according to the Bluetooth set of specifications or another standard or proprietary wireless communications protocol. In another example, network 130 may also include communications over cellular networks, including, for example, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), or Enhanced Data for Global Evolution (EDGE) networks.

[0037] Environment 100 illustrates one example of a system architecture and components that can be used to train and execute one or more AI models, such as AI model 110c. Specifically, as shown in FIG. 1A and described herein, analytics server 110a can train AI model 110c using data acquired from ego aircraft 140 (e.g., using data streams 172 and 174) using methods described herein. Once AI model 110c is trained, each ego aircraft 140 can access and execute trained AI model 110c. For example, ego aircraft 140a with ego computing device 141a can send its camera footage to trained AI model 110c and generate a graph (e.g., data stream 174) defining lane segments in the environment. Additionally, data captured and / or predicted by AI model 110c for ego aircraft 140 (during inference) can also be used to improve AI model 110c. Thus, environment 100 illustrates a continuous loop that can periodically improve the accuracy of AI model 110c. Furthermore, environment 100 illustrates a loop that can use received data by ego machine 140 in a training phase in addition to an inference phase.

[0038] Analytics server 110a may be configured to collect, process, and analyze navigation data (e.g., images captured while navigating) and various sensor data collected from ego aircraft 140. The collected data may then be processed and prepared into a training dataset. The training dataset may then be used to train one or more AI models, such as AI model 110c. Analytics server 110a may also be configured to collect visual data from ego aircraft 140. Using AI model 110c (trained using the methods and systems described herein), analytics server 110a may generate a dataset and / or an occupancy map for ego aircraft 140. Analytics server 110a may display the occupancy map on ego aircraft 140 and / or transmit the occupancy map / dataset to ego computing device 141, administrator computing device 120, and / or server 160.

[0039] Although in FIG. 1A, the AI ​​model 110c is shown as a component of the system database 110b, the AI ​​model 110c may be stored in a different or separate component, such as cloud storage or any other data repository accessible to the analytics server 110a.

[0040] The analytic server 110a may also be configured to display an electronic platform showing various training attributes for training the AI ​​model 110c. The electronic platform may be displayed on the administrator computing device 120 to allow an analyst to monitor the training of the AI ​​model 110c. An example of an electronic platform generated and hosted by the analytic server 110a may be a web-based application or website configured to display the training dataset collected from the ego machine 140 and / or the training status / metrics of the AI ​​model 110c.

[0041] The analytic server 110a may be any computing device comprising a processor and non-transitory machine-readable storage capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include a workstation computer, a laptop computer, a server computer, etc. Although the environment 100 includes a single analytic server 110a, the environment 100 may include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0042] Ego aircraft 140 may represent various electronic data sources that transmit data related to a previous or current navigation session to analytics server 110a. Ego aircraft 140 may be any device configured for navigation, such as vehicle 140a and / or truck 140c. Ego aircraft 140 is not limited to being a vehicle and may also include robotic devices. For example, ego aircraft 140 may include robot 140b, which may represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. Robot 140b may be equipped with software that enables balance, navigation, perception, or interaction with the physical world. Robot 140b may also include various cameras configured to transmit visual data to analytics server 110a.

[0043] Although referred to herein as an “ego aircraft,” ego aircraft 140 may or may not be an autonomous device configured for automatic navigation. For example, in some embodiments, ego aircraft 140 may be controlled by a human operator or a remote processor. Ego aircraft 140 may include various sensors, such as those shown in FIG. 1B . The sensors may be configured to collect data as ego aircraft 140 navigates various terrains (e.g., roads). Analytics server 110a may collect data provided by ego aircraft 140. For example, analytics server 110a may obtain navigation session and / or road / terrain data (e.g., images of ego aircraft 140 navigating roads) from various sensors, such that the collected data is ultimately used by AI model 110c for training purposes.

[0044] As used herein, a navigation session corresponds to a trip in which ego aircraft 140 travels a route, regardless of whether the trip was autonomous or human-controlled. In some embodiments, the navigation session may be for data collection and model training purposes. However, in some other embodiments, ego aircraft 140 may refer to a vehicle purchased by a consumer, and the purpose of the trip may be classified as everyday use. A navigation session may begin when ego aircraft 140 travels more than a threshold distance (e.g., 0.1 miles, 100 feet) from a non-moving location or exceeds a threshold speed (e.g., greater than 0 mph, greater than 1 mph, greater than 5 mph). A navigation session may end when ego aircraft 140 is returned to a non-moving location and / or turned off (e.g., when the driver exits the vehicle).

[0045] The ego aircraft 140 may represent a collection of ego aircraft monitored by the analytics server 110a to train the AI ​​model 110c. For example, the drivers of the vehicles 140a may authorize the analytics server 110a to monitor data associated with their respective vehicles. As a result, the analytics server 110a can utilize various methods described herein to collect sensor / camera data and accordingly generate a training dataset for training the AI ​​model 110c. The analytics server 110a can then execute the trained AI model 110c to analyze data associated with the ego aircraft 140 and predict an occupancy map for the ego aircraft 140. Additionally, additional / ongoing data associated with the ego aircraft 140 can also be processed and added to the training dataset, and the analytics server 110a can recalibrate the AI ​​model 110c accordingly. Thus, the environment 100 illustrates a loop in which the AI ​​model 110c can be trained using navigation data received from the ego aircraft 140. The ego aircraft 140 may include a processor that executes the trained AI model 110c for navigation purposes. During navigation, ego aircraft 140 can collect additional data about their navigation session and can use the additional data to calibrate AI model 110c. That is, ego aircraft 140 represents an ego aircraft that can be used to train, run / use, and recalibrate AI model 110c. In a non-limiting example, ego aircraft 140 represents a vehicle purchased by a customer that can use AI model 110c to navigate autonomously while simultaneously improving AI model 110c.

[0046] Ego aircraft 140 may be equipped with various technologies that enable it to gather data from its surroundings and (potentially) navigate autonomously. For example, ego aircraft 140 may be equipped with an inference chip for running self-driving software.

[0047] Various sensors on each ego aircraft 140 may monitor and transmit collected data related to different navigation sessions to analytics server 110a. FIGS. 1B and 1C show block diagrams of sensors incorporated within ego aircraft 140, according to one embodiment. The number and location of each sensor described with respect to FIGS. 1B and 1C may depend on the type of ego aircraft described in FIG. 1A. For example, robot 140b may include different sensors than vehicle 140a or truck 140c. For example, robot 140b may not include airbag activation sensor 170q. Additionally, the sensors on vehicle 140a and truck 140c may be in different locations than those shown in FIG. 1C.

[0048] As described herein, various sensors incorporated within each ego aircraft 140 may be configured to measure various data associated with each navigation session. Analytics server 110a may periodically collect the data monitored and collected by these sensors, which is processed according to methods described herein and used to train and / or run AI model 110c to generate occupancy maps.

[0049] 1B, ego aircraft 140 may include user interface 170a. User interface 170a may refer to the user interface of an ego computing device (e.g., ego computing device 141 of FIG. 1A). User interface 170a may be implemented as a display screen integrated into or coupled to a vehicle's interior, a head-up display, a touchscreen, etc. User interface 170a may include input devices such as a touchscreen, knobs, buttons, a keyboard, a mouse, a gesture sensor, a steering wheel, etc. In various embodiments, user interface 170a may be adapted to provide user input (e.g., as types of signals and / or sensor information) to other devices or sensors of ego aircraft 140, such as controller 170c (e.g., the sensors shown in FIG. 1B).

[0050] User interface 170a may also be implemented using one or more logic devices that may be adapted to execute instructions, such as software instructions, that implement any of the various processes and / or methods described herein. For example, user interface 170a may be adapted to form a communication link, send and / or receive communications (e.g., sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods. In another example, a driver may use user interface 170a to control the temperature of ego vehicle 140 or activate its functions (e.g., autonomous driving system or steering system 170o). Thus, user interface 170a may monitor and collect driving session data in conjunction with other sensors described herein. User interface 170a may also be configured to display various data generated / predicted by analytics server 110a and / or AI model 110c.

[0051] Orientation sensor 170b may be implemented as one or more of a compass, a float, an accelerometer, and / or other digital or analog device capable of measuring the orientation of ego aircraft 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more orientations of reference, such as gravity and / or magnetic north). Orientation sensor 170b may be adapted to provide orientation measurements of ego aircraft 140. In other embodiments, orientation sensor 170b may be adapted to provide roll, pitch, and / or yaw rate of ego aircraft 140 using a time series of orientation measurements. Orientation sensor 170b may be positioned and / or adapted to provide orientation measurements relative to a particular coordinate system of ego aircraft 140.

[0052] Controller 170c may be implemented as any suitable logic device (e.g., a processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions that implement control loops for controlling various operations of ego aircraft 140. Such software instructions may also implement methods for processing sensor signals, determining sensor information, providing user feedback (e.g., via user interface 170a), interrogating devices for operating parameters, selecting operating parameters for devices, or performing any of the various operations described herein.

[0053] Communications module 170e may be implemented as any wired and / or wireless interface configured to communicate sensor data, configuration data, parameters, and / or other data and / or signals to any feature shown in FIG. 1A (e.g., analytics server 110a). As described herein, in some embodiments, communications module 170e may be implemented in a distributed manner, such that portions of communications module 170e are implemented within one or more elements and sensors shown in FIG. 1B. In some embodiments, communications module 170e may delay communication of sensor data. For example, if ego aircraft 140 does not have network connectivity, communications module 170e may store sensor data in temporary data storage and transmit the sensor data when ego aircraft 140 is identified as having adequate network connectivity.

[0054] Speed ​​sensor 170d may be implemented as an electronic pitot tube, a measuring gear or wheel, a water speed sensor, a wind speed sensor, a wind velocity sensor (e.g., direction and magnitude), and / or other device capable of measuring or determining the linear velocity of ego vehicle 140 (e.g., within the surrounding medium and / or aligned with the longitudinal axis of ego vehicle 140) and providing such measurement as a sensor signal that may be communicated to various devices.

[0055] Gyroscope / accelerometer 170f may be implemented as an electronic sextant, a semiconductor device, an integrated chip, an accelerometer sensor, or other system or device capable of measuring angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of ego aircraft 140 and providing such measurements as sensor signals that can be communicated to other devices, such as analytics server 110a. Gyroscope / accelerometer 170f may be positioned and / or adapted to make such measurements relative to a particular coordinate system of ego aircraft 140. In various embodiments, gyroscope / accelerometer 170f may be implemented in a common housing and / or module with other elements shown in FIG. 1B to ensure a common frame of reference or to ensure a known transformation between frames of reference.

[0056] Global navigation satellite system (GNSS) 170h may be implemented as a global positioning satellite receiver and / or other device capable of determining the absolute and / or relative position of ego aircraft 140, for example, based on radio signals received from space-based and / or terrestrial sources, and providing such measurements as sensor signals that can be communicated to various devices. In some embodiments, GNSS 170h may be adapted to determine the speed, velocity, and / or yaw rate of ego aircraft 140 (e.g., using a time series of position measurements), such as the absolute velocity and / or the yaw component of the angular velocity of ego aircraft 140.

[0057] Temperature sensor 170i may be implemented as a thermistor, an electrical sensor, an electrical thermometer, and / or other device capable of measuring a temperature associated with ego aircraft 140 and providing such a measurement as a sensor signal. Temperature sensor 170i may be configured to measure an environmental temperature associated with ego aircraft 140, such as the temperature of a cockpit or dashboard, which may be used to estimate the temperature of one or more elements of ego aircraft 140.

[0058] Humidity sensor 170j may be implemented as a relative humidity sensor, an electrical sensor, an electrical relative humidity sensor, and / or other device capable of measuring the relative humidity associated with ego machine 140 and providing such measurement as a sensor signal.

[0059] Steering sensor 170g may be adapted to physically adjust the orientation of ego aircraft 140 according to one or more control signals provided by a logic device, such as controller 170c, and / or user input. Steering sensor 170g may include one or more actuators and control surfaces (e.g., rudder or other types of steering or trim mechanisms) of ego aircraft 140 and may be adapted to physically adjust the control surfaces to various positive and / or negative steering angles / positions. Steering sensor 170g may also be adapted to sense the current steering angles / positions of such steering mechanisms and provide such measurements.

[0060] Propulsion system 170k may be implemented as a propeller, turbine, or other thrust-based propulsion system, a mechanical wheeled and / or towed propulsion system, a wind / sail-based propulsion system, and / or other type of propulsion system that can be used to provide propulsion to ego vehicle 140. Propulsion system 170k may also monitor the direction of motive force and / or thrust of ego vehicle 140 relative to a reference frame of ego vehicle 140. In some embodiments, propulsion system 170k may be coupled to and / or incorporated with steering sensor 170g.

[0061] Occupant restraint sensor 170l may monitor the seat belt detection and lock / unlock assembly, as well as other occupant restraint subsystems. Occupant restraint sensor 170l may include various environmental and / or status sensors, actuators, and / or other devices that facilitate operation of safety mechanisms associated with the operation of ego aircraft 140. For example, occupant restraint sensor 170l may be configured to receive movement and / or status data from other sensors shown in FIG. 1B. Occupant restraint sensor 170l may determine whether a safety measure (e.g., a seat belt) is engaged.

[0062] Camera 170m may refer to one or more cameras integrated into ego aircraft 140, as shown in FIG. 1C, and may include multiple cameras integrated (or retrofitted) into ego aircraft 140. Camera 170m may be a camera facing the interior or exterior of ego aircraft 140. For example, as shown in FIG. 1C, ego aircraft 140 may include one or more inward-facing cameras capable of monitoring and collecting video footage of the occupants of ego aircraft 140. Ego aircraft 140 may include eight outward-facing cameras. For example, ego aircraft 140 may include a front camera 170m-1, a forward-facing side camera 170m-2, a forward-facing side camera 170m-3, a rearward-facing side camera 170m-4 on each front fender, a camera 170m-5 on each side (e.g., integrated into a B-pillar), and a rear camera 170m-6.

[0063] 1B, radar 170n and ultrasonic sensor 170p may be configured to monitor the distance of other objects from ego aircraft 140, such as other vehicles or immovable objects (e.g., trees or garage doors). Ego aircraft 140 may also include an automated driving or steering system 170o configured to autonomously navigate ego aircraft 140 using data collected via various sensors (e.g., radar 170n, speed sensor 170d, and / or ultrasonic sensor 170p).

[0064] Thus, the automated driving or steering system 170o may analyze various data collected by one or more sensors described herein to identify driving data. For example, the automated driving or steering system 170o may calculate the risk of a forward collision based on the speed of the ego aircraft 140 and its distance to another vehicle on the road. The automated driving or steering system 170o may also determine whether the driver is touching the steering wheel. The automated driving or steering system 170o may transmit the analyzed data to various features described herein, such as an analytics server.

[0065] Airbag activation sensor 170q may predict or detect a crash to cause the activation or deployment of one or more airbags. Airbag activation sensor 170q may transmit data regarding the deployment of the airbags, including data related to the event that caused the deployment.

[0066] 1A , administrator computing device 120 may represent a computing device operated by a system administrator. Administrator computing device 120 may be configured to display data obtained or generated by analytics server 110a (e.g., various analytics metrics and risk scores), allowing the system administrator to monitor various models utilized by analytics server 110a, review feedback, and / or facilitate training of AI models 110c maintained by analytics server 110a.

[0067] Ego aircraft(s) 140 may be any device configured to navigate various routes, such as vehicle 140a or robot 140b. As described with respect to FIGS. 1B-1C, ego aircraft 140 may include various remotely operated sensors. Ego aircraft 140 may also include ego computing device 141. Specifically, each ego aircraft may have its own ego computing device 141. For example, truck 140c may have ego computing device 141c. For simplicity, ego computing devices are collectively referred to as ego computing device 141. Ego computing device 141 may control the presentation of content on an infotainment system of ego aircraft 140, process commands related to the infotainment system, aggregate sensor data, manage communication of data to electronic data sources, receive updates, and / or send messages. In one configuration, ego computing device 141 communicates with an electronic control unit. In another configuration, ego computing device 141 is an electronic control unit. Ego computing device 141 may include a processor and a non-transitory machine-readable storage medium capable of performing various tasks and processes described herein. For example, AI model 110c described herein may be stored and executed (or directly accessed) by ego computing device 141. Non-limiting examples of ego computing device 141 may include a vehicle multimedia and / or display system.

[0068] In one example of training AI model 110c, analytics server 110a may collect data from ego aircraft 140 to train AI model 110c. Before executing AI model 110c to generate or predict graphs defining lane segments, analytics server 110a may train AI model 110c using various methods. Through training, AI model 110c can incorporate data from one or more cameras of one or more ego aircraft 140 (without having to receive radar data) and predict occupancy data around the ego aircraft. The operations described in this example may be performed by any number of computing devices (e.g., processors of ego aircraft 140) operating in the distributed computing system described in FIGS. 1A and 1B.

[0069] To train the AI ​​model 110c, the analytics server 110a may first use one or more ego aircraft 140 to drive a particular route. While driving, the ego aircraft 140 may use one or more of the ego aircraft's sensors (including one or more cameras) to generate navigation session data. For example, one or more ego aircraft 140 equipped with various sensors may navigate a specified route. As the one or more ego aircraft 140 traverse terrain, the ego aircraft's sensors may capture continuous (or periodic) data about their surroundings. The sensors may indicate occupancy conditions around the one or more ego aircraft 140. For example, the sensor data may indicate various objects having mass around the one or more ego aircraft 140 as they navigate the route.

[0070] During operation, as one or more ego aircraft 140 navigate, their sensors collect data and transmit the data to analysis server 110a, as shown in data stream 172. In some embodiments, one or more ego aircraft 140 may include one or more high-resolution cameras that capture a continuous stream of visual data from around the one or more ego aircraft 140 as the one or more ego aircraft 140 navigate a route. In that case, analysis server 110a may use the camera footage to generate a second dataset, in which visual elements / representations of different voxels around the one or more ego aircraft 140 are included in the second dataset. During operation, as one or more ego aircraft 140 navigate, the ego aircraft's cameras collect data and transmit the data to analysis server 110a, as shown in data stream 172. For example, ego computing device 141 may transmit image data to analysis server 110a using data stream 172.

[0071] Analytics server 110a may generate a training dataset using data collected from ego aircraft 140 (e.g., camera footage received from ego aircraft 140). The training dataset may identify or include a set of examples. Each example may identify or include input data and expected output data from the input data. In each example, the input may include collected data such as sensor data (e.g., video or images from one or more cameras) and map data (e.g., a navigation map) from ego aircraft 140. The output may include, among other things, environmental features (e.g., attributes collected from sensor data), map features (e.g., attributes in a navigation map such as topological features and road layouts), classifications (e.g., types of topology), and output tokens (e.g., combinations of environmental features, map features, and classifications) included in a graph defining lane segments. In some embodiments, the output may be created by a human reviewer examining the input data.

[0072] Using the training dataset, the analytics server 110a may provide a set of training datasets to the AI ​​model 110c and obtain a set of predicted outputs (e.g., environment features, map features, classifications, and output tokens). The analytics server 110a may then compare the predicted data with ground truth data to determine differences and train the AI ​​model 110c by adjusting the internal weights and parameters of the AI ​​model 110c proportional to the determined differences according to a loss function. The analytics server 110a may train the AI ​​model 110c in a similar manner until the predictions of the trained AI model 110c are accurate up to a certain threshold (e.g., recall or precision).

[0073] In some embodiments, the analytics server 110a may use supervised training methods. For example, using ground truth data and received visual data, the AI ​​model 110c may train itself to predict outputs. As a result, once trained, the AI ​​model 110c may receive sensor data and map data, analyze the received data, and generate tokens. In some embodiments, the analytics server 110a may use unsupervised methods in which the training dataset is unlabeled. Because labeling the data in the training dataset can be time-consuming and require excessive computing power, the analytics server 110a may utilize unsupervised training techniques to train the AI ​​model 110c.

[0074] Upon establishment of AI model 110c, analysis server 110a can send, transmit, or otherwise distribute weights for AI model 110c to each ego computing device 141a-141c. Upon receipt, ego computing devices 141a-141c can store and maintain AI model 110c in local storage. Once stored and loaded, ego computing devices 141a-141c can use the weights in processing newly acquired data (e.g., sensor and map data) to create graphs defining lane segments for autonomously navigating each ego vehicle 140a-140c through the environment. From time to time, analysis server 110a can send, transmit, or otherwise distribute updated weights for AI model 110c to update the instances of AI model 110c on ego computing devices 141a-141c.

[0075] 2, a non-limiting example flow diagram of a method 200 for identifying a subset of data in a dataset is shown. Method 200 may be implemented using any of the components described herein, such as one or more components of an analytic server (e.g., an analytic server the same as or similar to analytic server 110a of FIG. 1A). Although method 200 includes the steps described herein, other embodiments may include additional or alternative steps or omit one or more steps.

[0076] In some implementations, method 200 is performed by an analytical server. However, one or more steps of method 200 may be performed by one or more other computing devices separate from and / or including the analytical server, such as another analytical server, one or more computing devices in at least one vehicle (e.g., ego aircraft 140), and / or one or more other computing devices operating in a distributed computing system (e.g., a distributed computing system the same as or similar to the distributed computing system described in FIGS. 1A-1C). For example, one or more computing devices in the ego aircraft may locally perform some or all of the steps described with respect to method 200.

[0077] In step 202, the analytics server receives data related to the operation of at least one vehicle. For example, the analytics server may receive (e.g., collect) data related to the operation of the at least one vehicle from one or more ego aircraft (e.g., one or more ego aircraft the same as or similar to ego aircraft 140 of FIG. 1 ). In some implementations, the at least one vehicle may be configured to collect data and transmit the collected data. The analytics server may receive the data related to the operation of the at least one vehicle periodically (e.g., at predetermined time intervals), when one or more conditions are met (e.g., when a system of the ego aircraft establishes a communications connection with the analytics server to transmit data to the analytics server, when a system of the ego aircraft determines that a predetermined amount of on-board storage has been consumed, and / or the like), or continuously (e.g., in real time). In these examples, data related to the operation of the at least one vehicle may be generated while the ego vehicle is operating within the environment, such as when traversing a drivable surface (for ego vehicles the same as or similar to ego vehicles 140a and 140c in FIG. 1A) and / or a traversable (e.g., walkable) surface (for ego vehicles the same as or similar to ego vehicle 140b in FIG. 1A).

[0078] As the ego vehicle operates within its environment (e.g., along drivable and / or traversable surfaces), a computing device of the ego vehicle (e.g., a computing device the same as or similar to vehicle computing device 171 of FIG. 1B ) may monitor the operation of one or more of the ego vehicle's components (e.g., one or more components shown in FIG. 1B ). The computing device may then generate data related to the vehicle's operation based on the operation of the one or more components. In this example, the computing device may be configured to communicate with one or more components and to generate and store data generated by the one or more components that is represented by at least a portion of the data related to the operation of at least one vehicle. This process may also be referred to as logging or data logging. It will be understood that the computing device may store the data in a local database (e.g., a database included in the computing device monitoring the one or more components or maintained in separate storage).

[0079] The computing device may monitor operation of one or more sensors of the vehicle and generate data related to operation of the at least one vehicle based on (e.g., by including) sensor data related to one or more sensors of the ego aircraft. The sensor data may include one or more sensor signals generated by one or more sensors during operation of the ego vehicle. In an example, the sensor data may include orientation sensor data related to sensor signals generated by one or more orientation sensors of the ego aircraft, speed sensor data related to sensor signals generated by one or more speed sensors of the ego aircraft, gyroscope / accelerometer sensor data related to sensor signals generated by one or more gyroscope / accelerometers of the ego aircraft, steering sensor data related to one or more sensor signals generated by steering sensors of the ego aircraft, GNSS sensor data related to one or more sensor signals generated by GNSS of the ego aircraft, temperature sensor data related to one or more sensor signals generated by temperature sensors of the ego aircraft, humidity sensor data related to one or more sensor signals generated by humidity sensors of the ego aircraft, and the like. the ego aircraft, humidity sensor data associated with one or more sensor signals generated by the ego aircraft, occupant restraint sensor data associated with one or more sensor signals generated by one or more occupant restraint sensors; camera data associated with one or more sensor signals generated by a camera (either outward-facing or inward-facing focused on the ego aircraft); radar data associated with sensor signals generated by one or more radars on the ego aircraft; ultrasonic sensor data associated with one or more sensor signals generated by an ultrasonic sensor on the ego aircraft; airbag deployment sensor data associated with sensor signals generated by an airbag deployment sensor on the ego aircraft; and / or any other sensor data described herein.

[0080] The data related to the operation of the at least one vehicle may include user interface data related to input / output provided to or by a passenger in the ego vehicle via a user interface (e.g., a display device, a torque sensor corresponding to a steering wheel, and / or the like). Additionally or alternatively, the data related to the operation of the at least one vehicle may include controller data related to a controller of the ego vehicle. In this example, the controller may control the operation of one or more devices in the ego vehicle (e.g., devices controlling vehicle acceleration, braking, windshield wipers, communication of user inputs, and / or the like) that are simultaneously included in the data related to the operation of the at least one vehicle. In some examples, the data related to the operation of the at least one vehicle may include data generated during operation of the communications module. For example, data transmitted and / or received from and / or via the communications module to a system of the ego vehicle may be included in the data related to the operation of the at least one vehicle. In some implementations, the ego vehicle may generate data related to the operation of the at least one vehicle based on operation of the propulsion system. For example, the data may include data generated by the propulsion system (e.g., control signals to cause the ego vehicle to increase or decrease speed and acceleration).

[0081] The data related to the operation of the at least one vehicle may include automated driving system data or steering system data related to the operation of the automated driving system or steering system. For example, during operation, the automated driving system or steering system may receive data (e.g., from one or more sensors, such as a camera, radar, and / or one or more other sensors described in connection with FIG. 1B ) and perform one or more actions. Such actions may include, for example, perception, planning, and control that support automated movement of the ego vehicle through its environment toward a destination. When the automated driving system or steering system performs one or more actions, data generated by the automated driving system or steering system during the execution of the actions may be included in the data related to the operation of the at least one vehicle. Additionally or alternatively, the data related to the operation of the at least one vehicle may include automated driving system data or steering system data generated during a simulation including the automated driving system or steering system. For example, at a first time point or period, the data related to the operation of the at least one vehicle may be provided to an automated driving system or steering system installed in the at least one vehicle, causing the automated driving system or steering system to provide as output the automated driving system data or steering system data used to update the data related to the operation of the at least one vehicle. At a second (later) time or period, the data related to the operation of the at least one vehicle may be provided to a different automated driving or steering system (e.g., a later version of the automated driving or steering system) while the automated driving or steering system is simulated (sometimes referred to as being "played back"). The data related to the operation of the at least one vehicle may again be updated to include data output by the automated driving or steering system.

[0082] Continuing with the discussion of perception, an automated driving or steering system may have a perception system receive data necessary to perceive objects and / or agents proximate to the ego vehicle and classify the objects or agents (e.g., as cones, traffic lights, pedestrians, bicyclists, and / or the like). This data may include camera data, radar data, ultrasound data, and / or the like. The perception system may then generate data associated with the classification of the objects and / or agents, and the computing system may include that data in data associated with the operation of the at least one vehicle. In some examples, the computing system may include the inputs, intermediate decisions, and outputs of the perception system in the data associated with the operation of the at least one vehicle.

[0083] With respect to planning, the automated driving system or steering system may provide data to the planning system. The data may include data generated by the perception system, map data associated with one or more maps, and / or data associated with (e.g., specification of) a route and / or a target destination. The planning system may then determine one or more maneuvers (e.g., lane changes, overtaking, yielding, and / or the like) to perform and / or one or more trajectories for the ego aircraft to travel. Once determined, the planning system may provide data related to the one or more trajectories to a controller to operate the ego aircraft according to the one or more trajectories. The computing system may include data related to the one or more trajectories output by the planning system along with data related to the operation of the at least one vehicle. In some examples, the computing system may include the inputs, intermediate decisions, and outputs of the planning system in the data related to the operation of the at least one vehicle.

[0084] With respect to control, the automated driving system or steering system may provide data to a control system. The data may include data provided to a planning system and / or data provided as output to the planning system. The control system may then determine data associated with one or more control signals based on the data provided to the control system. The one or more control signals may be configured to actuate one or more systems of the ego vehicle (e.g., a propulsion system the same as or similar to propulsion system 170k of FIG. 1B, a steering system involved in controlling actuators that change the steering angle of the ego vehicle, and / or the like). The computing system may include data associated with the one or more control signals with data related to the operation of at least one vehicle. In some examples, the computing system may include inputs, intermediate decisions, and outputs of the control system in data related to the operation of at least one vehicle.

[0085] The data related to the operation of the at least one vehicle may be generated during at least one time period. For example, the data related to the operation of the at least one vehicle may be associated with (e.g., coincide with) a time period during which the at least one vehicle is operating. In such an example, the data may be further associated with one or more timestamps. The one or more timestamps may indicate one or more points in time at which a particular portion of the data related to the operation of the at least one vehicle was generated. In one illustrative example, as the vehicle operates, camera data related to one or more sensor signals generated by one or more cameras may be included in the data related to the operation of the at least one vehicle. The camera data may be further associated with one or more timestamps corresponding to the points in time at which the one or more sensor signals were generated.

[0086] Data related to the operation of the at least one vehicle is transmitted to a vehicle data management system (sometimes referred to as a vehicle data management service). For example, a vehicle (e.g., an ego vehicle) that generates data related to the operation of the at least one vehicle may transmit the data to the vehicle data management system as data related to the operation of the at least one vehicle (sometimes referred to as vehicle data). In some examples, the vehicle data management system is the same as or similar to an analytics server. Additionally or alternatively, the vehicle data management system may be associated with (e.g., cooperate with) the analytics server to perform one or more of the operations described herein.

[0087] The vehicle data management system may be configured to receive (e.g., collect) data related to the operation of at least one vehicle. In examples, at least one vehicle (e.g., at least one vehicle computing device) may be configured to collect, store, and periodically, conditionally, or continuously transmit data related to the operation of the at least one vehicle to the vehicle data management system. In some implementations, the vehicle data management system is configured to receive data related to the operation of the at least one vehicle based on the vehicle computing system determining that one or more events are represented by the data related to the operation of the at least one vehicle. As described below, the at least one vehicle computing system may be configured to determine whether one or more events have occurred (e.g., based on one or more triggers corresponding to the events, scenarios, locations based on geographic coordinates or other identifiers, and / or the like) and store data related to the operation of the at least one vehicle based on whether the one or more events have occurred.

[0088] In step 204, the analytics server determines a plurality of events. For example, the analytics server may determine a plurality of events, where each event of the plurality of events is associated with (e.g., occurs at) one or more time points (e.g., time points, time periods, and / or the like). In some examples, the analytics server may determine an event type for each event of the plurality of events. The analytics server may determine the event type based on the analytics server processing data related to the operation of the at least one vehicle. In one example, the analytics server may process the data related to the operation of the at least one vehicle by converting the data into an array of sensor signals. When processing the data, the analytics server may associate a tag with each event of the plurality of events based on the analytics server determining that the one or more events occurred at one or more time points, where the tag is associated with (e.g., represents) an event type for each event of the plurality of events.

[0089] In an illustrative example, the analytics server may process sensor signals generated by one or more cameras of at least one vehicle. In that case, the analytics server may generate arrays shown in Tables 1a, 1b, and 1c below, which represent the status of one or more other components of at least one vehicle. As shown in Table 1a, a vehicle speed signal is represented by an array of timestamps and vehicle speed. As shown in Table 1b, a vehicle battery status signal is represented by an array of timestamps and vehicle battery status. As shown in Table 1c, a vehicle heater operation signal is represented by an array of timestamps and vehicle heater operation. In some examples, these arrays can be stored as metadata. For example, each image can include information corresponding to each image as metadata (e.g., a first image can be captured at 1:56 PM with at least one vehicle traveling at 60 mph, a second image can be captured at 1:57 PM with at least one vehicle traveling at 66 mph, and so on). Note that Tables 1a, 1b, and 1c are examples, and the present disclosure is not limited thereto. [Table 1a] [Table 1b] [Table 1c]

[0090] The analytics server may also process data related to the operation of at least one vehicle by grouping the data based on sensor signals. In some embodiments, each of the received sensor signals may be included in multiple arrays, with each array representing a sensor signal such as those shown in the table above. In these embodiments, the arrays may be grouped based on the vehicle signal they represent. For example, the sensor signals received from vehicle 1, vehicle 2, and vehicle 3 may be processed as arrays such as an array of vehicle speed versus time and an array of vehicle jerk events versus time. In this example, the arrays of vehicle speed versus time associated with vehicle 1, vehicle 2, and vehicle 3 may be grouped. Thus, if a request involves the operation of at least one vehicle at high speed (e.g., above a predetermined speed), the analytics server may scan for sensor signals that meet this requirement while refraining from scanning other sensor signals. This may conserve computing resources that would be required to scan the entire data related to the operation of at least one vehicle when responding to a request.

[0091] When processing the data related to the operation of the at least one vehicle, the analytics server may identify the occurrence of each of the plurality of events during that time period. Additionally or alternatively, the analytics server may determine the event type based on one or more tags associated with the data related to the operation of the at least one vehicle. In this example, a computing system associated with each of the at least one vehicle may associate one or more tags with one or more time points based on the computing system determining that one or more events occurred at the one or more time points. It will be appreciated that the analytics server may process the events by converting the data related to the operation of the at least one vehicle into corresponding arrays, each array representing (by way of example) the time the event occurred, the speed at which the vehicle was traveling, the status of one or more components of the vehicle (e.g., heater and / or the like), and / or the like.

[0092] The event type may be associated with a change in the state of at least one vehicle. For example, the event type may be associated with a state change in which a driver engages or disengages one or more systems of the vehicle, such as windshield wipers, headlights, accelerator pedal, or brakes, and / or the like. Additionally or alternatively, the event type may be associated with a state change in which one or more systems of the vehicle (e.g., heating and air conditioning systems) automatically engage or disengage. In some implementations, the analytics server determines that an event has occurred based on the analytics server determining that a change in state has occurred. Additionally or alternatively, the at least one vehicle computing system may determine that an event has occurred based on the computing device monitoring other systems of the vehicle and may include a tag indicating that the event has occurred. In this example, the analytics server may determine that an event has occurred based on a tag included in data related to the operation of the at least one vehicle. As described herein, the vehicle computing device may determine that an event has occurred based on a trigger.

[0093] In some examples, an event type may represent one or more scenarios. For example, an event type may be associated with a scenario such as a cut-in, in which a vehicle moves from an adjacent lane of travel into the lane of at least one vehicle. In other examples, an event type may be associated with a scenario such as J-walking, in which an agent (e.g., a pedestrian) crosses the lane of travel of a vehicle in an unguarded area (e.g., an area other than a crosswalk or an area where pedestrians are allowed to cross the lane of travel). In some implementations, the analysis server determines that an event has occurred based on the analysis server determining that one or more objects and / or agents were present and / or that one or more objects or agents were moving through the environment according to the corresponding event type. Additionally or alternatively, the computing device of at least one vehicle may determine that an event has occurred based on the computing device monitoring other systems of the vehicle and including a tag indicating that an event has occurred.

[0094] In further examples, the event type may represent the presence of one or more agents and / or objects. For example, the event type may be associated with the presence of one or more objects (e.g., traffic cones, plastic or concrete median barriers, traffic signals and / or signs, parked vehicles, and / or the like). In some implementations, the analytics server determines that an event has occurred based on the analytics server determining that one or more objects and / or agents were present at one or more time points. Additionally or alternatively, the computing system of at least one vehicle may monitor other systems of the vehicle and determine that an event has occurred based on a computing device including a tag indicating the presence of one or more agents and / or objects at one or more time points.

[0095] The analytics server may receive data related to the operation of the at least one vehicle based on the analytics server providing the at least one vehicle with data related to the trigger. For example, the analytics server may receive input related to the trigger (e.g., via a user, such as an autonomous system developer and / or the like). In one example, the analytics server may receive input indicating that data related to the classification of one or more objects and / or agents should be associated with the trigger. In this example, the analytics server may generate data related to the trigger, and the data related to the trigger may be configured to store the data related to the trigger on a computing device of the at least one vehicle and / or transmit the data related to the trigger to the analytics server. The data related to the trigger may include data generated by one or more components of the ego vehicle when the autonomous driving system and / or steering system classify one or more objects and / or agents in the at least one vehicle's environment.

[0096] The analytics server may determine whether the trigger satisfies a probability threshold. For example, the analytics server may determine whether the trigger satisfies a probability threshold, where the probability threshold represents the extent to which the trigger identifies criteria that result in the correct identification of one or more events. In an example, the analytics server may determine whether the trigger satisfies the probability threshold based on the analytics server applying the trigger to previously received data related to the operation of at least one vehicle. If the probability threshold is met (e.g., if a predetermined number of events corresponding to the trigger are correctly identified using the trigger, if a percentage of events corresponding to the trigger are correctly identified, and / or the like), the analytics server may provide (e.g., transmit and / or otherwise make available) data related to the trigger to one or more vehicles. The data related to the trigger may be configured to cause a computing device of one or more vehicles to implement the trigger. Alternatively, if the probability threshold is not met, the analytics server may provide an output related to an indication that the probability threshold was not met. In this example, the analytics server may receive subsequent input (e.g., input via a user) to update one or more aspects of the trigger, after which the analytics server may re-determine whether the updated trigger satisfies the probability threshold. In such an example, if the analytics server determines that the updated trigger satisfies the probability threshold, the analytics server may provide data related to the updated trigger to one or more vehicles. In this example, the data related to the updated trigger may be transmitted to a computing device of at least one vehicle.

[0097] In step 206, the analytics server receives data associated with the request. For example, the analytics server may receive data associated with the request from a computing device associated with a user, such as an autonomous system developer and / or the like. In some examples, the request may be generated based on a computing device associated with a user receiving input from the user specifying one or more event types. The one or more event types may be specified by identifying one or more predefined event types and / or one or more criteria corresponding to the event types (e.g., event types defined by the user in the request). In some embodiments, a user (sometimes referred to as a third party) provides input that is received by the analytics server via a request, where the input is represented as a logical expression, for example, by utilizing Python code and / or any other suitable expression.

[0098] The analytics server may analyze the request. For example, the analytics server may analyze (e.g., process) the request to identify one or more aspects of the request. In some embodiments, the request may be represented as a logical expression based, for example, on Python code. In one illustrative example, if a user provides input represented as a logical expression requesting data related to the operation of at least one vehicle whose speed exceeds a 60 mph speed limit, the analytics server may analyze the input (as represented by the request) and scan one or more metadata arrays, such as Table 1a. The analytics server may then provide corresponding data related to the operation of at least one vehicle whose speed exceeded 60 mph. The corresponding data may be provided to a computing device associated with the user who sent the request.

[0099] The analytics server may receive the request, which is associated with one or more triggers (sometimes referred to as trigger events). In these embodiments, the input may be in the form of computer-executable data, such as an executable script, provided as input by a user. For example, the trigger may be associated with at least one vehicle operating parameter, such as a determination that the vehicle's location is within defined geographic boundary information (e.g., enters a geofence) or a determination that the vehicle's operating parameter is above a preset threshold. In some embodiments, the analytics server may perform debugging of the input and provide whether there are any errors.

[0100] The analytics server may determine a trigger based on the request. For example, the analytics server may determine an event type associated with the request, and the analytics server may identify a corresponding trigger for the event type. Additionally or alternatively, the analytics server may compare one or more criteria corresponding to the event type with one or more criteria corresponding to the one or more triggers and determine the trigger based on the comparison. As described above, the trigger may be configured to cause one or more vehicle computing devices to store and / or transmit data related to the operation of at least one vehicle in the environment. In an example, the computing device may transmit data related to the operation of the at least one vehicle determined based on the trigger to the analytics server. In some implementations, the analytics server provides data associated with the trigger to at least one vehicle (e.g., the computing device of the at least one vehicle) to implement and provide the trigger to the at least one vehicle. This may then cause the computing device of the at least one vehicle to transmit data related to the operation of the at least one vehicle in the environment. In this aspect, the analytics server may cause one or more vehicles to provide data related to the operation of the at least one vehicle in the environment in response to the request.

[0101] In step 208, the analytics server determines a set of events. For example, the analytics server may determine the set of events from among a plurality of events. In some implementations, the plurality of events are events stored in an event database associated with the analytics server. For example, the analytics server may determine a plurality of events corresponding to data related to the operation of at least one vehicle collected before and / or after receiving the request. In some implementations, the analytics server processes (e.g., scans) the data to determine which portions of the data are associated with the plurality of events specified by the request.

[0102] The analytics server may determine the set of events based on scanning data related to the operation of the at least one vehicle to identify data corresponding to the request. The analytics server may be configured to scan metadata of the data related to the operation of the at least one vehicle. In some embodiments, the analytics server may set scan parameters based on the request. For example, if the request specifies a vehicle speed, the analytics server may scan only data related to the specified vehicle speed. In this example, the metadata may provide an index for each array of stored data, and the analytics server may reference the index based on the scan parameters. In some embodiments, a user may select the scan parameters. For example, a user may select an array of data and / or a maximum amount of data to return, such as an array of image data or video data.

[0103] The analytics server may scan the data based on a trigger associated with the request. If the scan results indicate that the trigger is satisfied, the analytics server may select data responsive to the request based on the trigger being satisfied. As described above, a trigger may specify one or more aspects of the operation of at least one vehicle for which the trigger may be satisfied. In this example, the analytics server may transmit the portion of the data that satisfies the request (e.g., that corresponds to the type of vehicle operating parameter specified by the request).

[0104] The analytics server may determine the set of events based on comparing one or more event types specified by the request with one or more event types corresponding to one or more tags of each event (in the plurality of events). In doing so, the analytics server may determine a set of events in which each event in the set of events corresponds to one or more time points within at least one time period in which the plurality of events occurred. In an example, the at least one time period may be specified in the request. Additionally or alternatively, the at least one time period may be determined based on data related to the operation of at least one vehicle in the environment available to the analytics server when responding to the request.

[0105] In step 210, the analytics server generates event data. For example, the analytics server may generate event data related to the operation of at least one vehicle in the environment responsive to the request. The event data may correspond to a portion (e.g., a copy) of data related to the operation of the at least one vehicle in the environment related to the request. In some examples, the event data is associated with (e.g., represents) each event in a set of events. In some implementations, the event data is generated based on the set of events and data generated during the operation of the at least one vehicle at times corresponding to the occurrence of each event in the set of events.

[0106] In step 212, the analytics server provides the event data. For example, the analytics server may provide (e.g., transmit and / or otherwise make available for download) the event data to the computing device that generated and / or sent the request to the analytics server. In some implementations, the analytics server may provide the event data via one or more application programming interfaces (APIs). For example, when the analytics server receives data related to a request via an API (referred to as an API request), the operations described in one or more steps herein may be performed and event data may be generated by the analytics server. In this example, the analytics server may provide the event data via the API (referred to as an API response). In some implementations, the data is provided in one or more formats (e.g., JSON, XML, and / or plain text format).

[0107] 3A and 3B, block diagrams of an implementation 300 relating to a process for managing vehicle data are shown. The implementation 300 may include any of the components described herein, such as a vehicle 302 (e.g., a vehicle the same as or similar to the ego aircraft 140 of FIG. 1A), a network service provider 304 (e.g., a computing device associated with (e.g., controlled by) an organization that manages at least a portion of the infrastructure involved in communication between devices over a network), a vehicle data management system 306 (e.g., a computing device the same as or similar to the analytics server 110a of FIG. 1), a database 308, and / or a computing device 310 (e.g., a computing device associated with one or more users described herein). While the implementation 300 may be performed by the network service provider 304 in cooperation with the vehicle data management system 306 as described herein, one or more aspects of the implementation 300 may be performed by any number of computing devices operating in a distributed computing system (e.g., a distributed computing system the same as or similar to that described in FIGS. 1A-1C). For example, one or more computing devices of the ego machine may perform some or all of the steps described in Figures 3A and 3B locally or in conjunction with an analytics server.

[0108] Vehicle 302, network service provider 304, vehicle data management system 306, database 308, and / or computing device 310 may be any computing device equipped with a processor and non-transitory machine-readable storage capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include workstation computers, laptop computers, server computers, and / or the like, which are not explicitly shown for clarity. Further, one or more of vehicle 302, network service provider 304, vehicle data management system 306, database 308, and / or computing device 310 may be interconnected with each other via a network (e.g., a network the same as or similar to network 130 of FIG. 1A).

[0109] 3A , as indicated by operation 320, the network service provider 304 collects vehicle data from the vehicles 302. The vehicle data may include data related to the operation of at least one of the vehicles 302. For example, the vehicle data may include camera data (sometimes referred to as vision system data) related to one or more sensor signals generated by one or more cameras, as described herein.

[0110] As indicated by operation 322, the network service provider 304 processes the vehicle data. For example, the network service provider 304 may have the vehicle data management system 306 process the vehicle data. In this example, the network service provider 304 may have the vehicle data management system 306 process the vehicle data based on (e.g., in response to) the network service provider 304 collecting the vehicle data. Although shown as a single network service provider 304, in some examples, multiple network service providers may be located in different geographic locations and configured to collect vehicle data related to the geographic locations. In some examples, the vehicle data management system processes the vehicle data by converting the vehicle data into an array of vehicle data signals. The vehicle data management system 306 may be configured to process the collected data into an array of signal data. For example, the vehicle data management system 306 may process one or more sensor signals generated by one or more cameras received from the vehicle (multiple sets of camera signals may be referred to as a video clip). The processing may include adapting the signals to a normalized format, such as the formats shown in Tables 1a, 1b, and 1c below.

[0111] [Table 1a]

[0112] [Table 1b]

[0113] [Table 1c]

[0114] As shown in Table 1a, the vehicle speed signal is represented by an array of timestamps and vehicle speed. As shown in Table 1b, the vehicle battery status signal is represented by an array of timestamps and vehicle battery status. As shown in Table 1c, the vehicle heater operation signal is represented by an array of timestamps and vehicle heater operation status. These arrays may be stored individually or as metadata (e.g., metadata corresponding to other vehicle data, such as camera data and / or the like). Tables 1a, 1b, and 1c are provided merely as examples, and the present disclosure is not limited thereto.

[0115] In some embodiments, the vehicle data management system 306 can also process vehicle data by grouping the vehicle data based on vehicle signals generated by a particular vehicle. In some embodiments, portions of vehicle data received from multiple vehicles can be stored as multiple arrays, each representing a vehicle signal, as shown in the table above. In these embodiments, the arrays can be grouped based on different representative vehicle signals. For example, vehicle data received from multiple vehicles (e.g., first, second, and third vehicles) can be processed as arrays, such as a vehicle speed vs. time array and a vehicle jerk event vs. time array, represented for each vehicle, and grouped with arrays generated according to other signals (e.g., vehicle speed signal, vehicle batter status signal, and vehicle heater operation signal). In this example, the vehicle speed vs. time arrays associated with the vehicle can be grouped. Thus, if a third party requests high-speed vehicle data, the network service provider can scan only the group of vehicle data indicating vehicle speed and other vehicle data corresponding to the vehicle data indicating such vehicle speed.

[0116] As indicated by operation 324, network service provider 304 causes vehicle data management system 306 to transmit vehicle data to database 308. In examples where multiple network service providers are located in different geographic locations, one or more databases may be collocated, located nearby, and / or in communication with one or more of the multiple network service providers. In this manner, a network (not explicitly shown) may be formed, as described below, allowing vehicle data to be obtained from network service providers located closer to a requesting computing device for some or all of the vehicle. As shown, optionally, database 308 may be included in network service provider 304 (e.g., implemented by the network service provider).

[0117] The database 308 stores the vehicle data, as indicated by operation 326. For example, the database 308 may store the vehicle data based on (e.g., in response to) receiving the vehicle data from the network service provider 304.

[0118] 3B , as indicated by operation 328, computing device 310 receives input from a user. The input may be associated with (e.g., represent) a request for at least a portion of vehicle data. The input may be expressed in any suitable format, such as by using Python code, JavaScript Object Notation (JSON), and / or the like.

[0119] As indicated by operation 330, the computing device 310 generates a request. For example, the computing device 310 may generate the request based on the computing device 310 receiving input from a user. The request may specify one or more events (e.g., one or more event types described herein) corresponding to the vehicle data to be obtained. As indicated by operation 332, the computing device 310 sends the request to the network service provider 304. This may cause the network service provider 304 to have one or more other devices (e.g., the vehicle data management system 306) process the request and return the vehicle data in response to the request.

[0120] As indicated by operation 334, the network service provider 304 retrieves the data from the database 308 in response to the request. For example, the network service provider 304 may retrieve the data in response to the request based on the network service provider 304 processing the request. Alternatively, data associated with the input from the user may be provided directly to the network service provider, and the network service provider may process the input to identify the third-party request. For example, the input may be provided as a logical representation of a query for vehicle data related to vehicle speeds above a 60 mph speed limit. In some embodiments, the input may be written in a customizable format, such as by including any query related to vehicle operating parameters, environment, driver behavior, etc.

[0121] In some implementations, the input may be analyzed by the vehicle data management system 306, which may receive the input as one or more triggers (sometimes referred to as trigger events). In these embodiments, the input may be in the form of computer-executable data, such as an executable script that triggers based on an event specified by a third-party input. For example, the trigger (e.g., a trigger action) may be based on a preset vehicle operating parameter, such as a determination that the vehicle's location is within defined geographic boundary information (e.g., entering a geofence) or a determination that a vehicle operating parameter is above a preset threshold. In some embodiments, the vehicle data management system 306 may perform debugging of the input and update the input if there is an error.

[0122] In some implementations, the vehicle data management system 306 scans the database 308 to provide vehicle data relevant to the third-party request. The vehicle data management system 306 can be configured to scan metadata stored in the database 308. In some embodiments, the vehicle data management system 306 may set scan parameters based on the identified input. For example, if the identified input is related to vehicle speed, the vehicle data management system 306 may scan only vehicle data related to vehicle speed. In this example, the metadata may provide an index for each stored array of data, and the vehicle data management system 306 may reference the index to determine the scan parameters. In some embodiments, the third party may select the scan parameters. For example, the third party may select the array of data and / or the maximum number of vehicle data, such as arrays of image data or video data.

[0123] In some embodiments, vehicle data management system 306 scans stored vehicle data based on a trigger provided as input. If the scan results indicate that the trigger is satisfied, vehicle data management system 306 can select vehicle data to be sent to a third party that requested vehicle data associated with the trigger. As described above, the trigger, in some embodiments, can specify particular data or data types to be sent to the third party. For example, the selected data can correspond to types of vehicle operating parameters, services offered, user preferences, service provider preferences, and / or the like. For example, the vehicle data can include software visioning information, diagnostic logs, sensor values, user information, language preferences, and / or the like.

[0124] As indicated by operation 336, the network service provider 304 transmits the data to the computing device 310 in response to the request.

[0125] 4A-4C , block diagrams of implementations 400 of a process for identifying a subset of data in a dataset responsive to a request are shown. Implementations 400 may include any of the components described herein, such as a vehicle 402, an analytics server 404, and a computing device 406. While implementations 300 may be performed by one or more of vehicle 402 (e.g., a vehicle that may be the same as or similar to ego aircraft 140 of FIG. 1A ), analytics server 404 (e.g., an analytics server the same as or similar to analytics server 110 a of FIG. 1A ), and computing device 406 (e.g., a computing device the same as or similar to administrator computing device 120 and / or server 160 of FIG. 1A ), one or more aspects of implementations 400 may be performed by any number of computing devices operating in a distributed computing system (e.g., a distributed computing system the same as or similar to that described in FIGS. 1A-1C ). For example, one or more computing devices of the ego machine may perform some or all of the steps described in FIGS. 4A-4C locally or in conjunction with an analytics server.

[0126] As illustrated by operation 420 in FIG. 4A , the analytics server 404 receives data related to the operation of at least one vehicle, including vehicle 402 (e.g., a vehicle the same as or similar to ego aircraft 140 in FIG. 1 ). In some implementations, the analytics server receives the data related to the operation of the at least one vehicle periodically, conditionally, or continuously.

[0127] As indicated by operation 422, the analysis server 404 determines multiple events. For example, the analysis server 404 determines a first event ("Event 1") and a second event ("Event 2"), where the first and second events are associated with the operation of a first vehicle ("Vehicle 1"). As shown, both events occur during a time period between T=0 and T=n, which corresponds to the period during which Vehicle 1 is operating. As further indicated by FIG. 4A, where the analysis server 404 does not determine any events during the operation of Vehicle 1 at other times, the analysis server 404 also determines a third event ("Event 3"), where the third event is associated with the operation of a second vehicle ("Vehicle 2").

[0128] 4B , the computing device 406 receives input from a user, as indicated by operation 424. The input may specify one or more parameters that can be used to generate the request. For example, the input may specify one or more parameters that describe vehicle operation above a threshold speed (e.g., 60 mph or greater). The computing device 406 generates the request, as indicated by operation 426. The request is generated based on the input received from the user and specifies one or more events (e.g., one or more event types associated with the one or more events). The computing device 406 may then generate data associated with the request, which can be sent to the analytics server 404. As indicated by operation 428, the computing device 406 sends the data associated with the request to the analytics server 404.

[0129] As indicated by operation 430, the analytic server 404 determines a set of events. In an example, the analytic server 404 determines the set of events based on the analytic server 404 comparing the request to a plurality of events determined by the analytic server 404. Optionally, if additional data is desired (e.g., if an adequate amount of events has not been identified and / or the analytic server 404 is unavailable), the analytic server 404 determines a trigger, as indicated by operation 432. In this example, the analytic server 404 determines a trigger based on the request. As indicated by operation 434, the analytic server 404 transmits data associated with the trigger to the vehicle 402. The data associated with the trigger may be configured to cause the vehicle 402 (e.g., a computing device associated with the vehicle 402) to monitor the vehicle's operation for events that satisfy the requirements of the trigger.

[0130] 4D , the analytics server 404 receives data related to the operation of at least one vehicle based on the trigger, as indicated by operation 436. The analytics server determines a plurality of events based on the data related to the operation of the at least one vehicle, as indicated by operation 438. In this example, the plurality of events occurs between time T=n and time T=n+1.

[0131] 4E , the analytic server 404 generates event data for one or more events, as indicated by operation 440. In this example, the analytic server 404 generates the event data based on the analytic server 404 determining a set of events. The event data represents the events included in the set of events. As indicated by operation 442, the analytic server 404 provides the event data to the computing device 406. The event data may be provided based on the analytic server 404 sending the event data to the computing device 406. As shown, the analytic server 404 provides the event data to the computing device 406 that generated the request.

[0132] 5A-5D, exemplary user interfaces for identifying a subset of data in a dataset are shown, according to one embodiment. User interface 500 may be displayed via one or more computing devices described herein. For example, user interface 500 may be displayed by a display device (e.g., a monitor and / or the like) of a computing device the same as or similar to one or more of analysis server 110a, administrator computing device 120, and / or server 160 of FIG. 1A.

[0133] 5A , a user interface 500 includes a query selection area 502. The query selection area 502 includes a first button 502a and a second button 502b. The first button 502a is associated with a first query type (e.g., a Python-based query), and the second button 502b is associated with a second query type (e.g., a JSON-based query). In some implementations, a computing device may receive input from a user selecting either the first button 502a or the second button 502b. In that case, the computing device may update the user interface 500 based on the computing device recognizing a selection of either the first button 502a or the second button 502b. In some implementations, the stated query may be the same as or similar to the request described herein.

[0134] Upon selection of the first button 502a or the second button 502b, the computing device may update the user interface 500 to include a query window 504. The query window 504 includes an area configured to receive input from a user when submitting one or more queries associated with a first query type or a second query type. More specifically, the area associated with the query window 504 may present a representation of the query as output that the user can review and edit. In some implementations, the area associated with the query window 504 may be associated with a web-based integrated development environment (IDE), such as Amazon's AWS Cloud9, GitPod, and / or the like. In this aspect, the user may provide input (e.g., input corresponding to Python-based code and / or any other suitable format), and the computing device may then generate a query based on the user input. In some examples, the computing device may receive input representing one or more script-based queries representing scenarios such as vehicle cut-in (e.g., a vehicle moving into the ego aircraft's driving lane), cut-out (e.g., a vehicle moving out of the ego aircraft's driving lane), etc.

[0135] The computing device may be configured to reactivate operation of one or more ego aircraft based on a query. For example, in response to a user providing input related to the ego aircraft operating under certain conditions (e.g., low traffic, high traffic, and / or the like), the computing device may obtain data responsive to the query. The computing device may then simulate operation of one or more components of the ego aircraft based on the data responsive to the query. In this example, the user may further update one or more parameters related to the ego aircraft's operation during the resimulation (e.g., by causing the computing device to simulate operation of the ego aircraft based on the data responsive to the query and the one or more updated parameters).

[0136] Input received and displayed via query window 504 may include a query that causes a computing device to retrieve data based on a predetermined data format (e.g., by communicating with a database (e.g., a storage unit such as an Amazon Web Services S3 storage unit (sometimes referred to as an S3 bucket), a Microsoft Azure Blob storage container, and / or the like), or by communicating with another computing device associated with the database, such as an analytics server the same as or similar to analytics server 110a of FIG. 1A). The predetermined data format may define an organizational system in which one or more sensor signals of the vehicle are organized as described herein and stored in a database for later retrieval in response to a query. In an example, the predetermined data format may be associated with one or more raw data formats (e.g., unprocessed sensor signals generated by components of the ego aircraft during operation). The raw data may be indexed, for example, based on a timestamp, etc.

[0137] The query window 504 can also present a representation of a query, the query including one or more scripts that can be executed by a computing device hosting data (e.g., data related to the operation of at least one vehicle). For example, when the first button 502a is selected, one or more queries expressed as Python-based scripts can be presented that the computing device uses to query a database, and when the second button 502b is selected, one or more queries expressed as JSON-based queries can be presented that the computing device uses to query the same database. In some implementations, the JSON-based queries can be the same as or similar to the triggers described herein. In some implementations, the computing device generates an HTTP range request based on the Python-based scripts and / or the JSON-based queries. In this aspect, the computing device can query the database to retrieve only data (e.g., a portion of the data related to the operation of at least one vehicle) in response to the query, rather than, for example, retrieving all data related to the at least one vehicle. This may be particularly useful, for example, when a user is creating a query for data related to the operation of a particular component of a vehicle (e.g., windshield wiper operation on an EGO aircraft), but does not wish to download other data related to the operation of the vehicle (e.g., kinematic data, radar data, and / or the like).

[0138] The user interface 500 includes a filter button 506. In response to receiving user input selecting the filter button 506, the computing device may display a first filter drop-down menu 506a. Additionally or alternatively, in response to receiving user input selecting the filter button 506, the computing device may display a second filter menu 506b (see FIG. 5B ). After displaying, the computing device may receive further input selecting one or more buttons on the first filter menu 506a or the second filter menu 506b. The first filter menu 506a and the second filter menu 506b may define one or more predefined filters that can be applied in addition to or instead of one or more queries. For example, a user may provide data related to a query formatted according to a first query type, and the user may select one or more predefined filters. The computing device may then apply the one or more predefined filters before or after querying a database based on the query formatted according to the first query type or the second query type. In an example where second filter menu 506b is shown, the computing device may update user interface 500 so that query window 504 is positioned below the area of ​​user interface 500 associated with second filter menu 506b.

[0139] 5C , in response to receiving a query, the computing device may update user interface 500 to display data in response to the query. For example, as shown in FIG. 5C , the computing device may update user interface 500 to include multiple regions corresponding to sensor signals identified as corresponding to the query. The sensor signals may be further associated with data related to the operation of at least one vehicle. For example, the computing device may receive one or more sensor signals responsive to the query and display representations of the sensor signals via user interface 500. In one illustrative example, the computing device may display representations of one or more sensor signals generated by a camera in region 508 of user interface 500. In another illustrative example, the computing device may display representations of one or more sensor signals related to an automated driving system or steering system in region 510. In the latter illustrative example, the representations may be representations of the ego aircraft, one or more trajectories associated with the ego aircraft (e.g., one or more proposed trajectories and / or executed trajectories at a point in time), the velocity of the ego aircraft, the geographic location of the ego aircraft, and / or the like.

[0140] Referring now to FIG. 5D , a computing device may alternatively present a user interface 500′ representing sensor signals responsive to a query. In one illustrative example, the computing device generates and provides a user interface 500′ in which sensor signals returned based on a query are sorted. In this example, the user interface 500′ includes a tree region 512 and a data representation region 514. The tree region may include one or more buttons corresponding to the sensor signals, which are sorted into sets of signals and corresponding sets of sub-signals. For example, as shown in FIG. 5D , the tree region includes a button 512a titled “pose_timestamp” that references the pose (position and orientation) of the ego aircraft involved in the query. In response to the computing device receiving input from a user selecting the button titled “pose_timestamp,” the computing device updates the tree region 512 to display multiple buttons corresponding to the sub-signals. The buttons may be titled "geographic_heading," which refers to the direction the ego aircraft is pointed in (e.g., north, northeast, east, and / or the like), "geographic_position_lat_degree," which refers to a latitude coordinate associated with the ego aircraft's position, "geographic_position_lon_degree," which refers to a longitude coordinate associated with the ego aircraft's position, etc. In response to a user providing input, such as selecting "pose_timestamp," the computing device may update data representation area 514 with a representation of one or more ego aircraft vehicle attitudes or multiple sensor signals over a period of time. The representation may be visual (e.g., a graph) and / or text-based (e.g., a table of values).

[0141] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure or the claims.

[0142] Some embodiments of the present disclosure are described herein in connection with a threshold value. As described herein, meeting a threshold value may refer to greater than the threshold, more than the threshold, higher than the threshold, equal to or greater than the threshold, less than the threshold, less than the threshold, lower than the threshold, less than the threshold, equal to or less than the threshold, and / or similar values. As used herein, "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more" and "at least one." Additionally, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and may be used interchangeably with "one or more" or "at least one." When only one item is intended, the term "a" or similar term is used. Additionally, as used herein, terms such as "has," "have," and "having" are intended to be open-ended terms. Additionally, the phrase "based on" is intended to mean "based at least in part on," unless otherwise specified.

[0143] Computer software-implemented embodiments may be implemented in software, firmware, middleware, microcode, hardware description languages, or any combination thereof. A code segment or machine-executable instruction may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0144] The actual software code or specialized control hardware used to implement these systems and methods is not a limitation of the claimed features or this disclosure. Accordingly, the operation and behavior of the systems and methods are described without reference to specific software code, and it will be understood that software and control hardware can be designed to implement the systems and methods based on the description herein.

[0145] If implemented in software, the functions may be stored as one or more instructions or code on a non-transitory, computer-readable, or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium. Non-transitory computer-readable or processor-readable media include both computer storage media and tangible storage media that facilitate transfer of a computer program from one place to another. Non-transitory processor-readable storage media may be any available medium that can be accessed by a computer. By way of example, and not limitation, such non-transitory processor-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or any other tangible storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer or processor. As used herein, disc and disk include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), Blu-ray discs, and floppy disks, with a "disk" typically reproducing data magnetically, while a "disk" reproducing data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media. Furthermore, the operations of a method or algorithm may reside as one or any combination or set of code and / or instructions on a non-transitory processor-readable medium and / or computer-readable medium, which may be incorporated into a computer program product.

[0146] The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the embodiments described herein and variations thereof. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the spirit or scope of the subject matter disclosed herein. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the following claims and the principles and novel features disclosed herein.

[0147] While various aspects and embodiments have been disclosed, other aspects and embodiments are contemplated. The various disclosed aspects and embodiments are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

1. 1. A system for identifying a subset of data in a dataset, comprising: receiving data relating to the operation of at least one vehicle within an environment, the operation of the at least one vehicle occurring during at least one time period; determining a plurality of events that occurred during the at least one time period based on the movement of the at least one vehicle within the environment, each event having an event type; receiving data from the computing device relating to a request specifying one or more event types; determining a set of events from among the plurality of events based on the one or more event types specified by the request, wherein each event from the set of events occurs at a time point associated with the at least one time period; generating event data relating to operation of the at least one vehicle within the environment at a time corresponding to each event of the set of events; and transmitting to the computing device associated with the request the event data configured to cause a display associated with the computing device to generate a user interface representing operation of the at least one vehicle. A system comprising at least one programmed processor.

2. the request specifies at least one event type corresponding to a change in a state of the at least one vehicle; When determining the plurality of events, the at least one processor: The system of claim 1 , programmed to determine a change in the condition of the at least one vehicle during the at least one period of time.

3. When determining the plurality of events, the at least one processor: The system of claim 1 , programmed to determine the presence of one or more scenarios, agents, or objects in the environment at a point in time during the at least one period of time.

4. When generating the event data related to operation of the at least one vehicle within the environment, the at least one processor:

2. The system of claim 1, further programmed to generate the event data based on the set of events and the data generated during operation of the at least one vehicle at times corresponding to the occurrence of each event in the set of events.

5. the at least one processor: determining, based on the request, a trigger configured to cause a computing device located in the at least one vehicle to store the data related to operation of the at least one vehicle in the environment; and providing data associated with the trigger to the at least one vehicle, the data being configured to cause the trigger to be implemented by the computing device located in the at least one vehicle; The system of claim 1 further comprising a program.

6. the at least one processor:

6. The system of claim 5, further programmed to receive the data related to operation of the at least one vehicle within the environment based on providing the data related to the trigger to the at least one vehicle.

7. the at least one processor: determining, based on the request, a trigger configured to cause a computing device located in the at least one vehicle to store the data related to operation of the at least one vehicle in the environment; and determining whether the trigger satisfies a probability threshold based on applying the trigger to the data related to operation of the at least one vehicle within the environment; The system of claim 1 further comprising a program.

8. the at least one processor: providing, based on determining that the trigger satisfies the probability threshold, data related to the trigger to the at least one vehicle configured to cause implementation of the trigger by the computing device located in the at least one vehicle; The system of claim 7 further comprising a program.

9. the at least one processor: updating the trigger based on determining that the trigger does not meet a probability threshold; and providing, to the at least one vehicle, the data related to the trigger, configured to cause implementation of the trigger by the computing device installed in the at least one vehicle, based on updating the trigger; The system of claim 7 further comprising a program.

10. receiving, by at least one processor, data relating to the operation of at least one vehicle within an environment, the operation of the at least one vehicle occurring during at least one time period; determining, by the at least one processor, a plurality of events that occurred during the at least one time period based on movement of the at least one vehicle within the environment, each event having an event type; receiving, by the at least one processor, data associated with a request from a computing device, the request specifying one or more event types; determining, by the at least one processor, a set of events from among the plurality of events based on the one or more event types specified by the request, wherein each event from the set of events occurs at a time point associated with the at least one time period; generating, by the at least one processor, event data relating to operation of the at least one vehicle within the environment at a time corresponding to each event of the set of events; transmitting, to the computing device associated with the request, the event data configured to cause a display associated with the computing device to generate a user interface representing operation of the at least one vehicle; 11. A computer-implemented method comprising:

11. the request specifies at least one event type corresponding to a change in a state of the at least one vehicle; determining the plurality of events The computer-implemented method of claim 10 , further comprising determining a change in the state of the at least one vehicle during the at least one period of time.

12. determining the plurality of events The computer-implemented method of claim 10 , further comprising determining the presence of one or more scenarios, agents, or objects in the environment at a point in time during the at least one period of time.

13. generating the event data related to operation of the at least one vehicle within the environment, 11. The computer-implemented method of claim 10, further comprising generating the event data based on the set of events and the data generated during operation of the at least one vehicle at times corresponding to the occurrence of each event in the set of events.

14. determining, by the at least one processor based on the request, a trigger configured to cause a computing device located in the at least one vehicle to store the data related to operation of the at least one vehicle in the environment; providing, by the at least one processor, data associated with the trigger to the at least one vehicle such that the trigger is implemented by the computing device located in the at least one vehicle; The computer-implemented method of claim 10 further comprising:

15. receiving, by the at least one processor, the data related to operation of the at least one vehicle within the environment based on providing the data related to the trigger to the at least one vehicle; The computer-implemented method of claim 14 further comprising:

16. determining, by the at least one processor based on the request, a trigger configured to cause a computing device located in the at least one vehicle to store the data related to operation of the at least one vehicle in the environment; determining, by the at least one processor, whether the trigger satisfies a probability threshold based on applying the trigger to the data related to operation of the at least one vehicle in the environment; The computer-implemented method of claim 10 further comprising:

17. providing, by the at least one processor, data related to the trigger to the at least one vehicle, such that the trigger is implemented by the computing device located in the at least one vehicle, based on determining that the trigger satisfies the probability threshold; The computer-implemented method of claim 16 further comprising:

18. updating, by the at least one processor, the trigger based on determining that the trigger does not meet a probability threshold; providing, by the at least one processor, the data related to the trigger to the at least one vehicle, such that the trigger is implemented by the computing device located in the at least one vehicle based on updating the trigger; The computer-implemented method of claim 16 further comprising:

19. A non-transitory machine-readable medium having stored thereon computer-executable instructions, the computer-executable instructions, when executed by one or more processors, causing the one or more processors to: receiving data relating to the operation of at least one vehicle within an environment, the operation of the at least one vehicle occurring during at least one time period; determining a plurality of events that occurred during the at least one time period based on the movement of the at least one vehicle within the environment, each event having an event type; receiving data associated with a request from a computing device, the request specifying one or more event types; determining a set of events from among the plurality of events based on the one or more event types specified by the request, wherein each event from the set of events occurs at a time point associated with the at least one time period; generating event data relating to operation of the at least one vehicle within the environment at a time corresponding to each event of the set of events; transmitting, to the computing device associated with the request, the event data configured to cause a display associated with the computing device to generate a user interface representing operation of the at least one vehicle; A non-transitory machine-readable medium for causing an operation to be performed, including

20. the request specifies at least one event type corresponding to a change in a state of the at least one vehicle; The computer-executable instructions for causing the one or more processors to determine the plurality of events may include causing the one or more processors to:

20. The non-transitory machine-readable medium of claim 19, causing a change in the state of the at least one vehicle during the at least one period of time to be determined.

Images