Device Message Framework

Through a hierarchical data structure, the equipment data is organized according to the topic and time, and the problems of low data access efficiency and waste of resources in the prior art are solved, and efficient data storage and access are achieved.

CN113168403BActive Publication Date: 2024-11-19祖克斯技术有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980082575.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-12-13
Filing Date
2019-12-10
Publication Date
2024-11-19
Estimated Expiration
2039-12-10

AI Technical Summary

Technical Problem

In the prior art, the large amount of data generated by the device has unnecessary waste of bandwidth, storage and computing resources during transmission and storage, and data access is inefficient, especially when analyzing specific data, traditional file structures lead to excessive data volume. Large, inflexible and difficult to access.

Method used

Using a hierarchical data structure to decompose device data into small files by topic and time, use a directory or a hierarchical data structure for organization and storage, supports efficient access and analysis, indexing and accessing through topic, time and device identifiers, providing intuitive And efficient data access method.

Benefits of technology

It realizes fast and efficient access and storage of device data, reduces unnecessary resource consumption, supports more complete and flexible data record management, and improves data access efficiency and storage efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113168403B_ABST
    Figure CN113168403B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a hierarchical data structure and systems and techniques for using the hierarchical data structure. In one instance, first device data may be received, the first device data comprising a plurality of files and corresponding to message data generated by one or more components of a device during a certain time period. The first plurality of files may be stored on one or more data stores and at a first location according to a hierarchical data structure, the hierarchical data structure being based at least in part on a message time, a data identifier, or a device identifier. Second device data associated with the certain time period and comprising a second plurality of files may be obtained. The second device data may then be stored at a second location on the one or more data stores according to the hierarchical data structure.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Citation of Related Applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 219,623, filed on December 13, 2018, entitled “Device Messaging Framework,” the entire disclosure of which is incorporated herein by reference in its entirety for all purposes. Background Art

[0003] Many devices, including robotics, computing, and autonomous or semi-autonomous vehicles, can generate large amounts of data that are typically analyzed and stored away from the device itself. When logged into a single file, such data can create very large files, especially given long time spans. Such large file sizes can cause problems with long-term recording of sequential messages and offline manipulation when transferred to a remote computing location, as they typically need to be copied locally before they can be opened. In addition, many times, only a small portion of the data is needed for analysis purposes, resulting in a large amount of data being transferred and accessed relative to the data needed. This, in turn, can lead to unnecessary use of bandwidth, storage, and computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The same reference numbers used in different drawings indicate similar or identical components and features.

[0005] Figure 1 An example system is shown in which an autonomous vehicle data services system receives data from autonomous vehicles and uses a hierarchical data structure to store, access, and perform analysis on the data.

[0006] Figure 2 Shown in Figure 1 An example diagram of how data is broken down into files in a hierarchical data structure.

[0007] Figure 3 An example system is shown in which an autonomous vehicle data services system receives data from autonomous vehicles and uses a hierarchical data structure to store, access, and perform analysis on the data.

[0008] Figure 4 Show autonomous vehicle services (such as Figure 3 Example communication between different aspects of a service).

[0009] Figure 5 An example of a hierarchical data structure including data layers is shown.

[0010] Figure 6 Show Figure 5 An instance of a hierarchical data structure of the contents of a file.

[0011] Figure 7 Shows how to Figure 5 An instance of a hierarchical data structure that accesses data from two related but distinct file trees or data structures.

[0012] Figure 8 Show Figure 5 Another instance of a hierarchical data structure of two related but different file trees or data structures.

[0013] Fig. 9 An example process for accessing and storing robotic device data in a hierarchical data structure is shown.

[0014] Fig.10 An example process for accessing and combining robotic device data using a hierarchical data structure is shown.

[0015] Fig.11 Example processes for receiving, storing, accessing, and combining data using a hierarchical data structure are shown.

[0016] Fig.12 Shows an example of a subsystem that provides messages and data to a data service system. DETAILED DESCRIPTION

[0017] The present disclosure describes methods, devices and systems for effectively organizing, accessing, and performing analysis and operations on data obtained from devices (such as, robotic devices or autonomous driving cars) using directories or hierarchical data structures. Devices (such as, autonomous vehicles) can generate multiple messages during normal operation, including log data, sensor data and decision points. As described herein, the device itself or the service receiving the data can write the data into a hierarchical data structure to enable faster access and more efficient and intuitive analysis and operations to be performed on the data. The hierarchical data structure can organize messages into different topics (data identifiers). Files in the hierarchical structure (which can be the lowest level in the data structure) can include one or more messages (e.g., logs or other data) for a specific topic. Each file can be associated with its record or the time to which it belongs, and can be less than a threshold size used as a limit on the file size. The time value can represent the second level of the data structure. In some aspects, the device or autonomous vehicle can organize the data created by the device or autonomous vehicle in this way so that certain data can be reported faster, more efficiently and more accurately to the linked service or system.

[0018] In some aspects, a data structure can be used to modify and store one or more files of a data structure, for example, in one or more layers of a data structure. In some instances, the data is stored back in the data structure, and in other instances, the data is stored in another data structure that organizes the data in the same manner. In one example, this can include bundling together certain topics from the same time window, excluding other topics, changing values ​​in one or more files, etc. Then, such modified data can be stored in a subdirectory or layer (e.g., a variant layer) in, for example, a data structure. In this way, analysis or preprocessing can be performed on different files and stored in the same data structure to enhance downstream processing and make access to modified data more efficient.

[0019] In some aspects, the data structure may include a level that specifies a device (such as a vehicle) to which one or more files belong, such as in the case of a multi-vehicle data management system or service. The root layer may be implemented to identify one or more other characteristics of the data, such as storage location, access permissions, or constraint information, etc. In this way, data may be organized, processed, and accessed intelligently and efficiently.

[0020] In some aspects, the lowest time level may represent minutes, although any other time intervals are contemplated (e.g., milliseconds, seconds, hours, etc.), such that each file may include, for example, data generated by a device at a one-minute interval, such as one or more messages generated on an autonomous vehicle. As described herein, files contain a small amount (less than a threshold amount) of data, and therefore may provide a number of specific benefits and advantages over current systems. The described data directory or hierarchical data structure may address one or more identified problems with previously used systems, including handling excessive amounts of data per one or more files, inflexibility, arbitrary topic groupings, difficulty in scaling data storage, slow access, dependency on files of a specific operating system (e.g., ROS), lack of a unified storage and access location, and no unified API access.

[0021] For example, some data runs or time spans using a logging system may be greater than 100GB, which would take 1.5 hours to download at 20MB / sec. By separating the data into shorter time periods and organizing by more customizable topics, the data can be downloaded or communicated to another system more quickly, and the relevant data can be accessed more quickly. The described data directory can provide small granular files that are easy to locate and access. The data directory can provide topic access directly by name, thereby providing an intuitive and efficient way to access data at a very granular level.

[0022] The described hierarchical data structure can store files organized at multiple levels for certain topics without the need to crop or otherwise exclude data from the record, and thus provide a more complete data record that can better support a wider variety of devices than existing methods. In some aspects, video files in the described hierarchical data structure can be processed through a random access API to further alleviate the inefficiency of storing and accessing large amounts of data.

[0023] The described data directory can interface with different systems and can support standard storage technologies.

[0024] As described in more detail below and used throughout the text, a message can represent the lowest level unit in a hierarchical data structure. In some aspects, a message can represent a message sent on a vehicle (or other device), and can include associated timestamps and metadata. In some aspects, a message can include log data (e.g., data from a sensor, from an intermediate process, from a component and / or subcomponent response of a device / subsystem of a system, etc.). A topic can be the type of message sent on a vehicle or device. A topic can be predefined in an autonomous vehicle service, or can be configurable. In some aspects, an example topic can include data related to or generated by a specific subsystem (such as a lidar subsystem, a radar subsystem, an object determination subsystem, or various other subsystems) of a robotic device or autonomous vehicle. In some other instances, a topic can include, for example, certain decisions made in response to a specific trigger, inputs from multiple subsystems for making one or more determinations or decisions, data related to keywords from one or more subsystems, etc.

[0025] A window may refer to a time period into which a topic is broken down. For example, in order to balance file size and data freshness, it may be beneficial to divide a topic into one-minute windows. A file may represent data related to a window of a topic. A file may contain one or more messages about a specific topic. In some instances, a one-minute window for a file may be equal to an average file size of less than 1MB (where larger files are in the range of 70 to 100MB), which may be particularly beneficial for quickly sending or streaming data and accessing data in response to requests. A layer or subdirectory may refer to a layer or set or group of data having a root, a variant identifier, and associated data or files, which may include one or more additional subdirectories into which the associated data or files are organized. Different layers or subdirectories may share part or all of the same level structure, including a time level, a topic level, a device identifier level, etc., but may contain different data or files. A variant or variant layer may be, for example, an alternative message set modified in some way based on data collected from a vehicle. Such variant layers can provide additional functionality (such as, but not limited to, associating a data set associated with one of multiple levels of access control), providing storage of certain portions on a variation system for faster access, providing data sets that have been independently modified (e.g., by testing algorithms) and / or combinations of data from other layers. In some aspects, variants can be optimized for performing customized analysis, more efficient data preprocessing, and / or offline use. In some aspects, the operation of an autonomous vehicle can assume a time measured in nanoseconds. During the retrieval of data from one or more data stores, such variants and themes can be used for access control purposes. As a non-limiting example, certain variant layers can be stored on one data store, while other variant layers can be stored in another data store. Data from one data store may be available only to one user, location, etc., based at least in part on the user, variant, location, etc. Further, when storing data, variant layers and themes can be selectively stored in different data stores for optimal retrieval. As a non-limiting example, frequently accessed data can be stored in a local root, while infrequently accessed associated variants and / or themes can be stored in a data store that is far away and / or has a longer access time. In some cases, different tiers of storage may be determined based on latency and / or frequency of access to data.

[0026] Figure 1 An example system 100 is shown in which a service 102 receives data from a device 106, which, as depicted, may include an autonomous vehicle that uses a directory structure to store, access, and perform analysis on the data. It should be understood that the device 106 is provided as an autonomous vehicle only as an example. The described techniques are applicable to a variety of devices, including devices having robotic components or features, such that a variety of other devices may be advantageously utilized in conjunction with the described hierarchical data structure.

[0027] The device 106 may generate various data regarding sensor readings, data from various subsystems of the device 106, decision points made, processing steps, etc. In some cases, the data may be very large. As shown, the data is represented by sensor data 110, which may include lidar data 112 and image data 114, and decision data 108, which may include various other data, including log data, etc. As described below, Fig.12 Detailing various subsystems in an autonomous or semi-autonomous vehicle as examples of devices 106, they may generate and report this or other types of data, which may be collectively referred to herein as device or vehicle data.

[0028] In some aspects, the data generated by the different subsystems of the device 106 can be in the form of messages or log data. These messages can be generated by the subsystem based on data obtained by one or more sensors of the device 106, for example. In some cases, one or more computing systems on the device 106 can encapsulate messages or generate messages from raw or unencapsulated data. In any of these cases, the device 106 or its computing system can group the messages into files according to the different topics to which the messages or data belong. In some aspects, the topics can include different operating parameters of the device, sensor readings, operational decisions, etc. The topics can be predefined for the service 102, can be predefined for a subset of the device 106 managed by the service 102, and / or can be customizable.

[0029] Device 106 can package raw data or messages into files for different topics. In some aspects, device 106 can package data into files at regular or variable intervals (e.g., at one-minute intervals). Device 106 can send files to service 102, and service 102 can store data in data store 104 according to a hierarchical data structure, which is indexed by at least topic and time.

[0030] The service 102 may include different computing systems (such as, one or more servers, virtual computing instances, data storage areas, etc. and combinations thereof) to facilitate the collection of device data including decision data 108 and sensor data 110 from multiple devices 106. In some aspects, the service 102 may include storage devices, one or more processors, memory, and an operating system. The storage devices, processors, memory, and operating system may be communicatively coupled via a communication infrastructure. Optionally, the computer system may interact with a user or environment via an input / output (I / O) device and one or more other computing devices (such as, devices 106 on a network) via a communication infrastructure. The operating system may interact with other components to control one or more applications. In some cases, the computer system may implement any hardware and / or software to implement the different subsystems described herein.

[0031] In some cases, data store 104 may be provided by or associated with service 102, or may be implemented by a separate system or service.

[0032] The service 102 can provide access to data via an indexing system keyed to a hierarchical data structure. In some aspects, the service 102, the device 106, or a combination thereof can attach or otherwise organize files indexed by time into different other hierarchical levels, such as vehicle or device identifiers, root values, and other identifiers. The hierarchical data structure or directory can be used by the service 102 to store and provide access to specific files by root, device, theme, and time. In some cases, the service 102 can further enable modification of one or more files in the hierarchical data structure. The service 102 can also store one or more modified files back into the hierarchical data structure, for example, by adding or changing the layer to which the file belongs. In this way, the service 102 can facilitate fast and efficient access to device data on the granularity of the theme indexed by time.

[0033] Figure 2 An example diagram 200 illustrates the manner in which device data is broken down into files in the described directory or hierarchical data structure.

[0034] Under the hierarchical data structure represented by the files 214 described herein, each file 214 represents one of a plurality of different topics 216, 218, 220, 222 for a given time period or window 224. In this way, each topic is contained in a series of files, separated by time intervals. In some aspects, a one-minute time interval of the window size 224 can provide a balance of useful information and manageable file size.

[0035] In some aspects, topics 216, 218, 220, 222 of a series of files may be defined in a more precise and narrow manner than topics 204, 206, 208, 210 of package 202 represented by another data organization scheme 212, such as including custom topics. In some examples, topics 216, 218, 220, 222 may be configured to correspond to any of a number of subsystems of an autonomous vehicle, operating standards, types of data or messages related to different decision points, and the like.

[0036] Figure 3 Another example system 300 is shown, in which a data service 302 receives data 310 from an autonomous or semi-autonomous vehicle 318 using a directory structure to store, access, and perform analysis on the data. The system 300 can be combined with the above reference Figure 1 For example, service 302 may incorporate one or more aspects of service 102. Figure 1 Similarly described, the service 302 can receive vehicle data 310 from one or more vehicles 318. The service 302 can receive data in a hierarchical data structure or classify the data into a hierarchical data structure, which is organized by at least one of a subject, time, vehicle, root, or other value. In some cases, data from multiple devices or vehicles 318 can be combined or merged into a single data structure using a hierarchical data structure, which will be described in more detail below. Amalgamation can take various forms and combinations of forms, such as by adding or combining data sets from different vehicles, performing calculations (e.g., statistical calculations, testing of additional / alternative processes to be run on the device (e.g., proposed new algorithms for perception, planning, prediction, etc.), etc.) to create data points based on data from multiple vehicles, etc. In the illustrated example, the service 302 can also include a permission manager 304, a storage manager or builder 306, and a data interface or reader 308. Each of the rights manager 304, storage manager 306, and data interface 308 may be a service running on or provided by the service 302, and / or may operate on a combined or different computing system, such as the service 302. As also shown in the example system 300, the data store 312 may be separate from the service 302 and hosted or provided by a data storage service 314 or other networked computing service provider or system. In this example, the data store 312 (which may be separate or integrated into the service 302) may store files according to a hierarchical data structure 316 that includes different levels for year / month / day, hour, minute, second, vehicle, and subject.

[0037] Based on the data structure 316, the storage manager or builder 306 can receive the vehicle data 310 and write the vehicle data 310 to the data store 312. In some aspects, the builder 306 can write the data to the data store 312 using the location and vehicle name or identifier to which the data will be written, for example, in the data store 312. In some aspects, the builder 306 can associate the vehicle data 310 with different layers and / or levels (such as multiple time levels and subject identifiers) of the hierarchical structure 316. It should be understood that the vehicle 318 and the builder 306 can both perform operations to associate the vehicle data 310 with various layers to be stored in the data store 312, either separately or in combination. In some cases, the builder 306 can maintain an index or other data structure that associates different files with different locations in the data store 312 to enable access to various files of the vehicle data 310. In some cases, this index can include a directory of all root names and layer indicators to facilitate easy access to the files of the vehicle data 310.

[0038] In order to facilitate preprocessing, reprocessing, adding, providing output from new intermediaries (e.g., testing new planning systems, such as planning system 1228 described below or other systems of vehicle 318), grouping, or other operations on data, builder 306 can also write the same or similar data to different layers. These different layers are referred to as variant layers in this article. These variant layers can be optimized for customized analysis or offline use, as will be described in more detail below.

[0039] In some aspects, the builder 306 may store different files in different locations to optimize various access requirements to the vehicle data 310. For example, the builder 306 may store some files of the vehicle data 310 to a local or easily accessible memory location or data store. This may be particularly useful for files that are accessed more frequently than other files. Similarly, for example, the builder 306 may store some files of the vehicle data 310 in a remote memory location to reduce the cost and resources required to store large amounts of vehicle data 310. In some aspects, storing the files of the vehicle data 310 in different locations may be further facilitated or performed in whole or in part by the data storage service 314.

[0040] The data interface / reader 308 can read or access vehicle data 310 stored in the data store 312. In some aspects, the reader 308 can access files in the data store based on the storage location and the vehicle name. In some cases, the reader 308 can facilitate running queries on the data stored in the data store 312 and accessing specific files of the vehicle data 310. The reader can receive information specifying what files to obtain, for example, via one or more interfaces provided by the autonomous vehicle service 302. In some aspects, files of vehicle tool data can be located and retrieved by searching for data ranges. The range can include a time range, such as a start and stop time value, or a duration with a start or end point. The range information can also include or be accompanied by a vehicle identifier and one or more topics.

[0041] In some aspects, the reader 308 may receive instructions to query and obtain a certain set of files, for example, identified by a range (including a time value, a vehicle, and a subject). In some aspects, the reader 308 may search, for example, an index of vehicle data 310 of stored data in a data store 312 maintained by the service 302 to identify a set of files corresponding to the range information. In this instance, the index may include an inventory of what data is stored in the data store 312 according to the various layers in the hierarchical data structure and / or the location of the stored data. In other cases, the data storage service 314 may identify the requested file based on the range information. In either case, the reader 308 or the data storage service 314 may obtain the requested vehicle data 310 from the data store 312. The reader 308 may then provide the requested file to the service 302 for responding to the initial request. In some aspects, for example, when a request or query specifies a subset of data contained in a file selected to comply with the request, at least a portion of the data in the selected file may be provided. In some cases, the messages within the file obtained in response to the query may be provided in the order in which they were recorded, for example, by a device or vehicle. This may include continuously streaming or otherwise providing the message in a file to re-record the message at a first instance (eg, playback of an event occurring at a device or vehicle).

[0042] In some aspects, variant or replacement layers may be created or stored back into the data structure 316. These variant layers may include combining files from different layers of the data structure, removing some files from a single layer, and other operations and modifications. Advantageously, variant layers may be read and written by simply specifying the variant layer name (e.g., using a URL) in the read or write instructions, without requiring other definitions or parameters.

[0043] In some examples, one or more files may be modified by builder 306 using a different variant layer identifier and stored back to data store 312 .

[0044] In some aspects, the data structure 316 and the data store 312 can support different programming languages ​​based primarily on a layer and file hierarchy (e.g., C++, Python, REST, etc.). Because there is a direct mapping from time, device (e.g., vehicle), and subject to file for efficiency, interfacing with the data store 312 and the data hierarchy 316 can be relatively easily configured for a variety of different languages, using different API calls, etc. In some instances, the data structure 316 can be stored in the data store 312 using different formats (including SSTable format), which can provide a well-tested API to further ensure efficient access to the data. In some cases, the data can be compressed when stored in the hierarchy 316, which can improve the communication of data via a limited bandwidth channel.

[0045] In some aspects, a layer may be stored, accessed, or called using a uniform resource locator (URL) or other uniform resource identifier (URI) that may specify a subject, variation, time, etc. In this way, a specific file from a specific time period generated by a specific vehicle or device may be easily located and recalled. In some cases, a layer itself or a layer referencing another layer may indicate to which file subject, time, device, etc. the data stored in the layer relates.

[0046] In some cases, if a request is received to access or recall one or more files using a time range or start and end values ​​for a file request, and the specified time is between two time subdirectories, the reader can select the earliest file at the beginning of the range and the latest file at the end of the range to provide a more complete response to the request.

[0047] In some cases, the service 302 may include a permissions manager 304 that can control access to different files of the vehicle data 310, for example, at the root level, the variant layer level, the subject level, the time level, the device identifier level, or a combination thereof. The permissions manager 304 may associate different permissions with different files to control what entities are allowed to access certain files of the vehicle data 310. In some cases, the permissions manager 304 may group files by subject, vehicle, or different variant layer attributes. The permissions manager 304 may then associate permissions with different data groups based on who may need access to a specific subset of the vehicle data 310, or to restrict access to one or more groups or accounts. For example, a variant layer may define certain operations performed on vehicle data (e.g., which may include a combination of different files or subjects), indicating one or more security measures implemented on the autonomous vehicle 318. It may be desirable to restrict who can access secure data, for example, if the data is confidential, or only a certain group may ever access the data.

[0048] The permissions manager 304 may associate one or more permissions with one or more files or layers and communicate the information to the builder 306 and / or the data storage service 314 to restrict or control future access to the data.

[0049] In some aspects, the permissions manager 304 may have or provide an administrative interface to enable an administrator to define, configure, or select certain permissions and associate those permissions with different groups of users, entities, IP addresses, or other characteristics. In some aspects, certain data may be associated with one or more variation layers to further enable more precise and granular access control to different data in the hierarchical data structure. For example, navigation control data may be generated by the vehicle 318. The navigation control data may be split and stored in different variation layers to enable different groups to more efficiently access different pieces of the navigation control data. For example, one group may be responsible for monitoring and improving a tracking subsystem, such as described below. Fig.12 The tracking subsystem 1226 described above may be responsible for monitoring to ensure that the perception and prediction subsystem 1212 is working correctly, and another group may be responsible for making improvements to the drive controller 1270. The navigation control data may be associated with and stored in different variant layers depending on what is useful to each group and the relevant data for each group. Each variant layer may be associated with access rights that specify a user, group, or other identifying information in the form of credentials that allow access to the data in each variant layer.

[0050] Access rights can enable groups to effectively access different subsets of navigation control data by limiting access to data that is not relevant to the different groups. In some cases, access rights can be used to prevent certain groups or users from accessing sensitive data, such as to comply with one or more regulations or rules imposed by an external organization, for data privacy reasons, etc.

[0051] Figure 4 Example communications 400 between autonomous vehicle services and different aspects of an autonomous vehicle are shown, such as those described above with reference to Figure 1 and 3 In some aspects, the vehicle sensors 402 and / or the onboard computing system 404 may be autonomous or semi-autonomous vehicles (such as, for example, as described above with reference to Figure 1 and Figure 3 A portion of the vehicle 106, 306) described herein, and / or including the following references Fig.12 In addition, vehicle data storage 410 may include the above-referenced Figure 1 and Figure 3 One or more aspects of the vehicle data stores 104 and 312 are described.

[0052] The vehicle sensor 402 can obtain various sensor data, as shown below. Fig.12 At operation 412, the sensor data may be communicated to one or more onboard computing systems 404. In some cases, as described below with reference to Fig.12 As described in more detail, the vehicle sensor 402 can communicate sensor readings and data 412 to one or more computing systems of one or more vehicle subsystems. In either case, the central onboard computing system 404 or one or more of its subsystems can process the sensor data at operation 414 during the operation of the vehicle. Operation 414 can generate operational data in the form of multiple messages about various aspects of the sensor data and the operation of the vehicle, for example, the operational data can be communicated between the various subsystems of the vehicle. In some aspects, the message can include log data. As a non-limiting example, such an onboard computing system 404 can perform operations such as detection, classification, object segmentation, positioning of the vehicle relative to a map, etc. based at least in part on sensor data from one or more sensors. Such detection, classification, segmentation, positioning, predicted object motion, etc. can be output in the form of one or more messages. Furthermore, in such an instance, other processes can consume such messages to create more messages (for example, the planning system described below can create messages to be consumed by the vehicle controller based on the above message data). Of course, any number of components and subcomponents can transmit data between each other. In any such instance, the message can have a corresponding timestamp as provided by the central timing server process.

[0053] In some aspects, at operation 416, the vehicle computing system 404 or one or more computing systems of one or more subsystems of the vehicle can encapsulate sensor data and operation data into a file, for example, according to a hierarchical data structure as described above. In some aspects, operations 414 and / or 416 may include the computing system 404 collecting and recording messages generated from multiple subsystems of the autonomous vehicle. The computing system 404 may group messages that may include log data into multiple different topics. Grouping can be performed in many different ways, including by the subsystem that generates the message, the task or process performed (which may span one or more subsystems), by keywords, or by other metrics or schemes. Message data can be decomposed into time periods. In some aspects, time periods or windows can be predefined, such as at one-minute intervals, but any other time periods are contemplated. In some cases, such time windows can be determined based on the frequency and amount of the data received to ensure that the amount of data contained therein does not exceed the threshold data limit. One or more of operations 412, 414, 416 and / or 418 can be performed at different times (such as, at regular intervals, at configurable and / or variable intervals, or when one or more events occur). The one or more events may include an on-board event such as triggered by one or more sensors or subsystems of the vehicle.

[0054] At operation 418, onboard computing system 404 and / or other components of the autonomous vehicle may communicate sensor and operational data to autonomous vehicle services 408, for example, over one or more networks. Where the vehicle computing system packages the data at operation 416, the data may be communicated to the autonomous vehicle in smaller files (e.g., most are less than 1MB, and some larger files are up to about 70 to 100MB). In some aspects, packaging and communicating the data at 432 represented by operations 412, 414, 416, and 418 in smaller windows may reduce the stress on onboard computing system 404 in sending data to autonomous vehicle services 408. In some aspects, the files packaged at operation 416 may be small enough that no data buffer is needed, or alternatively, a small or large data buffer may be used.

[0055] Upon receiving the sensor and operational data at operation 418, autonomous vehicle service 408 may organize the data into a hierarchical data structure at operation 420 and store the data in vehicle data store 422 at operation 422. In some aspects, operation 420 may include associating files of vehicle data (which may already be associated with a time value and a subject) with additional levels in the data hierarchy (such as a vehicle identifier, a root value, and / or one or more variant layers). Multiple files of different subjects may be grouped or organized into sub-directories or layers, organized by time value.

[0056] In some aspects, at operation 424, autonomous vehicle service 408 may receive one or more instructions to access and / or modify some vehicle data. In some aspects, operation 422 may include receiving instructions to modify data, including creating a new subset of data represented by a variant layer having one or more characteristics of a file or theme included therein, which may be changed or modified from the original data obtained. In response, at operation 426, autonomous vehicle service 408 may request and obtain data from vehicle data store 410. In some cases, autonomous vehicle service 408 may modify data according to instructions received at operation 424, at operation 428, and store the modified data to data store 410 in a hierarchical data structure at operation 430. In this way, communication 400 may facilitate easy access and manipulation of vehicle data.

[0057] In some aspects, the instructions received at operation 424 to modify the data may include a number of different processes to be performed on the data, including adding or removing certain files from one or more subdirectories of the hierarchical data structure to create new subdirectories.

[0058] Figure 5 An example of a file structure is shown, which may be referred to as a subdirectory, tree, or layer 500 in a directory data structure, including multiple different levels of data. The tree 500 may include different levels organized in a hierarchical manner to more efficiently store data received from one or more devices (such as, autonomous or semi-autonomous vehicles). The tree structure 500 is described in relational terms to facilitate understanding. It should be understood that this is given for explanation purposes only and does not represent how the data is physically stored.

[0059] The tree can have a root level 502, which specifies one or more properties of the tree, including one or more access rights associated with the tree, data storage or memory location of the tree, etc. As a non-limiting example, such a root level can specify that data exists in a remote, cloud-based local, compressed, uncompressed server that can be accessed via a specific protocol, etc. Next, the specific date and time to which the file of the tree 500 belongs can include one or more levels. In the illustrated example, the first level 504 represents the year, the next level 506 represents the month, the next level 508 represents the day, and the next level 510 represents the hour, minute and second. Each of the time levels 504, 506, 508 and 510 can be expanded to include any time period. As shown in the figure, the time per minute is represented by new lowest level time 512, 520, 524, etc. It should be understood that the levels and other time divisions of other numbers can be used similarly.

[0060] Below one or more time levels, a device identification level 512 may be included. As a non-limiting example, data generated and collected from a 5th autonomous vehicle in a fleet of autonomous vehicles may be assigned a level such as "AV_5". A plurality of different topic files 514, 516, 518, etc. may be included under each time window for each device. In the illustrated example, topic 1 514 is included as files 514, 522, and 526 in each of the time levels 512, 520, and 524. In some examples, files representing the same topic may be included in or present in different time intervals when data falling within those topics (e.g., one or more messages belonging to that topic) is generated by vehicles in corresponding time intervals.

[0061] Structuring vehicle or other equipment data in this manner can provide easy and efficient access to vehicle data at a granular level. In addition, the data structure supports readers (such as reader 206 described above) reading from multiple hierarchical roots. The same structure or format can be used across multiple trees to help provide more efficient data access or recall.

[0062] Figure 6 Show Figure 5 An instance of the contents of a file 600 of a directory data structure. In one instance, the subject file 600 can be stored in a tabular format (such as the SSTable format of Google LevelDB), but other formats are contemplated. Each row 602 in the table can include a message number 604 / message 606 pair. In one instance, the message numbers 604 can be sequential in time, so that 2 represents the time after 1, and so on. Messages can be in many different formats (such as ROS, protocol buffers, JSON, XML, etc.), and any topic can contain multiple message types. Each message 606 may include a type identifier 608 and a data field 610, which type identifier can identify the type of content of the message (e.g., ROS message, protocol buffer, etc.), origin, or one or more attributes. In some aspects, space can be reserved by an additional size field 612 for another format or field of data 614 in a message (e.g., in message 616). In some aspects, the additional data 614 may include a timestamp 618 or other time identifier (where the time may be reported in nanoseconds, in Unix time or UNIX Epcoh time) and a frame identifier 620 (eg, to describe the source of the additional data).

[0063] In one example, a row in file 600 (the last row shown) may include a metadata record 622 describing the type and other metadata. As shown, metadata record 622 may include various fields and pieces of information about file 600, such as a version, name, time information, and information about any data format included in file 600.

[0064] Figure 7 Show Figure 5 700. A first tree 702, labeled root_1, includes files for topic_1 704, 706, 708 relating to a particular vehicle 710 at a particular time 712, 714, 716. A second tree 718, labeled root_2, may store files for topic_2 720, 724, and 728 and topic_3 722, 726, and 730 relating to the time period or window 710, 714, and 716 of the same vehicle 710. In this manner, files for different topics for a vehicle or other robotic device may be stored in separate trees or subdirectories.

[0065] The ability to store robot or vehicle data in separate trees or subdirectories according to a unified data hierarchy can be beneficial and allow for efficient local caching of certain files that are accessed more frequently than other files. It can also be beneficial to store certain topics in separate trees, for example, to facilitate simultaneous access to different data, and for many other reasons. In some cases, for example, as described above with reference to Figure 2 As described in the permission manager 204, for different topics, storing different files in different trees can be used to associate different permission schemes with different data.

[0066] It should be understood that trees 702 and 718 may include any number of files for different topics, for variations in time length, and include time windows of different sizes.

[0067] In some aspects, the levels of data trees 702 and 718 may have a defined order, and for each file, the first file found is retrieved when requested. Thus, in the illustrated example, although tree 2718 may have data corresponding to topic_1 according to the example access scheme, a request for files for topic_1 retrieves files 704, 706, and 708.

[0068] Figure 8 Show Figure 5 Another example of two related but different file trees or data structures of the directory data structure 800. Figure 8In the example of , two different trees 802 and 804 are shown, each associated with root_2. For time period 810 and for vehicle 812, tree 802 includes files 814 associated with theme 1. For time period 810 and for vehicle 812, tree 804 includes files 816 and 818 associated with themes 2 and 3. Tree 802 has two additional layers 806, 808 that specify that the tree is associated with a variant layer. Variant layer identifiers 806 and 808 can be used to modify the files stored in the tree in one way or another, and the modified files, or different combinations of files, or newly generated files are saved to individually addressable trees. It should be understood that trees 802 and 804 can include any number of files for different themes, for variants of time length, and include time windows of different sizes. In some aspects, tree 804 can also store files 814 associated with theme 1, wherein configurable access preferences (e.g., access the theme from tree 802, first, if not present, access from tree 804).

[0069] The hierarchical data structure or directory described here, in part, enables easy and efficient modification, reorganization, comparison and addition of data to be performed by organizing files according to subject, time and vehicle. Variant layers, or layers of customizable content in the form of different files, external data, data derived from other files, etc., can be easily created and accessed via the described directory. Variant layers can be created in a variety of ways, including by combining files of different layers to form new layers by associating variant layer identifiers with new layers. Layers can be formed by combining files of different themes aligned in time according to the hierarchical data structure. This can include combining files associated with different themes but the same time period or window into layers with one or more time levels. In this way, data can be combined in almost unlimited ways to enable faster access to data that may be related but from different sources (such as, different subsystems of autonomous vehicles).

[0070] In another example, a variant layer may be created by associating other data, including external data and / or other data related to the device. In some examples, data may be derived from data generated by the device and then combined with files stored in a hierarchical data structure. This may include accessing a first set of files of device data, abstracting or obtaining some subset of the information contained in those files, and then combining the data with a second set of files of device data, and using the hierarchical data structure to store the newly generated set of data.

[0071] For example, the first set of files may include different sensor data and subsystem (such as the following reference Fig.12Other messages and / or log data related to the sensor calibration subsystem 1230 described above. It may be desirable to extract some information from the data generated by the subsystem 1230 so as to associate the extracted information with data generated by another subsystem (such as the road navigation system 1242). Data related to the calibration of one or more sensors (such as cameras, radars, lidars, etc.) or some other operating characteristics may be particularly useful when compared to certain navigation decisions made by the road navigation system 1242 and are temporally correlated. In this scenario, files associated with the sensor calibration subsystem (e.g., the first subject) and a specific vehicle over a period of time can be accessed via a hierarchical data structure, as described herein. Feature information can be extracted from some or all of the accessed files (e.g., if the data is contained in each of the files). Files associated with the road navigation system 1242 can also be accessed for the same time period and specific vehicle. The abstract data can be organized into files, then combined with the accessed road navigation system files, and stored in a new layer of the hierarchical data structure for further analysis and use, etc.

[0072] In some examples, the abstract data can be organized into new files associated with new topics and time values. These new files can then be combined with files of road navigation system files by associating them with files of road navigation system files by aligning the two different sets of files according to time and vehicle identifiers (e.g., in a new layer or an existing layer). By using the described hierarchical data structure, data from various subsystems of a vehicle or robotic device can be easily and efficiently combined and aligned in time.

[0073] In other instances, different types of information can be derived from the data generated by the device or autonomous vehicle by calculating values ​​or performing other operations on the data. These operations can include mathematical operations (calculating averages, accelerations, special relationships, etc.), Boolean operations, linguistic operations (associating certain keywords with values ​​or with other keywords), and essentially any defined operation that uses the underlying data in some way. In this regard, the described hierarchical data structure is a very powerful tool for analyzing, preprocessing, organizing, and storing data for a wide range of analyses.

[0074] In some aspects, data obtained from the vehicle can be aggregated so that the size of a large amount of data can be reduced by extracting or modifying the underlying data for a specific purpose and saved in a variant layer.

[0075] In other cases, data not generated by the device or vehicle or data external to the device can be combined in a similar manner. For example, other data that may correspond to the operation of the vehicle at a particular time (e.g., external sensor data, weather data, traffic data, etc.) can be associated with the vehicle-generated data to derive or infer the impact of environmental factors on, for example, the operation or functioning of one or more subsystems of the vehicle.

[0076] Similarly, a variant layer can be generated by removing data in the form of files of one or more subjects from an existing layer. This may be particularly beneficial in situations where a subject normally contains a large amount of data or files, but for some specific purpose, only a small subset of the data or files is needed. By creating a variant layer that includes only the relevant data, and storing the variant layer in an easily accessible data storage location, for example, the data can be accessed more quickly and use fewer resources (e.g., processing, memory, and network resources).

[0077] In some cases, variant layers can be used to provide a more customizable access scheme, where certain variant layers of files with different themes can be associated with access permissions grouped or defined by any number of different characteristics, including entities that will generally have access to certain data (sensitive data that only certain entities are allowed to see), to comply with data privacy concerns, etc. In this way, access to various segments of data can be controlled in a very precise manner.

[0078] In some instances where data tiering is used, variant tiers can be used to group data that is accessed more frequently than other data and store it in a location that provides fast access to the data. This both reduces the amount of fast access data storage required and reduces the time to access data by minimizing the amount of data that must be searched. Using variant tiers in this manner to better organize data based on access can also be used to optimize data storage costs by duplicating data and storing it in a second location to increase fault tolerance, etc.

[0079] In yet other instances, variant layers can be used to record and compare data generated by running simulations on the operation of a device or vehicle or its subsystems (or otherwise testing new algorithms, processes, etc. on previously recorded data). For example, different inputs and / or different algorithms can be used to test new designs or account for different situations, environments, etc. in one or more subsystems of a device or vehicle. The outputs can be organized into variant layers and compared with data obtained from the device or vehicle using other inputs and / or other processing algorithms. As a non-limiting example, a new computer vision detector (a process for identifying objects in image data) can be run against previously recorded images obtained while driving a vehicle. In such an instance, the output from the first detector algorithm can be directly compared to the detection from the original detector. In this way, variant layers can be used to test and effectively compare the results of different design changes that can be implemented in a device or vehicle, or one or more subsystems thereof. In a more specific instance, one or more decision points or algorithms used in the planning subsystem 1228 of the vehicle 1200 can be advantageously tested (see below). Fig.12 1228 ), and output the results to one or more variant layers. These new variant layers can then be effectively compared to the existing layers output from the planning subsystem 1228 using other decision points or algorithms to help improve these decision points or algorithms and optimize the performance of the subsystem 1228 and / or vehicle 1200. Such use of variant layers can be particularly useful for testing new design features, code for implementing one or more aspects of one or more subsystems of the vehicle, new or changes to existing algorithms for making decisions for operating the vehicle, etc. In some cases, data can be derived from vehicle-generated data, modified (e.g., one or more operations performed thereon), and then stored in variant layers to enable comparison of the derived metrics or data.

[0080] In another example, one or more variant layers may be used to store specific sensor and other data that may be useful to a third party. In this case, the camera or other sensor 1250 data may be easily retrieved and stored in one or more new variant layers for efficient access to only the relevant data by such necessary third parties. Similarly, other data that is not required (e.g., decision messages, etc.) may not be accessible to such third parties because it may contain proprietary information. In other cases, data may be written directly to the new variant layer.

[0081] In another example, one or more components may be replaced or added to a vehicle (such as one or more detectors, sensors, etc.) By storing new data obtained from the replaced or new components in one or more variant layers, the performance of the new components can be accessed and easily compared to the old components (e.g., stored in another layer) to help modify and optimize the performance of the vehicle or equipment.

[0082] Fig. 9 An example process 900 for accessing and storing robotic device data in a hierarchical data structure is shown. Process 900 may be performed, for example, by a robotic device or an autonomous vehicle, or a subsystem thereof (such as an onboard vehicle computing system 404), as described above with reference to Figure 4 In other examples, process 900 may be performed by a robotic device or autonomous vehicle (such as the one described above). Figure 1 and 3 In some other cases, the different operations of process 900 can be performed by a robotic device or service.

[0083] The process may begin at operation 902, where first robotic device data including a plurality of files organized in a hierarchical data structure may be obtained. In some cases, each file in the plurality of files is based on at least one corresponding message generated by a robotic device. In still other cases, the files may be sequential files, and / or associated with a first subject and a first time value. In some cases, a robotic device service may obtain the first robotic device data. At operation 904, the service may also obtain additional robotic device data related to the operation of the robotic vehicle.

[0084] At operation 906, the service may store the additional robotic device data in the additional file set according to the hierarchical data structure. In some aspects, the additional robotic device data may include at least one of a second set of files organized in the hierarchical data structure, data derived from files stored in the hierarchical data structure, or data external to the robotic device, as described above with reference to Figure 8 In some aspects, storing the additional robot information may include combining it with some or all files of the first robot data. In yet other instances, the additional robot device data may be appended or added to the first robot device data at a new temporal level, such as after the files of the first robot device data. In some aspects, multiple files of the first robot device data may be organized in a first layer, and the additional file set may be written to a new variant layer.

[0085] In some aspects, the first file and the additional file can be combined by organizing the first file and the additional file in a layer according to the time values ​​associated with the respective files. In some aspects, the time values ​​of different files can overlap partially or completely, so that one or more files from the first set of files can be stored in the same time level as one or more files from the offset of the additional files.

[0086] Fig.10 An example process 1000 for accessing and combining robotic device data using a hierarchical data structure is shown. Process 1000 may be performed, for example, by a robotic device or an autonomous vehicle, or a subsystem thereof (such as an onboard vehicle computing system 404), as described above with reference to Figure 4 In other examples, process 1000 may be performed by a robotic device or autonomous vehicle (such as described above with reference to Figure 1 and 3 In some other cases, different operations of process 1000 can be performed by a robotic device or service.

[0087] Process 1000 may begin at operation 1002, where robotic device data organized in files in a hierarchical data structure as described throughout may be obtained. The robotic device data may be obtained from a robotic device or autonomous vehicle 106, 318 or a corresponding service 102, 302, as described above with reference to Figure 1 and Figure 3 At operation 1004, for example, in response to a request to access data received by autonomous vehicle service 102, 302 or reader 308, at least one first file from a subdirectory or layer of a hierarchical data structure may be accessed. Similarly, at operation 1006, at least one second file from a second subdirectory or layer of the hierarchical data structure may be accessed.

[0088] Next, at operation 1008, the at least one first file and the at least one second file may be combined to generate a new or third subdirectory or layer. At operation 1010, the third layer may be stored in the hierarchical data structure. In some aspects, operations 1008 and 1010 may be performed by the autonomous vehicle service 102, 302 or the builder 306.

[0089] Fig.11 An example process 1100 is shown for receiving, storing, accessing, and combining autonomous vehicle data using a hierarchical data structure. Process 1100 may be performed, for example, by an autonomous vehicle service, such as described above with reference to Figure 1 and 3 Described services 102, 302.

[0090] Process 1100 may begin at operation 1102, where an autonomous vehicle service (such as service 102, 302) may obtain or receive data generated by one or more subsystems of an autonomous vehicle, a timestamp, and a subject from an autonomous vehicle. Next, at operation 1104, a portion of the data including a plurality of files may be sent to one or more data stores, such as data store 104 or 312 of data storage service 314. The one or more data stores may be configured to store the files in, for example, one or more subdirectories or layers of a hierarchical data structure or directory 316 based at least in part on a timestamp, subject, or an identifier of the autonomous vehicle.

[0091] At operation 1106, a request for data (such as, including a first topic, a second topic, a start time and an end time or a time range) may be received, for example, by an autonomous vehicle service. In response to the request, at operation 1108, at least one first file may be accessed according to at least the first topic and in some cases, according to a specified time range. Similarly, at operation 1110, at least one second file may be accessed according to at least the second topic and in some cases, according to a specified time range. Then, at operation 1112, at least one first file and at least one second file may be combined in a new layer or subdirectory and provided in response to the request. In some aspects, a hierarchical data structure may be used to store the new layer or subdirectory, as described throughout the present invention.

[0092] In some aspects, the request or the account associated with the request may be associated with a first permission level. In some cases, a service (such as, service 302 and / or permission manager 304) may determine whether the permission level associated with the request meets or exceeds the permission level associated with one or more files in the requested file. In the case where the leveled permissions associated with the file, subject, level, root or layer specified in the request are higher than the permission level requested, the service may deny access to one or more files, levels, subdirectories, etc. As a further example, different variant layers may be restricted to different user teams and / or groups. As a non-limiting example, certain sensor data may be made available to software developers, while information about (for example) personally identifiable information may not be available. As such, different user groups may have access (according to permissions set on different variant layers). In at least some instances, a profile may be associated with one or more computing systems. Such a profile may determine any one or more users who may access different variant layers and / or retrieve data in a preferred order. As a non-limiting example, a profile may specify data of a preferred variant "A" relative to variant "B". In such an example, when data is requested from multiple sources (eg, both variant layers "A" and "B"), the request will retrieve data in order of preference and according to the permission set indicated in the profile.

[0093] Fig.12 An example of elements that may be used in accordance with the architecture 1200 of an autonomous vehicle is shown. An autonomous vehicle may be characterized as having an autonomous vehicle operating system 1202 coupled to various controllers, which in turn are coupled to various components of the autonomous vehicle to handle motion, power management, etc. The elements of the autonomous vehicle operating system 1202 provide a computing system for implementing object identification and environmental analysis, as described herein. These elements may be used in other applications outside of autonomous vehicles.

[0094] Architecture 1200 may specify one or more computer systems including various hardware, software, firmware, etc. to implement various aspects of the systems, methods, and devices described herein. For example, autonomous vehicle operating system 1202 may include environment analysis system 1203 and other components that may be used in various aspects of an autonomous vehicle. Environment analysis system 1203 may be used to obtain information that autonomous vehicle operating system 1202 may use to operate controllers for motors, steering, object avoidance, etc.

[0095] The environment analysis system 1203 can be organized into multiple subsystems to simplify implementation. In some instances, the subsystems are implemented independently, while in other instances, more than one subsystem is partially or fully integrated together. The subsystems can include a lidar subsystem, a camera subsystem, a radar subsystem, a sonar subsystem, a perception and prediction subsystem 1212, a positioning and mapping subsystem 1214, a tracking subsystem 1226, a planning subsystem 1228, a sensor calibration subsystem 1230, and possible other subsystems 1234. The perception and prediction subsystem 1212 performs object detection, segmentation, and classification of objects from various types of sensor data including lidar, radar, and visual images.

[0096] A given subsystem may be implemented in program code or hardware for communicating with other subsystems to receive inputs and provide outputs. Some inputs may come from sensors. In some descriptions herein, for readability, a subsystem may be described as including a sensor from which the subsystem obtains data or signals and / or a transmitter to which the subsystem outputs data or signals. For example, a sonar subsystem may be described as having an ultrasonic sensor or may be described as receiving a signal from an ultrasonic sensor. As another example, a camera subsystem may be described as having a camera and a display, or may be described as receiving a signal or data from the camera and sending the signal or data to a display.

[0097] although Fig.121203. It is not shown, but it should be understood that communication between subsystems can be provided as needed. A given subsystem can communicate with another subsystem by sending data directly to another subsystem via a channel, or the environment analysis system 1203 may include a bus subsystem or communication infrastructure, wherein the subsystems can communicate via the bus subsystem or communication infrastructure by passing data and / or signals therebetween. The environment analysis system 1203 can also be configured to receive external data and communicate information to the outside of the environment analysis system 1203.

[0098] A given subsystem may have some of its own computing processing, which may be performed by hardware dedicated to the given subsystem, or may be performed by a processor or circuit assigned to perform the computations of the subsystem, such as may be the case where the subsystem is implemented entirely in software and executed by one or more processors 1236 using memory 1238 (e.g., program code memory and data storage memory). Memory can be used for temporary storage of variables and data, such as RAM, as well as memory for permanent storage (i.e., data that persists without refresh, power, etc. over a period of time), and even if not explicitly mentioned, it should be implied where indicated. For example, when a subsystem is described as operating on or storing data on a database, there will be some form of memory for storing data in an electronically readable form. In some cases, the database or data storage in the memory is not specific and internal to a subsystem. In these cases, the memory can be accessed by more than one subsystem. For example, one subsystem can create records based on sensor data obtained by the subsystem and write those records to a database or other data structure, and in turn, another subsystem can read and use the data. Where a subsystem is implemented in software, the subsystem may include program code coupled to a processor dedicated to that subsystem or a more general program code memory and processor.

[0099] In some cases, an environment analysis system 1203 is employed in an autonomous vehicle. In some cases, the environment analysis system 1203 can provide perception and planning functions for the autonomous vehicle. Generally speaking, the environment analysis system 1203 can provide lidar perception, radar perception, visual (camera) perception, acoustic perception, segmentation and classification, tracking and fusion, and prediction / planning, as well as interfaces with other controllers (such as drive controllers, power controllers, environmental controllers, and communication controllers).

[0100] Autonomous vehicle operating system 1202 may include a road navigation system 1242, an inventory manager 1244, and an audit / fault logger 1246. Autonomous vehicle operating system 1202 may also include or interface with various sensors 1250 and transmitters 1252.

[0101] The autonomous vehicle operating system 1202 may interface to a drive controller 1270 that interacts with the motor 1280, steering 1282, brakes 1284, and suspension 1286, a power controller 1272 that interacts with the battery 1288 and inverter / charger 1290, an environmental controller 1274 that interacts with heating, ventilation, and air conditioning (HVAC) components 1292 and lighting 1294, a communication controller 1276 to handle communications between the autonomous vehicle and devices used with the autonomous vehicle, and external devices such as via a network, cellular channel, or Wi-Fi channel 1296. The combination of the autonomous vehicle operating system 1202, controllers, and vehicle components installed in an autonomous vehicle may provide a vehicle that is capable of safely navigating without the need for constant human intervention.

[0102] Autonomous vehicle operating system 1202 may include any number or type of sensors suitable for an autonomous vehicle other than those shown. Various sensors 1250 may include, but are not limited to, ultrasonic transducers, wheel encoders, environmental sensors, microphones, inertial measurement units (IMUs), accelerometers, gyroscopes, magnetometers, temperature sensors, humidity sensors, light sensors, global positioning system (GPS) sensors, pressure sensors, position sensors, etc.

[0103] The lidar subsystem may include one or more lidar sensors to capture lidar data for segmentation, as described herein, and may include any one or more depth sensors as described in detail herein. In some cases, the lidar subsystem 1204 may include functionality for combining or synthesizing lidar data from multiple lidar sensors to generate a meta-spin of lidar data, where the meta-spin of lidar data may refer to the lidar data based on multiple lidar sensors. In the case of a meta-spin of lidar data, the lidar subsystem may include functionality for determining a virtual origin of the meta-spin data (e.g., a coordinate reference system shared by all lidar sensors) and performing a data transformation so that the lidar data from each of the one or more lidar sensors is expressed relative to the virtual origin. As may be understood in the context of the present disclosure, the lidar subsystem may capture data and may transmit data sets to other subsystems for subsequent processing.

[0104] The camera subsystem may include or interface to one or more camera sensors to capture visual data for image segmentation and / or classification. The camera subsystem may include any number and type of camera sensors. For example, the camera subsystem may include any color camera, monochrome camera, depth camera, RGB-D camera, stereo camera, infrared (IR) camera, ultraviolet (UV) camera, etc. As can be understood in the context of the present disclosure, the camera subsystem can capture data and can transfer data sets to other subsystems for subsequent processing. For example, data from the camera subsystem can be included as one or more channels of a multi-channel image, which is thereby processed by another subsystem.

[0105] The radar subsystem may include one or more radar sensors to capture the range, angle, and / or velocity of objects in the environment. As can be understood in the context of the present disclosure, the radar subsystem may capture data and may transmit data sets to other subsystems for subsequent processing. For example, data from the radar subsystem may be included as one or more channels of a multi-channel image provided to another subsystem.

[0106] The sonar subsystem may include or interface to one or more speakers or sound transmitters and one or more microphones (such as a microphone array) to capture acoustic information from objects in the environment. Additionally or alternatively, such a sonar subsystem may include various ultrasonic transducers. For example, the sonar subsystem may cause the ultrasonic transducer to emit sound pulses and may listen to the echoes to determine the position and / or motion information associated with the objects in the environment. As can be understood in the context of the present disclosure, the sonar subsystem may capture data and may transmit the data set to other subsystems for subsequent processing. For example, another subsystem may fuse the data obtained from the sonar subsystem with the data obtained from the lidar subsystem to more accurately segment the object and / or determine information about the object, or for other purposes.

[0107] In some cases, the lidar subsystem, camera subsystem, radar subsystem, and / or sonar subsystem may provide one or more data sets to other subsystems of the environment analysis system for combined and / or synthesized data.

[0108] The environmental analysis system 1203 may also include storage for simulated data generated by a computer simulation algorithm for use in part in testing. In some cases, the simulated data may include any type of simulated data, such as camera data, lidar data, radar data, sonar data, inertial data, GPS data, etc. In some instances, the environmental analysis system 1203 may modify, transform, and / or perform the conversion operations described herein on the simulated data for verification operations and / or for training machine learning algorithms, as described herein. For example, in order to test some functions in a laboratory environment, simulated sensor data / signals may be provided to a subsystem as if it were actual sensor data to test the performance of some subsystems. In some embodiments, the simulated data may include messages, events, and system alarms sent to the autonomous vehicle operating system 1202.

[0109] The positioning and mapping subsystem 1214 may include functionality to convert or map data to a voxel map. For example, the positioning and mapping subsystem 1214 may receive lidar data, camera data, radar data, sonar data, etc., and map, convert, or associate individual data points to a voxel map representing a three-dimensional space in the environment. Voxel space is a logical representation of a three-dimensional environment (such as the space around an autonomous vehicle) represented as discrete small volumes (e.g., voxels). The voxel map provides data or values ​​for corresponding voxels in the voxel space. As a representation of the three-dimensional environment, the voxel map may be stored in a memory and manipulated by a processor.

[0110] In some instances, the positioning and mapping subsystem 1214 may define the dimensions of the voxel space, including the length, width, and height of the voxel space. Further, the positioning and mapping subsystem 1214 may determine the size of individual voxels. In some instances, the voxels may be of uniform size and shape throughout the voxel space, while in some instances, the size and / or density of the voxels may vary based on the relative position in the voxel space. For example, the size of the voxel may increase or decrease in proportion to the distance of the voxel from the origin or center of the voxel space. Additionally or alternatively, such a positioning and mapping subsystem 1214 may include a transformation between a virtual origin and the origin of the voxel space. In some instances, the positioning and mapping subsystem 1214 may include a function for generating a sparse voxel space, wherein voxels that do not include data or include an amount of data below a data threshold do not need to exist in the voxel map and the values ​​of those voxels may be assumed or ignored. In such instances, the voxel map may be organized as an octet, a voxel hash, or the like. In some instances, the localization and mapping subsystem 1214 may include functionality for reducing the amount of noise in the data of the voxel map or the data used to generate the voxel map by filtering the data as it is mapped to the voxel space and stored in the voxel map. For example, filtering may include removing data that is below a threshold amount of data per voxel (e.g., the number of lidar data points associated with a voxel) or exceeds a predetermined number of voxels (e.g., the number of lidar data points associated with the number of neighboring voxels). In some instances, the localization and mapping subsystem 1214 may update the voxel map as data is collected over time and / or in response to a vehicle tool navigating within a real-world environment to which the voxel space corresponds. For example, as an autonomous vehicle navigates the environment, the localization and mapping subsystem 1214 may add data and / or discard data from the voxel map.

[0111] In some cases, the localization and mapping subsystem 1214 can initialize the voxel map and other voxel space parameters (such as voxel size, orientation, and extent), treat the initial voxel map as representing empty space, and the localization and mapping subsystem 1214 can build a representation of the object as the lidar data is captured over time. In other instances, the localization and mapping subsystem 1214 can initialize the voxel map and voxel space parameters using global mapping data.

[0112] Tracking subsystem 1226 may include functionality to receive indications of one or more dynamic objects and perform additional processing to track the objects. For example, tracking subsystem 1226 may determine the velocity of a dynamic object and / or may determine and store the trajectory of a dynamic object over time. In some cases, tracking subsystem 1226 may be programmed to execute a prediction algorithm that may predict the path of an object to be tracked based on the object's previous motion. In various instances, tracking subsystem 1226 performs data association that an object is identical to a previously detected object. In some instances, the tracking subsystem maintains range, calculates velocity, and determines the orientation of an object.

[0113] Planning subsystem 1228 may include functionality to receive segmentation data and / or indications of a ground plane, static objects, and / or dynamic objects to determine a trajectory for an autonomous vehicle. For example, planning subsystem 1228 may receive segmentation information identifying a ground plane and may generate a trajectory for the autonomous vehicle to follow.

[0114] The sensor calibration subsystem 1230 may include functionality to calibrate one or more sensors 1250 based at least in part on segment information determined relative to the environment. For example, while sensor data from a lidar, camera, radar, and / or sonar sensor or subsystem may be used to estimate position and / or orientation (e.g., using simultaneous localization and mapping (SLAM)), the autonomous vehicle may also include additional sensors (such as an inertial measurement unit (IMU) and / or a GPS unit) for determining the position of the autonomous vehicle in the environment. In some cases, the IMU may indicate that the autonomous vehicle is at a first location, while analysis of the lidar data discussed herein indicates that the vehicle is at a second location different from the first location. The sensor calibration subsystem 1230 may determine the difference in location and may adjust or calibrate one or more sensors to update the position of the autonomous vehicle or one or more sensor intrinsic or extrinsic characteristics.

[0115] For example, characteristics of a camera sensor may include focal length, image center, distortion parameters, shutter speed, resolution, and spectrum. Characteristics of a radar may include output power and input sensitivity. Characteristics of a lidar sensor may include resolution and sampling rate. An exemplary neural network may pass input data through a series of connected layers to produce an output. An example of a neural network may include a convolutional neural network, or CNN. Each layer in a CNN may also include another CNN, or may include multiple layers. As may be understood in the context of the present invention, a neural network may utilize machine learning, which may refer to a large class of such algorithms that generate outputs based on learned parameters.

[0116] Although discussed in the context of neural networks, many types of machine learning can be used consistent with the present disclosure. For example, machine learning algorithms can include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimated scatter plot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomizer 3 (ID3), chi-squared automatic interaction detection (CHAID), decision stakes, conditional decision trees), Bayesian algorithms (e.g., natural Bayes, Gaussian natural Bayes, multinomial natural Bayes, average single dependency estimation AODE, Bayesian Belief Network (BNN), Bayesian Network), Clustering Algorithms (e.g., k-means, k-median, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning Algorithms (e.g., Perceptron, Back Propagation, Hopfield Network, Radial Basis Function Network (RBFN)), Deep Learning Algorithms (e.g., Deep Boltzmann Machine (DBM), Deep Bayesian Network (DBN), Convolutional Neural Network (CNN), Stacked Autoencoder), Dimensionality Reduction Algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Map, Multidimensional Scaling (MDS), Projection Proximity (Prx)), Pursuit), Linear Discriminant Analysis (LDA), Mixed Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), Overall Algorithms (e.g., Boosting, Bootstraped Aggregation (Bagging), AdaBoost, Stacked Generalization (Hybrid), Gradient Boosting Machine (GBM), Gradient Boosting Regression Tree (GBRT), Random Forest), SVM (Support Vector Machine), Supervised Learning, Unsupervised Learning, Semi-supervised Learning, etc.

[0117] Fig.12 The environment depicted in may be implemented with one or more computer systems including storage devices, one or more processors, memory, and possibly an operating system.

[0118] The systems and methods described herein can be implemented with software or hardware or any combination thereof. The systems and methods described herein can be implemented using one or more computing devices that may or may not be physically or logically separated from each other. These methods can be performed by components arranged as local hardware, local virtual systems, or hosted private instances. In addition, different aspects of the methods described herein can be combined or incorporated into other functions.

[0119] The exemplary environment and computer system for implementing the system and method may include a processor or computer system that can be configured to perform some or all of the methods described herein in particular. In some instances, the method may be partially or completely autonomous by one or more computers or processors. The system and method described herein may be implemented using a combination of any one of hardware, firmware and / or software. The present system and method described herein (or any part or function thereof) may be implemented using hardware, software, firmware or a combination thereof, and may be implemented in one or more computer systems or other processing systems. In some instances, the illustrated system elements may be combined into a single hardware device or separated into multiple hardware devices. If multiple hardware devices are used, the hardware devices may be physically located adjacent to or away from each other. The examples of the described and illustrated methods are intended to be illustrative rather than restrictive. For example, in different instances, some or all steps of these methods may be combined, rearranged, and / or omitted.

[0120] In one example, the systems and methods described herein may involve one or more computer systems capable of performing the functions described herein. Example computing devices may be, but are not limited to, running any operating system (such as, but not limited to, OS X). TM 、iOS TM , Linux TM 、Android TM and Microsoft TM Windows TM ) personal computer (PC) system. However, the system and method described herein may not be limited to these platforms. On the contrary, the system and method described herein may be implemented on any suitable computer system running any suitable operating system. Other components of the system and method described herein (such as but not limited to, computing devices, communication devices, mobile phones, smart phones, telephone devices, phones, personal digital assistants (PDAs), personal computers (PCs), handheld PCs, interactive televisions (iTVs), digital video recorders (DVDs), client workstations, thin clients, thick clients, proxy servers, network communication servers, remote access devices, client computers, server computers, routers, network servers, data, media, audio, video, telephone or streaming technology servers, etc.) may also be implemented using computing devices. For example, but not limited to interactive televisions (iTVs), video on demand systems (VODs) and on-demand services provided by digital video recorders (DVRs) or other on-demand viewing systems may be used.

[0121] The system may include one or more processors. The processor may be connected to a communication infrastructure, such as, but not limited to, a communication bus, a crossbar, or a network. The process and the processor need not be located at the same physical location. In other words, the process may be executed at one or more geographically remote processors, for example, via a LAN or WAN connection. The computing device may include a display interface that may forward graphics, text, and other data from the communication infrastructure for display on a display unit.

[0122] The computer system may also include, but is not limited to, a main memory, a random access memory (RAM), and an auxiliary memory, etc. The auxiliary memory may include, for example, a hard disk drive and / or a removable storage drive, such as a compact disk drive CD-ROM, etc. The removable storage drive may read from and / or write to a removable storage unit. As may be appreciated, a removable storage unit may include a computer-usable storage medium having computer software and / or data stored therein. In some instances, a machine-accessible medium may refer to any storage device for storing data accessible by a computer. Examples of machine-accessible media may include, for example, but not limited to, a magnetic hard disk; a floppy disk; an optical disk, such as a compact disk read-only memory (CD-ROM) or a digital versatile disk (DVD); a magnetic tape; and / or a memory chip, etc.

[0123] The processor may also include or be operably coupled to communicate with one or more data storage devices for storing data. As non-limiting examples, such data storage devices may include magnetic disks (including internal hard disks and removable disks), magneto-optical disks, optical disks, read-only memory, random access memory, and / or flash memory. Storage devices suitable for tangibly embodying computer program instructions and data may also include all forms of non-volatile memory, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0124] The processing system may communicate with a computerized data storage system. The data storage system may include a non-relational or relational data storage, such as MySQL TM or other relational databases. Other physical and logical database types may be used. The data store may be a database server, such as Microsoft SQL Server TM 、Oracle TM 、IBMDB2 TM 、SQLITE TM, or any other relational or other form of database software. The data storage area can store information identifying grammatical tags and any information required for grammatical tag operations. In some instances, the processing system can use object-oriented programming and can store data in objects. In these instances, the processing system can use an object-relational mapper (ORM) to store data objects in a relational database. The systems and methods described herein can be implemented using any number of physical data models. In one instance, a relational database management system (RDBMS) can be used. In those instances, the tables in the RDBMS can include columns representing coordinates. The tables can have predefined relationships between them. The tables can also have auxiliary objects associated with the coordinates.

[0125] In alternative examples, auxiliary storage may include other similar devices for allowing computer programs or other instructions to be loaded into a computer system. For example, such devices may include removable storage units and interfaces. Such examples may include program cartridges and cartridge interfaces (such as, but not limited to, those found in video game devices), removable memory chips (such as, but not limited to, erasable programmable read-only memory (EPROM) or programmable read-only memory (PROM) and associated sockets), and other removable storage units and interfaces that may allow software and data to be transferred from a removable storage unit to a computer system.

[0126] The computing device may also include input devices, such as, but not limited to, voice input devices (such as, microphones), touch screens, gesture recognition devices (such as, cameras), other natural user interfaces, mice or other pointing devices (such as, digitizers), and keyboards or other data input devices. The computing device may also include output devices, such as, but not limited to, displays and display interfaces. The computing device may include input / output (I / O) devices, such as, but not limited to, communication interfaces, cables, and communication paths, etc. These devices may include, but are not limited to, network interface cards and modems. The communication interface may allow software and data to be transmitted between the computer system and one or more external devices.

[0127] In one or more instances, the computing device may be operably coupled to an automotive system. Such an automotive system may be manually operated, semi-autonomous, or fully autonomous. In such instances, the input and output devices may include one or more image capture devices, controllers, microcontrollers, and / or other processors to control automotive functions (such as, but not limited to, acceleration, braking, and steering). Further, the communication infrastructure in such instances may also include a controller area network (CAN) bus.

[0128] In one or more instances, the computing device can be operably coupled to any machine-based vision system. For example, such machine-based vision systems include, but are not limited to, manually operated, semi-autonomous, or fully autonomous industrial or agricultural robots, domestic robots, inspection systems, security systems, etc. That is, the examples described herein are not limited to one particular context and may be applicable to any application utilizing machine vision.

[0129] In one or more instances, the present examples may be practiced in the context of one or more computer networks. The network may include a private network, or a public network (e.g., the Internet as described below), or a combination of both. The network may include hardware, software, or a combination of both.

[0130] From a telecommunications-oriented point of view, a network can be described as a set of hardware nodes interconnected by communication facilities, with one or more processes (hardware, software, or a combination thereof) running at each such node. These processes can communicate and exchange information with each other via communication paths between them using inter-process communication paths. On these paths, appropriate communication protocols are used. The system receives operating commands (remote operation) via cellular, infrared, radio, or wireless networks.

[0131] The exemplary computer and / or telecommunication network environment according to this example may include a node, which may include hardware, software, or a combination of hardware and software. The nodes may be interconnected via a communication network. Each node may include one or more processes executable by a processor incorporated into the node. For example, a single process may be run by multiple processors, or multiple processes may be run by a single processor. In addition, each node may provide an interface point between the network and the outside world, and may merge a collection of subnetworks.

[0132] In an example, processes can communicate with each other through inter-process communication paths that support communication through any communication protocol. These paths can function sequentially or in parallel, continuously or intermittently. In addition to the standard parallel instruction sets used by many computers, the paths can use any of the communication standards, protocols, or technologies described herein with respect to the communication network.

[0133] A node may include any entity capable of performing a processing function. Examples of such nodes that may be used with the examples include computers (such as personal computers, workstations, servers, or mainframes), handheld wireless devices, and wired devices (such as personal digital assistants (PDAs), modem cellular phones with processing capabilities, including BlackBerry TMA node may be a wireless email device (such as a wireless email device of a device), a document processing device (such as a scanner, printer, fax machine, or multifunction document machine), or a complex entity (such as a local area network or a wide area network) connecting a collection of processors, as described. For example, in the context of the present disclosure, a node itself can be a wide area network (WAN), a local area network (LAN), a private network (such as a virtual private network (VPN)), or a collection of networks.

[0134] Communication between nodes can be achieved through a communication network. The nodes can be connected to the communication network continuously or intermittently. For example, in the context of the present disclosure, the communication network can be a digital communication infrastructure that provides sufficient bandwidth and information security.

[0135] The communication network may include wired communication capabilities, wireless communication capabilities, or a combination of the two, at any frequency, using any type of standards, protocols, or technologies. Additionally, in this example, the communication network may be a private network (e.g., a VPN) or a public network (e.g., the Internet).

[0136] A non-inclusive list of exemplary wireless protocols and technologies used by the communication network may include Bluetooth TM , General Packet Radio Service (GPRS), Cellular Digital Packet Data (CDPD), Mobile Solutions Platform (MSP), Multimedia Messaging (MMS), Wireless Application Protocol (WAP), Code Division Multiple Access (CDMA), Short Message Service (SMS), Wireless Markup Language (WML), Handheld Device Markup Language (HDML), Binary Runtime Environment for Wireless (BREW), Radio Access Network (RAN), and Packet Switched Core Network (PS-CN). Different generations of wireless technologies are also included. An exemplary non-inclusive list of the main wired protocols and technologies used by the communication network includes asynchronous transfer mode (ATM), enhanced interior gateway routing protocol (EIGRP), frame relay (FR), high-level data link control (HDLC), Internet control message protocol (ICMP), interior gateway routing protocol (IGRP), Internet packet exchange (IPX), ISDN, point-to-point protocol (PPP), transmission control protocol / Internet protocol (TCP / IP), routing information protocol (RIP), and user datagram protocol (UDP). As will be appreciated by the technician, any other known or expected wireless or wired protocols and technologies may be used.

[0137] Examples of the present disclosure may include devices for performing the operations described herein. An apparatus may be specially constructed for the intended purposes, or it may include a general purpose device selectively activated or reconfigured by a program stored in the device.

[0138] In one or more instances, the present example is embodied in machine executable instructions. Instructions can be used to enable a processing device (e.g., a general or special processor) programmed with instructions to perform the steps of the present disclosure. Alternatively, the steps of the present disclosure can be performed by a specific hardware component containing a hard-wired logic for performing the steps, or by any combination of programmed computer components and custom hardware components. For example, as described above, the present disclosure can be provided as a computer program product. In an environment, an example can include a machine-readable medium on which instructions are stored. Instructions can be used to program any processor or multiple processors (or other electronic devices) to perform a process or method according to the present example. In addition, the present disclosure can also be downloaded and stored on a computer program product. Here, the program can be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem or a network connection) by means of a data signal embodied in a carrier or other propagation medium, and eventually such a signal can be stored on a computer system for subsequent execution.

[0139] The method may be implemented in a computer program product accessible from a computer usable or computer readable storage medium that provides program code for use by or in conjunction with a computer or any instruction execution system. A computer usable or computer readable storage medium may be any device that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0140] The data processing system suitable for storing and / or executing corresponding program code can include at least one processor that is directly or indirectly coupled to computerized data storage device (such as, memory element). Input / output (I / O) equipment (including but not limited to keyboard, display, pointing device etc.) can be coupled to system. Network adapter can also be coupled to system, so that data processing system can become coupled to other data processing systems or remote printers or storage devices through intermediary private or public network. In order to provide interaction with user, can have display device (such as, LCD (liquid crystal display)) or be used for displaying another type of monitor of information to user and user can provide input keyboard and input device (such as, mouse or trackball) to computer by it and realize these features.

[0141] A computer program may be a set of instructions that can be used directly or indirectly in a computer. The systems and methods described herein may be implemented using programming languages ​​such as CUDA, OpenCL, Flash TM , JAVA TM , C++, C, C#, Python, VisualBasicTM , JavaScript TM PHP, XML, HTML, etc. or a combination of programming languages, including compiled or interpreted languages), and can be deployed in any form, including as a stand-alone program or as a subsystem, component, subroutine, or other unit suitable for use in a computing environment. Software may include, but is not limited to, firmware, resident software, microcode, etc. Protocols such as SOAP / HTTP may be used to implement interfaces between programming subsystems. The components and functions described herein may be implemented using any programming language suitable for software development (including, but not limited to, different versions of Microsoft Windows TM , Apple TM Mac TM 、iOS TM , Unix TM / X-Windows TM , Linux TM 、VMS TM , Android, QNX, etc.) on any desktop operating system executed in a virtualized or non-virtualized environment. The system can be implemented using a web application framework such as Ruby on Rails.

[0142] Suitable processors for executing instruction programs include, but are not limited to, general and special purpose microprocessors, and the sole processor or one of multiple processors or cores of any type of computer. The processor can receive and store instructions and data from a computerized data storage device (e.g., a read-only memory, a random access memory, both, or any combination of data storage devices described herein). The processor may include any processing circuit or control circuit operable to control the operation and performance of an electronic device.

[0143] The systems, subsystems, and methods described herein can be implemented using any combination of software or hardware elements. The systems, subsystems, and methods described herein can be implemented using one or more virtual machines operating alone or in combination with each other. Any applicable virtualization solution can be used to encapsulate a physical computer platform into a virtual machine executed under the control of virtualization software running on a hardware computing platform or host. A virtual machine can have both virtual system hardware and client operating system software.

[0144] The systems and methods described herein can be implemented in a computer system that includes back-end components such as a data server, or includes middleware components such as an application server or an Internet server, or includes front-end components (such as a client computer with a graphical user interface or an Internet browser, or any combination thereof). The components of the system can be connected by any form or medium of digital data communication (such as a communication network). Examples of communication networks include, for example, LANs, WANs, and the computers and networks that form the Internet.

[0145] One or more examples of the present disclosure may be practiced with other computer system configurations, including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The systems and methods described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a network.

[0146] The terms "computer program medium" and "computer readable medium" may be used to broadly refer to media such as, but not limited to, removable storage drives, hard disks installed in hard disk drives. These computer program products may provide software to a computer system. The systems and methods described herein may be directed to such computer program products.

[0147] Here, "one embodiment," "an embodiment," "an exemplary embodiment," "various embodiments," and the like may indicate that an embodiment of the present disclosure may include specific features, structures, or characteristics, but not every embodiment must include the specific features, structures, or characteristics. Further, repeated use of the phrase "in one embodiment" or "in an exemplary embodiment" does not necessarily refer to the same embodiment, although they may be the same. Similarly, reference to an "example" may indicate that various examples of the present disclosure may include certain features, structures, or characteristics, but not every example must include the certain features, structures, or characteristics. Further, repeated use of the phrase "in some examples" does not necessarily refer to the same example, although it may.

[0148] An algorithm may generally be considered to be a self-consistent sequence of acts or operations leading to a desired result. These include physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven sometimes, primarily for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be understood, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.

[0149] Unless expressly stated otherwise, it should be understood that throughout this specification, terms such as “process,” “compute,” “calculate,” “determine,” etc. refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulates and / or transforms data represented as physical quantities (such as electronic quantities) in the registers and / or memories of the computing system into other data similarly represented as physical quantities in the memories, registers, or other such information storage, transmission, or display devices of the computing system.

[0150] In a similar manner, the term "processor" may refer to any device or part of a device that processes electronic data from registers and / or memory and converts the electronic data into other electronic data that can be stored in registers and / or memory. As a non-limiting example, a "processor" may be a central processing unit (CPU) or a graphics processing unit (GPU). A "computing platform" may include one or more processors. As used herein, a "software" process may include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Moreover, each process may refer to a plurality of processes for executing instructions sequentially or in parallel, continuously or intermittently. The terms "system" and "method" are used interchangeably herein, as long as a system can embody one or more methods and these methods can be considered as a system.

[0151] In this article, reference may be made to obtaining, acquiring, receiving analog or digital data or inputting analog or digital data into a subsystem, a computer system or a computer-implemented machine. The process of obtaining, acquiring, receiving or inputting analog or digital data can be completed in various ways, such as by receiving data as a parameter of a function call or a call to an application programming interface. In some implementations, the process of obtaining, acquiring, receiving or inputting analog or digital data can be completed by transmitting data via a serial or parallel interface. In another implementation, the process of obtaining, acquiring, receiving or inputting analog or digital data can be completed by transmitting data from a providing entity to an acquiring entity via a computer network. Reference may also be made to providing, outputting, transmitting, sending or presenting analog or digital data. In various instances, the process of providing, outputting, transmitting, sending or presenting analog or digital data can be completed by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface or an interprocess communication mechanism.

[0152] While one or more embodiments have been described, various alterations, additions, permutations, and equivalents thereof fall within the scope of the invention.

[0153] In the description of the embodiments, reference is made to the accompanying drawings forming a part thereof, which illustrate by way of illustration a specific embodiment in which the claimed subject matter can be implemented. It should be understood that other embodiments can be used, and changes or alterations, such as structural changes, can be made. Such embodiments, changes or alterations do not necessarily deviate from the scope of the subject matter for which protection is intended. Although the steps herein can be presented in a particular order, in some cases, the order can be changed so that specific inputs are provided at different times or in different orders without changing the functions of the described systems and methods. The disclosed processes can also be performed in different orders. In addition, it is not necessary to perform different calculations in the disclosed order herein, and other embodiments using alternative ordering of calculations can be easily implemented. In addition to being reordered, calculations can also be decomposed into sub-calculations with the same results. In some instances, the operations shown can be performed in parallel and substantially simultaneously. In general, although a series of blocks can be shown, not all steps may be required to perform the processes shown.

[0154] Although the above discussion sets forth example implementations of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of the invention. In addition, although a specific distribution of responsibilities is defined above for purposes of discussion, the various functions and responsibilities may be distributed and divided in different ways depending on the circumstances.

[0155] Furthermore, although the subject matter has been described using language specific to structural features and methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, these specific features and acts are disclosed as example forms of implementing the claims.

[0156] Sample Clauses

[0157] Embodiments of the present disclosure may be described in terms of the following:

[0158] 1. A system comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the system to: receive a message including data, a timestamp, and a subject generated by one or more subsystems of an autonomous vehicle; store a plurality of files in a first directory of the one or more data stores, a first file in the plurality of files including at least a portion of the data and stored in a first subdirectory of the first directory according to a hierarchical structure, the hierarchical structure being based at least in part on a timestamp, a subject, or an identifier; store a plurality of second files in a second directory of the one or more data stores, a second file in the plurality of second files being based at least in part on a portion of the data and stored in a second subdirectory of the second directory according to a hierarchical structure; receive a request including a request subject, a start time, or an end time; and provide one or more of the first file or the second file based at least in part on the request subject, the start time, or the end time.

[0159] 2. The system of clause 1, wherein providing one or more of the first file or the second file is based at least in part on whether the first file or the second file is associated with the requesting subject and a profile indicating a catalog preference.

[0160] 3. A system as described in clause 1 or 2, wherein the first directory is associated with a first access permission and the second directory is associated with a second access permission that is more restrictive than the first permission, wherein the request is associated with a request permission level, and wherein providing one or more of the first file or the second file includes providing the first file based at least in part on the first access permission, the second access permission, and the request permission level.

[0161] 4. A system as described in any of clauses 1 to 3, wherein the first subject is associated with a first subsystem of an autonomous vehicle and the second subject is associated with a second subsystem of the autonomous vehicle, and wherein the request includes a request for a set of change data determined at least in part based on data from the first subsystem of the autonomous vehicle and data from the second subsystem of the autonomous vehicle between a start time and an end time.

[0162] 5. A system as described in any of clauses 1 to 4, wherein the first directory includes a first root directory indicating a first storage location on one or more data stores, and the second directory includes a second root directory indicating a second storage location on one or more data stores, the first storage location being different from the second storage location.

[0163] 6. The system of clause 5, wherein the first storage location is located in a local data store optimized for fast access.

[0164] 7. The system of any one of clauses 1 to 6, wherein the first file and the second file are smaller than a threshold size.

[0165] 8. A method comprising: receiving first device data, the first device data comprising a plurality of files and corresponding to message data generated by one or more components of a device during a certain time period; storing the first plurality of files on one or more data stores and at a first location according to a hierarchical data structure, the hierarchical data structure being based at least in part on message time, a data identifier, or a device identifier; obtaining second device data comprising a second plurality of files, the second device data being associated with a time period; and storing the second device data at a second location on the one or more data stores according to the hierarchical data structure.

[0166] 9. The method of clause 8, wherein the second device data comprises data derived from the first plurality of files.

[0167] 10. The method of clause 8 or 9, wherein a file in the first plurality of files consists of a plurality of messages of message data, the file being smaller than a threshold size.

[0168] 11. The method of clause 10, wherein a message in the plurality of messages comprises: a message type, a timestamp of when the message was created, and a message number.

[0169] 12. The method of any of clauses 8 to 11, wherein the first plurality of files are stored in a first data store, and wherein the second plurality of files are stored in a second data store based on latency or frequency of use of the second plurality of files.

[0170] 13. The method of any of clauses 8 to 12, wherein the first location is associated with a first access right and the second location is associated with a second access right, the second access right being more restrictive than the first access right.

[0171] 14. The method as described in clause 13 also includes: receiving a request, the request including a request data identifier, a start time, an end time, and a requested permission level; and transmitting a first file in the first plurality of files or a second file in the second plurality of files based at least in part on the request, the first access permission, the second access permission, and a profile, wherein the profile provides instructions for accessing the first plurality of files or the second plurality of files based at least in part on the request data identifier.

[0172] 15. The method of any of clauses 8 to 14, wherein the first device data is obtained from a first subsystem of the autonomous vehicle and the second device data is obtained from a second subsystem of the hosted vehicle.

[0173] 16. A system comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the system to: obtain a query, the query comprising a first data identifier, a second data identifier, and a query access level; retrieve a first plurality of files from a first location and based at least in part on the query and a profile, the first plurality of files being arranged at the first location according to a hierarchical data structure; retrieve a second plurality of files from a second location and based at least in part on the query and the profile and arranged at the second location according to the hierarchical data structure; and transmit the first plurality of files and the second plurality of files, wherein the profile comprises instructions for retrieving the first plurality of files from the first location based at least in part on the first data identifier, and instructions for retrieving the second plurality of files from the second location based at least in part on the second data identifier, and wherein the first plurality of files and the second plurality of files comprise files that are less than or equal to a threshold file size.

[0174] 17. The system of clause 16, wherein the hierarchical data structure comprises a plurality of sub-directories based at least in part on a plurality of timestamps, a plurality of data identifiers, and a plurality of device identifiers, and wherein a timestamp in the plurality of timestamps meets or exceeds a minimum time difference.

[0175] 18. A system as described in clause 16 or 17, wherein the first location is associated with a first access level and the second location is associated with a second access level, wherein retrieval of the first plurality of files is based at least in part on the first access level and the query access level, and wherein retrieval of the second plurality of files is based at least in part on the second access level and the query access level.

[0176] 19. The system of clause 17 or 18, wherein a file in the first plurality of files or the second plurality of files comprises a plurality of messages, a message in the plurality of messages comprising one or more of: a message type, a message size, or message data.

[0177] 20. The system of any of clauses 17 to 19, wherein the first location is a location on a remote data store and the second location is a location on a local data store.

Claims

1. A system, include: one or more processors; as well as a memory storing instructions that, when executed by the one or more processors, cause the system to: receiving a message including data, a timestamp, and a topic generated by one or more subsystems of an autonomous vehicle; storing a plurality of files in a first directory of one or more data stores, a first file of the plurality of files comprising at least a portion of the data and stored in a first subdirectory of the first directory according to a hierarchical structure, the hierarchical structure being based at least in part on the timestamp, the subject, or the identifier; storing a plurality of second files in a second directory of the one or more data stores, second files of the plurality of second files being based at least in part on the portion of the data and stored in a second subdirectory of the second directory according to the hierarchical structure; receiving a request, the request including a request subject, a start time, or an end time; and One or more of the first file or the second file is provided based at least in part on the request subject, the start time, or the end time.

2. The system according to claim 1, in, Providing one or more of the first file or the second file is based at least in part on whether the first file or the second file is associated with the requesting subject and a profile indicating a catalog preference.

3. The system according to claim 1, in, The first directory is associated with a first access right and the second directory is associated with a second access right that is more restrictive than the first access right, wherein the request is associated with a request permission level, and Wherein providing one or more of the first file or the second file comprises providing the first file based at least in part on the first access right, the second access right, and the requested permission level.

4. The system according to claim 2, in, The first directory is associated with a first access right and the second directory is associated with a second access right that is more restrictive than the first access right, wherein the request is associated with a request permission level, and Wherein providing one or more of the first file or the second file comprises providing the first file based at least in part on the first access right, the second access right, and the requested permission level.

5. The system according to claim 1, in, The first topic is associated with a first subsystem of the autonomous vehicle and the second topic is associated with a second subsystem of the autonomous vehicle, and wherein the request includes a request for a variable data set determined at least in part based on data from the first subsystem of the autonomous vehicle and data from the second subsystem of the autonomous vehicle between the start time and the end time.

6. The system according to claim 2, in, The first topic is associated with a first subsystem of the autonomous vehicle and the second topic is associated with a second subsystem of the autonomous vehicle, and wherein the request includes a request for a variable data set determined at least in part based on data from the first subsystem of the autonomous vehicle and data from the second subsystem of the autonomous vehicle between the start time and the end time.

7. The system according to claim 3, in, The first topic is associated with a first subsystem of the autonomous vehicle and the second topic is associated with a second subsystem of the autonomous vehicle, and wherein the request includes a request for a variable data set determined at least in part based on data from the first subsystem of the autonomous vehicle and data from the second subsystem of the autonomous vehicle between the start time and the end time.

8. The system according to claim 4, in, The first topic is associated with a first subsystem of the autonomous vehicle and the second topic is associated with a second subsystem of the autonomous vehicle, and wherein the request includes a request for a variable data set determined at least in part based on data from the first subsystem of the autonomous vehicle and data from the second subsystem of the autonomous vehicle between the start time and the end time.

9. A system according to any preceding claim, in, The first directory includes a first root directory indicating a first storage location on the one or more data stores, and the second directory includes a second root directory indicating a second storage location on the one or more data stores, the first storage location being different from the second storage location.

10. The system according to claim 9, in, The first storage location is located in a local data store optimized for fast access.

11. A method, include: receiving first device data comprising a first plurality of files and corresponding to message data generated by one or more components of a device during a period of time; storing the first plurality of files on one or more data stores and at a first location according to a hierarchical data structure, the hierarchical data structure being based at least in part on message times, data identifiers, or device identifiers; obtaining second device data comprising a second plurality of files, the second device data being associated with the time period; as well as The second device data is stored at a second location on the one or more data stores according to the hierarchical data structure.

12. The method according to claim 11, in, The second device data includes data derived from the first plurality of files.

13. The method according to claim 11, in, A file in the first plurality of files is comprised of a plurality of messages of the message data, the file being smaller than a threshold size.

14. The method according to claim 12, in, A file in the first plurality of files is comprised of a plurality of messages of the message data, the file being smaller than a threshold size.

15. The method according to claim 13, in, The messages of the plurality of messages include: Message type, The timestamp of when the message was created, and Message number.

16. The method according to claim 14, in, The messages of the plurality of messages include: Message type, The timestamp of when the message was created, and Message number.

17. The method according to claim 11, in, The first plurality of files are stored in a first data store, and wherein the second plurality of files are stored in a second data store based on a latency or frequency of use of the second plurality of files.

18. The method according to claim 12, in, The first plurality of files are stored in a first data store, and wherein the second plurality of files are stored in a second data store based on a latency or frequency of use of the second plurality of files.

19. The method according to claim 13, in, The first plurality of files are stored in a first data store, and wherein the second plurality of files are stored in a second data store based on a latency or frequency of use of the second plurality of files.

20. The method according to claim 14, in, The first plurality of files are stored in a first data store, and wherein the second plurality of files are stored in a second data store based on a latency or frequency of use of the second plurality of files.

21. The method according to claim 15, in, The first plurality of files are stored in a first data store, and wherein the second plurality of files are stored in a second data store based on a latency or frequency of use of the second plurality of files.

22. The method according to claim 16, in, The first plurality of files are stored in a first data store, and wherein the second plurality of files are stored in a second data store based on a latency or frequency of use of the second plurality of files.

23. The method according to any one of claims 11 to 22, in, The first location is associated with a first access right, and the second location is associated with a second access right, the second access right being more restrictive than the first access right.

24. The method according to claim 23, further comprising: include: receiving a request, the request comprising a request data identifier, a start time, an end time, and a request permission level; as well as transmitting a first file in the first plurality of files or a second file in the second plurality of files based at least in part on the request, the first access permission, the second access permission, and a profile, Wherein the profile provides instructions to access the first plurality of files or the second plurality of files based at least in part on the request data identifier.

25. The method according to any one of claims 11 to 22, in, The first device data is obtained from a first subsystem of an autonomous vehicle, and the second device data is obtained from a second subsystem of the autonomous vehicle.

26. The method according to claim 23, in, The first device data is obtained from a first subsystem of an autonomous vehicle, and the second device data is obtained from a second subsystem of the autonomous vehicle.

27. The method according to claim 24, in, The first device data is obtained from a first subsystem of an autonomous vehicle, and the second device data is obtained from a second subsystem of the autonomous vehicle.

28. One or more non-transitory computer-readable media having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 11 to 27.