Data manager

Abstract

To provide technology related to data managers.SOLUTION: A system and a method include the steps of: monitoring an available storage capacity of a vehicle's onboard data store and annotated data stored in the data store, wherein the annotated data stored in the data store includes event data associated with the operation of the vehicle; automatically deleting at least a portion of the data from the data store in response to an indicator of at least one of the available storage capacity of the data store and the annotated data stored in the data store; and generating an output including an indicator of completion of automatically deleting the at least a portion of the data.SELECTED DRAWING: Figure 4

Description

[Background technology]

[0001] An autonomous vehicle is a motor vehicle that can perform one or more required driving functions without the input of a human driver, and generally includes Level 2 or higher capabilities as generally described in SAE International's J3016 standard, and in certain embodiments, includes self-driving trucks that include sensors, devices, and systems that may function together to generate sensor data indicative of various parameter values ​​related to the vehicle's position, speed, operating characteristics, and state of the vehicle, including data generated in response to various objects, situations, and environments that the autonomous vehicle encounters during its operation.

[0002] A vast amount of sensor and other vehicle data for understanding the state of the road, the vehicle's condition, and the environment around the vehicle may be generated and stored on-board an autonomous vehicle during the vehicle's on-road operation (i.e., one or more driving sessions, trips, etc.). Typically, engineers and other involved entities must wait until the vehicle has completed its operational execution and returned to a facility where the data storage device (e.g., disk drive, etc.) that stores the data can be removed from the vehicle and the data can be retrieved therefrom, or the stored data can be uploaded from the vehicle for further inspection, processing, and analysis.

[0003] In some instances, for example, it may be desirable or necessary to observe or evaluate the performance of an autonomous vehicle during and after the vehicle is operating. As such, there is a need for robust systems and methods for efficiently and reliably managing data associated with the operation of an autonomous vehicle that is continuously generated and stored on-board the autonomous vehicle during driving operation of the vehicle, for example, to ensure secure storage of desired data. [Brief description of the drawings]

[0004] The features and advantages of the illustrative embodiments, and the manner in which they are accomplished, will become more readily apparent by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0005] [Figure 1] FIG. 1 is an example block diagram of a control system that may be deployed in a vehicle, according to an example embodiment.

[0006] [Figure 2A] 1 is an example depiction of an exterior view of a semi-truck in accordance with an example embodiment; [Figure 2B] 1 is an example depiction of an exterior view of a semi-truck in accordance with an example embodiment; [Figure 2C] 1 is an example depiction of an exterior view of a semi-truck in accordance with an example embodiment;

[0007] [Diagram 3] 1 is an exemplary depiction of a communication environment in which an autonomous vehicle may operate, in accordance with an illustrative embodiment.

[0008] [Figure 4] FIG. 1 is an exemplary block diagram illustrating aspects of a system, according to an exemplary embodiment.

[0009] [Diagram 5] FIG. 1 illustrates an example flow diagram of a process in accordance with an example embodiment.

[0010] [Figure 6] FIG. 11 is an example flow diagram of another process, according to an example embodiment.

[0011] [Figure 7] 1 is an example depiction of a data table corresponding to a configuration file, according to an example embodiment.

[0012] [Figure 8] 1 is an exemplary block diagram of a computing system in accordance with an exemplary embodiment.

[0013] Throughout the drawings and detailed description, the same drawing reference numbers will be understood to refer to the same elements, features, and structures unless otherwise stated. The relative size and depiction of these elements may be exaggerated or adjusted for clarity, illustration, and / or convenience. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In the following description, specific details are described to provide a thorough understanding of various exemplary embodiments. It is to be understood that various modifications to the embodiments will be readily apparent to those skilled in the art, and that the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Moreover, in the following description, many details are described for the purpose of explanation. However, those skilled in the art should understand that the embodiments may be practiced without using these specific details. In other instances, well-known structures and processes are not shown or described so as not to obscure the description with unnecessary detail. Thus, the present disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0015] A number of terms are used herein for convenience and ease of presentation. For example, the term "semi-truck" is used to refer to a vehicle in which the system of the exemplary embodiments may be used. The terms "semi-truck," "truck," "tractor," "vehicle," and "semi" may be used interchangeably herein. Additionally, embodiments of the present invention may be used in conjunction with other types of vehicles, as will be apparent to one of ordinary skill in the art after reading this disclosure. In general, embodiments may be used with desirable results in conjunction with any vehicle that tows a trailer or hauls cargo over long distances.

[0016] FIG 1 illustrates a control system 100 that may be deployed in and included in an autonomous vehicle (AV), such as, for example, a semi-truck 200 illustrated in FIGs. 2A-2C, according to an example embodiment. With reference to FIG 1, the control system 100 may include sensors 110 that collect data and information that are provided to a computer system 140 to perform operations including, for example, control operations to control components of the vehicle via a gateway 180. According to some embodiments, the gateway 180 is configured to enable the computer system 140 to control different components from different manufacturers.

[0017] Computer system 140 may be configured with one or more central processing units (CPUs) 142 to perform processing, including processing to implement features of embodiments of the present invention as described elsewhere herein, and to receive sensor data from sensors 110 for use in generating control signals to control one or more actuators or other controllers associated with systems of the vehicle in which control system 100 is deployed (e.g., actuators or controllers that enable control of throttle 184, steering system 186, brakes 188, and / or other devices and systems). In general, control system 100 may be configured to operate a vehicle (e.g., semi-truck 200) in an autonomous (or semi-autonomous) mode of operation.

[0018] For example, the control system 100 may be operated to capture images from one or more cameras 112 mounted at various locations on the semi-truck 200 and perform processing (e.g., image processing) on ​​these captured images to identify objects proximate to or within the path of the semi-truck 200. In some aspects, one or more lidar 114 and radar 116 sensors may be disposed on the vehicle to sense or detect the presence and volume of objects proximate to or within the path of the semi-truck 200. Other sensors may also be disposed or mounted at various locations on the semi-truck 200 to capture other information, such as position data. For example, the sensors may include one or more satellite positioning sensors and / or an inertial navigation system, such as the GNSS / IMU 118. The Global Navigation Satellite System (GNSS) is a space-based system of satellites that provides position information (longitude, latitude, altitude) and time information in all weather conditions on or near the Earth to devices called GNSS receivers. GPS is the most used GNSS system in the world and may be used interchangeably with GNSS herein. An inertial measurement unit ("IMU") is an inertial navigation system. In general, an inertial navigation system ("INS") measures and integrates the orientation, position, velocity, and acceleration of a moving object. The INS integrates the measurement data, and the GNSS is used as a correction to the integration errors of the INS orientation calculation. Any number of different types of GNSS / IMU 118 sensors may be used in conjunction with features of the present invention.

[0019] Data collected by each of the sensors 110 may be processed by the computer system 140 to generate control signals that may be used to control the operation of the semi-truck 200. For example, image and position information may be processed to identify or detect objects around or within the path of the semi-truck 200, and control signals may be sent via the controller 182 to adjust the throttle 184, steering 186, and / or brakes 188 as necessary to safely operate the semi-truck 200 in an autonomous or semi-autonomous manner. Although exemplary sensors, actuators, and other vehicle systems and devices are shown in FIG. 1, one of ordinary skill in the art will understand after reading this disclosure that other sensors, actuators, and systems may also be included in the system 100 consistent with the present disclosure. For example, in some embodiments, actuators may also be provided that provide a mechanism to allow control of the transmission of the vehicle (e.g., the semi-truck 200).

[0020] The control system 100 may include a computer system 140 (e.g., a computer server) configured to provide a computing environment in which one or more software, firmware, and control applications (e.g., items 160-182) may execute to perform at least some of the processes described herein. In some embodiments, the computer system 140 includes components deployed on the vehicle (e.g., deployed in a system rack 240 located within a sleeper compartment 212 of the semi-truck, as shown in FIG. 2C). The computer system 140 may communicate with other computer systems (not shown) that may be local to the semi-truck 200 and / or remote from the semi-truck (e.g., the computer system 140 may communicate with one or more remote ground or cloud-based computer systems via a wireless communication network connection).

[0021] According to various embodiments described herein, computer system 140 may be implemented as a server. In some embodiments, computer system 140 may be configured using any of a number of computing systems, environments, and / or configurations, including, but not limited to, a personal computer system, a cloud platform, a server computer system, a thin client, a thick client, a handheld or laptop device, a tablet, a smartphone, a database, a multiprocessor system, a microprocessor-based system, a set-top box, a programmable consumer electronics device, a network PC, a minicomputer system, a mainframe computer system, a distributed cloud computing environment, and the like, which may include any of the above systems or devices, and the like.

[0022] Different software applications or components may be executed by the computer system 140 and the control system 100. For example, as shown in active learning component 160, an application performing active learning machine processing may be provided to process images captured by one or more cameras 112 and information acquired by LIDAR 114. For example, the image data may be processed using a deep learning segmentation model 162 that identifies objects of interest (e.g., other vehicles, construction signs, etc.) in the captured images. In some aspects herein, deep learning segmentation may be used to identify lane points in a LIDAR scan. As an example, the system may use an intensity-based voxel filter to identify lane points in a LIDAR scan. The LIDAR data may be processed by a machine learning application 164 that draws or identifies bounding boxes on the image data to identify objects of interest located by the LIDAR sensor.

[0023] Information output from the machine learning applications may be provided as input to Object Fusion 168 and Vision Map Fusion 170 software components, which may perform processing to predict the behavior of other road users, fuse local vehicle pose with global map geometry in real time, and enable on-the-fly map correction. Output from the machine learning applications may be captured with information from Radar 116 and Map Localization 166 application data (as well as positioning data). In some aspects, these applications enable the control system 100 to be less reliant on maps and more able to handle constantly changing road environments. Furthermore, by correcting any map errors on the fly, the control system 100 may facilitate safer, more scalable, and more efficient operation compared to alternative map-centric approaches.

[0024] The information is provided to a predictive planning application 172, which provides input to components of a trajectory plan 174, allowing a trajectory to be generated in real-time by a trajectory generation system 176 based on interactions and predicted interactions between the semi-truck 200 and other associated vehicles in the truck operating environment. In some embodiments, for example, the control system 100 generates a 60 second planning period and analyzes the relevant actors and available trajectories. The plan that best meets multiple criteria (including safety, comfort, and route preferences) may be selected, and any associated control inputs required to implement the plan are provided to a controller 182 to control the movement of the semi-truck 200.

[0025] In some embodiments, these disclosed applications or components (as well as other components or flows described herein) may be implemented in hardware, a computer program executed by a processor, firmware, or a combination of the above, unless otherwise specified. In some cases, the computer program may be embodied on a computer-readable medium, such as a storage medium or storage device. For example, the computer program, code, or instructions may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disk read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art.

[0026] The non-transitory storage medium may be coupled to the processor such that the processor may read information from and write information to the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). In alternative embodiments, the processor and the storage medium may reside as separate components. For example, FIG. 1 illustrates an exemplary computer system 140 that may represent or be integrated with any of the components disclosed below. Thus, FIG. 1 is not intended to suggest any limitation on the scope of use or functionality of the embodiments of the systems and methods disclosed herein. The computer system 140 may be implemented and / or perform any of the functionality disclosed herein.

[0027] Computer system 140 may be described in the general context of computer system executable instructions, such as program modules, executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer system 140 may also be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including non-transitory memory storage devices.

[0028] 1, computer system 140 is shown in the form of a general-purpose computing device. Components of computer system 140 may include, but are not limited to, one or more processors (e.g., CPU 142 and GPU 144), a communication interface 146, one or more input / output interfaces 148, and one or more storage devices 150. Although not shown, computer system 140 may also include a system bus that couples various system components, including system memory, to CPU 142. In some embodiments, input / output (I / O) interface 148 may also include a network interface. For example, in some embodiments, some or all of the components of control system 100 may communicate via a controller area network (CAN) bus or the like that interconnects various components within the vehicle in which control system 100 is deployed and associated.

[0029] In some embodiments, storage device 150 may include various types and forms of non-transitory computer-readable media. Such media may be any available media accessible by a computer system / server and may include both volatile and non-volatile media, removable and non-removable media. The system memory, in one embodiment, implements the processes represented by the flow diagrams of other figures herein. The system memory may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. As another example, storage device 150 may read from and write to a non-removable non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, storage device 150 may include one or more removable non-volatile disk drives, such as magnetic, tape, or optical disk drives. In such a case, each may be connected to the bus by one or more data media interfaces. Storage device 150 may include at least one program product having a set (e.g., at least one) of program modules, codes, and / or instructions configured to perform the functions of various embodiments of the application.

[0030] 2A-2C are exemplary depictions of exterior views of a semi-truck 200 that may be associated with or used in accordance with exemplary embodiments. The semi-truck 200 is shown for illustrative purposes only. Thus, one skilled in the art will understand after reading this disclosure that the embodiments may be used in conjunction with many different types of vehicles and are not limited to the types of vehicles shown in FIGS. 2A-2C. The exemplary semi-truck 200 shown in FIGS. 2A-2C is one style of truck configuration common in North America, including an engine 206, a steering axle 214, and two drive axles 216, located forward of the cab 202. A trailer (not shown) may be attached to the semi-truck 200 via a fifth wheel trailer coupling, typically provided on a frame 218 and located on the drive axles 216. As shown in FIGS. 2A and 2C, a sleeper compartment 212 may be located behind the cab 202. FIGS. 2A-2C further show a number of sensors located at different locations on the semi-truck 200. For example, one or more sensors may be mounted on the roof of the cab 202 on a sensor rack 220. Sensors may also be mounted on the side mirrors 210, and other locations on the semi-truck. Sensors may be mounted on the bumper 204, as well as on the side of the cab 202, and other locations. For example, in FIG. 2A, rear-facing radar 236 is shown mounted on the side of the cab 202. Embodiments may be used with other configurations of trucks and other vehicles (e.g., semi-trucks having cab-over or cab-forward configurations, etc.). In general, without limiting the embodiments of the present disclosure, the features of the present invention may be used with desired results in vehicles that haul cargo over long distances, such as long-distance semi-truck routes.

[0031] FIG. 2B is a front view of the semi-truck 200, showing a number of sensors and sensor locations. A sensor rack 220 may mount and position several sensors on the windshield 208, including a long-range lidar 222, a long-range camera 224, a GPS antenna 234, and a mid-range front-facing camera 226. The side mirrors 210 may provide mounting locations for a rear-facing camera 228 and a mid-range lidar 230. A front radar 232 may be mounted on the bumper 204. Other sensors (some shown and some not shown) may be mounted or located in other locations on the semi-truck 200. Thus, the locations and mountings shown in FIGS. 2A-2C are for illustrative purposes only.

[0032] 2C, there is shown a partial view of the semi-truck 200 illustrating the interior of the cab 202 and some aspects of the sleeper compartment 212. In some embodiments, a portion of the control system 100 of FIG. 1 is mounted in a system rack 240 within the sleeper compartment 212, allowing easy access to the components of the control system 100 for maintenance and operation.

[0033] Certain aspects of the present disclosure relate to methods and systems that provide a framework or architecture for specifying triggers that indicate whether an incident of interest has occurred, and further provide a mechanism for capturing and reliably uploading specified datasets for immediate access in real-time automatically as an autonomous vehicle (AV) (e.g., a truck similar to those disclosed in Figures 1 and 2A-2C) is operating (e.g., driving). The disclosed framework may be used to provide insight into the status of an AV and its surrounding environment as it is operating, as well as more immediate (i.e., real-time) access to certain datasets captured by the AV. Aspects of the present disclosure generally provide a framework for accessing specific datasets corresponding to predefined events as the specific datasets are sensed, generated, or collected at the AV during operation of the AV.

[0034] FIG. 3 is an example depiction of an environment in which an autonomous vehicle may operate, according to an example embodiment. As shown in FIG. 3, system 300 may include one or more AVs (e.g., semi-trucks) 305a-305n that communicate with remote monitoring system 310 via networks 320, 325, and 330. In the example of FIG. 3, AV 305a communicates with remote monitoring system 310 via network 325 (e.g., a first cellular network or cell) and network 320, and AV 305n communicates with remote monitoring system 310 via network 330 (e.g., a second cellular network or cell) and network 320. In some instances, network 320 may include one or more different networks (not shown), and these different communication networks are collectively represented by network 320, as the number or type of networks may change, for example, as AVs 305a-n traverse different roads.

[0035] In some cases, the AV may record all of the sensor and other system data generated by its built-in sensors (e.g., cameras, lidar, radar, etc.), control systems (e.g., steering, braking, etc.), and processing systems (e.g., planning software, predictive software, perception software, etc.) in an on-board memory device (e.g., semi-truck storage 150 of FIG. 1 ), generally referred to herein as a data store, which may include one or more disk drives, memory, files, etc. The amount of all of this data is typically very large (e.g., 150 MBps) and may be stored on the AV until its operation is complete. In some aspects, embodiments disclosed herein are configured to reliably transmit data associated with the operation of the AV (e.g., sensor and other data generated and stored on-board the AV) to a remote monitoring system as the data is being captured (i.e., during operation of the AV) or at some point thereafter, for example, by transmitting certain data sets to the remote monitoring system 310 or other infrastructure for inspection, analysis, and further processing. In some cases, the remote monitoring system or other infrastructure may include a cloud computing infrastructure.

[0036] In some aspects, the particular set of data transmitted to the remote monitoring system may be a particular defined subset of data generated and stored on-board the AV. In some aspects, the present disclosure relates to systems and methods that efficiently and reliably manage data generated by and stored in a data store of the AV to, for example, ensure that sufficient free space is maintained in the data store to accommodate the data generated, as well as new data that may be generated during operation (e.g., continued operation) of the AV. In some embodiments, as described in more detail below, the present disclosure includes systems and methods for managing data generated by and stored on an AV, including monitoring available storage capacity of a data store on-board the AV and automatically (i.e., without human intervention) deleting at least a portion of the data generated by and stored on the AV. In some embodiments, the disclosed systems and methods may be capable of automatically performing the functions that they are configured to perform without requiring human interaction.

[0037] Referring now to FIG. 4, an exemplary system architecture block diagram 400 depicts a data manager 405 associated with an AV (e.g., AV 305 of FIG. 3). In one practical application, the data manager 405, data annotator system 410, data store 415, and data transmitter 420 shown in FIG. 4 may be deployed in a vehicle in conjunction with other vehicle sensors and systems, such as those shown in the semi-truck 200 disclosed in FIGS. 2A-2C, and the control system 100 of FIG. 1. In some embodiments, the data manager 405, data annotator system 410, data store 415, and data transmitter 420 may be executed by a primary vehicle computing, and these systems need not be attached to the vehicle's autonomous driving code. The primary vehicle computing may be on during operation of the AV.

[0038] In some aspects, data manager 405 may cooperatively interact with data annotator system 410, data store 415, and data transmitter 420 to effectively manage data generated and stored by the AV herein. Data annotator system 410 may also cooperatively interface with other vehicle sensors and systems. In general, data annotator system 410 tracks all of the data being created and recorded by the AV and annotates the data, including, for example, annotating the data with a priority and an indication of whether the data should be deleted.

[0039] The data manager 405 may generally manage data stored by an AV (e.g., data files or records) by deleting particular files or records (i.e., portions) of the data based on a combination of factors. The combination of factors may include, for example, the available storage capacity of the data store (i.e., storage utilization), the lifespan of the data (i.e., expiration date), and annotations associated with (e.g., assigned to) the data. In some cases, the lifespan of a particular file or record of data (e.g., a file representing event data generated by one or more sensors and devices of an AV herein, where the event data may be formatted as a bag file) may be represented by an annotation associated with the file or record, which may include an indication of the lifespan of the file or record. In some cases, annotations associated with a file or record of data generated by an AV may include an indication of whether the file or record should be deleted or not deleted. The indication of whether the file or record should be deleted or not deleted may be represented by an explicit instruction to delete the file or a flag, value, or other indicia that may be interpreted or determined to indicate whether the file or record should be deleted or not deleted. In some cases, the interpretation or determination of the meaning of a flag, value, or other indicia may be specified by a rule, specification, lookup table, or analysis of the flag, value, or other indicia.

[0040] The data transmitter 420 may generally operate to upload or otherwise transmit data (e.g., one or more data files or records) designated for transmission to a remote monitoring system or other remote infrastructure (e.g., cloud 490). In some embodiments, the data transmitter 420 may read data from the data store 415 (and drives therein) and upload or transmit it to the remote monitoring system or other remote infrastructure (e.g., cloud 490). Data may be uploaded or transmitted based on the available storage capacity of the data store (i.e., storage utilization), the lifespan (i.e., expiration date) of the data, and annotations associated with (e.g., assigned to) the data.

[0041] In some cases, an embodiment of the data store 415 may be implemented with one or more storage devices, such as drive A (417) through drive N (419). In one embodiment, the one or more drives may collectively have a storage capacity of about 36 terabytes (TB), although other storage capacities may be provided. The one or more drives comprising the data store may be implemented using different types of non-volatile memory to hold data generated by the AV herein. For example, in some embodiments, the drives of the data store may include memory of Serial Advanced Technology Attachment (SATA) Solid State Drives (SSD), Non-Volatile Express (NVMe), and other form factors and communication interfaces that may have write speeds of at least a minimum (e.g., ≧300 MB / s, and in some cases ≧600 MB / s). In some cases, the data store may be implemented to be tamper-resistant to protect the data stored therein.

[0042] In some aspects, the data manager 405 may operate to ensure that all AV data that is offloaded from the vehicle is retained on-board the vehicle until it can be successfully transmitted to a remote monitoring system or other designated destination (e.g., cloud computing infrastructure 490 providing resources to an enterprise's back-end infrastructure), where the data manager may retain certain data, but the data manager may also operate to delete other AV data from the on-board data store in accordance with predetermined data retention policies.

[0043] In some embodiments, the data annotation system 410 is configured to annotate or associate a label with the data file or record (e.g., a flag or other indicia in the header of the data file or record) that indicates whether the data file or record should be deleted or not deleted. In some aspects, the data annotation system 410 may include or have access to a bag file database 440, where the data annotation system tracks all of the data associated with the AV that is generated on the AV (and stored as a bag file in some embodiments). The data annotation system 410 may further include an application programming interface (API) or direct access interface 440 to facilitate communication between the data annotation system and the data manager. In some embodiments, the data file may be annotated for deletion, and the absence of a particular annotation for deletion may be interpreted or determined to indicate that this particular data file should be retained. In some other embodiments, data files may be annotated for retention, and the absence of a specific annotation for retention may be interpreted or determined to indicate that the subject data file may be deleted, if desired (e.g., data files that are not specifically marked for retention may be deleted when the storage capacity of the AV's data store exceeds a specified threshold or when data files are deleted to adhere to the AV's data retention policy).

[0044] Referring again to the data manager 405, an emergency data manager process 455 may be provided to the API 440 to monitor or poll for indications that the storage utilization capacity of the data store has exceeded its specified storage utilization threshold. In some embodiments, the emergency data manager process 455 may directly interface with the data store 415 (and drives therein) to determine whether the data store has exceeded its specified storage utilization threshold. For example, the emergency data manager process 455 may be monitoring one or more drives on which AV-related data is recorded to be more than 90% full (i.e., the specified storage utilization threshold of the exemplary AV data store). In response to an indication via a message at 445 that one or more drives (or the data store 415 as an entity) have exceeded the specified storage utilization threshold, the emergency data manager process 455 may initiate a query to the API or direct access interface 440, asking the API or direct access interface 440 to inform it of the highest priority bag file (i.e., the highest priority bag file for deletion) on the current disk that is recording AV-related data. The priority definitions may be managed by one or a combination of bag file database 435 and data annotator system 410. In some aspects, emergency data manager process 455 may delete the indicated highest priority bag file from data store 415 via file deletion interface 460. In some aspects, data manager 405 may notify the data annotation system via message 450 that it has deleted the bag file notified via message 445, and may request the data annotation system to provide an indication of the next bag file to delete if necessary to adhere to data storage usage policies.In some instances, the data annotation system and the data manager may iteratively interact with each other to delete data files until the available storage capacity of the data store (or one or more of its drives) falls below a threshold specified by a data storage usage policy implemented by the data annotation system. For example, in the current example where the data storage usage policy specifies a maximum data store (or its drives) usage threshold of 90%, bag files may continue to be identified by the data annotation system for deletion, and the data manager may continue to delete such identified data files until the data store (or its drives) has at least 10% of its storage capacity available as free space. In some embodiments, the data annotation system and the data manager may continue to delete data files until the specified usage threshold and an additional amount of free data storage capacity (e.g., +5% of the total storage capacity of the data store or its drives) is achieved. In response to the data manager deleting enough files to satisfy the data storage usage policy, the emergency data manager process 455 may return to a "sleep" state at least periodically polling the API or direct access interface 440 rather than actively deleting files.

[0045] In some embodiments, the data manager 405 may operate to manage data stored on-board the AV along with annotations or labeling provided by the data annotation system 410. For example, if a data file is annotated for deletion by the data annotation system, the data manager 405 proceeds to delete the data file. If a data file is not annotated for deletion by the data annotation system, the data manager may not delete the data file.

[0046] In some embodiments, the data manager 405 may also operate to manage data stored on-board the AV in accordance with a data storage usage policy or specification. The data storage usage policy may be defined to specify that the AV's data store will not exceed a threshold occupancy capacity (e.g., 90% full). That is, the data storage usage policy may indicate that the data store will maintain at least 10% of free or available space for data related to future operation of the AV. In some aspects, the data storage usage policy may specify an expiration or lifespan limit for data that does not otherwise need to be retained by the AV data store. In an illustrative example, the data storage usage policy may specify that AV data that does not otherwise need to be retained that exceeds a maximum threshold lifespan (e.g., 72 hours) is deleted. In this example, the data manager 405 here may proceed to delete AV data files that are older than a specified maximum threshold lifespan (e.g., 72 hours, but this may be a configurable variable that can have other alternative values ​​set by an authorized entity or personnel) and that are not otherwise marked or annotated for retention. In some embodiments, data files, records, or other data structures marked, annotated, or otherwise designated for transmission are retained at least until transmitted from the system by data transmitter 420. In some aspects, the data storage usage policy may specify at least one of (i) one or more types of data to delete, and (ii) an ordered sequence in which to delete data of the one or more specified types. For example, the data storage usage policy may specify that certain data stored by the AV may be deleted (e.g., data older than a specified maximum threshold age and not otherwise marked for transmission, no context data, product metric data, etc.), and the ordered sequence in which different types of data should be deleted.For example, a data storage usage policy may specify that data is to be deleted in the following order: (1) data older than a specified maximum threshold age and not otherwise marked for transmission (e.g., do not delete product metric data, triage data, and other data automatically marked for transmission), (2) contextless event data, and (3) product metric data. In some embodiments, a data storage usage policy may specify that certain data stored by the AV is not to be deleted. That is, a data storage usage policy may specify that certain types of data stored by the AV are not to be intentionally deleted by the data manager 405. In some embodiments, the data annotation system 410 may operate not to annotate certain types of data for deletion (e.g., AV intervention data, AV on-demand request data, etc.) in accordance with the data storage usage policy. Thus, the data manager 405, which operates to delete data based on annotations associated with the data, may not delete certain types of data that are not specifically annotated for deletion by the data annotation system.

[0047] In some embodiments, data successfully uploaded from an AV to the remote monitoring system 310 or other designated infrastructure (e.g., cloud 490) may be deleted from the data store 415. This aspect of managing the AV's data may be included in or independent of a data retention policy. In some embodiments, when a data file, record, or other data structure is transmitted from the AV by the data transmitter 420, the data transmitter may provide a record or indication (e.g., annotation, acknowledgment, etc.) that the data file, record, or other data structure was successfully transmitted. Based on such record or indication provided by the data transmitter, the data manager may proceed to delete the previously transmitted data file, record, or other data structure.

[0048] In some aspects, the data manager in some embodiments herein may not operate to record or transmit data, as these functions may be performed by the data annotation system and the data transmitter, respectively, herein. In some aspects, the data manager herein may delete (or not delete) data files based on annotations associated with the data files and the storage usage policy of the AV, where the data files may be annotated by the data annotation system and the data transmitter, where the storage usage policy may be implemented by the data annotation system executing or referencing a configuration file that specifies the storage usage policy. For example, the data annotation system 410 herein may annotate for deletion data files that are older than an expiration date specified in the storage usage policy and are not otherwise marked or specified for (automatic) transmission, and the data manager 405 deletes the data files annotated for deletion.

[0049] 4, data files representing AV event data created on-board the AV and associated with the operation of the AV are stored in the data store 415. In some instances, data files formatted as bag files 421 may be persisted across one or more of the AV's on-board drives 417 and 419. The data manager 405 may be configured to automatically delete one or more of the data files based on a combination of a specified data expiration date, a storage usage policy, and a data upload status of the data. In some embodiments, the storage usage policy may be defined in a configuration file utilized by an execution of the data annotation system 410, and the data expiration date may be specified in the configuration file (optionally as part of the storage usage policy). In some aspects, the data annotation system 410 may send a pointer to the bag file 421 when a particular bag file is older than a specified expiration date (e.g., 72 hours). For example, when a bag file is not marked (annotated) for transmission and is older than a specified expiration date, the bag file may be pushed to a queue 425 that may be monitored by the expired data manager process 430 (e.g., a worker thread, a method, a service, etc.). In some aspects, the expired data manager process 430 may monitor or poll the queue 425 for an indication of an expired data file. When the data annotation system 410 provides an indication that a data file is expired, the expired data manager process 430 retrieves the data file from the data store and proceeds with deleting the file. The expired data manager process 430 may utilize a file deletion interface 460 to delete the target data file (e.g., a bag file) from the data store 415.

[0050] With regard to some aspects of the file deletion interface 460, the file deletion interface may be implemented as a process, worker thread, method, or service defined such that both the expired data manager process 430 and the urgent data manager process 455 may call it directly from within their respective processes or otherwise invoke it. In some aspects, the file deletion interface 460 provides, in part, a mechanism for the data manager process 430 and the urgent data manager process 455 to lock a bag file for deletion. By taking an exclusive lock on the data file, the data manager process 430 and the data manager process 455 may proceed to delete the data file without interference from other processes or systems. In one embodiment, a process, worker thread, or service may be defined that is called or executed to obtain an exclusive lock on the data file to be deleted by the data manager process 430 or the urgent data manager process 455. For example, when the data manager is deleting data (via the data manager process 430 or the urgent data manager process 455), the data manager 405 attempts to obtain an exclusive lock on the particular bag file being deleted. When the data manager 405 is able to obtain an exclusive lock on a particular bag file, the data manager system can delete the target file while preventing other processes or systems from accessing or modifying the target file. In some aspects, the particular process, technique, or logic used to implement the exclusive lock may vary so long as the desired functionality of the exclusive lock as disclosed herein is achieved.

[0051] In some aspects, the data transmitter 420 provides, in part, a mechanism for the data transmitter 420 to lock the bag file for its transmission in some embodiments. By taking an exclusive lock on the data file, the data transmitter may proceed to transmit the data file without interference from other processes or systems. In one embodiment, a process, worker thread, or service may be defined that is called or executed to obtain an exclusive lock on the data file to be transmitted by the data transmitter 420. For example, when the data transmitter 420 is uploading or transmitting data, the data transmitter 420 attempts to obtain an exclusive lock on the particular bag file being transmitted. When the data transmitter 420 is able to obtain an exclusive lock on the particular bag file, the data transmitter system may transmit the subject file while preventing other processes or systems from accessing or modifying the subject file. After transmitting the subject file, the data transmitter 420 may proceed to delete the file. In some aspects, the particular process, technique, or logic used to implement the exclusive lock may vary, so long as the desired functionality of the exclusive lock as disclosed herein is achieved.

[0052] In some embodiments, if the data manager 405 (through the data manager process 430 or the urgent data manager process 455) or the data transmitter system 420 is unable to obtain an exclusive lock on a particular data file, the data manager 405 or the data transmitter system 420 learns that the other system owns an exclusive lock on the target data file. In such a scenario, the system that was unable to obtain an exclusive lock assumes that the other system owns an exclusive lock on the target data file and further terminates its own processing to delete or transmit the data, respectively. The system that was unable to obtain an exclusive lock terminates the process of deleting or transmitting the data since the other system with exclusive rights to the data file will process the data file. For example, if the data manager 405 (through the data manager process 430 or the urgent data manager process 455) is unable to obtain an exclusive lock on a particular data file, the data manager terminates the process of deleting the particular file since the data transmitter system 420 has an exclusive lock on the particular data file, transmits the particular data file, and deletes the particular data file (e.g., requires a bag) after transmission. In one embodiment, the data transmitter system 420 may indicate to the data manager 405 (or another system, device, worker thread, or method) that the data transmitter system is to send certain data files so that they may be deleted. Alternatively, for example, if the data transmitter system 420 is unable to obtain an exclusive lock on a particular data file, it stops the process of sending the particular file because the data manager 405 (via the data manager process 430 or the emergency data manager process 455) has an exclusive lock on the particular data file and will delete the particular data file accordingly. Thus, the data transmitter need not be involved in sending a particular file that has been locked by the data manager for deletion.In some aspects, the act of obtaining an exclusive lock and deleting a file by the file deletion interface 460 is illustrated by a connection 465 between the file deletion interface and the data store 415. If the file deletion interface 460 is able to successfully delete the file after obtaining the lock, the file deletion interface 460 may publish a message to the removal file queue 470 indicating that the file has been deleted. The message published to the removal file queue 470 indicating that the file has been deleted may be picked up by an API updater process 475 which may further update the remote API 480 that the data file (e.g., bag file) has been deleted on the AV. In some embodiments, the remote monitoring system represented by the cloud computing infrastructure 490 of FIG. 4 may include a remote database 485 which may include data files transmitted from the AV by the data transmitter 420 as well as reporting and analysis data regarding the files deleted by, for example, the data manager 405.

[0053] FIG. 5 is an example flow diagram of a process according to an example embodiment. In some embodiments, some aspects of the process 500 may be implemented using a framework or system architecture disclosed herein. In operation 505, one or more of a system, message, cue, or signal in the AV may be monitored for available storage capacity of the vehicle's on-board data store and an indication of annotated data stored in the data store. With reference to the example system shown in FIG. 4 and the corresponding disclosure above in connection therewith, the data manager 405, including the expired data manager process 430 and the emergency data manager process 455, may monitor the data annotator system 410 for annotated data including event data associated with the operation of the vehicle. In some aspects, the annotated data may include an indication of whether the subject data should be deleted, the lifespan of the annotated data, and other data annotations. In some aspects, the data manager 405 may obtain or determine available storage capacity of the vehicle storage's on-board data store (or one or more drives including the data store) from the AV's data store.

[0054] In operation 510, at least a portion of the data from the data store may be automatically deleted in response to at least one indication of the available storage capacity of the data store and the annotated data stored therein. For example, if monitoring of the data annotator system reveals that the available storage capacity of the data store exceeds a maximum threshold specified by a storage usage policy for the AV, an action may be invoked to delete at least a portion of the data stored in the data store such that the available storage capacity of the data store complies with the storage usage policy. As another example, if monitoring of the data annotator system reveals that the life span of the data exceeds a specified value, an action may be invoked to delete at least a portion of the expired data stored in the data store such that the available storage capacity of the data store is further freed up to accommodate data files created and stored in the future by the AV. Yet another example includes monitoring of annotated data from a data transmitter of the AV, which may annotate data previously uploaded or transmitted from the AV. In some aspects, data already transmitted from the AV to a remote monitoring system or other authorized infrastructure may not need to be retained within the AV. Thus, data that is annotated to indicate that it has already been successfully transmitted to the remote monitoring system (where it may be securely retained by the remote monitoring system) may be deleted from the AV's data store.

[0055] Continuing with process 500 at operation 515, an output is generated that includes an indication of completion of automatically deleting at least a portion of the data. The generated output may be configured as a data signal, message, data record, or other data structure that can be interpreted and understood by the systems herein (e.g., system 400 of FIG. 4 and one or more subsystems and devices therein), as well as systems, devices, or entities interfacing with the systems herein. The output generated at operation 515 may be transmitted to a remote monitoring system, as noted at operation 520. In some aspects, the remote monitoring system referenced at operation 520 may include, at least in part, a cloud computing infrastructure or environment.

[0056] With reference to FIG. 6, the framework or system architecture disclosed herein may be used to implement some aspects of a process 600 to provide efficient management of data stored in an AV's on-board data store, according to an example embodiment. At operation 605, event data created and stored on the AV may be monitored. The data may be monitored for indicators regarding available storage capacity of the vehicle's on-board data store, and annotated data stored therein. At operation 610, a determination is performed to determine whether the monitored data is marked for deletion. In some aspects, the data may be marked or annotated for deletion by a data annotator (e.g., data annotator system 410 of FIG. 4) based on one or more factors, such as the data has expired or is not the type of data the system was designed to retain for an extended period of time in its on-board data store. In some instances, if the data is successfully transmitted or uploaded from the AV to a remote monitoring system or other infrastructure that can securely hold the data such that the data no longer needs to be held onboard the AV, the data may be marked for deletion or otherwise annotated by a data transmitter (e.g., data transmitter 420) of the AV. If the monitored data is not annotated for deletion as determined in operation 610, process 600 may return to operation 605. If the monitored data is marked for deletion as determined in operation 610, process 600 (e.g., via expired data manager process 430 of FIG. 4 ) may proceed to operation 615 to obtain an exclusive lock on the data (e.g., one or more data files). If an exclusive lock cannot be obtained in operation 615, the process of deleting the subject data files may be stopped and process 600 may return to operation 605.If the exclusive lock is successfully obtained at operation 615, the process of deleting the target data file may proceed to deleting the target file at operation 620, and may report that the target file was successfully deleted at operation 625 and return to operation 605.

[0057] Referring again to operation 605, the event data created and stored on the AV may be monitored for indicators related to the available storage capacity of the vehicle's on-board data store. At operation 630, a determination is performed to determine whether an on-board data store usage policy defined for the AV has been exceeded. For example, the on-board data store usage policy may specify that neither the data store nor the drive containing the data store is utilized at more than 90% of its storage capacity. If the specified data store usage is not exceeded, as determined at operation 630, process 600 may return to operation 605 from operation 630. If the specified data store usage is exceeded, as determined at operation 630, process 600 (e.g., via emergency data manager process 455 of FIG. 4) may proceed to operation 635 to obtain an exclusive lock on the data (e.g., one or more data files). If an exclusive lock cannot be obtained at operation 635, the process of deleting the target data files to comply with the data store usage policy may be terminated, and process 600 may return to operation 605 from operation 635. If the exclusive lock is successfully obtained at operation 635, the process of deleting the target data file may proceed to deleting the target file at operation 640, and may report that the target file was successfully deleted at operation 645 and return to operation 605.

[0058] FIG. 7 is an exemplary depiction of a table 700 representation of a configuration file that may be referenced by a data annotator herein to annotate data, which may be monitored or otherwise referenced by a data manager herein and deleted by the data manager in response to annotations associated with the data. As shown, table 700 includes a list of requirements 705, as well as criteria 710 and notes 715 associated with each requirement. For example, a data expiration requirement specifies that data that is older than 72 hours and not marked for transmission should be deleted, the criteria of which are defined by the data annotator. Also, as illustrated in the example of FIG. 7, a requirement regarding transmitted data specifies that data transmitted to the cloud infrastructure should be deleted, the criteria of this requirement are defined by the data transmitter. Also, as shown in table 700 of FIG. 7, a requirement regarding data store usage specifies that if data store usage exceeds a specified threshold, a portion of the data should be deleted until compliance with a data store usage policy. The criteria of this requirement may be specified by the data annotator, including the value of the threshold, the order and type of data to delete.

[0059] In some aspects, the configuration files herein may be written as configurable code that is deployed to the vehicle. In some instances, the configuration files (or other data structures) referenced and utilized by the data annotator system herein may be deployed to the AV separately from other code and configuration files. In this manner, this separate deployment of the configuration files for the data annotator system may be managed independently of other configuration files and process deployments, including the lifecycle of each deployment. For example, the configuration files for the data annotator system are deployed separately from the autonomy code installed on the AV, and updates to the data annotator system's configuration files may not require changes to the AV's autonomy stack.

[0060] FIG. 8 illustrates a computing system 800 that may be used in any of the architectures or frameworks (e.g., computer 140 of FIG. 1 , computing system 400 of FIG. 4 ) and processes (e.g., FIGS. 5 and 6 ) disclosed herein, according to an example embodiment. FIG. 8 illustrates a block diagram of a system embodying a primary computing resource of an AV, including a data manager, according to some embodiments. Computing system 800 may include a general-purpose computing device and may execute program code to perform any of the functions described herein. Computing system 800 may include other, not shown, elements, according to some embodiments.

[0061] The computing system 800 includes a processing unit 810 operably coupled to a communication device 820, a data storage device 830, one or more input devices 840, one or more output devices 850, and a memory 860. The communication device 820 may facilitate communication with external devices, such as an external network, a data storage device, or other data sources. The input device 840 may include, for example, a keyboard, a keypad, a mouse or other pointing device, a microphone, a knob or switch, an infrared (IR) port, a docking station, and / or a touch screen. The input device 840 may be used, for example, to input information into the computing system 800. The output device 850 may include, for example, a display (e.g., a display screen), a speaker, and / or a printer.

[0062] The data storage device 830 may include any suitable persistent storage device including combinations of magnetic storage devices (e.g., magnetic tape, hard disk drives, and flash memory), optical storage devices, read-only memory (ROM) devices, etc., while the memory 860 may include random access memory (RAM).

[0063] The application servers 832 may each include program code that is executed by the processor 810 to cause the computing system 800 to perform any one or more of the processes described herein. The embodiments are not limited to the execution of these processes by a single computing device. The data storage device 830 may also store data and other program code for providing additional functionality and / or necessary for the operation of the computing system 800, such as device drivers, operating system files, etc. The data manager engine 834 may include program code that is executed by the processor 810 to delete certain data stored in the AV's data store, as disclosed in one or more processes and various embodiments herein. The configuration file 836 may be referenced by the data annotator in some embodiments herein to efficiently and accurately manage (e.g., delete, etc.) data created and stored by the AV. The configuration file may be defined to specify boundary constraints and values ​​of one or more data management related requirements, including, for example, data store data usage policies, data retention and expiration policies, and other data storage criteria. Results generated by the data manager engine 834 may be stored in the database management system node 840.

[0064] As will be understood based on the foregoing specification, the above-mentioned examples of the present disclosure may be implemented using computer programming or engineering techniques, including computer software, firmware, hardware, or any combination or subset thereof. Such resulting programs having computer readable code may be embodied or provided in one or more non-transitory computer readable media, thereby creating a computer program product, i.e., an article of manufacture, according to the discussed examples of the present disclosure. For example, the non-transitory computer readable medium may be, but is not limited to, a fixed drive, a diskette, an optical disk, a magnetic tape, a flash memory, an external drive, a semiconductor memory such as a read-only memory (ROM), a random access memory (RAM), and / or any other non-transitory transmitting and / or receiving medium, such as the Internet, cloud storage, the Internet of Things (IoT), or other communication networks or links. An article of manufacture including the computer code may be created and / or used by executing the code directly from one medium, by copying the code from one medium to another, or by transmitting the code over a network.

[0065] A computer program (also referred to as a program, software, software application, "app", or code) may include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine code. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, cloud storage, Internet of Things, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. However, "machine-readable medium" and "computer-readable medium" do not include transitory signals. The term "machine-readable signal" refers to any signal that may be used to provide machine instructions and / or any other type of data to a programmable processor.

[0066] The above description and illustration of processes herein should not be construed as implying a fixed order for performing process steps. Rather, process steps may be performed in any order practicable, including simultaneous performance of at least some steps. Although the present disclosure has been described in conjunction with specific examples, it should be understood that various changes, substitutions, and alterations apparent to those skilled in the art may be made to the disclosed embodiments without departing from the spirit and scope of the present disclosure as set forth in the appended claims.

Claims

1. 1. A vehicle computing system, comprising: a memory for storing computer instructions; a data storage device disposed in the vehicle and configured to store event data associated with operation of the vehicle; a processor communicatively coupled to the memory for executing the computer instructions, the processor configured to: monitoring event data generated and stored in the vehicle; determining while monitoring the event data that the event data should be deleted; acquiring an exclusive data lock on the event data based on the determination; in response to obtaining the exclusive data lock on the event data, deleting the event data; and generating an output including an indication of completion of the deletion of the event data.

2. the event data is monitored for metrics related to available storage capacity of an on-board data store of the vehicle; The vehicle computing system of claim 1 , wherein the determination that the event data should be deleted comprises a determination that the data store has exceeded a storage utilization policy.

3. The vehicle computing system of claim 2 , wherein the storage usage policy specifies that certain types of data cannot be intentionally deleted.

4. 3. The vehicle computing system of claim 2, wherein the storage utilization policy specifies at least one of: (i) one or more types of data to delete and (ii) an ordered sequence for deleting the one or more types of data if the available storage capacity of the data store exceeds a first threshold.

5. The vehicle computing system of claim 1 , wherein the determination that the event data should be deleted comprises a determination that the event data is marked for deletion.

6. 6. The vehicle computing system of claim 5, wherein the event data is marked for deletion based on one or more of a specified data expiration date, a specified data type, a lifespan of the event data, and an upload status of the event data.

7. The vehicle computing system of claim 1 , wherein the event data is generated by at least one device on-board the vehicle.

8. monitoring event data generated and stored in a vehicle; determining while monitoring the event data that the event data should be deleted; acquiring an exclusive data lock on the event data based on the determination; in response to obtaining the exclusive data lock on the event data, deleting the event data; generating an output comprising an indication of completion of the deletion of the event data.

9. the event data is monitored for metrics related to available storage capacity of an on-board data store of the vehicle; The method of claim 8 , wherein the determination that the event data should be deleted comprises a determination that the data store has exceeded a storage utilization policy.

10. The method of claim 9 , wherein the storage usage policy specifies that certain types of data cannot be intentionally deleted.

11. 10. The method of claim 9, wherein the storage utilization policy specifies at least one of: (i) one or more types of data to delete, and (ii) an ordered sequence for deleting the one or more types of data if the available storage capacity of the data store exceeds a first threshold.

12. The method of claim 8 , wherein the determination that the event data should be deleted comprises a determination that the event data is marked for deletion.

13. The method of claim 12 , wherein the event data is marked for deletion based on one or more of a specified data expiration date, a specified data type, a lifespan of the event data, and an upload status of the event data.

14. The method of claim 8 , wherein the event data is generated by at least one device on-board the vehicle.

15. A non-transitory medium having processor-executable instructions stored thereon, the instructions, when executed by a processor, causing a computer to perform a method, the method comprising: monitoring event data generated and stored in a vehicle; determining while monitoring the event data that the event data should be deleted; obtaining an exclusive data lock on the event data based on the determination; and in response to obtaining the exclusive data lock on the event data, deleting the event data; generating an output including an indication of completion of the deletion of the event data; The event data is generated by at least one device on-board the vehicle, the non-transitory medium.

16. the event data is monitored for metrics related to available storage capacity of an on-board data store of the vehicle; The non-transitory medium of claim 15 , wherein the determination that the event data should be deleted comprises a determination that the data store has exceeded a storage utilization policy.

17. The non-transitory media of claim 16 , wherein the storage usage policy specifies that certain types of data cannot be intentionally deleted.

18. 20. The non-transitory media of claim 16, wherein the storage utilization policy specifies at least one of: (i) one or more types of data to delete and (ii) an ordered sequence for deleting the one or more types of data if the available storage capacity of the data store exceeds a first threshold.

19. The non-transitory medium of claim 15 , wherein the determination that the event data should be deleted comprises a determination that the event data is marked for deletion.

20. 20. The non-transitory media of claim 19, wherein the event data is marked for deletion based on one or more of a specified data expiration date, a specified data type, a lifespan of the event data, and an upload status of the event data.

Images