Automotive data processing system with efficient metadata generation and export

Abstract

An automotive data processing system (20) includes a storage subsystem (44) and a processor (48). The storage subsystem is located in a vehicle and is configured to store at least one data (45) generated by one or more data sources (24, 28, 30, 32, 34) of the vehicle. The processor is installed in the vehicle and is configured to apply at least one model to data stored in or en route to the storage subsystem, the at least one model identifying one or more specified features of interest in the data, thereby generating metadata (46) that marks the occurrence of the specified features of interest in the stored data, and the processor is configured to export at least a portion of the metadata to an external system located outside the vehicle.

Description

[0001] Priority requirements

[0002] This patent application claims the benefit of U.S. Provisional Patent Application 62 / 948,027, filed December 13, 2019, the disclosure of which is incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to automotive data processing, and particularly to methods and systems for generating and exporting metadata in vehicles. Background Technology

[0004] Autonomous vehicles and other vehicles generate and process massive amounts of data during a day's driving, sometimes in the terabyte range. Data sources in autonomous vehicles can include, for example, cameras and other sensors, advanced driver assistance systems (ADAS), telematics control units (TCUs), infotainment systems, and various electronic control units (ECUs). Typically, these vast amounts of data are periodically uploaded to remote processors.

[0005] The above description is provided as an overview of the relevant technology in the field and should not be construed as an admission that any information contained herein constitutes prior art relative to this patent application. Summary of the Invention

[0006] The embodiments described herein provide an automotive data processing system including a storage subsystem and a processor. The storage subsystem is disposed in a vehicle and configured to store at least one data source generated by the vehicle from one or more data sources. The processor is disposed in the vehicle and configured to apply at least one model to data stored in or en route to the storage subsystem, the model identifying one or more specified features of interest in the data, thereby generating metadata that tags the occurrence of the specified features of interest in the stored data, and exporting at least a portion of the metadata to an external system located outside the vehicle.

[0007] In some embodiments, the processor is configured to identify the presence of the same feature of interest in data generated from two or more different data sources within the data source, and to tag the data in the generation of the metadata, thereby linking the presence of the same feature of interest presented in the data generated from the two or more different data sources within the data source. In one embodiment, the data generated from the two or more data sources within the data source is of the same type. In another embodiment, the data generated from the two or more data sources within the data source is of different types. In yet another embodiment, the two or more data sources within the data source are located at different locations within the vehicle.

[0008] In some embodiments, the processor is further configured to: receive, in response to the exported metadata, a request from the external system for one or more selected portions of the data relating to the feature of interest; and provide at least the selected portions of the data to the external system, but less than the entire data. In an exemplary embodiment, the processor is configured to provide the external system with two or more selected portions of the data, which are generated by two or more different data sources in the data source and associated with the same feature of interest.

[0009] In the disclosed embodiments, the processor is configured to select the portion of the metadata to be exported based on selection criteria. In another embodiment, the processor is configured to export the metadata without simultaneously exporting the data corresponding to that metadata.

[0010] In an exemplary embodiment, the storage subsystem is configured to store at least data generated by multiple independent data sources within the vehicle. In another embodiment, the storage subsystem is configured to store at least data generated by multiple data sources installed at different locations within the vehicle. In another embodiment, the storage subsystem is configured to store two or more different types of data.

[0011] In some embodiments, the processor is configured to receive a model from an external system. In one embodiment, the processor is configured to export metadata as the vehicle connects to a stop. In another embodiment, the processor is configured to export metadata independently of whether the vehicle is connected to a stop.

[0012] In some embodiments, the model includes an artificial intelligence (AI) inference model, and the processor is configured to generate metadata by applying the AI ​​inference model to the data. In one embodiment, the processor is configured to receive a pre-trained AI inference model from the external system and update the training of the AI ​​inference model in the vehicle.

[0013] In some embodiments, the storage subsystem includes a centralized storage device configured to store data generated from multiple data sources. In one embodiment, the centralized storage device is configured to store different types of data generated from two or more of the data sources.

[0014] In an embodiment, the processor is further configured to: apply a second model, different from the first model, to the data, the second model identifying one or more second features of interest in the data; add a second occurrence of the second feature of interest to the metadata according to the second model; and provide the external system with one or more of the following: (i) metadata relating only to the occurrence of the feature of interest, (ii) metadata relating only to the second occurrence of the second feature of interest, and (iii) metadata relating to both the occurrence and the second occurrence. In an embodiment, the processor is configured to store the structured metadata in the storage subsystem.

[0015] Furthermore, according to the embodiments described herein, a vehicle data processing method is also provided, comprising storing data generated from one or more data sources of the vehicle into a storage subsystem disposed within the vehicle. Using a processor installed in the vehicle, at least one model is applied to the data stored in or en route to the storage subsystem, the model identifying one or more specified features of interest in the data, thereby generating metadata that tags the occurrence of the specified features of interest in the stored data. At least a portion of this metadata is exported to an external system located outside the vehicle.

[0016] Furthermore, according to the embodiments described herein, an automotive data processing system is also provided, comprising multiple data sources and a processor. The data sources are installed in a vehicle and configured to generate data. The processor is installed in the vehicle and configured to: collect the data generated by the data sources and locally store the data in a storage device located within the vehicle, without exporting the data outside the vehicle; identify one or more portions of interest in the locally stored data; and, while the vehicle is connected to a docking station coupled to an external system located remotely relative to the docking station, upload at least the identified portions of interest to the external system, less than the total amount of the locally stored data.

[0017] In some embodiments, the processor is configured to collect and store the data and identify the portion of interest independently of whether the vehicle is connected to a stop. In one embodiment, the processor is configured to collect data from two or more different data sources that produce the same type of data. In another embodiment, the processor is configured to collect data from two or more different data sources that produce different types of data.

[0018] In yet another embodiment, the processor is configured to collect data from two or more different data sources located at different locations within the vehicle. In the disclosed embodiment, the processor is configured to identify and upload two or more portions of interest generated by two or more different data sources and associated with the same feature of interest.

[0019] According to the embodiments described herein, a vehicle data processing method is also provided, comprising generating data from multiple data sources installed in a vehicle. Using a processor installed in the vehicle, the data generated from the data sources is collected and locally stored in a storage device located in the vehicle without exporting the data outside the vehicle; one or more portions of interest in the locally stored data are identified; and while the vehicle is connected to a docking station coupled to an external system located remotely relative to the docking station, at least the identified portions of interest are uploaded to the external system, less than the entirety of the locally stored data.

[0020] According to the embodiments described herein, an automotive data processing system is also provided, comprising multiple data sources, a packet network, and a processor. The data sources are distributed at different locations within the vehicle and configured to generate data. The packet network is located within the vehicle and configured to transfer data from the multiple data sources to a central storage location within the vehicle. The processor is installed in the vehicle and configured to: generate metadata corresponding to the data transferred from the multiple data sources and stored in the central storage location; and transfer one or more selected portions of the sensor data to an external system located outside the vehicle, the selected portions being selected based on the metadata corresponding to the data transferred from the multiple data sources and stored in the central storage location.

[0021] In one embodiment, the plurality of data sources includes two or more different data sources configured to produce different types of data. In another embodiment, the data produced by the two or more data sources are of different types. In the disclosed embodiment, the processor is configured to select one or more portions of the sensor data, at least in part, in response to an indication of one or more features of interest received from an external system.

[0022] In an exemplary embodiment, the processor is configured to generate the metadata by applying an artificial intelligence (AI) inference model to data stored at the central storage location. In one embodiment, during the generation of the metadata, the processor is configured to tag one or more occurrences of one or more features of interest in the data. In another embodiment, during the generation of the metadata, the processor is configured to tag the occurrence of the same feature of interest in two or more different types of data or in data generated from two or more different data sources.

[0023] Furthermore, according to the embodiments described herein, a vehicle data processing method is also provided, comprising: generating data from multiple data sources distributed at different locations within the vehicle; and transmitting the data from the multiple data sources to a central storage location within the vehicle via a packet network configured within the vehicle. Using a processor installed in the vehicle, metadata corresponding to the data transmitted from the multiple data sources and stored at the central storage location is generated; and one or more selected portions of the sensor data, the selected portions being chosen based on the metadata corresponding to the data transmitted from the multiple data sources and stored at the central storage location, are transmitted to an external system located outside the vehicle.

[0024] According to the embodiments described herein, an automotive data analysis system is also provided, comprising an interface for communicating with a vehicle and a computer. The computer is configured to: define a model that identifies one or more specified features in data generated from one or more data sources in the vehicle; provide the model to a processor installed in the vehicle; receive metadata from the processor in the vehicle, wherein the metadata is generated by the processor using the model and marks the occurrence of the specified features in the data; and analyze the received metadata.

[0025] In one embodiment, the interface is configured to receive the metadata while the vehicle is connected to a stop. In another embodiment, the interface is configured to receive metadata from the vehicle independently of whether the vehicle is connected to a stop.

[0026] In some embodiments, the model includes an artificial intelligence (AI) inference model, and the computer is configured to train the AI ​​inference model and provide the trained AI inference model to a processor in the vehicle. In the disclosed embodiments, the computer is configured to, in response to received metadata, send a request to the processor in the vehicle for one or more selected portions of stored data, receive the requested one or more selected portions from the processor, and analyze the one or more selected portions of the data.

[0027] In another embodiment, the computer is configured to request two or more selected portions of stored data generated by two or more different data sources in the data source. In yet another embodiment, the computer is configured to request two or more selected portions of stored data associated with the same feature of interest.

[0028] Furthermore, according to embodiments disclosed herein, a vehicle data analysis method is also provided, comprising: defining a model in a computer located outside the vehicle, the model identifying one or more specified features in data generated by one or more data sources of the vehicle; and providing the model to a processor installed in the vehicle. In the computer located outside the vehicle, metadata is received from the processor in the vehicle, wherein the metadata is generated by the processor using the model and marks the occurrence of the specified features in the data. The received metadata is analyzed.

[0029] This disclosure will be more fully understood through the following detailed description of embodiments of the disclosure in conjunction with the accompanying drawings, in which: Attached Figure Description

[0030] Figure 1 This is a schematic block diagram of an automotive data processor system according to an embodiment described herein;

[0031] Figure 2 The embodiments described herein are illustrated schematically. Figure 1 A flowchart illustrating the methods for generating and processing metadata in automotive systems. Detailed Implementation

[0032] Modern vehicles generate massive amounts of data of various types from a variety of data sources. For example, autonomous vehicles can generate one or more video feeds from video cameras, various types of sensor data from sensor streams located at various locations within the vehicle, radar and lidar information, driver behavior information, information from infotainment subsystems and user applications, and positioning and navigation information, such as GPS data and maps. In a typical day of driving, an autonomous vehicle can generate approximately four terabytes of data.

[0033] Data generated by vehicles during operation can provide invaluable information if properly analyzed. For example, GPS data and vehicle driving data can be used to update commuting prediction models; continuous camera footage can be used to optimize the autonomous driving capabilities and safety processes of self-driving cars; and driving behavior information can be used by insurance companies to determine driver insurance premiums—these are just a few examples.

[0034] The computing power required to perform such analysis typically far exceeds the capabilities of current technology or the affordability of onboard computing resources. Therefore, in various embodiments, to effectively analyze the data, at least some of the raw data is exported to an external system, such as a cloud-based data processing system. On the other hand, limitations regarding time, bandwidth, and power consumption make it impractical to transfer all the raw data from the vehicle to an external system. These constraints limit the effectiveness of analyzing data generated in the vehicle and the effectiveness of extracting valuable and actionable information from it.

[0035] In view of the foregoing, the embodiments described herein provide improved methods and systems for processing, communicating, and analyzing data generated in vehicles.

[0036] In some disclosed embodiments, an automotive data processing system is deployed in a vehicle. This automotive data processing system includes a storage subsystem coupled to a processor. During vehicle operation, the storage subsystem stores raw data generated from various data sources. The processor preprocesses the raw data by identifying specified features of interest and generates metadata that marks the presence of the specified features of interest in the raw data. This metadata, or at least a selected portion of it, is exported from the vehicle to an external system.

[0037] This external system typically analyzes metadata to identify specific portions of the raw data of interest and retrieves these specific portions from the vehicle for analysis. In this embodiment, the vehicle data processing system is configured to communicate with the external system while the vehicle is connected to a stop. In this embodiment, the processor continuously stores the metadata locally in the storage subsystem. When the vehicle stops, i.e., when it is parked and connected to a stop, the processor transfers the accumulated metadata to the external system.

[0038] In some embodiments, the onboard processor identifies features of interest and generates metadata by applying a pre-trained artificial intelligence (AI) inference model to the raw data. This AI model may be provided by, for example, an external system. In some embodiments, the processor applies several different AI models, for example, to identify different features of interest, or to identify the same features of interest in different types of raw data. Several examples of implementations and illustrative uses of the disclosed techniques will be described below.

[0039] In some embodiments, the same feature of interest (e.g., an object or event) may be presented in two or more different raw data streams generated by two or more data sources. For example, the two or more data sources may be sensors of the same type but located at different locations in the vehicle, or they may be sensors of different types located at the same or different locations in the vehicle. In some embodiments, the onboard processor identifies the feature of interest (e.g., an object or event) in the different raw data streams and generates metadata that marks the parts of the raw data where the same feature of interest is common or shared in the different streams. Other aspects of the generation and processing of metadata related to different media objects are described in U.S. Patent Application Publication 2020 / 0042548 entitled “Metadata Generation for Multiple Object Types,” the disclosure of which is incorporated herein by reference.

[0040] Typically, metadata is several orders of magnitude smaller, simpler, and has a structured format than the raw data. Accordingly, metadata can be transmitted to external systems within a short timeframe using moderate bandwidth. The specific portion of the raw data subsequently requested by the external system based on the metadata is also much smaller than the total amount of raw data. Therefore, the disclosed technique provides an optimal "workload partitioning" that (i) overcomes the communication bottleneck between the in-vehicle system and the external system, and (ii) does not compromise the quality of data analysis.

[0041] Figure 1 This is a schematic block diagram of an automotive data processor system 20 according to an embodiment described herein. The system 20 is installed in a vehicle and includes various sensors 24, multiple electronic device control units (ECUs) 32, an advanced driver assistance system (ADAS) 28, an infotainment system 30, and a central computer 34.

[0042] For example, sensor 24 may include a video camera, speed sensor, accelerometer, audio sensor, infrared sensor, radar sensor, lidar sensor, ultrasonic sensor, rangefinder or other proximity sensor, or any other suitable sensor type. In this example, each ECU 32 (sometimes referred to as a “zone ECU”) is connected to a sensor installed in a corresponding zone of the vehicle. Each ECU 32 typically controls its corresponding sensor 24 and collects data from that sensor. In an embodiment, one or more of the sensors 24 (e.g., image sensors) are directly connected to ADAS 28 without going through ECU 32.

[0043] In this context, any element in system 20 that generates data (e.g., including sensor 24, ECU 32, ADAS 28, infotainment system 30, and central computer 34) is considered a “data source.” ECU 32, ADAS 28, infotainment system 30, and central computer 34 are considered examples of “electronic subsystems” of the vehicle. It can be seen that a system component can act as both a data source and an electronic subsystem.

[0044] In some embodiments, various electronic subsystems of system 20 are deployed at various locations within the vehicle and communicate via a packet network installed in the vehicle. In this example, the packet network includes an Ethernet network, but other suitable network protocols may also be used. The network includes multiple Ethernet links 36 and one or more Ethernet switches 40. In various embodiments, the bit rate used in the network may be 10 Gbps according to IEEE 802.3ch, 1000 Mbps according to IEEE 802.3bp, 100 Mbps according to IEEE 802.3bw, 10 Mbps (10Base-T1s) according to IEEE 802.3cg, or any other suitable bit rate. For example, link 36 may include a twisted-pair copper link or any other type of link suitable for Ethernet communication.

[0045] exist Figure 1 In the example, system 20 also includes a centralized storage device 44 (also referred to as a “storage subsystem”) and a processor 48. For example, aspects of a centralized storage device in an automotive network are described in U.S. Patent Application 17 / 094,844, filed November 11, 2020, entitled “Automotive Network with Centralized Storage,” the disclosure of which is incorporated herein by reference.

[0046] Storage device 44 stores raw data 45 provided by various data sources (e.g., sensors and / or electronic subsystems) within the vehicle. In some embodiments, processor 48 identifies specified features of interest in the raw data 45, generates metadata 46 for the occurrence of the specified features of interest in the raw data, and stores the metadata 46 in storage device 45. In embodiments, processor 48 identifies features of interest by applying a computational artificial intelligence (AI) engine to the raw data 45. Processor 48 exports at least a portion of the metadata to an external system located outside the vehicle, such as a cloud-based data processing system or another suitable computer located outside the vehicle.

[0047] In this embodiment, the centralized storage device 44 includes a solid-state drive (SSD) directly connected to a packet network installed in the vehicle. As can be seen from the figures, in this example, the centralized storage device 44 includes two interfaces connected via corresponding links 36 to two different ports of two different switches 40. This arrangement provides a degree of redundancy because a failure of a single interface (or its corresponding link) will not disconnect the centralized storage device 44 from the network.

[0048] In this example, processor 48 and storage device 44 communicate with each other via a high-speed computer bus, such as a high-speed peripheral component interconnect (PCIe) bus. Alternatively, in some embodiments, processor 48 may be independently connected to the packet network, in which case processor 48 and storage device 44 communicate with each other through the network. Processor 48 is configured to communicate using data link 52 with a docking station or any other suitable interface (not shown in the figures) to which the vehicle is connected for data transfer.

[0049] Typically, the data source (sensors and / or electronic subsystems) sends raw data 45 to the centralized storage device 44 via this network. In various embodiments, the data source and the centralized storage device 44 can communicate using any suitable protocol, such as the Non-Volatile Memory Express Over Fabrics (NVMe-oF) protocol or NVMe over TCP. Other aspects of a centralized storage device that can be applied in system 20 are described in U.S. Patent Application 17 / 094,844, filed November 11, 2020, entitled “Automotive Network with Centralized Storage,” the disclosure of which is incorporated herein by reference.

[0050] like Figure 1 The configuration of the communication system 20 and its components (e.g., various devices and electronic subsystems and / or centralized storage device 44) shown is merely an exemplary configuration depicted for clarity. In alternative embodiments, any other suitable configuration may be used. For example, system 20 may include any other suitable type of electronic subsystems and / or devices laid out and connected in any other suitable manner. As another example, the packet network (including link 36 and switch 40) may have any other suitable topology.

[0051] Different elements of system 20 and its components may be implemented using dedicated hardware or firmware, for example, using hard-wired or programmable logic in, for example, application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). Alternatively or additionally, some functions of components of system 20 (e.g., ECU 32, storage device 44, and / or processor 48) may be implemented in software and / or using a combination of hardware and software elements. For clarity, elements not essential for understanding the disclosed technology have been omitted from the accompanying drawings.

[0052] In some embodiments, some functions of the ECU 32, storage device 44, and / or processor 48 may be implemented in one or more programmable processors, which are software-programmed to perform the functions described herein. This software may be downloaded electronically to any processor via a network, or it may be alternatively or additionally provided and / or stored on a non-volatile tangible medium such as magnetic, optical, or electronic memory.

[0053] As noted above, various data sources within the vehicle (e.g., sensors and electronic subsystems) generate and transmit raw data 45 during the vehicle's operation for storage in a centralized storage device 44. For example, raw data 45 may include video data, audio data, sensor data, radar and / or lidar data, driver behavior data, data from the infotainment subsystem and / or user applications, positioning and / or navigation data such as GPS data and maps, and / or any other suitable type of data.

[0054] As can be seen, at least some of the raw data (e.g., video) can be unstructured. Raw data typically has multiple different types (modalities) and usually originates from multiple independent data sources of multiple different types installed at multiple different locations within the vehicle. In some embodiments, multiple raw data streams are generated by multiple sensors of the same type (e.g., multiple video cameras or multiple LiDARs), each of which is positioned at a different location within the vehicle. In other embodiments, multiple instances of the same type of sensors are positioned at the same location within the vehicle to provide redundant streams of the same type of raw data.

[0055] Processor 48 scans raw data 45 to identify one or more specified features of interest, generates a database of metadata 46 that tags the occurrence of the specified features of interest in the raw data, and stores the metadata 46 in storage device 45. In various embodiments, for example, processor 48 can search for any suitable feature of interest, such as:

[0056] • Objects or environmental conditions near the vehicle that are visible in the video or detectable in sensor data (e.g., obstacles, traffic signs, another vehicle, or pedestrians approaching the vehicle).

[0057] • Events related to driver behavior (e.g., vehicle deceleration, acceleration in response to traffic lights, steering to avoid accidents or obstacles, or responding to traffic signs).

[0058] • Events related to the vehicle's road behavior (e.g., autonomous driving effects).

[0059] • Events related to the presence or behavior of nearby vehicles, such as congested or smooth traffic situations.

[0060] • Events related to driver intervention in the autonomous operation of a vehicle.

[0061] • Communication with external components (e.g., other vehicles, traffic lights, road sensors).

[0062] • Mechanical symptoms in the vehicle (e.g., changes in responsiveness or noise due to tire pressure loss, wear of one or more brake pads, loss of visibility due to burnt-out headlights or other lights, decreased connectivity due to in-vehicle network problems).

[0063] The list of features of interest above is provided for illustrative purposes only. Alternatively, processor 48 may identify any other suitable features that may be of interest for analysis.

[0064] As noted above, in some cases, a feature of interest (e.g., an object or event) appears in raw data from multiple data sources. These multiple data sources (and thus the raw data streams) can be of different types (e.g., video, audio, LiDAR, radar, etc.) or can be multiple instances of the same type of raw data (e.g., video captured at different locations within a vehicle). For example, a road accident or mechanical damage or malfunction may be visible in a video feed, audible in audio data, sensed by various sensors, and / or reported by one or more electronic subsystems. The correlation of such occurrences of a given feature of interest in the raw data from different data sources is valuable in itself. Therefore, in some embodiments, processor 48 uses the same label to tag the occurrences of such features of interest (e.g., objects or events), or otherwise tags them in a way that links their occurrences in the data from different sensors.

[0065] In various embodiments, processor 48 may use various techniques to identify specified features of interest in the raw data 45. In some embodiments, processor 48 runs a computational artificial intelligence (AI) engine configured to compile and apply a pre-trained AI inference model to identify features of interest in the raw data 45. In some embodiments, processor 48 applies several different AI models to the raw data, for example, to identify different features of interest, or to identify the same features of interest in different types of raw data.

[0066] In one embodiment, processor 48 receives one or more pre-trained artificial intelligence (AI) models from an external system, for example, via data link 52 when the vehicle connects to a stop. During the operation of the vehicle, processor 48 applies the one or more pre-trained AI models to raw data 45 provided by various data sources.

[0067] Processor 48 uses one or more AI models to identify the presence of features of interest in the raw data 45 and generates metadata 46 that tags these presences. Processor 48 may apply AI models to the raw data already stored in storage device 44 and / or to the raw data en route to storage device 44. Metadata 46 is temporarily stored in storage device 44. At appropriate times, such as when the vehicle connects to a stop, processor 48 transmits the metadata 46 calculated in the vehicle to an external system, typically without transmitting the entire raw data.

[0068] Note that because the metadata 46 is relatively small (compared to the original data), in some embodiments, it is not necessary for the vehicle to use its connection to the dock to transmit metadata to an external system. In embodiments, the processor 48 may transmit the metadata 46 to an external system during vehicle operation, independent of whether the vehicle is connected to the dock, for example, via a suitable wireless link, such as a cellular link, mobile Wi-Fi connectivity coupling, or other suitable wireless link. Similarly, in embodiments, the external system may send the AI ​​model to the processor 48, independent of whether the vehicle is connected to the dock, for example, via a suitable wireless link. Generally, communication via the dock (e.g., receiving the AI ​​model and deriving portions of the metadata and / or the requested original data) is considered an offline process.

[0069] In some embodiments, processor 48 transmits all of metadata 46 to an external system. In other embodiments, processor 48 uses selection criteria to select only a portion of metadata 46 and sends only the selected portion of the metadata to the external system.

[0070] The disclosed techniques can be used to analyze a wide range of features across various use cases. For example, consider a use case where an external system analyzes the behavior of an autonomous vehicle when an animal is presented on or near a road. To perform this type of analysis, in an embodiment, the external system trains an AI model to identify animals in raw video data. Processor 48 receives the pre-trained AI model, compiles it, and applies it to video camera feeds from various cameras in the vehicle. Based on the presence of identified features of interest (the presentation of an animal in the video), processor 48 generates metadata that tagged the video frames presenting the animal. The tagged frames may be located in video streams from two or more video sensors positioned at different locations within the vehicle. Processor 48 uploads the metadata to the external system for initial analysis.

[0071] In one embodiment, the external system can then request the portion of interest in the raw data 45 relating to the identified occurrence, such as a video segment that begins slightly before the animal's presentation. Note that the requested portion of interest in the raw data may not necessarily be exactly the identified occurrence. For example, the external system could record the time from identification to animal in the video and request sensor data corresponding to those times. In another embodiment, the processor 48 can upload the portion of interest in the raw data without request from the external system, for example, along with metadata.

[0072] As another example, consider the use of an external system to analyze the behavior of an autonomous vehicle in the presence of pedestrians under rainy conditions. To perform this type of analysis, in this embodiment, the external system provides processor 48 with multiple pre-trained AI models, such as a model for identifying rainy scenes in a video, another model for identifying pedestrians in the video, and other possible models for identifying rainy conditions in the output of other types of sensors (e.g., environmental condition sensors). Using these models, processor 48 generates metadata that labels the time intervals in the video where it is raining and the time intervals in the video where pedestrians are shown, and it is possible to do so in the raw data from other sensors. The external system can later look for intersections between the various types of intervals (i.e., time intervals identifying rainy conditions that include pedestrians) and request sensor data and / or subsystem data corresponding to these time intervals.

[0073] The two examples above are highly simplified and do not constitute a limitation in any way, and are given solely to demonstrate the effectiveness of the disclosed techniques. Other examples, for instance, could involve a much larger number of different types of data sources.

[0074] Figure 2This is a flowchart schematically illustrating a method for generating and processing metadata according to embodiments described herein. The left side of the diagram depicts actions performed by an external system (e.g., a cloud-based data processing system). The right side of the diagram depicts actions performed by system 20 within the vehicle.

[0075] In model training operation 60, an external system trains an AI inference model to identify a feature of interest in some type of raw data 45 (e.g., identifying people in video data, identifying mechanical faults in audio data, or sensor data). For example (but not necessarily), the trained model is provided to processor 48 while the vehicle is connecting to a stop. During compilation operation 64, processor 48 compiles the AI ​​model to run on the processor's computational AI engine. In embodiments, processor 48 may update the training of the AI ​​model to take local input into account, for example, performing additional training using data acquired in a particular vehicle. Such updates can be useful, for example, to better fit the area that a particular vehicle typically travels in. This additional training can be performed at any stage, for example, while the vehicle is connecting to a stop and / or while the vehicle is in motion.

[0076] In data collection operation 68, various data sources in the vehicle generate raw data 45 and send it for storage in storage device 44. In identification operation 72, processor 48 identifies the presence of features of interest in the raw data 45 by applying an AI model.

[0077] In metadata generation operation 76, processor 48 generates metadata 46 that tags the identified occurrences. Each tag in metadata 46 typically indicates the identified feature of interest and the location of that occurrence in the raw data (e.g., a frame number in a video or another location depending on the type of raw data). Processor 48 can apply AI models to the raw data already stored in storage device 44 and / or to the raw data en route to storage device 44.

[0078] As noted above, metadata generation can be applied to multiple sets of raw data, and metadata generation can involve checking the labels of the same features of interest (e.g., objects or events) presented in different sets of raw data provided by different sensors of the same or different types.

[0079] In metadata storage operation 80, processor 48 stores metadata 46 in storage device 44. In notification operation 84, processor 48 notifies an external system that metadata 46 is available. In metadata retrieval operation 88, the external system retrieves metadata 46 from the vehicle's storage device 44. In various embodiments, metadata 46 can be "pulled" from the vehicle by the external system, or the vehicle can actively "push" metadata to the external system. For example, processor 48 can periodically push metadata to the external system or when the vehicle connects to a stop.

[0080] In initial analysis operation 92, the external system performs an appropriate initial analysis on the metadata. This initial analysis typically identifies one or more portions of interest in the raw data 45 regarding features of interest based on the metadata. In data retrieval operation 96, the external system retrieves the portions of interest from the vehicle's storage device 44. Subsequently, in analysis operation 100, the external system analyzes the portions of interest in the raw data.

[0081] In this embodiment, the process described above can be repeated using different AI models. These different AI models are remapped by the processor 48 to the computational AI engine and applied to label different features of interest (or the same features of interest in different types of raw data). The processor 48 can add new metadata to the existing metadata database or generate a separate database of metadata 46 and store it in the storage device 44.

[0082] While the embodiments described herein can process raw data acquired in automotive systems, the methods and systems described herein can also be used in other applications, such as for processing various other types of unstructured raw data, such as video library analysis, voice call analysis, etc.

[0083] It should be noted that the embodiments described above are illustrated by way of example, and the present invention is not limited to the specific content shown and described above. Rather, the scope of the present invention includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of these various features that would occur to those skilled in the art upon reading the foregoing description and that are not disclosed in the prior art. Documents incorporated herein by reference should be considered an integral part of this application, unless any term is defined in such incorporated documents in a manner that contradicts the definitions expressly or implicitly made in this specification, in which case only the definitions in this specification should be considered.

Claims

1. A vehicle data processing system, comprising: A storage subsystem is provided in the vehicle and configured to store at least one data source generated by the vehicle; as well as A processor, which is installed in the vehicle and configured to: At least one model is applied to the data stored in or en route to the storage subsystem, the model identifying one or more specified features of interest in the data, thereby generating metadata that marks the occurrence of the specified features of interest in the stored data; At least a portion of the metadata is exported to an external system located outside the vehicle via a data link, wherein the metadata has been marked for the presence of the specified features of interest before the data containing the one or more specified features of interest is exported; After the metadata is exported to the external system, a request for one or more selected portions of the data is received from the external system, wherein the request is determined by the external system based on the parsing of the metadata, and the selected portions are related to the feature of interest; as well as In response to the request, a selected portion of the data is provided to the external system via the data link, wherein the selected portion includes one or more specified features of interest, and the selected portion provided is less than the entire data, in order to reduce the amount of data transmitted via the data link.

2. The vehicle data processing system of claim 1, wherein the processor is configured to identify the appearance of the same feature of interest in the data generated by two or more different data sources in the data source, and to mark the data in the generation of the metadata, thereby linking the appearance of the same feature of interest presented in the data generated by the two or more different data sources in the data source to each other.

3. The vehicle data processing system of claim 2, wherein the data generated by the two or more data sources in the data source is data of the same type.

4. The vehicle data processing system of claim 2, wherein the data generated by the two or more data sources in the data source are data of different types.

5. The vehicle data processing system according to claim 2, wherein the two or more data sources are located at different locations within the vehicle.

6. The automotive data processing system of claim 1, wherein the processor is configured to provide two or more selected portions of the data to the external system, the two or more selected portions being generated by two or more different data sources in the data source and associated with the same feature of interest.

7. The vehicle data processing system according to any one of claims 1-5, wherein the processor is configured to select the portion of the metadata to be exported according to a selection criterion.

8. The vehicle data processing system according to any one of claims 1-5, wherein the processor is configured to export the metadata without simultaneously exporting data corresponding to the metadata.

9. The vehicle data processing system according to any one of claims 1-5, wherein the storage subsystem is configured to store at least data generated by a plurality of independent data sources in the vehicle.

10. The vehicle data processing system according to any one of claims 1-5, wherein the storage subsystem is configured to store at least data generated by a plurality of data sources installed at different locations in the vehicle.

11. The vehicle data processing system according to any one of claims 1-5, wherein the storage subsystem is configured to store two or more different types of said data.

12. The vehicle data processing system according to any one of claims 1-5, wherein the processor is configured to receive the model from the external system.

13. The vehicle data processing system according to any one of claims 1-5, wherein the processor is configured to export the metadata while the vehicle is connected to a stop.

14. The vehicle data processing system according to any one of claims 1-5, wherein the processor is configured to export the metadata independently of whether the vehicle is connected to a stop.

15. The vehicle data processing system according to any one of claims 1-5, wherein the model includes an artificial intelligence (AI) inference model, and wherein the processor is configured to generate the metadata by applying the AI ​​inference model to the data.

16. The vehicle data processing system of claim 15, wherein the processor is configured to receive the pre-trained AI inference model from the external system and update the training of the AI ​​inference model in the vehicle.

17. The vehicle data processing system according to any one of claims 1-5, wherein the storage subsystem includes a centralized storage device configured to store the data generated by the plurality of said data sources.

18. The vehicle data processing system of claim 17, wherein the centralized storage device is configured to store different types of data generated by two or more of the data sources.

19. The vehicle data processing system according to any one of claims 1-5, wherein the model is a first model, and the processor is further configured to: A second model, different from the first model, is applied to the data, wherein the second model identifies one or more second features of interest in the data; According to the second model, add a second occurrence of the second feature of interest to the metadata; and Provide the external system with one or more of the following: (i) metadata relating only to the occurrence of the feature of interest, (ii) metadata relating only to the second occurrence of the second feature of interest, and (iii) metadata relating to both the occurrence and the second occurrence.

20. The vehicle data processing system according to any one of claims 1-5, wherein the processor is configured to store the metadata in a structured form in the storage subsystem.

21. A vehicle data processing method, comprising: At least one data source generated by the vehicle shall be stored in a storage subsystem set up in the vehicle; as well as Use the processor installed in the vehicle: At least one model is applied to the data stored in or en route to the storage subsystem, the model identifying one or more specified features of interest in the data, thereby generating metadata that marks the occurrence of the specified features of interest in the stored data; At least a portion of the metadata is exported to an external system located outside the vehicle via a data link, wherein the metadata has been marked for the presence of the specified features of interest before the data containing the one or more specified features of interest is exported; After the metadata is exported to the external system, a request for one or more selected portions of the data is received from the external system, wherein the request is determined by the external system based on the parsing of the metadata, and the selected portions are related to the feature of interest; as well as In response to the request, a selected portion of the data is provided to the external system via the data link, wherein the selected portion includes one or more specified features of interest, and the selected portion provided is less than the entire data, in order to reduce the amount of data transmitted via the data link.

22. The vehicle data processing method of claim 21, wherein applying the model includes identifying the appearance of the same feature of interest in the data generated from two or more different data sources in the data source, and wherein generating the metadata includes tagging the data such that the appearance of the same feature of interest presented in the data generated from the two or more different data sources in the data source is mutually linked.

23. The vehicle data processing method of claim 22, wherein identifying the occurrence of the same feature of interest includes identifying the occurrence in data provided by two or more data sources located at different locations in the vehicle.

24. The vehicle data processing method of claim 21, wherein providing the selected portions includes providing two or more selected portions of the data to the external system, the two or more selected portions being generated by two or more different data sources in the data source and associated with the same feature of interest.

25. The vehicle data processing method according to any one of claims 21-23, wherein exporting the metadata includes exporting the metadata separately from data corresponding to the metadata, selecting one or more portions of interest of the data based on the metadata, and exporting the selected one or more portions of interest of the data when the vehicle is connected to a stop.

26. A vehicle data processing system, comprising: Multiple data sources are installed in the vehicle and configured to generate data; as well as A processor, which is installed in the vehicle and configured to: The data generated by the data source is collected and stored locally in a storage device installed in the vehicle, without exporting the data outside the vehicle; At least one model is applied to locally stored data to identify one or more specified features of interest, thereby generating metadata that tags the occurrence of the specified features of interest in the stored data; At least a portion of the metadata is exported to an external system located outside the vehicle via a data link; After the metadata is exported to the external system, a request for one or more selected portions of the data is received from the external system, wherein the request is determined by the external system based on the parsing of the metadata, and the selected portions are related to the feature of interest; and In response to the request, while the vehicle is connected to a stop and the stop is coupled to an external system located at a remote location relative to the stop, a selected portion of the data is uploaded to the external system via the data link, wherein the selected portion includes one or more specified features of interest, and the uploaded selected portion is less than the total amount of data stored locally, in order to reduce the amount of data transmitted via the data link.

27. The vehicle data processing system of claim 26, wherein the processor is configured to collect and store the data and identify the portion of interest, independent of whether the vehicle is connected to a stop.

28. The automotive data processing system of claim 26, wherein the processor is configured to collect the data from two or more different data sources that generate the same type of data from the data source.

29. The automotive data processing system of claim 26, wherein the processor is configured to collect the data from two or more different data sources that generate different types of data from the data source.

30. The vehicle data processing system of claim 26, wherein the processor is configured to collect the data from two or more different data sources located at different locations in the vehicle.

31. The vehicle data processing system according to any one of claims 26-30, wherein the processor is configured to identify and upload two or more portions of interest generated by two or more different data sources in the data sources and associated with the same feature of interest.

32. A vehicle data processing method, comprising: Data is generated from multiple data sources installed in the vehicle; as well as Use the processor installed in the vehicle: The data generated by the data source is collected and stored locally in a storage device installed in the vehicle, without exporting the data outside the vehicle; At least one model is applied to locally stored data to identify one or more specified features of interest, thereby generating metadata that tags the occurrence of the specified features of interest in the stored data; At least a portion of the metadata is exported to an external system located outside the vehicle via a data link; After the metadata is exported to the external system, a request for one or more selected portions of the data is received from the external system, wherein the request is determined by the external system based on the parsing of the metadata, and the selected portions are related to the feature of interest; as well as In response to the request, while the vehicle is connected to a stop and the stop is coupled to an external system located at a remote location relative to the stop, a selected portion of the data is uploaded to the external system via the data link, wherein the selected portion includes one or more specified features of interest, and the uploaded selected portion is less than the total amount of data stored locally, in order to reduce the amount of data transmitted via the data link.

33. The vehicle data processing method of claim 32, wherein the collection and storage of the data and the identification of the portion of interest are performed independently of whether the vehicle is connected to a stop.

34. The vehicle data processing method of claim 32, wherein collecting the data includes collecting the data from two or more different data sources that generate the same type of data from the data source.

35. The vehicle data processing method of claim 32, wherein collecting the data includes collecting the data from two or more different data sources that generate different types of data from the data source.

36. The vehicle data processing method of claim 32, wherein collecting the data includes collecting the data from two or more different data sources located at different locations in the vehicle.

37. The vehicle data processing method according to any one of claims 32-36, wherein identifying and uploading the portion of interest includes identifying and uploading two or more portions of interest generated by two or more different data sources in the data source and associated with the same feature of interest.

38. A vehicle data processing system, comprising: Multiple data sources, distributed at different locations within the vehicle and configured to generate data; A packet network is provided in the vehicle and configured to transfer the data from the plurality of data sources to a central storage location in the vehicle; as well as A processor, which is installed in the vehicle and configured to: At least one model is applied to the data stored at the central storage location, the model identifying one or more specified features of interest in the data, thereby generating metadata that marks the occurrence of the specified features of interest in the data; At least a portion of the metadata is exported to an external system located outside the vehicle via a data link; After the metadata is exported to the external system, a request for one or more selected portions of the data is received from the external system, wherein the request is determined by the external system based on the parsing of the metadata, and the selected portions are related to the feature of interest; as well as In response to the request, one or more selected portions of the data are transmitted to the external system via the data link. The selected portions are chosen based on the metadata and are less than the entire data to reduce the amount of data transmitted via the data link.

39. The vehicle data processing system of claim 38, wherein the plurality of data sources includes two or more different data sources configured to generate different types of data.

40. The vehicle data processing system of claim 38, wherein the data generated by two or more of the data sources are of different types.

41. The automotive data processing system according to any one of claims 38-40, wherein the processor is configured to select the one or more selected portions of the data in at least part of response to an indication of one or more features of interest received from the external system.

42. The automotive data processing system according to any one of claims 38-40, wherein the processor is configured to generate the metadata by applying an artificial intelligence (AI) inference model to the data stored at the central storage location.

43. The vehicle data processing system according to any one of claims 38-40, wherein in generating the metadata, the processor is configured to mark one or more occurrences of one or more features of interest in the data.

44. The vehicle data processing system according to any one of claims 38-40, wherein, when generating the metadata, the processor is configured to mark the occurrence of the same feature of interest in two or more different types of said data or in data generated from two or more different data sources.

45. A vehicle data processing method, comprising: Data is generated from multiple data sources located at different locations within the vehicle; The data is transmitted from the plurality of data sources to a central storage location within the vehicle via a packet network configured within the vehicle; and Use the processor installed in the vehicle: At least one model is applied to the data stored at the central storage location, the model identifying one or more specified features of interest in the data, thereby generating metadata that marks the occurrence of the specified features of interest in the stored data; At least a portion of the metadata is exported to an external system located outside the vehicle via a data link; After the metadata is exported to the external system, a request for one or more selected portions of the data is received from the external system, wherein the request is determined by the external system based on the parsing of the metadata, and the selected portions are related to the feature of interest; as well as In response to the request, one or more selected portions of the data are transmitted to the external system via the data link. The selected portions are chosen based on the metadata and are less than the entire data to reduce the amount of data transmitted via the data link.

46. ​​The vehicle data processing method of claim 45, wherein the plurality of data sources includes two or more different data sources configured to generate different types of data.

47. The vehicle data processing method of claim 45, wherein the data generated by two or more of the data sources are of different types.

48. The vehicle data processing method according to any one of claims 45-47, wherein the selection of the one or more selected portions of the data is performed at least in part in response to an instruction for one or more features of interest received from the external system.

49. The vehicle data processing method according to any one of claims 45-47, wherein generating the metadata includes applying an artificial intelligence (AI) inference model to the data stored at the central storage location.

50. The vehicle data processing method according to any one of claims 45-47, wherein generating the metadata includes tagging one or more occurrences of one or more features of interest in the data.

51. The vehicle data processing method according to any one of claims 45-47, wherein generating the metadata includes marking the occurrence of the same feature of interest in two or more different types of said data or in data generated from two or more different data sources.

52. A vehicle data analysis system, comprising: An interface used for communicating with the vehicle; as well as The computer is configured as follows: Define a model that identifies one or more specified features in data generated from one or more data sources of the vehicle; The model is provided to the processor installed in the vehicle; The processor in the vehicle receives metadata, wherein the metadata is generated by the processor using the model and marks the occurrence of the specified features in the data; Analyze the received metadata; Based on the analysis of the metadata, a request for one or more selected portions of the data is sent to the processor in the vehicle, wherein the selected portions are related to features marked by the metadata; as well as The processor receives a selected portion of the requested data, wherein the received selected portion is less than the total data generated by one or more data sources of the vehicle.

53. The vehicle data analysis system of claim 52, wherein the interface is configured to receive the metadata while the vehicle is connected to a stop.

54. The vehicle data analysis system of claim 52, wherein the interface is configured to receive the metadata from the vehicle regardless of whether the vehicle is connected to a stop.

55. The vehicle data analysis system according to any one of claims 52-54, wherein the model includes an artificial intelligence (AI) inference model, and wherein the computer is configured to train the AI ​​inference model and provide the trained AI inference model to the processor in the vehicle.

56. The vehicle data analysis system according to any one of claims 52-54, wherein the computer is configured to send a request to the processor in the vehicle for one or more selected portions of stored data in response to received metadata, receive the requested one or more selected portions from the processor, and analyze the one or more selected portions of the data.

57. The automotive data analysis system according to any one of claims 52-54, wherein the computer is configured to request two or more selected portions of the stored data generated by two or more different data sources among the data sources.

58. The automotive data analysis system according to any one of claims 52-54, wherein the computer is configured to request two or more selected portions of stored data associated with the same features of interest.

59. A method for analyzing automotive data, comprising: In a computer located outside the vehicle, a model is defined that identifies one or more specified features in data generated from one or more data sources of the vehicle; The model is provided to the processor installed in the vehicle; In the computer located outside the vehicle, metadata is received from the processor in the vehicle, wherein the metadata is generated by the processor using the model and the occurrence of the specified features in the data is marked; as well as Analyze the received metadata; Based on the analysis of the metadata, a request for one or more selected portions of the data is sent to the processor in the vehicle, wherein the selected portions are related to features marked by the metadata; as well as The processor receives a selected portion of the requested data, wherein the received selected portion is less than the total data generated by one or more data sources of the vehicle.

Images