Method of operating data recording apparatus and system
By introducing a correlation determination unit into the data recording device, data correlation determination is carried out, the high cost problem caused by redundant data recording is solved, and an efficient and flexible data recording method is realized.
Patent Information
- Application Number
- CN202411275724.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-09-14
- Filing Date
- 2024-09-12
- Publication Date
- 2025-03-14
AI Technical Summary
In autonomous driving and advanced driver assistance system applications, data recording devices record excessive redundant data, resulting in significant storage and review costs.
By introducing a correlation determination unit into the data recording device, the correlation determination is performed using a supervised and unsupervised prior, only the data deemed to be related is stored in the persistent storage area.
An efficient and cost-effective data recording method is realized, reducing the storage of redundant data, reducing storage and review costs, while improving the flexibility and adaptability of data recording.
Smart Images
Figure CN119625860A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to a method of operating a data recording device and a method of operating a data recording system. Background Art
[0002] In modern data logging sessions for autonomous driving (AD) or advanced driver assistance systems (ADAS) applications in vehicles, an excessive amount of data is recorded by the corresponding data logging devices. A large portion of the recorded data may be redundant and does not provide any added value. Despite this, the data is still recorded and stored. This incurs significant costs, both visible costs, such as storage costs, and hidden costs, such as the engineering effort to review and filter irrelevant data.
[0003] Therefore, there is a need for an efficient and cost-effective data recording method that omits the recording of redundant data. Summary of the invention
[0004] The present disclosure provides in a first aspect and a second aspect a computer implemented method of operating a data recording device according to the independent claims. Embodiments are given in the dependent claims, the description and the drawings.
[0005] In a first aspect, the present disclosure relates to a computer-implemented method of operating a data logging device for autonomous driving (AD) or advanced driver assistance system (ADAS) applications of a vehicle, the data logging device comprising: a data logging unit, a computing unit, a storage unit and a correlation determination unit comprising at least one correlation determination module, the storage unit comprising a circular buffer and a persistent storage area, wherein the method comprises the following steps: the data logging unit and the computing unit retrieve sensor data, in particular acquired by a sensor in the vehicle; the data logging unit stores the sensor data in the circular buffer; the computing unit analyzes the sensor data and uses the sensor data to derive a perception result and / or a neural network embedding from the retrieved sensor data; the correlation determination unit retrieves and stores the perception result and / or the neural network embedding; the correlation determination unit determines whether the retrieved perception result and / or the neural network embedding is considered to be relevant or irrelevant; in the event that the perception result and / or the neural network embedding corresponding to the sensor data is considered relevant by the correlation determination unit, the correlation determination unit triggers the transmission of the sensor data from the circular buffer to the persistent storage area for persistent storage of the sensor data.
[0006] The present disclosure provides a method for correlation-based data logging that uses supervised and / or unsupervised priors for correlation determination. The method provides an efficient and cost-effective way to log data that is very flexible and easily adaptable to different applications.
[0007] In a typical data recording session, data is collected while a vehicle equipped with sensors for data acquisition is driving in different environments. The acquired data may include information about the surrounding environment and the host system, and may include a list of relevant objects, such as a detection list of road users and traffic signs in the external perception AD or ADAS context or a list of detected passengers in the internal AD or ADAS context. The information on the host system may include, for example, speed and yaw rate from the own vehicle and similar parameters.
[0008] During a data logging session, data is collected by a data logging device attached to or as part of a vehicle. Sensors provide raw sensor data, such as camera images, LiDAR point clouds, or GPS measurements. The raw sensor data is transmitted to a data logging unit (e.g., by using a VIGEM logger). The data logging unit continuously transmits the data to a circular buffer within a storage unit. Newly acquired data continuously enters the circular buffer, and once the memory of the circular buffer is full, the oldest data stored in the circular buffer is overwritten. Depending on the specific logging requirements, typical buffer sizes may be in the range of 1 to 5 minutes of continuous data logging.
[0009] During the data acquisition phase, some of the data acquired in the circular buffer may be considered relevant. If considered relevant, the specific data is moved to a persistent storage area within the storage unit, or alternatively, is marked as belonging to the persistent storage area for permanent storage. Thus, data in the persistent storage area is prevented from being overwritten later by newer data entering the circular buffer.
[0010] In order to be able to decide which data to record, i.e. which data to keep in a persistent storage area, based on relevance, the computing unit continuously processes the incoming raw sensor data, providing perception results and / or neural network embeddings from the sensor data.
[0011] The perception results may include information about the surrounding environment and the host system, and may include a list of relevant objects, for example, a detected list of road users and traffic signs or a list of detected passengers. The information on the host system may include, for example, speed and yaw rate from the host vehicle and similar parameters.
[0012] Neural networks are typically used to perform detection, classification or segmentation tasks in typical AD or ADAS classification scenarios. Neural network embeddings, also referred to below as embeddings or embedding vectors, involve an intermediate processing step within the signal processing chain of a neural network. In mathematical terms, an embedding is an n-dimensional vector that represents the projection of the input data into a high-dimensional neural network latent space. In this regard, an embedding vector is an abstract representation of the input data in the context of a neural network. This embedding vector representation can therefore be used to assess the novelty or uniqueness of the observed and collected sensor data. Input sensor data that is mapped to a similar embedding will be less novel than input data that is mapped to a unique, previously unobserved vector embedding.
[0013] The computing unit transmits its perception results and / or neural network embeddings to the relevance determination unit. The relevance determination unit may include a relevance status module and one or more independent relevance determination modules. The relevance determination unit may be integrated in the computing unit or may be a separate component. Each relevance determination module may consider the incoming data to be relevant or irrelevant. If one of the independent relevance determination modules considers that specific incoming sensor data is relevant, the relevance status module changes the state of the sensor data from irrelevant to relevant. This triggers the transfer of the corresponding sensor data from the circular buffer to the persistent storage area and protects the sensor data from being overwritten at a subsequent time.
[0014] If at any time, at least one correlation determination module deems particular sensor data to be relevant, then that data will be recorded and transferred from the circular buffer to a persistent storage area. The data transferred to the persistent storage area may include data acquired during an offset time prior to the correlation determination. The data acquired during an offset time prior to the correlation determination will therefore be part of the recorded data and will also be retained in the persistent storage area. The offset time may be flexible and may be selectively selected. The offset time may therefore vary between different correlation modules and may also depend on the particular recorded data.
[0015] As long as at least one relevance determination module remains in the relevant state, data recording continues.Thereby, all corresponding tags and meta information can be attached to the corresponding data record, as described below.
[0016] When a relevance determination module transitions from irrelevant to relevant, it can augment this determination with additional information, such as a label indicating why the particular sensor data is considered relevant. The additional information may also include additional meta-information and offset time lengths included in the recording of the particular sensor data. When a relevance determination module transitions from irrelevant to relevant, metadata may be recorded to record the internal reasoning of the corresponding relevance determination module. When a relevance determination module transitions from irrelevant to relevant, a human observer may be prompted to agree or disagree with the determination. Feedback collected in this manner can be fed back to the data recording device to improve and / or fine-tune the thresholds of individual thresholds or triggers.
[0017] If all relevance determination modules are switched to the irrelevant state, recording continues if at least one of the relevance determination modules has requested a certain post-recording time span. If all relevance determination modules are switched to the irrelevant state and no relevance determination module has requested an additional post-recording time, recording (i.e. the transfer of data from the circular buffer to the persistent storage area) stops. The recording length thus depends on the relevance decision provided by each of the at least one relevance determination module. Thus, a record does not have a fixed size, but has a flexible length.
[0018] According to an embodiment of the first aspect, at least one relevance determination module determines whether the retrieved perceptual results and / or the neural network embeddings are considered to be relevant or irrelevant by applying a predefined set of rules to the retrieved perceptual results.
[0019] The predefined data set may include, for example, the number of objects of a certain class detected (such as motorcycles or pedestrians), or a specific subject speed reached. The rule set may be provided to the relevance determination unit, for example, as python code. The relevance determination module compares the predefined rule set with the perception results retrieved from the computing unit, and makes a decision based on the comparison whether the corresponding data is relevant or not.
[0020] According to an embodiment of the first aspect, the method comprises the following additional steps: the computing unit reduces the dimensionality of the neural network embedding, in particular by principal component analysis (PCA); the computing unit transmits the reduced neural network embedding to a correlation determination unit; and the correlation determination unit determines the correlation based on the reduced neural network embedding.
[0021] Based on the trade-off between accuracy and speed, the dimensionality of the neural network embedding can be reduced to an arbitrary dimension. Higher dimensional embedding vector spaces enhance accuracy, but also require more powerful computing resources. Reducing the dimensionality of the embedding vector space can enhance the computing speed and can enable weaker computing resources to process the data. Therefore, using lower dimensional embeddings allows the method to be deployed also in vehicles equipped with weaker computing resources, for example, consumer cars instead of dedicated development cars. By exploiting the dimensionality reduction of the embedding space, the method can easily adjust specificity, storage requirements and computational workload. If dimensionality reduction is applied to the embedding vector, the linear projection matrix that maps the corresponding embedding vector to a compressed state will be added to the relevance determination unit as an internal state.
[0022] According to an embodiment of the first aspect, the data recording device further comprises a display unit provided with a display and a touch panel, and the method comprises the following additional steps: the computing unit transmits the perception result to the display unit; the display unit displays the perception result on the display; the display unit recognizes a manual user interaction on the touch panel in response to the perception result displayed on the display; and the display unit, when recognizing the manual user interaction, transmits information that the perception result must be considered relevant to the determination module.
[0023] The display unit can be used to visualize the incoming raw sensor data and the perception results transmitted by the computing unit. The display unit can be used by a human observer, for example, the driver of the vehicle or an additional observer traveling with the vehicle, to monitor the recording and collection of data. In the case where the human observer considers a particular situation to be relevant, a manual interaction with the touch panel can be performed. This causes the display unit to forward information that the corresponding data or situation is considered relevant to the relevance determination unit. The human observer's decision that a particular situation is relevant may be based on a predetermined scene catalog, for example. The recording of the data can be ended in various ways, for example, by another manual interaction with the touch panel. The human observer can, for example, also select a predetermined time span, after which the recording ends.
[0024] According to an embodiment of the first aspect, the method comprises the following additional steps: a computing unit receives a text-based user query; the computing unit encodes the retrieved text-based user query into a neural network embedding; the computing unit transmits the neural network embedding to a relevance determination unit; and at least one relevance determination module determines whether the neural network embedding derived from the sensor data is considered relevant or irrelevant based on a similarity measure between the neural network embedding derived from the sensor data and the encoded user query neural network embedding, in particular using cosine similarity.
[0025] The computation unit comprises at least two data encoders, one encoder for encoding text-based user queries and one encoder for encoding sensor data, in particular image data. In the present context, encoding means computing a neural network embedding from text-based data, image data, or other types of sensor data received by the computation unit.
[0026] An example of such a data encoder is provided by the OpenAI CLIP model, which provides an embedding vector representation of image data that is independent of specific camera settings and camera configurations. CLIP embeddings capture much of the semantic content of an image and thus provide a good mathematical representation of the semantics contained in the image.
[0027] The text-based user query may include a positive query, indicating data of interest that should be included in the query search. The text-based user query may also include a negative query, indicating data that should be excluded from the query search. If a determination is made that specific sensor data should be excluded based on the corresponding embedding vector, the relevance determination module will be prohibited from transitioning to the relevance state regardless of other queries (particularly positive queries).
[0028] The text-based query may be provided to the computing unit as a query vector. Specific examples of such text queries are "car approaching a roundabout" or "motorcycle standing at a red traffic light". The query vector is then encoded into an embedding vector using a text encoder of the computing unit. The text-based user query may also be transmitted to the computing unit via a user query device, which may be a component of the data recording device or a separate component.
[0029] The neural network encoder of the computation unit is trained to match image / text pairs to close embedding vectors based on cosine similarity and map unmatched pairs to more distant embedding vectors. After the image and text have been encoded into the respective embeddings, the similarity between the image encoding and the text encoding is calculated. This is done, for example, by calculating the dot product between the two vectors x and y of the neural network embeddings and calculating the similarity according to This is achieved by calculating the cosine similarity SC(x,y).
[0030] Before each data logging session, a compilation of user-defined text-based queries can be provided, which are encoded into corresponding embedding vectors. These vectors can also be classified using a classifier (e.g., a support vector machine trained on the corresponding text-based query data). During driving, i.e., during the data logging session, the classifier can be used to classify the embeddings from the incoming sensor data (e.g., camera image data). The method does not rely on labeled data, as similarity measures are used for correlation calculations, in particular utilizing combined text and image embeddings. The method is therefore very flexible.
[0031] According to an embodiment of the first aspect, at least one determination module decides whether the retrieved perceptual result and / or the neural network embedding is to be considered relevant or irrelevant based on a comparison of the neural network embedding with neural network embeddings previously stored in a relevance determination unit.
[0032] Recording data with very similar embeddings to previously recorded data may not provide additional information. Therefore, it is necessary to estimate whether the new incoming embedding vector is already adequately represented in the embedding database already stored in the relevance determination unit.
[0033] Just after initialization, the embedding database of recorded data will be empty. During the recording session, a growing database of previously observed embeddings is maintained while recording. From this, density estimation is used to determine the individual novelty of newly acquired embedding vectors.
[0034] The density of the embedding vectors previously stored in the relevance determination unit can be determined at two possible time scales, a long time scale and a short time scale. In the long time scale, the density of all previously stored data is considered. This will prioritize scenes that are underrepresented in the recorded data. In the short time scale, the density estimate can only include the embedding vectors that have been recently stored and added to the database, or a higher weight can be given to the embedding vectors that have been recently observed. In this way, the recording of recently observed novel data is prioritized, while recording is stopped when driving in a monotonous environment that produces data with similar embedding vectors. This allows for the collection of diversified data during each individual recording session.
[0035] According to an embodiment of the first aspect, at least one relevance determination module performs the following steps:
[0036] a) retrieving a predetermined number of neural network embeddings, particularly wherein the neural network embeddings are derived from sensor data acquired in continuous time;
[0037] b) for each of the neural network embeddings, determining a nearest neighbor within the entire data set of the neural network embeddings retrieved by the relevance determination module;
[0038] c) determining, for each of the neural network embeddings, a distance to a nearest neighbor, in particular by applying a “k-nearest neighbor” density estimation method, and / or evaluating the density of the neural network embedding within the entire data set of neural network embeddings retrieved by the relevance determination module;
[0039] d) calculating an average distance and / or an average density for a time-coherent subset of a predetermined number of neural network embeddings, in particular wherein the average distance and / or the average density is calculated as a moving average of the determined distance and / or density measures for a smaller subset of the predetermined number of neural network embeddings;
[0040] e) evaluating subsets having an average distance greater than a distance threshold and / or an average density less than a density threshold, and considering the corresponding subsets to be relevant; and
[0041] f) Continue with step a).
[0042] The above scheme optimizes the diversity of the data selected and recorded. During the recording session, the computing unit continuously provides new vector embeddings or "signatures" to the relevance determination unit. After a predetermined number N (e.g., N=10000) of signatures are received by the relevance determination unit, the nearest neighbor within the entire signature data set already present in the relevance determination unit is determined for each of the signatures.
[0043] In the next step, a time-coherent subsample of K signatures is selected from these N signatures. Time coherence means that these K signatures are from consecutive times rather than randomly picked from the entire data set. This allows to assign a density estimate to each consecutive subsample of the vector embedding received from the computing unit. K can be a positive number greater than a minimum value. The specific value and the minimum number of K are not fixed, but flexible, and may, for example, depend on the recording requirements. The minimum number may, for example, depend on the minimum recording time required for a specific data type. The time-coherent subsample of the relevant data is achieved by determining the first signature to be retained and estimating the relevance of the signature given that the next (K-1) consecutive signatures are also considered relevant and recorded.
[0044] This is achieved, for example, by a sliding window approach, where for each possible starting position the average distance and / or average density of all K signatures is evaluated, and windows where the average distance is greater than a distance threshold or the average density is less than a density threshold are marked as relevant and selected for recording. This ensures that data of a given size, i.e. K signatures, is always recorded. It also ensures that K signatures are selected from N signatures each time. The specific values of the distance threshold and the density threshold are not fixed, but flexible, and may, for example, depend on the recording requirements. Based on the principal components calculated using the entire database, dimensionality reduction may be applied to the original signatures, for example via PCA. Dimensionality reduction may help to provide better density estimates and improve the quality of the selection process.
[0045] In a second aspect, the present disclosure relates to a computer-implemented method of operating a system comprising a data center and a plurality of data recording devices, the data recording devices being configured to perform the method of the first aspect or any one of its embodiments, the data center comprising a global database and a global relevance determination unit, wherein the method comprises the following steps:
[0046] The data center exchanges data with each of the plurality of data recording devices, in particular via an over-the-air (OTA) transmission capability;
[0047] the data center queries and retrieves sensor data and / or perception results and / or neural network embeddings from each of the plurality of data recording devices at regular time intervals, in particular in a continuous manner, and / or the data center queries and retrieves, at regular time intervals, in particular in a continuous manner, an updated set of predefined rules for determining whether the perception results and / or neural network embeddings are considered relevant or irrelevant;
[0048] The data center stores the data in a global database as global sensor data and / or global perception results and / or global neural network embedding;
[0049] The global relevance determination unit analyzes the global database, and the global relevance determination unit provides an estimate for the global perceptual density and / or the global embedding density;
[0050] The system determines whether the retrieved perception results and / or neural network embeddings of a specific data recording device are considered relevant or irrelevant based on the perception results and / or neural network embeddings of the specific data recording device, the global perception results and / or global neural network embeddings, and / or an updated predefined rule set.
[0051] Using multiple vehicles, for example, a fleet of vehicles equipped with data recording devices, obviously increases the speed of data collection. However, using multiple vehicles does not guarantee that data is collected and recorded efficiently, because much of the data collected by the vehicles may be very similar and redundant.
[0052] Therefore, synchronizing data recording between multiple vehicles can improve the efficiency of data collection, because the recording of redundant data can be omitted. This is particularly beneficial if the goal is to record as diverse data as possible, because it can be ensured that each data recording device is aware of all data that other data recording devices have recorded.
[0053] In order to synchronize data acquisition across multiple vehicles (such as a fleet of vehicles), the correlation determination unit of each of the data recording devices is connected to a data center. The data recording devices of the fleet of vehicles can exchange data with the data center over the air, for example using a mobile connection according to the 5G standard. The data center stores the received data as global sensor data and / or global perception results and / or global neural network embeddings in a global database. The global correlation determination unit analyzes the global database and provides a central solution, which includes an estimate of the global perception density and / or the global embedding density. The central solution can be cloud-based.
[0054] The central solution may include an index or PCA-matrix of all vector embeddings observed so far and a set of rules for updating the rule-based relevance determination. To compute the updates, the global relevance determination unit monitors a global database and continuously updates the global representation based on the data available there, i.e. global sensor data and / or global perception results and / or global neural network embeddings. For example, if a k-nearest neighbor density estimation is used to decide whether a particular embedding is relevant or not, the index tree of all observed embeddings and optionally the PCA decomposition matrix are regularly updated.
[0055] While driving, vehicles can use over-the-air (OTA) connections to transmit compressed versions of their data in real time. For example, a vehicle can use an OTA channel to transmit embedding vectors of all data recorded by the vehicle so far to a data center without uploading all the corresponding raw sensor data. The advantage of this is that the embedding vectors provide a highly compressed representation of the data content. Because the embedding vectors are ultimately used for relevance determination, this information is sufficient to synchronize all vehicles with respect to the global data acquisition of all vehicles within a fleet of vehicles.
[0056] A determination is then made whether a particular data segment acquired by a data recording device of a particular vehicle is considered relevant or irrelevant based on the perception results and / or neural network embedding of that particular data, and a central solution including global perception results and / or global embedding density provided by a data center.
[0057] Thus, the determination may be made locally by a relevance determination unit within the data recording device of the specific vehicle, or may be made by a global relevance determination unit in a data center, which then transmits its determination, in particular the determination that a particular data segment is relevant, to the data recording device of the specific vehicle.
[0058] According to an embodiment of the second aspect, the method includes the following additional steps: the relevance determination unit of the specific data recording device synchronizes the predefined rule set and / or the global perceptual density and / or the global embedding density with the global relevance determination unit; and the relevance determination unit of the specific data recording device, together with at least one relevance determination module according to the first aspect or any embodiment thereof, determines whether the perceptual results and / or the neural network embedding of the specific data recording device are considered to be relevant or irrelevant by using the updated predefined rule set and / or the global perceptual density and / or the global embedding density.
[0059] In this embodiment, the correlation determination is made locally by a correlation determination unit within the data recording device of a specific vehicle. The correlation determination unit can, for example, run as a docker container within the data recording device. Thus, no air communication is required during the determination phase. At regular intervals, for example, every 15, 30 or 60 minutes, the data recording device can query the data center for an updated correlation model that includes an updated set of rules and / or global perception density and / or global embedding density. The data recording device of the specific vehicle then obtains the latest docker container from the data center and uses it for all further correlation determinations. This allows the correlation model to be changed quickly while the vehicle is still on site and acquiring data. Whenever new data is incorporated into the global database, or if the search priority and corresponding rule set change, the docker container can be created in a fully automatic manner as part of the continuous integration / continuous deployment (CI / CD) pipeline.
[0060] The method is very flexible and can be easily adapted to any similar data, especially automotive data. Moreover, it does not rely on expensive computational power, thanks to the use of possible dimensionality reduction for the embedding space, optimized search algorithms, and over-the-air capabilities.
[0061] According to an embodiment of the second aspect, the method includes the following additional steps: a global correlation determination unit, together with at least one correlation determination module according to the first aspect or any embodiment thereof, determines whether the perception results and / or neural network embeddings of a specific data recording device are considered to be relevant or irrelevant by using an updated set of predefined rules and / or global perception density and / or global embedding density; and the global correlation determination unit transmits the correlation determination to the correlation determination unit of the specific data recording device.
[0062] In this embodiment, the relevance decision is made by a global relevance determination unit in the data center, which then transmits its decision, in particular the decision that a particular data segment is relevant, to the data recording device of the specific vehicle. The relevance decision is based on a continuously updated global database, which includes, for example, all relevant embedding vectors received from all vehicles in the fleet. Thus, similar to the vehicle-specific relevance determination unit of the previous embodiment, the global relevance determination unit can be run as a docker container within the data center.
[0063] According to an embodiment of the second aspect, the data center further comprises an upload unit, which queries and receives sensor data that is considered relevant and stored in a persistent storage area of each of a plurality of data recording devices, and transmits the data to a global database, in particular wherein the sensor data is queried after a predetermined time interval, in particular wherein the predetermined time interval is 12, 24, 48 or 72 hours.
[0064] When data acquisition ends or is interrupted (e.g. during nighttime braking), the recorded data can be uploaded and added to the global database using a dedicated uploader service and possibly an available high-bandwidth fixed Internet connection.
[0065] In a third aspect, the present disclosure relates to a data recording device for autonomous driving (AD) or advanced driver assistance system (ADAS) applications of a vehicle, the data recording device being configured to perform the method of the first aspect or any embodiment thereof.
[0066] The data recording device comprises a data logging unit, a computing unit, a storage unit and a correlation determination unit comprising at least one correlation determination module, wherein the storage unit comprises a circular buffer and a persistent storage area, the data logging unit and the computing unit are configured to retrieve sensor data, in particular acquired by sensors within a vehicle, the data logging unit is configured to store the sensor data in the circular buffer, the computing unit is configured to analyze the sensor data and use the sensor data to derive perception results and / or neural network embeddings from the retrieved sensor data, the correlation determination unit is configured to retrieve and store the perception results and / or neural network embeddings, the correlation determination unit is configured to determine whether the retrieved perception results and / or neural network embeddings are considered relevant or irrelevant, and in the case where the perception results and / or neural network embeddings corresponding to the sensor data are considered relevant by the correlation determination unit, the correlation determination unit is further configured to trigger the transmission of the sensor data from the circular buffer to the persistent storage area for persistent storage of the sensor data.
[0067] At least one relevance determination module may be configured to determine whether retrieved perceptual results and / or neural network embeddings are considered relevant or irrelevant by applying a predefined set of rules to the retrieved perceptual results.
[0068] The computing unit may be configured to reduce the dimensionality of the neural network embedding, in particular by principal component analysis (PCA), transmit the reduced neural network embedding to the correlation determination module, and the correlation determination unit may be configured to determine the correlation based on the reduced neural network embedding.
[0069] The data recording device may further comprise a display unit provided with a display and a touch panel, wherein the computing unit is configured for transmitting the perception result to the display unit, wherein the display unit is configured for displaying the perception result on the display, wherein the display unit is configured for identifying a manual user interaction on the touch panel in response to the perception result displayed on the display, and wherein the display unit is configured for transmitting information that the perception result must be considered relevant to the relevance determination unit when a manual user interaction is identified.
[0070] The computing unit may be configured for receiving a text-based user query, for encoding the retrieved text-based user query into a neural network embedding, and for transmitting the neural network embedding to a relevance determination unit, wherein at least one relevance determination module is configured for determining whether the neural network embedding derived from the sensor data is considered relevant or irrelevant based on a similarity measure between the neural network embedding derived from the sensor data and the encoded user query neural network embedding, in particular using cosine similarity.
[0071] At least one relevance determination module is configured to determine whether the retrieved perceptual result and / or the neural network embedding is considered relevant or irrelevant based on a comparison of the neural network embedding with neural network embeddings previously stored in the relevance determination unit.
[0072] At least one relevance determination module is configured to perform the following steps:
[0073] a) retrieving a predetermined number of neural network embeddings, particularly wherein the neural network embeddings are derived from sensor data acquired in continuous time;
[0074] b) for each of the neural network embeddings, determining a nearest neighbor within the entire data set of the neural network embeddings retrieved by the relevance determination module;
[0075] c) determining, for each of the neural network embeddings, a distance to a nearest neighbor, in particular by applying a “k-nearest neighbor” density estimation method, and / or evaluating the density of the neural network embedding within the entire data set of neural network embeddings retrieved by the relevance determination module;
[0076] d) calculating an average distance and / or an average density for a time-coherent subset of a predetermined number of neural network embeddings, in particular wherein the average distance and / or the average density is calculated as a moving average of the determined distance and / or density measures for a smaller subset of the predetermined number of neural network embeddings;
[0077] e) evaluating subsets having an average distance greater than a distance threshold and / or an average density less than a density threshold, and considering the corresponding subsets to be relevant; and
[0078] f) Continue with step a).
[0079] In a fourth aspect, the present disclosure relates to a system configured to perform the method of the second aspect or any embodiment thereof.
[0080] The system comprises a data center and a plurality of data recording devices, wherein the data center comprises a global database and a global correlation determination unit, wherein the data center is configured to exchange data with each of the plurality of data recording devices, in particular via an over-the-air (OTA) transmission capability, wherein the data center is configured to query and retrieve sensor data and / or perception results and / or neural network embeddings from each of the plurality of data recording devices at regular time intervals, in particular in a continuous manner, and / or query and retrieve updated predefined rule sets for determining perception results and / or neural network embeddings at regular time intervals, in particular in a continuous manner is considered relevant or irrelevant, wherein the data center is configured to store the data as global sensor data and / or global perception results and / or global neural network embeddings in a global database, wherein the global relevance determination unit is configured to analyze the global database and provide an estimate for the global perception density and / or the global embedding density, and wherein the system is configured to determine whether the retrieved perception results and / or neural network embeddings of a specific data recording device are considered relevant or irrelevant based on the perception results and / or neural network embeddings of the specific data recording device, the global perception results and / or the global neural network embeddings and / or an updated predefined rule set.
[0081] The relevance determination unit of a specific data recording device may be configured to synchronize the global perceptual density and / or the global embedding density with the global relevance determination unit, and together with at least one relevance determination module according to the first aspect or any embodiment thereof, determine whether the perceptual results and / or the neural network embedding of the specific data recording device are considered to be relevant or irrelevant by using an updated set of predefined rules and / or the global perceptual density and / or the global embedding density.
[0082] The global relevance determination unit may be configured to, together with at least one relevance determination module according to the first aspect or any embodiment thereof, determine whether the perception results and / or neural network embeddings of a specific data recording device are considered to be relevant or irrelevant by using an updated set of predefined rules and / or a global perception density and / or a global embedding density, and wherein the relevance determination is transmitted to the relevance determination unit of the specific data recording device.
[0083] The data center further comprises an upload unit configured for querying and receiving sensor data considered relevant and stored in a persistent storage area of each of the plurality of data recording devices, and transmitting the data to the global database, in particular wherein the sensor data is queried after a predetermined time interval, in particular wherein the predetermined time interval is 12, 24, 48 or 72 hours.
[0084] The present disclosure also relates to a use of the data recording device of the third aspect for obtaining a correlation-based data set for training a neural network.
[0085] The present disclosure further relates to a use of the system of the fourth aspect for obtaining a correlation-based data set for training a neural network. BRIEF DESCRIPTION OF THE DRAWINGS
[0086] Exemplary embodiments and functionality of the present disclosure are described herein in conjunction with the following drawings, which schematically illustrate:
[0087] Figure 1 is an embodiment of a data logging device for AD or ADAS applications in a vehicle, and
[0088] Figure 2 is an embodiment of a data recording system for AD or ADAS applications in a vehicle. DETAILED DESCRIPTION
[0089] Figure 1 An embodiment of a data recording device 10 for AD or ADAS applications of a vehicle is depicted. The data recording device 10 is attached to or is part of a vehicle not shown. The data recording device 10 comprises a data log unit 12, a calculation unit 14, a storage unit 16 and a correlation determination unit 18 comprising at least one correlation determination module 20. The data recording device 10 further comprises a display unit 22 provided with a display and a touch panel. The storage unit 16 comprises a circular buffer 24 and a persistent storage area 26.
[0090] During a data logging session, in-vehicle sensors 28 provide raw sensor data, such as camera images, LiDAR point clouds, or GPS measurements. The raw sensor data is transmitted to the data logging unit 12 (e.g., using a VIGEM logger). The data logging unit 12 continuously transmits the data to a circular buffer 24 within the storage unit 16. Newly acquired data continuously enters the circular buffer 24, overwriting the oldest data stored in the circular buffer 24 once the memory of the circular buffer 24 is full.
[0091] During the data acquisition phase, some of the acquired data in the circular buffer 24 may be considered relevant. If considered relevant, the particular data is moved to the persistent storage area 26 within the storage unit 16, or alternatively, is marked as belonging to the persistent storage area 26 for permanent storage. Thus, the data in the persistent storage area is prevented from being overwritten later by newer data entering the circular buffer.
[0092] In order to be able to decide which data to record, ie which data to keep in the persistent storage area 26 , based on relevance, the computing unit 14 continuously processes the incoming raw sensor data, providing perception results and / or neural network embeddings from the sensor data.
[0093] The computation unit 14 transmits its perception results and / or neural network embeddings to the relevance determination unit 18. The relevance determination unit 18 includes a relevance state module 30 and a plurality of independent relevance determination modules 20. Each relevance determination module 20 may consider the incoming data to be relevant or irrelevant. If one of the independent relevance determination modules 20 considers a particular incoming sensor data to be relevant, the relevance state module 30 changes the state of the sensor data from irrelevant to relevant. This triggers the transfer of the corresponding sensor data from the circular buffer 24 to the persistent storage area 26 and protects the sensor data from being overwritten at a subsequent time.
[0094] If at any time, at least one relevance determination module 20 deems particular sensor data to be relevant, then that data will be recorded and transferred from the circular buffer 24 to the persistent storage area 26. The data transferred to the persistent storage area 26 may also include data acquired during an offset time before the relevance determination module 20 made a relevance determination that the particular data was relevant. Data recording will continue as long as at least one relevance determination module 20 remains in the relevant state. Thus, tags and meta information may be attached to the corresponding data records.
[0095] If all relevance determination modules 20 transition to the irrelevant state, recording continues if at least one of the relevance determination modules 20 has requested a certain post-recording time span. If all relevance determination modules 20 transition to the irrelevant state and no relevance determination module 20 requests additional post-recording time, recording (i.e., data transfer from the circular buffer 24 to the persistent storage area 26 or setting a flag indicating that data belongs to the persistent storage area 26) stops.
[0096] The relevance determination may be triggered by a human observer using the display unit 22 to monitor the recording and collection of data. In the event that the human observer deems a particular situation to be relevant, a manual interaction with the touch panel may be performed. This causes the display unit 22 to forward information to the relevance determination unit 18 that the corresponding data or situation is considered relevant.
[0097] The data recording device 10 is configured to use supervised and / or unsupervised priors on the collected data to determine in an automated manner whether the data is considered relevant or irrelevant. This method of operating the data recording device 10 provides an efficient and cost-effective method for recording data with a large variety. The method is very flexible and easily adaptable to different applications.
[0098] According to the supervised data recording method, the relevance determination module 20 determines whether the perception results and / or neural network embeddings transmitted by the computing unit 14 are considered relevant or irrelevant by applying a predefined rule set to the retrieved perception results. The predefined rule set may include, for example, the number of objects of a certain category (such as motorcycles or pedestrians) detected, or a specific subject speed reached. The rule set may be provided to the relevance determination unit 18, for example, as python code. The relevance determination module 20 compares the predefined rule set with the perception results retrieved from the computing unit 14, and determines whether the corresponding data is relevant or irrelevant based on the result of the comparison.
[0099] According to another supervised data recording method, the computing unit 14 receives a text-based user query and encodes the retrieved text-based user query into a neural network embedding. The computing unit 14 then transmits the neural network embedding to the relevance determination unit 18, and the relevance determination module 20 determines the relevance of a particular sensor data segment based on a similarity metric using a cosine similarity between the neural network embedding of the sensor data and the encoded user query neural network embedding. Before each data recording session, a user-defined compilation of text-based queries can be provided, which are encoded into corresponding embedding vectors. The text-based query can be provided to the computing unit 14 as a query vector. Specific examples of such text queries are "car approaching a roundabout" or "motorcycle standing at a red traffic light". The query vector is then encoded into an embedding vector using a text encoder of the computing unit 14. The text-based user query can also be transmitted to the computing unit 14 by a user query device not shown, which can be a component of the data recording device 10 or a separate component.
[0100] According to the unsupervised data recording method, the data is selected based on the optimized data diversity. During the recording session, the computing unit 14 continuously provides new vector embeddings or signatures to the relevance determination unit 18. After a predetermined number N (e.g., N=10000) of signatures are received by the relevance determination unit 18, the nearest neighbor that already exists in the entire signature data set in the relevance determination unit 18 is determined for each of the signatures.
[0101] In the next step, a time-coherent subsample of K signatures is selected from these N signatures. Time coherence means that these K signatures are from consecutive times rather than randomly picked from the entire data set. This allows to assign a density estimate to each consecutive subsample of the vector embedding received from the computational unit. K can be a positive number greater than a minimum number. The specific value and the minimum number of K are not fixed, but flexible, and may, for example, depend on the recording requirements. The minimum number may, for example, depend on the minimum recording time required for a specific data type. The time-coherent subsample of the relevant data is achieved by determining the first signature to be retained and estimating the relevance of this signature given that the next (K-1) consecutive signatures are also considered relevant and recorded.
[0102] The average distance and / or average density of all K signatures is evaluated using a sliding window calculation for each possible starting position, and windows in which the average distance is greater than a distance threshold or the average density is less than a density threshold are marked as relevant and selected for recording. This ensures that data of a given size, i.e., K signatures, is always recorded. This also ensures that K signatures are selected from N signatures each time. The specific values and of the distance threshold and the density threshold are not fixed, but flexible, and may, for example, depend on the recording requirements. Based on the principal components calculated using the entire database, dimensionality reduction may be applied to the original signature, for example, via PCA. Dimensionality reduction may help provide better density estimates and improve the quality of the selection process.
[0103] Figure 2 An embodiment of a data recording system 40 for AD or ADAS applications in a vehicle is depicted. The system 40 includes a data center 42 and a plurality of Figure 1 Each data recording device 10 is associated with a vehicle in the vehicle fleet. For a clearer representation, Figure 2 Only one of the data recording devices 10 is shown in FIG.
[0104] For synchronizing the data acquisition, the relevance determination unit 18 of each of the data recording devices 10 is connected to a data center 42. The data recording devices 10 of the vehicle fleet exchange data with the data center 42 over the air, for example using a mobile connection according to the 5G standard. The data center 42 stores the received data as an updated rule set and / or global sensor data and / or global perception results and / or global neural network embedding in a global database 44. The global relevance determination unit 46 analyzes the global database 44 and provides a central solution, which includes an estimate of the global perception density and / or the global embedding density.
[0105] A determination is then made as to whether a particular data segment acquired by a data recording device of a particular vehicle is considered relevant or irrelevant based on the perception results and / or neural network embedding of that particular data, as well as a central solution including global perception results and / or global embedding density provided by data center 42 .
[0106] Thus, the determination may be made locally by the relevance determination unit 18 within the particular vehicle's data recording device 10 or may be made by the global relevance determination unit 46 of the data center 42 which then communicates its determination to the particular vehicle's data recording device 10 .
[0107] In the case where the relevance determination is made locally by the relevance determination unit 18 within the data recording device 10 of a particular vehicle, the relevance determination unit 18 runs as a docker container within the data recording device 10. At regular intervals, for example, every 15, 30 or 60 minutes, the data recording device 10 queries the data center 42 for an updated relevance model, which includes an updated rule set and / or global perception density and / or global embedding density. The data recording device 10 of the particular vehicle then obtains the latest docker container from the data center 42 and uses it for all further relevance determinations. This allows the relevance model to be changed quickly while the vehicle is still on site and acquiring data. Docker containers can be created in a fully automatic manner every time new data is included in the global database 44, or if the search priority and corresponding rule set change.
[0108] Reference numerals list
[0109] 10 Data recording equipment
[0110] 12 Data logging unit
[0111] 14 Computational Unit
[0112] 16 Storage Units
[0113] 18 Relevance determination unit
[0114] 20 Relevance determination module
[0115] 22 Display unit
[0116] 24 Circular Buffer
[0117] 26 Persistent Storage Area
[0118] 28 Sensors
[0119] 30 Correlation Status Module
[0120] 40 Data Recording System
[0121] 42 Data Center
[0122] 44 Global Database
[0123] 46 Global correlation determination unit
[0124] 48 Upload Unit
Claims
1. A method of operating a data recording device (10) for autonomous driving (AD) or advanced driver assistance system (ADAS) applications of a vehicle, the data recording device (10) comprising: A data log unit (12), a calculation unit (14), a storage unit (16), and a correlation determination unit (18) comprising at least one correlation determination module (20), wherein the storage unit comprises a circular buffer (24) and a persistent storage area (26), Wherein, the method comprises the following steps: The data log unit (12) and the computing unit (14) retrieve sensor data, in particular acquired by sensors (28) in the vehicle; The data logging unit (12) stores the sensor data in the circular buffer (24); The computing unit (14) analyzes the sensor data and uses the sensor data to derive perception results and / or neural network embeddings from the retrieved sensor data; The relevance determination unit (18) retrieves and stores the perception results and / or neural network embeddings; The relevance determination unit (18) determines whether the retrieved perceptual results and / or neural network embeddings are considered relevant or irrelevant; In case the perception results and / or neural network embeddings corresponding to the sensor data are considered relevant by the relevance determination unit (18), the relevance determination unit (18) triggers the transfer of the sensor data from the circular buffer (24) to the persistent storage area (26) for persistent storage of the sensor data.
2. The method according to claim 1, characterized in that The at least one relevance determination module (20) determines whether the retrieved sensory results and / or neural network embeddings are considered to be relevant or irrelevant by applying a predefined set of rules to the retrieved sensory results.
3. The method as claimed in claim 1, comprising the following additional steps: the computing unit (14) reduces the dimension of the neural network embedding, in particular by principal component analysis PCA; the computing unit (14) transmits the reduced neural network embedding to the correlation determination unit (18); and the correlation determination unit (18) determines the correlation based on the reduced neural network embedding.
4. The method according to any one of claims 1 to 3, characterized in that The data recording device (10) further comprises a display unit (22) provided with a display and a touch panel, and wherein the method comprises the following additional steps: the computing unit (14) transmits the perception result to the display unit (22); the display unit (22) displays the perception result on the display; the display unit recognizes a manual user interaction on the touch panel in response to the perception result displayed on the display; and the display unit (22) transmits information that the perception result must be considered relevant to the relevance determination unit (18) when the manual user interaction is recognized.
5. The method according to any one of claims 1 to 3, characterized in that The method comprises the following additional steps: the computing unit (14) receives a text-based user query; the computing unit (14) encodes the retrieved text-based user query into a neural network embedding; The computing unit (14) transmits the neural network embedding to the relevance determination unit (18); and the at least one relevance determination module (20) determines whether the neural network embedding derived from the sensor data is considered relevant or irrelevant based on a similarity measure between the neural network embedding derived from the sensor data and the encoded user query neural network embedding, in particular using cosine similarity.
6. The method according to any one of claims 1 to 3, characterized in that The at least one relevance determination module (20) determines whether the retrieved perceptual results and / or neural network embeddings are considered relevant or irrelevant based on a comparison of the neural network embeddings with neural network embeddings previously stored in the relevance determination unit (18).
7. The method according to claim 6, characterized in that The at least one relevance determination module (20) performs the following steps: a) retrieving a predetermined number of neural network embeddings, in particular wherein the neural network embeddings are derived from sensor data acquired in continuous time; b) for each of the neural network embeddings, determining a nearest neighbor within the entire data set of the neural network embeddings retrieved by the relevance determination module; c) determining, for each of said neural network embeddings, a distance to said nearest neighbor, in particular by applying a "k-nearest neighbor" density estimation method, and / or evaluating the density of said neural network embedding within said entire data set of neural network embeddings retrieved by said relevance determination module; d) calculating an average distance and / or an average density for a temporally coherent subset of the predetermined number of neural network embeddings, in particular wherein the average distance and / or average density is calculated as a moving average of the determined distance and / or density measures for a smaller subset of the predetermined number of neural network embeddings; e) evaluating subsets having an average distance greater than a distance threshold and / or an average density less than a density threshold, and considering the corresponding subsets to be relevant; f) Continue with step a).
8. A method of operating a system (40) comprising a data center (42) and a plurality of data recording devices (10), the data recording devices (10) being configured to perform the method according to at least one of claims 1 to 7, and the data center (42) comprising a global database (44) and a global relevance determination unit (46), wherein: The method comprises the following steps: The data center (42) exchanges data with each of the plurality of data recording devices (10), in particular via an over-the-air (OTA) transmission capability; The data center (42) queries and retrieves sensor data and / or perception results and / or neural network embeddings from each of the plurality of data recording devices (10) at regular time intervals, in particular in a continuous manner, and / or the data center (42) queries and retrieves, at regular time intervals, in particular in a continuous manner, an updated set of predefined rules for determining whether the perception results and / or neural network embeddings are considered relevant or irrelevant; The data center (42) stores the data as global sensor data and / or global perception results and / or global neural network embedding in the global database (44); The global relevance determination unit (46) analyses the global database (44), and the global relevance determination unit (46) provides an estimate for a global perceptual density and / or a global embedding density; The system (40) determines whether the perception result and / or neural network embedding retrieved from the specific data recording device (10) is considered relevant or irrelevant based on the perception result and / or neural network embedding of the specific data recording device (10), the global perception result and / or global neural network embedding and / or the updated predefined rule set.
9. The method according to claim 8, characterized in that The method comprises the following additional steps: the relevance determination unit (18) of the specific data recording device (10) synchronizes the predefined rule set and / or the global perceptual density and / or the global embedding density with the global relevance determination unit (46); and the relevance determination unit (18) of the specific data recording device (10), together with at least one relevance determination module (20) according to at least one of claims 2 and 5 to 7, determines whether the perceptual result and / or the neural network embedding of the specific data recording device (10) is considered to be relevant or irrelevant by using the updated predefined rule set and / or the global perceptual density and / or the global embedding density.
10. The method according to claim 8, characterized in that The method comprises the following additional steps: the global relevance determination unit (46) together with at least one relevance determination module according to at least one of claims 2 and 5 to 7, determines whether the perception results and / or neural network embeddings of the specific data recording device (10) are considered to be relevant or irrelevant by using the updated predefined rule set and / or the global perception density and / or the global embedding density, and the global relevance determination unit (46) transmits the relevance determination to the relevance determination unit (18) of the specific data recording device (10).
11. The method according to claim 8, characterized in that The data center (42) further comprises an uploading unit (48) which queries and receives sensor data considered relevant and stored in the persistent storage area (26) of each of the plurality of data recording devices (10), and transmits the data to the global database (46), in particular wherein the sensor data is queried after a predetermined time interval, in particular wherein the predetermined time interval is 12, 24, 48 or 72 hours.
12. A data recording device (10) configured for performing the method according to at least one of claims 1 to 7.
13. A system (40) configured to perform the method of at least one of claims 8 to 11.
14. Use of a data recording device (10) as claimed in claim 12 for obtaining a correlation-based data set for training a neural network.
15. Use of the system (40) as claimed in claim 13 for obtaining a correlation-based data set for training a neural network.