Data logging for testing and validation of advanced driver assistance systems

Abstract

Data logging for advanced driver assistance system testing and validation. A system for evaluating the progress of a data logging activity performed to collect sensor data recorded by a sensor data recorder installed on a vehicle is provided. The system includes a metadata generation device configured to process data acquired by a sensor module to generate metadata for the sensor data, the metadata including classifications of attributes of a driving environment in which the vehicle was located during acquisition of the sensor data into corresponding classes within a predetermined set of classes of the attributes, and to transmit the metadata to a remote metadata processing device configured to determine whether the metadata includes at least a predetermined number of classifications according to predetermined classes within the predetermined set of classes, and based on the determination, generate an indicator for use in the data logging activity.

Description

Technical Field

[0001] Example aspects herein relate generally to the field of advanced driver assistance systems (ADAS) testing and evaluation, and more particularly, to systems for evaluating the progress of data logging activities performed to collect sensor data used in, for example, testing and validating an ADAS of a vehicle. Background Art

[0002] Testing and validating ADAS performance is complicated by the varying road and traffic conditions under which the host vehicle's ADAS must operate reliably. ADAS performance can vary depending on characteristics of the vehicle's environment, such as the number of vehicles in the vicinity of the host vehicle, weather and lighting conditions, the type of background environment (such as urban or rural environments), and variations in traffic signs in different countries. In addition, large amounts of real-world sensor data are required to train the artificial intelligence (AI) algorithms used by the ADAS electronic control units (ECUs). Typically, to create such large datasets, data recording campaigns known as mother of all road trips (MOARTs) are performed using more than one recording vehicle. To account for different road conditions, such data collection typically follows strictly defined requirements that specify the number of kilometers that should be recorded under specific road conditions.

[0003] An alternative approach to data collection for ADAS validation is to perform event-based data logging, where, instead of continuously logging data, only situations of interest are logged, such as when the ADAS makes a mistake and the driver needs to take over, for example.

[0004] The recorded data (which is typically video footage, but may also or alternatively include other types of data, such as radar data, lidar data, ultrasonic data, data from various vehicle sensors, time data, and / or GPS data) is typically stored in a data log, which is typically divided into smaller segments (e.g., 1GB to 2GB in size). At the end of the recording session, the data from all vehicles is copied to a temporary data storage device and then transported to a data storage center. Due to the size of the recorded data, it is normally not possible to stream the data via an internet connection or the like.

[0005] Current methods of collecting data for validating and testing ADAS ECUs under development have many shortcomings.

[0006] First, meeting the stringent requirements for various real-world road scenarios often requires recording hundreds of thousands of kilometers of driving. Furthermore, the recorded data cannot be verified online in real time; instead, it must be stored on-board storage and only later transferred to a data center for processing. The time between data recording and its availability for analysis is typically measured in weeks.

[0007] Secondly, recording activity can require a large amount of storage space. However, in practice, only a portion of the recorded data is used for ADAS testing and validation. In particular, a large amount of redundant video data may be recorded (for example, when the host vehicle is driven on a highway at night or in heavy traffic, where there is little or no overtaking and other vehicles near the host vehicle maintain their relative positions).

[0008] Third, the route plans of the fleet used for the recording campaign are usually prepared in advance. This means that missing and redundant data are only discovered when the recording campaign is completed and the recorded data is analyzed in the data center.

[0009] Fourth, the type of event-based recording described above requires that the ADAS ECU be in a mature state to enable it to determine which situations are worth recording, which is often not the case for early ADAS projects where the ECU is under development.

[0010] Furthermore, errors in the installation of data recording equipment (mixed signals, missing signals, crashed sensors, miscalibration, etc.) can only be detected when the recorded data is analyzed at a data center (usually after a delay of several weeks).

[0011] Finally, only a small subset of the recorded data is selected as the ground truth data for training the artificial intelligence algorithms (i.e., neural networks) of the ADAS. Labeling the ground truth data is a very time-consuming activity. For this reason, the best choice of data for forming the ground truth dataset involves selecting a minimal but sufficient amount of data. Summary of the Invention

[0012] In light of the foregoing, according to a first aspect of this disclosure, the inventors have devised a system for evaluating the progress of a data logging activity. The system performs data logging while a vehicle is being driven to collect sensor data recorded by a sensor data recorder installed in the vehicle. The sensor data is used in testing and validating an advanced driver assistance system (ADAS) of the vehicle. The ADAS is configured to provide driving assistance by processing sensor data acquired by sensor modules installed in the vehicle while the vehicle is being driven. The system includes a metadata generation device and a remote metadata processing device. The metadata generating device is configured to provide metadata when it is installed on a vehicle and used for data recording activities, and the metadata generating device includes: a data processing device, which is configured to process data acquired by the sensor module to generate metadata for the sensor data, the metadata including a classification of at least one of the following items: classifying attributes of the driving environment in which the vehicle is located during the acquisition of the sensor data into corresponding classes in a predetermined set of classes of attributes; and classifying driving scenarios involving the vehicle that occur during the acquisition of the sensor data into corresponding classes in a predetermined set of driving scenario classes associated with different driving scenarios, wherein each driving scenario is associated with a corresponding driving maneuver performed by the vehicle or a driving maneuver performed by a second vehicle relative to the vehicle; and a communication device, which is configured to send the metadata to a remote metadata processing device. The remote metadata processing device includes: a data storage unit configured to store metadata transmitted by a communication device; and an indicator generator module configured to determine whether the received metadata includes at least a predetermined number of classifications according to at least one of the following classes: if the metadata includes classifications of an attribute, a predetermined class from a predetermined set of classes of the attribute; and if the metadata includes classifications of a driving scene, a predetermined driving scene class from a predetermined set of driving scene classes associated with different driving scenes. The indicator generator module is further configured to generate an indicator for use in the data recording activity based on the determination.

[0013] Furthermore, according to a second aspect of the present invention, a metadata generation device is provided for use in a system for evaluating the progress of a data recording activity, the data recording activity being performed while a vehicle is being driven to collect sensor data recorded by a sensor data recorder mounted on the vehicle, the system comprising a remote metadata processing device. The sensor data is used in an advanced driver assistance system (ADAS) of a test vehicle, wherein the ADAS is configured to provide driving assistance by processing sensor data acquired by sensor modules mounted on the vehicle while the vehicle is being driven. The metadata generation device is configured to generate metadata when mounted on the vehicle and used for the data recording activity, and includes a data processing device configured to process the data acquired by the sensor modules to generate metadata for the sensor data, the metadata comprising at least one of the following: classifying an attribute of a driving environment in which the vehicle was located during the acquisition of the sensor data into a corresponding class from a predetermined set of classes of the attribute; and classifying driving scenarios involving the vehicle during the acquisition of the sensor data into corresponding classes from a predetermined set of driving scenario classes associated with different driving scenarios, wherein each driving scenario is associated with a corresponding driving maneuver performed by the vehicle or a driving maneuver performed by a second vehicle relative to the vehicle. The metadata generating device also includes: a communication device that is operable to send metadata to a remote metadata processing device and receive an indicator from the remote metadata processing device for use in a data recording activity, the indicator indicating a determination result by the remote metadata processing device as to whether the metadata received by the remote metadata processing device includes at least a predetermined number of classifications according to at least one of the following classes: in the case where the metadata includes classification of an attribute, a predetermined class in a predetermined set of classes of the attribute; and in the case where the metadata includes classification of a driving scene, a predetermined driving scene class in a predetermined set of driving scene classes associated with different driving scenes.

[0014] According to a third aspect of the present disclosure, a metadata processing device is provided for use in a system for evaluating the progress of a data recording activity performed while a vehicle is being driven to collect sensor data recorded by a sensor data recorder installed on the vehicle. The sensor data is used in an advanced driver assistance system (ADAS) of a test vehicle, which is configured to provide driving assistance by processing sensor data acquired by sensor modules installed on the vehicle while the vehicle is being driven. The metadata processing device includes a data storage unit operable to store metadata received from a metadata generation device, the metadata being based on the data acquired by the sensor modules and comprising at least one of the following: classifying an attribute of a driving environment in which the vehicle was located during acquisition of the sensor data by the sensor modules into a corresponding class within a predetermined set of classes of the attribute; and classifying driving scenarios involving the vehicle that occurred during acquisition of the sensor data into corresponding classes within a predetermined set of driving scenario classes associated with different driving scenarios, wherein each driving scenario is associated with a corresponding driving maneuver performed by the vehicle or a driving maneuver performed by a second vehicle relative to the vehicle. The metadata processing device further includes an indicator generator module configured to determine whether the received metadata includes at least a predetermined number of classifications according to at least one of the following classes: if the metadata includes classifications of an attribute, a predetermined class from a predetermined set of classes of the attribute; and if the metadata includes classifications of a driving scene, a predetermined driving scene class from a predetermined set of driving scene classes associated with different driving scenes. The indicator generator module is further configured to generate an indicator for use in the data recording activity based on the determination.

[0015] In an example embodiment of any of the above aspects, the sensor module may include: a first sensor configured to acquire first sensor data while the vehicle is being driven; and a second sensor configured to acquire second sensor data while the vehicle is being driven, the second sensor being different from the first sensor. Furthermore, the ADAS may be configured to provide driving assistance based on the first sensor data, and the metadata generation device may be configured to generate metadata by processing the second sensor data. Each of the first and second sensors may include a camera, a radar sensor, or a lidar sensor.

[0016] In another example embodiment of any of the above aspects, the sensor module may include a single sensor configured to acquire sensor data, and the ADAS and metadata generation device may be configured to process the sensor data acquired by the single sensor. Furthermore, the single sensor may include a camera sensor, a radar sensor, or a lidar sensor.

[0017] In any example embodiment of the system of the first or third aspect or its example embodiments set forth above, the sensor data may include a plurality of sensor data measurements, the plurality of sensor data measurements being acquired at different times during the data recording activity, and the metadata may include a plurality of metadata data groups, the data processing device being configured to generate respective metadata data groups for respective sensor data measurements from the plurality of sensor data measurements and to associate the metadata data groups with the respective sensor data measurements, wherein the respective metadata data groups are generated by processing data acquired by the sensor module when the sensor data measurements are acquired. Furthermore, the respective metadata data groups may include: if the metadata includes classifications of attributes, classifying the attributes of the driving environment in which the vehicle was located during the acquisition of the sensor data measurements into respective classes within a predetermined set of classes for the attributes; and if the metadata includes classifications of driving scenarios, classifying the driving scenarios involving the vehicle during the acquisition of the sensor data measurements into respective classes within a predetermined set of driving scenario classes associated with different driving scenarios. The indicator generator module may be configured to determine whether the stored metadata includes at least a predetermined number of categories by determining whether a predetermined number of metadata data packets have corresponding categories that satisfy a predetermined condition of at least one of the following: in the case where the metadata includes categories of attributes, attributes of the driving environment; and in the case where the metadata includes categories of driving scenes, driving scenes. In response to determining that the predetermined number of metadata data packets have corresponding categories that satisfy the predetermined condition, the indicator generator module may be configured to generate a first instruction as an indicator instructing the sensor data recorder to stop recording sensor data, the remote metadata processing device may be configured to send the first instruction to the vehicle, and the sensor data recorder may be configured to respond to the first instruction by stopping recording sensor data.

[0018] Furthermore, the indicator generator module may be configured to determine whether the metadata data packet stored in the data storage unit satisfies a predetermined condition of at least one of the following: an attribute in the case where the metadata includes a classification of the attribute; and a driving scene in the case where the metadata includes a classification of the driving scene, and in response to determining that the metadata data packet does not meet the predetermined condition, generate a second instruction as an indicator instructing the sensor data recorder to delete the stored sensor data measurement results associated with the metadata data packet. The remote metadata processing device may be configured to send the second instruction to the vehicle, and the sensor data recorder may be configured to respond to the second instruction by deleting the stored sensor data measurement results associated with the metadata data packet.

[0019] The sensor module may also be configured to acquire vehicle status data including measurement results of one or more of the following items during the acquisition of sensor data by the sensor module: the speed of the vehicle, the yaw rate of the vehicle, the throttle setting of the vehicle, and the position information of the vehicle. The data processing device may be configured to generate metadata (M) in a manner that also includes the vehicle status data. The remote metadata processing device may also include an anomaly detection module, which is configured to use one or more predetermined criteria to search for anomalies in the vehicle status data, and the indicator generator module may also be configured to generate a second indicator for use in the data recording activity, which second indicator indicates that an anomaly has been detected in the vehicle status data.

[0020] The remote metadata processing device may further include a dataset selection module configured to select a subset of sensor data measurement results from the plurality of sensor data measurement results based on the metadata and predetermined requirements, wherein the predetermined requirements specify at least one of the following items: in the case where the metadata includes a classification of attributes, a required number of sensor data measurement results having corresponding metadata data groups classified into predetermined attribute classes; and in the case where the metadata includes a classification of driving scenes, a required number of sensor data measurement results having corresponding metadata data groups classified into predetermined classes of driving scenes.

[0021] The dataset selection module can be configured to select a subset of sensor data measurements based on the metadata and the predetermined requirements by solving a system of linear equations subject to mathematical constraints defined according to the predetermined requirements using integer programming, wherein each linear equation in the system of linear equations is formed based on a classification included in a corresponding metadata data grouping set of the plurality of corresponding metadata data groups. Where the metadata includes a classification of an attribute, at least one variable of each linear equation corresponds to a predetermined class of the attribute, and a coefficient of the at least one variable of each linear equation corresponds to the number of classes in the predetermined class of the attribute indicated by the metadata data grouping set. Where the metadata includes a classification of a driving scene, at least one variable of each linear equation corresponds to a predetermined driving scene class, and a coefficient of the at least one variable of each linear equation corresponds to the number of classes in the predetermined driving scene class indicated by the metadata data grouping set.

[0022] The attributes of the vehicle's driving environment may include at least one of the following: a type of one or more objects in the vehicle's driving environment, wherein the data processing device is configured to process data acquired by the sensor module to detect one or more objects in the vehicle's driving environment and classify the detected one or more objects according to the object type; a number of objects of predetermined one or more types in the vehicle's driving environment, wherein the data processing device is configured to process data acquired by the sensor module to detect one or more objects of predetermined one or more types and classify the detected one or more objects according to the number of the detected one or more objects; an environment type of the vehicle's driving environment, wherein the data processing device is configured to acquire information about the vehicle's driving environment and classify the driving environment according to the environment type; a time of day of the driving environment, wherein the data processing device is configured to acquire the vehicle's local time and classify the driving environment according to the local time; and a weather type of the vehicle's driving environment, wherein the data processing device is configured to acquire information about weather conditions of the vehicle's driving environment and classify the driving environment according to the weather type using the acquired information about the weather conditions.

[0023] The set of predetermined driving scenario classes associated with different driving scenarios includes at least one of the following: an overtaking class associated with an overtaking maneuver performed by the vehicle; an overtaken class associated with a vehicle being overtaken by a second vehicle; an emergency braking class associated with emergency braking performed by the vehicle; a merging class associated with a second vehicle on the left or right side of the vehicle moving into the same lane as the vehicle; a yaw rate-related maneuver class associated with a yaw rate of the vehicle to achieve a predetermined condition; and a speed-related maneuver class associated with a speed of the vehicle to achieve a predetermined condition.

[0024] The indicator generator module may also be configured to display the indicator on the display. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Example embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which, unless otherwise indicated, the same reference numerals appearing in different figures may indicate identical or functionally similar elements.

[0026] Figure 1 is a schematic illustration of a system for evaluating the progress of a data logging activity performed to collect sensor data for use in testing and validating an ADAS of a vehicle according to a first example embodiment.

[0027] Figure 2is a schematic illustration of programmable signal processing hardware that can be configured to perform the functions of the modules of the data processing apparatus or remote metadata processing apparatus of the first example embodiment.

[0028] Figure 3 This example shows that Figure 1 Examples of attributes of the driving environment processed by the metadata generating device, and examples of predetermined classes of the respective attributes.

[0029] Figure 4 An example of a set of predetermined driving scene classes associated with different driving scenes that may be used to classify driving scenes to generate metadata is illustrated.

[0030] Figure 5 This example shows that Figure 1 Frames of images processed by a metadata generating device with annotations added to illustrate examples of the classification described herein.

[0031] Figure 6 is an illustrative graph of example distribution data in the form of a histogram generated by the distribution data generator module of the remote metadata processing device of the first example embodiment.

[0032] Figure 7 is a method for evaluating the progress of a data logging activity according to a second example embodiment Figure 1 Schematic illustration of a variation of the shown system. DETAILED DESCRIPTION

[0033] Figure 1 is a schematic illustration of a system 1 for evaluating the progress of a data logging activity according to a first example embodiment. Figure 1 As shown, a vehicle 5 includes an advanced driver assistance system (ADAS) 10, a sensor module 20 mounted on the vehicle 5 and including a first sensor 22 and a second sensor 24, a sensor data recorder 30, and a metadata generation device 40. The metadata generation device 40 includes a data processing device 42 and a communication device 44. The ADAS 10 is configured to provide driving assistance by processing sensor data acquired by the sensor module 20 while the vehicle 5 is being driven.

[0034] Figure 1Also illustrated is a remote metadata processing device 60, which is remote from the vehicle 5 and wirelessly communicates with the vehicle 5. The metadata processing device 60 includes a data storage unit 61 configured to store metadata M received from the metadata generation device 40, and an indicator generator module 62. As in the present embodiment, the remote metadata processing device 60 may also include a distribution generator module 64 and an anomaly detection module 66. All component modules of the metadata processing device 60 are communicatively coupled to enable communication with one another.

[0035] While the vehicle 5 is being driven, data logging activities are performed to collect sensor data recorded by the sensor data recorder 30 installed on the vehicle 5. The recorded sensor data can be used to test and verify the ADAS 10 of the vehicle 5. It should be understood that the system according to the embodiment includes the sensor data recorder 30 as described herein. Figure 1 The data processing device 42 and the remote metadata processing device 60 described herein may further include Figure 1 One or more of the other components described.

[0036] Figure 2is a schematic illustration of programmable signal processing hardware 200 that can be configured to perform the operations of one or more of the modules 61 to 66 of the data processing device 42, ADAS 10, or remote metadata processing device 60 of this example embodiment. The programmable signal processing device 200 includes a communication interface (I / F) 210 for receiving input data to be processed and outputting data generated by the processing. The signal processing device 200 also includes: a processor (e.g., a central processing unit, CPU) 220 for processing the input data, a working memory 230 (e.g., a random access memory), and an instruction storage unit 240 storing a computer program 245, the computer program 245 including computer-readable instructions that, when executed by the processor 220, cause the processor 220 to perform the functions of one of the aforementioned components of the system 1 described herein. The working memory 230 stores information used by the processor 220 during execution of the computer program 245. The instruction storage unit 240 may include a ROM preloaded with computer-readable instructions (for example, in the form of an electrically erasable programmable read-only memory (EEPROM) or flash memory). Alternatively, the instruction storage unit 240 may include a RAM or similar type of memory, and the computer-readable instructions of the computer program 245 may be input to the instruction storage unit from a computer program product (such as a non-transitory computer-readable storage medium 250 in the form of a CD-ROM, DVD-ROM, etc., or a computer-readable signal 260 carrying computer-readable instructions). In any case, when the computer program 245 is executed by the processor 220, the processor 220 performs the processing performed by one or more of the modules 61 to 66 of the data processing device 42, ADAS 10, or remote metadata processing device 60 of this example embodiment. However, it should be noted that each of these components of the system 1 may alternatively be implemented in non-programmable hardware (such as an application-specific integrated circuit (ASIC)).

[0037] Refer again Figure 1 , the first sensor 22 is configured to acquire first sensor data S1 by sensing the external (driving) environment of the vehicle 5 while the vehicle 5 is being driven. In this example embodiment, the first sensor 22 and the second sensor 24 may each be a camera sensor that senses visible light emitted / reflected by objects in the vehicle's environment. Furthermore, the second sensor 24 is configured to acquire second sensor data S2 while the vehicle 5 is being driven, the second sensor data indicating the environment of the vehicle 5. As in this embodiment, the second sensor 24 may be configured to acquire the second sensor data S2 while the first sensor 22 is acquiring the first sensor data S1, such that these camera sensors operate simultaneously.

[0038] Thus, in this embodiment, the first sensor data S1 and the second sensor data S2 respectively include frames of video clips obtained by the sensors 22 and 24 recording the driving environment of the vehicle 5 while the vehicle is being driven during the data recording activity.

[0039] However, it should be noted that the first sensor 22 and the second sensor 24 are not limited to both being in the form of camera sensors and may take other (possibly different) forms. For example, each of the first sensor 22 and the second sensor 24 may alternatively include a radio detection and ranging (radar) sensor (which is configured to acquire radar data as sensor data) or a light detection and ranging (lidar) sensor (which is configured to acquire lidar data as sensor data). Thus, for example, the first sensor 22 may be one of a camera sensor, a radar sensor, and a lidar sensor, and the second sensor 24 may be one of a camera sensor, a radar sensor, and a lidar sensor. In this embodiment, the fields of view of both sensors 22 and 24 are oriented in a generally similar manner relative to the vehicle 5 so as to record video of the road and / or at least some of the environment external to the vehicle 5.

[0040] exist Figure 1 , the ADAS 10 is arranged to provide driving assistance to the driver of the vehicle 5 based on first sensor data S1, which includes video frames recorded by the first (camera) sensor 22. The ADAS 10 may, for example, include one or more driving assistance functions under development, such as adaptive cruise control, emergency brake assist, forward collision warning, lane departure warning, and lateral control and lane change assist.

[0041] Moreover, if Figure 1 As illustrated, during the data recording activity, while the vehicle 5 is being driven, the sensor data recorder 30 records the sensor data S1 captured by the first sensor 22 for use in testing and validating the ADAS 10. Furthermore, while the first sensor 22 is capturing the first sensor data S1, the output O of the ADAS 10 (e.g., a decision made by the ADAS 10) may also be recorded by the sensor data recorder 30. Furthermore, the ADAS output O may be synchronized with the captured first sensor data S1 in time to allow the ADAS output O to be validated against the first sensor data S1. It should be noted that during the early stages of ADAS development, the ADAS 10 may not be fully functional. That is, although the first sensor 22 is capable of recording the sensor data S1, the ADAS 10 may not be able to perform any detection or classification on the recorded sensor data S1.

[0042] exist Figure 1In this example embodiment, the metadata generating device 40 is configured to record metadata M when installed on the vehicle 5 and used for data recording activities. In this example embodiment, the data processing device 42 of the metadata generating device 40 is configured to process the second sensor data S2 acquired by the second sensor 24 to generate metadata M for the first sensor data S1. In this embodiment, the metadata M for the first sensor data S1 may be generated approximately simultaneously with the acquisition of the second sensor data S2 by the data processing device 42 to generate the metadata M. The metadata M may thus include a classification of attributes of the driving environment in which the vehicle 5 was located during the acquisition of the first sensor data S1 into corresponding classes within a predetermined set of classes of attributes. Alternatively or additionally, the metadata M may include a classification of driving scenarios involving the vehicle 5 that occurred during the acquisition of the sensor data S1 into corresponding classes within a predetermined set of driving scenario classes associated with different driving scenarios. Each driving scenario includes a corresponding driving maneuver performed by the vehicle 5 or a driving maneuver performed by a second vehicle relative to the vehicle 5.

[0043] For example, in this example embodiment, the second sensor data S2 may include a plurality of second sensor data measurements, each of which is acquired at different times during the data recording activity while the vehicle is being driven. In this example, the plurality of second sensor data measurements may be a plurality of images acquired at different times by the second sensor 24 (referred to herein as a plurality of image acquisitions). The metadata generation device 40 may be configured to process the image data of each of these image acquisitions to generate a corresponding metadata data grouping, the corresponding metadata data grouping including, if the metadata M includes a classification of an attribute, a classification of an attribute of the vehicle environment into a class from the plurality of predetermined classes of the attribute. The metadata generation device 40 may also be configured to process the image data to generate metadata data grouping including, if the metadata M includes a classification of a driving scene, a classification of driving scenes involving the vehicle 5 that occurred during the acquisition of the sensor data measurements into a class from a predetermined set of driving scene classes associated with different driving scenes. In embodiments where the metadata M includes both classifications of attributes of the driving environment and classifications of driving scenes, the classifications may be included in the same metadata data group, but may alternatively be included in separate metadata data groups.

[0044] In embodiments where the metadata includes classifications of attributes of the driving environment, the image data for each image acquisition may be a single frame of a video clip captured by the second sensor 24, and the data processing device 42 may be configured to process each of the frames to generate corresponding metadata data packets. However, it should be understood that each of the image acquisitions performed by the second sensor 24 need not correspond to a single video frame, but may instead include multiple frames. Additionally, the second sensor 24 may not acquire image data (more generally, second sensor data) continuously, but may instead acquire second sensor data measurements only at predetermined intervals or based on the detection of certain events, for example.

[0045] In an embodiment where the metadata includes a classification of driving scenes involving the vehicle 5, the data processing device 42 can be configured to process image data comprising multiple video frames so as to generate metadata data groupings of the multiple video frames, which metadata data groupings include the classification of driving scenes into driving scene classes from a plurality of predetermined driving scene classes.

[0046] Furthermore, in this example embodiment, the first sensor data S1 may include a plurality of image acquisitions taken by the first sensor 22 at different times during the data recording activity while the vehicle 5 was being driven, wherein each image acquisition comprises one or more frames of a video clip captured by the first sensor 22. Each metadata data packet generated by the metadata generation device 40 is associated with a corresponding one of the image acquisitions taken by the first sensor 22, the association being based on a comparison of timestamps, video frame identifiers, and the like, which allows the metadata data packets to be temporally correlated with the image acquisitions taken by the first sensor 22 such that, where the metadata M includes classifications of attributes, each metadata data packet includes a corresponding classification of an attribute of the driving environment in which the vehicle 5 was located during the associated image acquisition taken by the first sensor 22. Alternatively or additionally, where the metadata M includes classifications of driving scenes, each metadata data packet may include a corresponding classification of a driving scene involving the vehicle 5 that occurred during the acquisition of the associated image acquisition taken by the first sensor 22.

[0047] Figure 3 This example shows that Figure 1 The metadata generating device 40 processes an example of the attributes 300 of the driving environment of the vehicle 5 to generate the metadata M. Figure 3 Some examples of predetermined classes 310 for various attributes 300 are also illustrated.

[0048] like Figure 3As shown in (a) of FIG. 5 , in this embodiment, driving environment attribute 300 includes the type of one or more objects in the driving environment of vehicle 5 . Data processing device 42 is configured to process data S2 acquired by second sensor 24 to detect one or more objects in the driving environment of vehicle 5 and classify the detected one or more objects according to object type. Specifically, classification of attribute 300 in metadata M may include classifying the type of one or more road objects in the driving environment of vehicle 5 during acquisition of first sensor data S1 into corresponding classes from a plurality of predetermined road object classes. Predetermined road object classes may include one or more of a vehicle class, a pedestrian class, a bicycle class, and a road traffic sign class. However, additional or alternative road object classes may be included in the predetermined road object classes.

[0049] Additionally or alternatively, the attribute 300 of the driving environment may include: a number of objects of one or more predetermined types in the driving environment of the vehicle 5, wherein the data processing device 42 is configured to process the data S2 acquired by the second sensor 24 to detect one or more objects of the predetermined one or more types, and classify the detected one or more objects according to the number of the detected one or more objects, such as Figure 3 As illustrated in (b) of FIG. In this example, the data processing device 42 may process each second sensor data measurement result (included in the second sensor data S2) by first detecting objects in the driving environment of the vehicle 5 based on the second sensor data measurement result. The data processing device 42 may also classify each detected object as belonging to an object class from among the plurality of predetermined object classes. The data processing device 42 may then determine the number of objects belonging to each of the plurality of predetermined object classes.

[0050] In this example, where the second sensor 24 comprises a camera sensor and the plurality of second sensor data measurements comprise a plurality of frames of recorded video of the driving environment of the vehicle 5, any one of a plurality of object recognition algorithms may be used to generate a metadata data packet (for each first sensor data measurement) comprising the type of object in the frame and the number of objects of each type in the frame. By way of example, the object recognition algorithm is one of a You Only Look Once (YOLO) network, a Region-based Convolutional Neural Network (R-CNN), or a Single Shot Detector (SSD) algorithm. However, another object detection algorithm known to those skilled in the art may alternatively be used. However, it should be noted that the data processing device 42 need not implement all perception functions similar to those of an ADAS, and may alternatively use analysis functions and information from a vehicle state machine.

[0051] While second sensor 24 is a camera sensor in this example, it should be noted that when second sensor data S2 is radar data acquired by a radar sensor or lidar data acquired by a lidar sensor, metadata M including a classification of road object types and the number of objects of each road object type can also be generated based on the second sensor data S2. For example, in embodiments where second sensor 24 is a radar sensor and second sensor data S2 is radar data, data processing device 42 can perform object tracking and object classification based on radar data acquired over an extended period (i.e., over a continuous observation period), for example, using a neural network, to identify and distinguish different types of road objects in the driving environment of vehicle 5. Furthermore, radar data can also be used to detect road obstacles and concrete curbs. For example, the radar data can include multiple radar data measurement sets, and metadata data groups can be generated for each radar data measurement set, each metadata data group including a classification of the object type and / or a classification of the number of objects of one or more predetermined types detected in the radar data measurement set. Although the second sensor 24 can generally extract more information about the driving environment when it takes the form of a camera sensor or a lidar sensor, a radar sensor can provide more robust performance under certain conditions, such as under varying ambient light conditions or in adverse weather conditions. Furthermore, a radar system can provide accurate measurements of the distance of objects from the ego vehicle.

[0052] In embodiments such as the present embodiment (where the second sensor 24 is a camera sensor), the attributes 300 of the driving environment may additionally or alternatively include background scene feature types, such as Figure 3More specifically, the classification of the attribute 300 of the driving environment of the vehicle 5 may further include: classifying the background scene features of the driving environment of the vehicle 5 detected by the camera sensor into one of a plurality of predetermined classes associated with different background scene feature types. The classification of the background scene feature types may, for example, include a corresponding determination of a confidence score for each of the plurality of background scene feature type classes. Each confidence score indicates the likelihood that the environment of the vehicle 5 includes a background scene feature belonging to a background scene feature class. In addition, the classification of the background scene features may include classifying the background scene features as belonging to one of the background scene feature classes.

[0053] By way of example, metadata generation device 40 may apply an image classification algorithm to at least a portion of a video frame captured by a camera sensor to determine, as metadata, confidence scores regarding the presence of various background feature types in the frame (such as trees, buildings, tunnels, and / or bridges). The confidence scores determined for each of a plurality of background scene feature classes may also be used to classify the image into one of a plurality of predetermined background scene feature classes, with the classification output being included in the metadata. The plurality of predetermined background feature classes may, for example, include: at least one or more of a building class indicating that the driving environment of vehicle 5 includes one or more buildings; a natural greenery class indicating that the driving environment of vehicle 5 includes greenery, such as one or more trees; a bridge class indicating that the driving environment of vehicle 5 includes one or more bridges; and a tunnel class indicating that the vehicle's environment includes one or more tunnels. However, other background feature classes may be included in the plurality of predetermined background feature classes. The image classification algorithm used to classify the image may, for example, be a convolutional neural network, such as a GoogleNet network. However, other forms of pre-trained neural networks may also be used. In some cases, an analysis function may be used in place of the image classification algorithm. For example, information about the time, date, and GPS position of the vehicle 5 may be used to calculate the height of the sun above the horizon.

[0054] The data processing device 42 may also be configured to monitor the background feature classification as described above over a sufficiently long period of time, or to obtain information about the driving environment of the vehicle 5 in other ways, and on this basis, classify the driving environment according to the environment type, such as Figure 3 The environment type may include, for example, an urban environment, a suburban environment, a rural environment, a coastal environment or an underground environment, such as Figure 3For example, a repeated classification indicating the presence of one or more buildings in the vehicle environment may cause the data processing device 42 to further classify the driving environment as belonging to the "urban environment" category of the environment type. Similarly, a repeated classification indicating the presence of one or more trees in the vehicle environment may cause the data processing device 42 to further classify the driving environment as belonging to the "rural environment" category of the environment type.

[0055] Additionally or alternatively, the attributes 300 of the driving environment may include the time of day of the driving environment, wherein the data processing device 42 is configured to classify the environment according to the time of day. For example, the metadata data generating device 40 may process each video frame to classify the image in the video frame as belonging to one of a plurality of predetermined classes associated with the time of day, such as a daytime class and a nighttime class, for example. Figure 3 Alternatively or additionally, the data processing device 42 may be arranged to obtain data indicative of the local time of the vehicle 5 from a clock in the sensor module 20 and to classify the environment according to the local time.

[0056] Additionally or alternatively, the attributes 300 of the driving environment may include a weather type of the driving environment of the vehicle 5, wherein the data processing device 42 is configured to classify the environment according to the weather type. For example, the metadata generating device 40 may classify each video frame as belonging to one of a plurality of predetermined classes associated with different weather conditions (e.g., using a neural network), such as a rainy condition class, a foggy condition class, and a snowy condition class. Alternatively or additionally, the data processing device 42 may be configured to acquire weather information about weather conditions in the environment of the vehicle 5 from the sensor module 20, and use the acquired weather information to classify the environment according to weather type, such as Figure 3 As illustrated in (e) in FIG. , the sensor module 20 may include, or communicate with, a rain sensor configured to detect rainfall on the windshield or other surfaces of the vehicle 5, and the sensor module 20 may generate weather information based on the detection results output by the rain sensor. The sensor module 20 may alternatively obtain weather information wirelessly from a weather forecast / monitoring service, for example.

[0057] Figure 4 An example of a set of predetermined driving scenario classes 410 associated with different driving scenarios that may be used to classify driving scenarios to generate metadata is illustrated.

[0058] like Figure 4As shown, the set of predetermined driving scenario classes 410 associated with different driving scenarios may include an overtaking class associated with an overtaking maneuver performed by vehicle 5. More specifically, when second sensor 24 is a camera sensor, a radar sensor, or a lidar sensor, data processing device 42 may process second sensor data S2 acquired over a certain period of time to determine a driving event in which vehicle 5 overtook a second vehicle. In particular, during the capture of second sensor data S2, data processing device 42 may use second sensor data S2 to perform object detection and / or object tracking of the second vehicle in the environment of vehicle 5. Based on the tracking of the second vehicle's position relative to the vehicle, an overtaking event may be determined, and metadata processing device 42 may generate metadata indicating the classification of the driving scenario into the overtaking class.

[0059] Alternatively or additionally, such as Figure 4 As shown, the set of predetermined driving scenario classes 410 may also include an overtaken vehicle class associated with vehicle 5 being overtaken by a second vehicle. In particular, the data processing device 42 may process the second sensor data S2 using the same object detection and / or object tracking methods described above to determine the occurrence of an event in which vehicle 5 is overtaken by a second vehicle.

[0060] Alternatively or additionally, such as Figure 4 As shown, the set of predetermined driving scenario classes 410 may also include an emergency braking class associated with emergency braking performed by vehicle 5. Data processing device 42 may determine to classify driving scenario 400 as the emergency braking class by processing second sensor data 24 using the aforementioned object detection and / or object tracking methods. However, data processing device 42 may alternatively determine the classification as the emergency braking class based on speed data of vehicle 5, such as obtained by a speed sensor or any other suitable means.

[0061] Alternatively or additionally, such as Figure 4 As shown, the set of predetermined driving scenario classes 410 may further include a lane merging class associated with a second vehicle on the left or right side of vehicle 5 moving into the same lane as vehicle 5. Specifically, the data processing device 42 may process the second sensor data S2 using the same object detection and / or object tracking methods described above to determine and classify the occurrence of an event in which a second vehicle on the left or right side of vehicle 5 moves into the same lane as vehicle 5.

[0062] Alternatively or additionally, in some embodiments, the data processing device 42 may be configured to receive measurements of the yaw rate of the vehicle 5 and, as such, Figure 4As shown, the set of predetermined driving scenario classes may also include a yaw-rate-related maneuvering class associated with the yaw rate of the vehicle 5 that meets a predetermined condition. For example, the data processing device 42 may be configured to receive yaw rate measurements of the vehicle 5 over a certain period of time and, upon determining that the yaw rate of the vehicle 5 meets a predetermined yaw rate condition, classify the driving scenario as belonging to the yaw-rate-related class. The predetermined condition may, for example, be the detection of a yaw rate value exceeding a predetermined yaw rate value for a period exceeding a predetermined period of time (e.g., as may occur when the vehicle is maneuvering around a sharp bend), but more generally may be any condition specified for the yaw rate.

[0063] Alternatively or additionally, in some embodiments the data processing device 42 may be configured to receive a measurement of the speed of the vehicle 5 and, as such, Figure 4 As shown, the set of predetermined driving scenario classes may also include a speed-related maneuvering class associated with the speed of vehicle 5 that achieves a predetermined condition. For example, data processing device 42 may be configured to receive speed measurements of vehicle 5 over a certain period of time and, upon determining that the speed of vehicle 5 meets a predetermined condition, classify driving scenario 400 into a speed-related class. The predetermined condition may be, for example, detecting that the speed of vehicle 5 exceeds a predetermined speed, but may be any condition defined for speed.

[0064] It should be noted that the different driving scenarios are not limited to the aforementioned examples and may include other driving scenarios corresponding to a driving maneuver performed by the vehicle 5 or a driving maneuver performed by a second vehicle relative to the vehicle 5 .

[0065] Figure 5 This example shows that Figure 1 The data processing device 42 processes the video frame to generate a classification of the attribute 300 of the driving environment. In this example embodiment, the video frame is processed by the data processing device 42 to generate the second sensor data measurement result of the metadata M (in the form of metadata data group) of the frame. Figure 5 As shown, the data processing device 42 can use the aforementioned technology to process the video frame to determine the use of Figure 3 When the data processing device 42 classifies the video frames into classes in a predetermined set of classes for a specific attribute of the driving environment, the respective classifications are generated. Figure 5 In the example of FIG. 4 , the data processing device 42 determines based on the illustrated video frame that the image contains Figure 3 Furthermore, the data processing device 42 determines that the vehicle in the frame belongs to an object of the "vehicle" class shown in (a). Figure 3 The “three vehicles” class is shown in (b) in the figure, and the background scene in the frame includes the classification of “natural greenery”, and the time of day is the daytime classification.

[0066] Refer again Figure 1 The metadata generation device 40 further includes a communication device 44 configured to transmit the generated metadata M to a remote metadata processing device 60 while the vehicle 5 is being driven during the data recording activity. In this example, the communication device 44 may include a transmitter configured to wirelessly transmit the metadata M to the metadata processing device 60 via a wireless network connected to the Internet. The wireless network may be, for example, a cellular network such as a 3G, 4G, or 5G network. The metadata processing device 60 is a remote computing device / system, which in this embodiment may be, for example, a cloud server of a cloud computing system, but may alternatively be a dedicated server configured to store and process the metadata M.

[0067] In this example embodiment, metadata data packets can be sent to a remote metadata processing device 60 in real time (e.g., via a live data stream) to allow real-time reporting of the type of driving environment in which the ADAS 10 under development is being tested, as well as the type of driving environment for which data has been recorded during the data recording activity. Furthermore, each metadata data packet sent to the remote metadata processing device 60 is preferably small enough in size to enable data transmission even over a low-bandwidth data connection. As a non-limiting example, each metadata data packet can be limited in size to less than 1 kilobyte. Thus, assuming that the second sensor 24 comprises a camera sensor that captures images at 30 frames per second, only 1.8 megabytes of metadata need be sent to the remote metadata processing device for approximately 60 seconds of video recorded by the sensor data recorder 30. Furthermore, the video frames captured by the second sensor 24 do not need to be stored in the sensor data recorder 30, and once the metadata data packets are generated based on the video frames, the data frames can be deleted.

[0068] exist Figure 1 In FIG. 5 , the remote metadata processing device 60 is connected to the Internet and comprises a data storage section 61 arranged to store metadata transmitted by the communication device 44 on the vehicle 5 .

[0069] like Figure 1As shown, the remote metadata processing device 60 further includes an indicator generator module 62 configured to determine whether the received metadata M includes at least a predetermined number of classifications according to at least one of the following classes: if the metadata M includes classifications of an attribute 300, a predetermined class from a set of predetermined classes 310 for the attribute 300; and if the metadata M includes classifications of a driving scene 400, a predetermined driving scene class from a set of predetermined driving scene classes 410 associated with different driving scenes. Based on this determination, the indicator generator module 62 further generates an indicator T for use in the data recording activity. The predetermined number of classifications for which the metadata M is evaluated may represent requirements defined at the remote metadata processing device 60 regarding the type of sensor data that needs to be captured by the sensor data recorder 30. These requirements may, for example, define the amount of sensor data (e.g., the number of video frames in this example) that needs to be recorded for each of a plurality of different driving environments and / or driving scenes.

[0070] In this example embodiment, the first sensor data S1 includes a plurality of first sensor data measurements acquired at different times during the data recording activity, and the second sensor data includes a plurality of second sensor data measurements. The metadata includes a plurality of metadata data packets generated for and associated with respective first sensor data measurements from the plurality of first sensor data measurements. Each metadata data packet is generated by processing a second sensor data measurement acquired by the sensor module at approximately the same time as the first sensor data measurement. Thus, each metadata data packet includes, if the metadata M includes classifications of attributes 300, a classification of each of one or more attributes of the environment in which the vehicle 5 was located during the acquisition of the first sensor data measurements into a respective class from a plurality of predetermined classes of the attributes. Alternatively or additionally, each metadata data packet may also include, if the metadata M includes classifications of driving scenarios 400, a classification of driving scenarios involving the vehicle 5 that occurred during the acquisition of the sensor data measurements into a class from a set of predetermined driving scenario classes 410 associated with different driving scenarios. In this embodiment, the indicator generator module 62 can determine whether the received metadata M includes at least a predetermined number of classifications of a specified class by determining whether a predetermined number of metadata data groups have classifications that satisfy a predetermined condition of at least one of the following items: in the case where the metadata M includes a classification of attribute 300, attribute 300 of the driving environment; and in the case where the metadata M includes a classification of driving scene 400, driving scene 400.

[0071] Where the metadata M includes classifications of attributes 300, in this example, the predetermined conditions for the one or more attributes 300 of the driving environment can be metadata data packets indicating specific classifications of the attributes. For example, assuming the requirement for the data recording activity is that the sensor data recorder 30 record at least X number of video frames, each of which has Y vehicles, Z pedestrians, and is recorded at night, the indicator generator module 62 can determine, based on the metadata M received at the remote metadata processing device 60, whether at least X number of metadata data packets (assuming one metadata data packet is generated for each frame) have the following classifications: Y vehicles (attribute 300 for the number of objects of the vehicle type), Z pedestrians (attribute 300 for the number of objects of the pedestrian type), and nighttime classification (attribute 300 for the time of day). While a specific example is used with respect to predetermined conditions for the one or more attributes, it should be understood that the requirements for the data recording activity can be defined in any appropriate manner for any number of attributes 300. Furthermore, the predetermined conditions for the attributes 300 need not include metadata data packets classified according to a specified class of attributes 300. Rather, the predetermined condition may instead be derived from the prescribed class 310 associated with the attribute 300. For example, instead of specifying that the metadata data packet should contain a class of W pedestrians, the predetermined condition may specify that the class included in the metadata data packet satisfies a condition of at least or at most W pedestrians.

[0072] Similarly, in the case where the metadata M includes a classification of the driving scene 400, in this example, the predetermined condition of the driving scene 400 may be a metadata data packet indicating a particular classification of the driving scene 400. For example, assuming that the requirement of the data recording activity is that the sensor data recorder 30 records at least a prescribed number of instances in which the recording vehicle 5 overtakes another vehicle during the recording activity, the indicator generator module 62 may determine, based on the metadata M received at the remote metadata processing device 60, whether at least a predetermined number of metadata data packets have a classification in the overtaking driving scene class associated with an overtaking maneuver performed by the vehicle 5.

[0073] Furthermore, in embodiments where the metadata M includes classifications of attributes 300 of the driving environment and classifications of driving scenes 400, a predetermined condition that must be satisfied by a predetermined number of metadata data packets may be defined based on both the attributes 300 and the driving scenes 400. Specifically, the predetermined condition may require that at least a predetermined number of metadata data packets satisfy both the predetermined condition for the attributes 300 of the driving environment and the predetermined condition for the driving scenes 400. The predetermined conditions defined for both the attributes 300 and the driving scenes 400 can be understood as corresponding to specific driving scenarios occurring in a specific driving environment. Specifically, the predetermined condition may specify at least a predetermined number of metadata data packets, each of which has a classification according to a predetermined class 410 of the driving scene 400 and a classification according to a predetermined class 310 of the attributes 300, wherein the classification according to the predetermined class 410 of the driving scene 400 and the classification according to the predetermined class 310 of the attributes 300 are determined by the data processing device 42 at least in part based on the common second sensor data S2. For example, where the second sensor is a camera and the second sensor data S2 is a video frame, the data processing device 42 should determine, based at least in part on one or more common video frames, a classification by the prescribed class 410 of the driving scene 400 and a classification by the prescribed class 310 of the attribute 300, in order to determine the presence of a driving scene corresponding to the prescribed class 410 of the attribute in the driving environment corresponding to the prescribed class 310 of the attribute. Non-limiting examples of predetermined conditions defined with respect to the attribute 300 of the driving environment and the driving scene 400 include a predetermined condition requiring at least a predetermined number of metadata packets, each of which includes a classification indicating an overtaking event occurring in an urban environment, or at least a predetermined number of metadata packets, each of which includes a classification indicating an emergency braking event occurring in rainy conditions. However, the predetermined conditions are by no means limited to the examples provided and may be based on any suitable criteria defined with respect to the attribute 300 and the driving scene 400.

[0074] Refer again Figure 1The (optional) distribution data generator module 64 is configured to, if the metadata M includes a classification of the attribute 300, generate distribution data D indicating the distribution of the classification of the attribute 300 across a plurality of predetermined classes 310 of the attribute 300. Alternatively or additionally, if the metadata M includes a classification of the driving scene 400, the distribution data generator module 64 is configured to generate distribution data D indicating the distribution of the classification of the driving scene 400 across a plurality of predetermined driving scene classes 410. This distribution data D can help a person monitoring the data recording activity to visualize the type of driving environment in which sensor data was collected during the data recording activity. For example, if the metadata M includes a classification of one or more road object types, the classification distribution module 64 can, for example, generate data indicating the number of classifications performed according to each of the plurality of predetermined road object type classes as the distribution data D. For example, in this embodiment, the distribution data D can also indicate the number of video frames containing objects classified by the metadata generation device 40 as belonging to a particular object class. For example, the distribution data may indicate that X video frames contain one or more cars, Y frames contain one or more pedestrians, and Z frames contain one or more bicycles.

[0075] Furthermore, in embodiments where the second sensor 2 is a camera sensor and the classification of the number of objects in each of the plurality of predetermined object classes is determined for each video frame captured by the camera sensor, the distribution data D may indicate the respective number of video frames classified as belonging to each of the predetermined object type classes and the "number of objects" class (e.g., a "video frame with two cars and two pedestrians" class, a "video frame with two cars and one pedestrian" class, a "video frame with three pedestrians and one bicycle" class, etc.). In embodiments where the metadata M includes a classification of background scene features, the distribution data D may also indicate the number of video frames classified as having background scene features belonging to a specified class in the background scene classes. In embodiments where the metadata M includes a classification of the time of day of the driving environment, the distribution data D may indicate the number of video frames classified as belonging to a specified class in the class associated with the time of day. In embodiments where the metadata M includes a classification of the weather type in the driving environment, the distribution data D may include data indicating the number of video frames acquired for each of a plurality of different weather type classes.

[0076] In some embodiments, the distribution data D may indicate the number of metadata data packets having a classification that satisfies a predetermined condition for an attribute of the driving environment (and therefore the number of first sensor data measurements). The predetermined condition for an attribute 300 may be a metadata data packet that indicates a specific classification of the attribute. However, in some embodiments, the predetermined condition may also be derived from a predetermined class associated with the attribute. For example, rather than specifying that a metadata data packet should contain a classification of W pedestrians, the predetermined condition may specify that the metadata data packet contain a classification that satisfies a requirement of at least W pedestrians. Additionally, the distribution data D may indicate the number of image frames that satisfy a plurality of predetermined conditions specified for a plurality of corresponding attributes 300. For example, the distribution data D may indicate the number of frames that have X vehicles, Y pedestrians, and a green scene as a background feature in each frame.

[0077] Furthermore, in embodiments where the metadata includes classifications of driving scenarios 400, the distribution data D may indicate the number of metadata data packets having classifications that satisfy a predetermined condition for driving scenario 400 (and therefore the number of first sensor data measurements). The predetermined condition for driving scenario 400 may be a metadata data packet that indicates a particular classification for that driving scenario 400. For example, the distribution data may indicate the number of metadata data packets that include classifications in the overtaking driving scenario class, the number of metadata data packets that include classifications in the emergency braking driving scenario class, and / or the number of metadata data packets that include classifications in the speed-related maneuver driving scenario class, as described in the previous examples. Furthermore, where the metadata includes classifications for both attribute 300 and driving scenario 400, the predetermined conditions for both attribute 300 and driving scenario 400 are defined in the same manner as discussed in the previous examples. For example, the distribution data D may indicate the number of metadata data packets that include classifications indicating an overtaking maneuver being performed by vehicle 5 in an urban setting, or the number of metadata data packets that include classifications indicating an emergency braking maneuver being performed in rainy conditions. It should be understood that the predetermined conditions are not limited to the foregoing and can be defined in any suitable manner based on the attributes 300 (more specifically, one or more predetermined classes of the attributes 300) and the driving scene 400 (specifically, one or more predetermined classes of the driving scene 400).

[0078] Figure 6 is an example diagram of distribution data 500 generated by the distribution data generator module 64 of the remote metadata processing device 60. Figure 6 In FIG. 5 , the distribution data 500 is presented in the form of a histogram, but may be presented graphically in any form suitable for displaying the distribution of categories among the various predetermined categories of the attributes 300. Figure 6The bins of the X-axis correspond to different driving environments / driving scenarios, which may directly correspond to predetermined classes of attributes / driving scenarios, but may also correspond to predetermined conditions based on predetermined classes of attributes / driving scenarios. Figure 6 In the example of , the Y axis represents the number of counts that achieve a specific driving environment specified by each box along the X axis. It should be understood that Figure 6 The different driving environment conditions / driving scenarios given by the boxes on the X-axis are merely examples, and alternatively, other driving conditions / driving scenarios may be specified instead.

[0079] In the case where the metadata includes classifications of attributes 300 of the driving environment, the indicator T generated by the indicator generator module 62 (when the indicator generator module 62 determines that the metadata includes at least a predetermined number of classifications for the predetermined classes 310 of the attributes 300) can, for example, indicate that sufficient sensor data has been recorded for the specific class 310 of the attributes 300. Furthermore, the indicator T can be sent, for example, to a display device of one or more individuals overseeing the data recording activity (e.g., the driver of the vehicle 5 or a person remotely monitoring the progress of the data recording activity). Similarly, in the case where the metadata includes classifications of the driving scene 400, the indicator T generated by the indicator generator module 62 (when the indicator generator module 62 determines that the metadata includes at least a predetermined number of classifications for the predetermined classes 410 of the driving scene 400) can, for example, indicate that sufficient sensor data has been recorded for the specific class 410 of the driving scene 400. Furthermore, the indicator T can be sent, for example, to a display device of one or more individuals overseeing the data recording activity.

[0080] By monitoring the distribution of the classifications obtained from the metadata M received from the vehicle 5, the data recording activity can be adjusted to allow for more efficient data collection. For example, if it has been determined (based on the distribution data D, or based on the generated indicator T) that sufficient sensor data has been acquired for a certain attribute of the driving environment or driving scenario, the driving route for the next portion of the data acquisition activity can be adjusted to focus on collecting sensor data for other types of driving environments or driving scenarios. In some embodiments, when it is determined that the metadata M includes at least a specified number of classifications for the predetermined classes 310 of the attribute 300, or includes at least a specified number of classifications for the predetermined driving scenario classes 410 of the driving scenario 400, the indicator generator module 62 can also be configured to generate an instruction I as the indicator T to instruct the sensor data recorder 30 to stop recording, and the remote metadata processing device 60 can be configured to send the instruction I to the vehicle 5. The aforementioned features are therefore advantageous because they allow for efficient execution of data recording activities while reducing the amount of distance that needs to be driven to collect the data required for testing and validating the ADAS 10. Thus, rather than defining requirements based on the distance that needs to be driven for a data logging activity, requirements may be defined based on the amount of data (eg, video frames) that needs to be captured for different driving situations and environments.

[0081] In some embodiments, the indicator generator module 62 may be configured to determine whether a metadata data packet received from the communication device 44 (in this example, a metadata data packet generated for a plurality of video frames captured by the first sensor) satisfies a predetermined condition for at least one of: attribute 300 (if the metadata M includes a classification of the attribute 300); and driving scene 400 (if the metadata M includes a classification of the driving scene 400). In response to determining that the metadata data packet does not meet the predetermined condition, the indicator generator module 62 may be configured to generate an instruction I as an indicator D instructing the sensor data recorder 30 at the vehicle 5 to delete the first sensor data S1 for which the metadata data packet has been generated, and the remote metadata processing device 60 may be configured to transmit the instruction I to the vehicle 5. The predetermined conditions for attribute 300 (if the metadata M includes a classification of the attribute) and / or driving scene 400 (if the metadata M includes a classification of the driving scene 400) may be the same as previously defined, i.e., the metadata data packet includes a specific classification of the attribute 300 and / or driving scene 400. However, more generally, where the metadata includes a classification of attributes 300, the predetermined condition may be a metadata data grouping that implements a predetermined condition based on a specific class 310 of attributes (e.g., instead of specifying W vehicles, the condition may specify at least W vehicles). Alternatively or additionally, the predetermined condition may be a metadata data grouping that implements a predetermined condition based on a specific driving scenario class 410. Moreover, in some embodiments, predetermined conditions may be defined for both attributes 300 and driving scenarios in the same manner as described in the previous embodiments. Thus, the remote metadata processing device 60 of this embodiment is capable of evaluating metadata data groups generated for image frames that have been recorded by the sensor data recorder 30 and determining whether the recorded frames contain a "scene of interest" of the type required for testing or training the ADAS 10. If the image frame does not contain a scene of the required type, the recorded frame can be immediately deleted from the sensor data recorder 30 to free up storage space (e.g., HDD space). This allows data recording activities to be performed more efficiently.

[0082] In addition to recording sensor data about the type of road conditions or driving environment in which the vehicle 5 is driven, it may also be necessary or desirable to accurately record information about the state of the vehicle 5 itself in order to obtain all necessary data for verification of the ADAS 10. Therefore, in some embodiments, the sensor module 40 may also include a vehicle data acquisition module ( Figure 1(not shown), the vehicle data acquisition module is configured to acquire one or more of the following vehicle status data during the acquisition of the first sensor data S1 by the first sensor 22: the speed of the vehicle 5, the yaw rate of the vehicle 5, the throttle setting of the vehicle 5, and the position information of the vehicle 5. The sensor data recorder 30 can also be configured to record the vehicle status data. In these embodiments, the metadata M can also include the vehicle status data. In addition, in these embodiments, the metadata processing device 60 can also include an anomaly detection module 66 (such as Figure 1 As shown), the anomaly detection module is configured to use one or more predetermined criteria to identify anomalies in the vehicle status data. The indicator generator module 62 can also be configured to generate a second indicator A for use in the data recording activity, which second indicator A indicates that an anomaly has been detected in the vehicle status data. The one or more predetermined criteria for identifying anomalies in the vehicle status data can, for example, be based on expected measurement results for the vehicle status data and can be defined in any suitable manner. The identified anomaly may be the result of an error in the installation of the equipment used to measure and record the vehicle status data, for example, an incorrect calibration of the measuring device, or a corrupted measurement result caused by mixed signals, hardware failure or human error. In this regard, by sending metadata M that also includes the aforementioned type of vehicle status data, anomalies or problems in the recorded vehicle status data can be identified with very little delay, and the indicator A can be generated and provided to relevant personnel (for example, the driver of the vehicle 5) to highlight the problem and allow the cause of the anomaly to be quickly resolved.

[0083] The metadata M received by the metadata processing device 60 can also be used to efficiently select ground truth data for training the ADAS 10. In particular, by automatically generating metadata data groups for each first sensor data measurement result (in this example, one or more video frames) during the data recording activity, the amount of data that needs to be manually annotated by a human is significantly reduced, and the metadata data groups generated for each of the first sensor data measurements can be used for the following purpose: selecting a set of first sensor data measurements that can be used to form a ground truth data set for training the ADAS 10.

[0084] Therefore, in some embodiments, the metadata generation device 60 may further include a data set selection module 68 (e.g., Figure 1), the data selection module is configured to select a first sensor data measurement subset from a plurality of first sensor data measurements based on the metadata M and a predetermined requirement. In the case where the metadata M includes a classification of an attribute 300, the predetermined requirement may specify a required number of sensor data measurements having corresponding metadata data groups that include a classification of a prescribed class 310 classified as the attribute 300. Alternatively or additionally, in the case where the metadata M includes a classification of a driving scene 400, the predetermined requirement may specify a required number of sensor data measurements having corresponding metadata data groups that include a classification of a prescribed class classified as the driving scene 400. In some embodiments, integer programming may be used to optimize the selection of the first sensor data measurement subset so as to select a minimum number of first sensor data measurements (i.e., a minimum number of video frames) that allows the predetermined requirement to be met.

[0085] More specifically, as in this example embodiment, the dataset selection module can be configured to select a subset of sensor data measurements based on the metadata and the predetermined requirements by solving a system of linear equations subject to mathematical constraints defined according to the predetermined requirements using integer programming. Each linear equation in the system of linear equations is formed based on a classification included in a corresponding set of metadata data groups of the plurality of corresponding metadata data groups. In particular, where the metadata M includes a classification of an attribute 300, at least one variable of each linear equation corresponds to a predetermined class of the attribute 300, and a coefficient of the at least one variable of each linear equation corresponds to the number of classifications in the predetermined class of the attribute 300 indicated by the set of metadata data groups. Furthermore, where the metadata M includes a classification of a driving scene 400, at least one variable of each linear equation corresponds to a predetermined driving scene class 410, and a coefficient of the at least one variable of each linear equation corresponds to the number of classifications in the predetermined driving scene class indicated by the set of metadata data groups.

[0086] As an example, in one embodiment where the metadata M includes a classification of an attribute of the environment, the plurality of sensor data measurements may be divided into subsets of sensor data measurements, each subset being referred to as a data log, and the dataset selection module 68 may generate classification distribution data for each data log based on the metadata data grouping of each sensor data measurement, the classification distribution data indicating the number of sensor data measurements classified in a predetermined class 310 having the attribute 310. The dataset selection module 68 may also use integer programming to select a set of data logs that meet predetermined requirements based on the classification distribution data. As a non-limiting example, z data logs Log1, Log2, ...Log z, the distribution data for each data log in may include a linear equation defining the number of frames having classification in two or more predetermined classes 310 (for one or more attributes 300), for example, as

[0087]

[0088] Wherein, p, b, c are variables representing the number of frames with one or more pedestrians, the number of frames with one or more bicycles, and the number of frames with both bicycles and pedestrians, respectively. Although three variables are defined in the system of linear equations shown above, it should be understood that other variables representing predetermined conditions based on predetermined classes 310 of one or more attributes 300 may also be used. As in this example, the predetermined requirement that the selected data logs need to meet may specify at least 10 frames with pedestrians and 50 frames with bicycles. The predetermined requirement may be defined as a mathematical constraint under which minimization of a function f(a, b, ..., z) is performed in order to select the minimum number of data logs that meet the predetermined requirement, where

[0089] f(a,b,…,z)=a*Log1+b*Log2+…+z*Log3

[0090] Where a, b, ..., z are binary values to be determined during the minimization process. In this embodiment, the solution to minimize min(f(a, b, ..., z)), which is an integer programming problem, can be determined using a simplex algorithm. The determined values a, b, ..., z indicate which data logs need to be selected so that the minimum number of selected data logs is used to meet the predetermined requirement.

[0091] In the aforementioned embodiment, the sensor module 20 includes a first sensor 22 and a second sensor 24, such that metadata M generated for the first sensor data S1 is based on processing of the second sensor data S2, which is acquired by the second sensor 24 at substantially the same time as the first sensor 22 acquires the first sensor data S1. However, it should be understood that the sensor module 20 need not include two sensors. In particular, in some embodiments, the sensor module 20 may instead include only the first sensor 22, and the ADAS 10 and the metadata generation device 40 may be configured to process the sensor data acquired by the first sensor 22.

[0092] Figure 7 is a schematic illustration of a system 1 ′ for evaluating the progress of a data logging activity according to a second example embodiment. Figure 7 System 1' and Figure 1The system 1 differs only in the configuration of the sensor module 20', which comprises a single sensor in the form of a first sensor 22 instead of Figure 1 Two sensors 22 and 24 are shown. Figure 7 The variant system 1 ' shown includes Figure 1 The same metadata generation device 40 and remote metadata processing device 60 are described, with the only difference being Figure 7 The metadata generating device 40 in the embodiment is configured to generate metadata M' of the first sensor data S1 by processing the first sensor data S1 provided by the first sensor 22. In particular, the data processing device 42 may process the first sensor data S1 in accordance with the metadata generated by the data processing device 42. Figure 1 In the same manner as the second sensor data S2 in the embodiment of FIG. 1 , the metadata M′ of the first sensor data S1 is generated by processing the first sensor data S1. In other words, Figure 7 In the example of the system 1', the first sensor 22, the ADAS 10, the sensor data recorder 30, the metadata generation device 40, and the remote metadata processing device 60 (including the indicator generator module 62, the distribution data generator module 64, the anomaly detection module 66, and the data set selection module 68) operate in exactly the same manner as described for the corresponding labeled components, and all examples and embodiments described with respect to these components should be understood as Figure 7 Furthermore, in addition to the metadata generation device 40 and the remote metadata processing device 60, the system 1' may also optionally further include Figure 7 One or more of any of the other components illustrated.

[0093] The example aspects described herein avoid specific computer-based limitations associated with the testing and validation of advanced driver assistance systems (ADAS) and autonomous driving applications. In particular, recording activities performed to collect data for ADAS validation require long driving distances in order to collect sufficient data to cover the diverse driving environments in which the ADAS will need to operate reliably. According to the example aspects described herein, when sensor data (such as image data, radar data, lidar data, etc.) is recorded during a data recording activity for validation of an ADAS under development, metadata for the sensor data is generated and transmitted to a remote metadata processing device. The metadata includes at least one of: classification of an attribute of the vehicle's driving environment into a class within a predetermined set of classes for the attribute; and classification of driving scenarios involving the vehicle that occurred during the acquisition of the sensor data into corresponding classes within a predetermined set of driving scenario classes associated with different driving scenarios. Furthermore, the remote metadata processing device includes an indicator generator module configured to determine whether the received metadata includes at least a predetermined number of classifications into specified classes within the predetermined set of classes, and based on this determination, generate an indicator for use in the data recording activity. The metadata received at the remote metadata processing device can be analyzed to determine the type of sensor data that has been recorded. This can improve the efficiency of the data collection process because the progress of the data recording activity can be monitored and adjustments can be made early in the data collection process (e.g., adjustments to the vehicle's planned route), thereby reducing the distance that needs to be driven and also improving data storage utilization at the vehicle. In addition, the availability of metadata for the collected sensor data reduces the amount of data that needs to be manually annotated by humans and allows for efficient selection of ground truth data for training ADAS. Moreover, based on the aforementioned computer technology-based capabilities of the example aspects described herein, the example aspects described herein improve computers and computer processing / functionality, and also improve the field of testing and validation of advanced driver assistance systems (ADAS) and autonomous driving applications.

[0094] In the foregoing description, example aspects have been described with reference to a number of example embodiments. Therefore, this description should be considered illustrative rather than restrictive. Similarly, the diagrams illustrated in the accompanying drawings, which highlight features and advantages of the example embodiments, are presented for illustrative purposes only. The architecture of the example embodiments is sufficiently flexible and configurable that it can be utilized (and navigated) in ways other than those shown in the accompanying drawings.

[0095] In one example embodiment, the software implementation of the examples presented herein can be provided as a computer program or software (such as one or more programs having instructions or sequences of instructions) that is included or stored in an article such as a machine-accessible or machine-readable medium, an instruction store, or a computer-readable storage device, each of which can be non-transitory. The program or instructions on a non-transitory machine-accessible medium, a machine-readable medium, an instruction store, or a computer-readable storage device can be used to program a computer system or other electronic device. The machine or computer-readable medium, the instruction store, and the storage device can include, but are not limited to, a floppy disk, an optical disk, and a magneto-optical disk, or other types of media / machine-readable media / instruction stores / storage devices suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They can find applicability in any computing or processing environment. As used herein, the terms "computer-readable," "machine-accessible medium," "machine-readable medium," "instruction storage," and "computer-readable storage" shall include any medium that is capable of storing, encoding, or transmitting instructions or sequences of instructions for execution by a machine, computer, or computer processor and that causes the machine / computer / computer processor to perform any of the methods described herein. Furthermore, it is common in the art to refer to software in one form or another (e.g., program, procedure, process, application, module, unit, logic, etc.) as taking an action or causing a result. Such expressions are merely a shorthand way of stating that execution of the software by a processing system causes the processor to perform an action to generate a result.

[0096] Some embodiments may also be implemented by the preparation of application specific integrated circuits, field programmable gate arrays, or by interconnecting an appropriate network of conventional component circuits.

[0097] Some embodiments include a computer program product. A computer program product can be a storage medium, instruction store, or storage device having instructions stored thereon or therein that can be used to control or cause a computer or computer processor to perform any of the processes of the example embodiments described herein. The storage medium / instruction store / storage device can include, for example, but not limited to, an optical disk, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory, flash memory cards, magnetic cards, optical cards, nanosystems, molecular-level memory integrated circuits, RAID, remote data storage / archiving / warehousing, and / or any other type of device suitable for storing instructions and / or data.

[0098] Some implementations stored on a computer-readable medium, instruction storage, or storage device include software for both controlling the system's hardware and enabling the system or microprocessor to interact with a human user or other mechanism using the results of the example embodiments described herein. Such software may include, but is not limited to, device drivers, operating systems, and user applications. Ultimately, such computer-readable media or storage devices also include software for executing the example aspects herein, as described above.

[0099] Included in the programming and / or software of the system are software modules for implementing the processes described herein. In some example embodiments herein, the modules include software, but in other example embodiments herein, the modules include hardware, or a combination of hardware and software.

[0100] Although various exemplary embodiments have been described above, it should be understood that they are presented by way of example and not limitation. It will be apparent to those skilled in the relevant art that various changes in form and detail may be made therein. Therefore, the present disclosure should not be limited by any of the exemplary embodiments described above, but should be defined solely in accordance with the appended claims and their equivalents.

[0101] Implementation of the metadata processing apparatus described in detail above is summarized in the following numbered clauses:

[0102] E1. A metadata processing device for use in a system for evaluating the progress of a data recording activity, the data recording activity being performed while a vehicle is being driven to collect sensor data recorded by a sensor data recorder mounted on the vehicle, the sensor data being used in testing an advanced driver assistance system (ADAS) of the vehicle, wherein the ADAS is configured to provide driving assistance by processing sensor data acquired by a sensor module mounted on the vehicle while the vehicle is being driven, the metadata processing device comprising:

[0103] a data storage unit operable to store metadata received from the metadata generating device, the metadata being based on the data acquired by the sensor module and including a classification of at least one of the following:

[0104] classifying attributes of a driving environment in which the vehicle is located during acquisition of the sensor data by the sensor module into corresponding classes in a predetermined set of classes of the attributes; and

[0105] classifying driving scenarios involving the vehicle occurring during acquisition of the sensor data into respective classes from a predetermined set of driving scenario classes associated with different driving scenarios, wherein each driving scenario is associated with a respective driving maneuver performed by the vehicle or a driving maneuver performed by a second vehicle relative to the vehicle;

[0106] an indicator generator module configured to determine whether the received metadata includes at least a predetermined number of classifications according to at least one of the following categories:

[0107] in case the metadata comprises a classification of the attribute, a predetermined class in the set of predetermined classes; and

[0108] In case the metadata comprises a classification of the driving scene, a predetermined driving scene class in the set of predetermined driving scene classes associated with different driving scenes,

[0109] The indicator generator module is further configured to generate an indicator for use in the data logging activity based on the determination.

[0110] E2. The metadata processing device of E1, wherein the sensor data comprises a plurality of sensor data measurements, each of the plurality of sensor data measurements being acquired at different times during the data recording activity, and the metadata comprises a plurality of metadata data packets, each metadata data packet being associated with a respective sensor data measurement of the plurality of sensor data measurements and being based on data acquired by the sensor module when the sensor data measurements were being acquired, and each metadata data packet comprising at least one of the following:

[0111] In the case where the metadata includes classifications of the attributes, classifying the attributes of the driving environment in which the vehicle is located during the acquisition of the sensor data measurement results into corresponding classes in the predetermined set of classes of the attributes; and

[0112] In case the metadata comprises classifications of the driving scenarios, the driving scenarios involving the vehicle occurring during the acquisition of the sensor data measurements are respectively classified into respective classes in the set of predetermined driving scenario classes associated with different driving scenarios.

[0113] E3. The metadata processing device of E2, wherein the indicator generator module is configured to determine whether the stored metadata includes at least the predetermined number of categories by determining whether a predetermined number of metadata data packets have corresponding categories that satisfy a predetermined condition of at least one of the following:

[0114] the attributes of the driving environment, in the case where the metadata includes a classification of the attributes; and

[0115] In case the metadata includes a classification of the driving scene, the driving scene.

[0116] E4. The metadata processing device according to E1, wherein

[0117] In response to determining that the predetermined number of metadata data packets have corresponding classifications that satisfy the predetermined condition, the indicator generator module is configured to generate a first instruction instructing the sensor data recorder to stop recording sensor data as the indicator, and

[0118] The metadata processing device is operable to send the first instruction to the vehicle.

[0119] E5. The metadata processing device according to any one of E2 to E4, wherein

[0120] The indicator generator module is arranged to determine whether a metadata data packet stored in the data storage portion satisfies a predetermined condition of at least one of the following:

[0121] In the case where the metadata includes a classification of the attribute, the attribute; and

[0122] In case the metadata includes a classification of the driving scene, the driving scene,

[0123] In response to determining that the metadata data packet does not satisfy the predetermined condition, generating as the indicator a second instruction instructing the sensor data recorder to delete the stored sensor data measurement associated with the metadata data packet, and

[0124] The metadata processing device is configured to send the second instruction to the vehicle.

[0125] E6. The metadata processing device according to any one of E1 to E5, wherein

[0126] The metadata further includes vehicle state data, the vehicle state data including one or more of the following items measured during acquisition of the sensor data by the sensor module: a speed of the vehicle, a yaw rate of the vehicle, a throttle setting of the vehicle, and position information of the vehicle,

[0127] The metadata processing device further comprises an anomaly detection module configured to search for anomalies in the vehicle status data using one or more predetermined criteria, and

[0128] The indicator generator module is further configured to generate a second indicator for use in the data logging activity, the second indicator indicating that an anomaly has been detected in the vehicle status data.

[0129] E7. The metadata processing device according to any one of E2 to E5, further comprising a dataset selection module configured to select a subset of sensor data measurements from the plurality of sensor data measurements based on the metadata and a predetermined requirement, wherein the predetermined requirement specifies at least one of the following:

[0130] In case the metadata comprises classifications of the attributes, having a required number of sensor data measurements comprising corresponding metadata data packets classified into predetermined classes of the attributes, and

[0131] In case the metadata comprises a classification of the driving scene, there is a required number of sensor data measurements comprising corresponding metadata data packets classified into a predetermined class of the driving scene.

[0132] E8. The metadata processing apparatus of E7, wherein the dataset selection module is configured to select the subset of sensor data measurements based on the metadata and the predetermined requirement by solving a system of linear equations subject to mathematical constraints defined according to the predetermined requirement using integer programming, each linear equation in the system of linear equations being formed based on a classification included in a respective metadata data group of the plurality of respective metadata data groups,

[0133] wherein, in the case where the metadata includes classifications of the attribute, at least one variable of each linear equation corresponds to a predetermined class of the attribute, and a coefficient of the at least one variable of each linear equation corresponds to the number of classifications under the predetermined class of the attribute indicated by the metadata data group, and

[0134] Wherein, in the case where the metadata includes the classification of the driving scene, at least one variable of each linear equation corresponds to a predetermined driving scene class, and the coefficient of the at least one variable of each linear equation corresponds to the number of classifications under the predetermined driving scene class indicated by the metadata data group.

[0135] E9. The metadata processing device according to any one of E1 to E8, wherein the attribute of the driving environment of the vehicle includes at least one of the following:

[0136] a type of one or more objects in a driving environment of a vehicle, wherein the data processing device is configured to process the data acquired by the sensor module to detect one or more objects in the driving environment of the vehicle and classify the detected one or more objects according to object type;

[0137] a number of objects of predetermined one or more types in the driving environment of the vehicle, wherein the data processing device is configured to process the data acquired by the sensor module to detect one or more objects of the predetermined one or more types, and classify the detected one or more objects according to the number of the detected one or more objects;

[0138] an environment type of the driving environment of the vehicle, wherein the data processing device is configured to obtain information about the driving environment of the vehicle and classify the driving environment according to an environment type;

[0139] a time of day of the driving environment, wherein the data processing device is configured to obtain a local time of the vehicle and classify the driving environment according to the local time; and

[0140] a weather type of the driving environment of the vehicle, wherein the data processing device is configured to obtain information about weather conditions of the driving environment of the vehicle and classify the driving environment according to weather types using the obtained information about the weather conditions, and

[0141] The predetermined driving scenario class set associated with different driving scenarios includes at least one of the following items:

[0142] an overtaking class associated with an overtaking maneuver performed by said vehicle;

[0143] an overtaken vehicle class associated with the vehicle being overtaken by a second vehicle;

[0144] an emergency braking class associated with emergency braking performed by the vehicle;

[0145] A second vehicle to the left or right of the vehicle moves into a merging class associated with the same lane as the vehicle;

[0146] a yaw rate related maneuvering class associated with a yaw rate of the vehicle that achieves a predetermined condition; and

[0147] A speed-dependent maneuvering class is associated with the speed of the vehicle to achieve a predetermined condition.

[0148] E10. The metadata processing device according to any one of E1 to E9, wherein the indicator generator module is further configured to display the indicator on a display.

Claims

1. A system for evaluating the progress of a data recording activity performed while a vehicle is driven to collect sensor data (S1) recorded by a sensor data recorder (30) installed on the vehicle (5), the sensor data (S1) being used in testing and validating an advanced driver assistance system (ADAS) (10) of the vehicle (5), wherein: The ADAS (10) is configured to provide driving assistance by processing sensor data (S1) acquired by a sensor module installed on the vehicle (5) while the vehicle (5) is being driven, the system comprising a metadata generating device (40) and a remote metadata processing device (60), wherein: The metadata generating device (40) is arranged to generate metadata (M) when mounted on the vehicle (5) and used for the data recording activity, the metadata generating device (40) comprising: A data processing device (42) is configured to process the data acquired by the sensor module to generate metadata (M) of the sensor data (S1), the metadata (M) including classification of the following items: Classifying driving scenarios (400) involving the vehicle occurring during acquisition of the sensor data (S1) into respective classes in a set of predetermined driving scenario classes (410) associated with different driving scenarios, wherein each driving scenario is associated with a respective driving maneuver performed by the vehicle (5) or a driving maneuver performed by a second vehicle relative to the vehicle (5), and wherein the set of predetermined driving scenario classes (410) associated with different driving scenarios comprises at least one of: an overtaking class associated with an overtaking maneuver performed by said vehicle (5); an overtaken vehicle class associated with the vehicle (5) being overtaken by the second vehicle; an emergency braking class associated with emergency braking performed by the vehicle (5); A second vehicle to the left or right of the vehicle (5) moves to a merging class associated with the same lane as the vehicle (5); a yaw rate related maneuvering class associated with a yaw rate of the vehicle that achieves a predetermined condition; and a speed-related maneuvering class associated with the speed of the vehicle to achieve a predetermined condition; and a communication device (44) arranged to transmit the metadata (M) to the remote metadata processing device (60), and The remote metadata processing device (60) comprises: a data storage unit (61) configured to store the metadata (M) transmitted by the communication device (44); and An indicator generator module (62) is arranged to determine whether the received metadata (M) comprises at least a predetermined number of classifications according to the following categories: a predetermined driving scenario class in the set of predetermined driving scenario classes (410) associated with different driving scenarios, Wherein the indicator generator module (62) is configured to generate an indicator (T) for use in the data logging activity based on the determination.

2. The system according to claim 1, wherein: The sensor module comprises: a first sensor (22), the first sensor being configured to acquire first sensor data (S1) when the vehicle (5) is driven; and a second sensor (24), the second sensor being configured to acquire second sensor data (S2) when the vehicle (5) is driven, the second sensor (24) being different from the first sensor (22). The ADAS (10) is configured to provide driving assistance based on the first sensor data (S1), and The metadata generating device (40) is configured to generate the metadata (M) by processing the second sensor data (S2).

3. The system according to claim 2, wherein: Each of the first sensor (22) and the second sensor (24) includes one of a camera, a radar sensor, and a lidar sensor.

4. The system according to claim 1, wherein: The sensor module includes a single sensor configured to acquire the sensor data (S1), and wherein the ADAS (10) and the metadata generating device (40) are configured to process the sensor data (S1) acquired by the single sensor.

5. The system according to claim 1, wherein The sensor data (S1) comprises a plurality of sensor data measurements, each of the plurality of sensor data measurements being acquired at a different moment during the data recording activity, and the metadata (M) comprises a plurality of metadata data groups, the data processing device (42) being configured to generate respective metadata data groups for respective ones of the plurality of sensor data measurements and to associate the metadata data groups with the respective sensor data measurements, wherein the respective metadata data groups are generated by processing data acquired by the sensor module when the sensor data measurements are acquired, and the respective metadata data groups comprise: In case the metadata (M) comprises a classification of an attribute (300) of the driving environment in which the vehicle (5) was located during the acquisition of the sensor data measurements, classifying the attribute (300) into a respective class from a set of predetermined classes (310) of the attribute (300); and In case the metadata (M) comprises classifications of driving scenarios (400) involving the vehicle occurring during the acquisition of the sensor data measurements, the driving scenarios (400) are respectively classified into respective classes in the set of predetermined driving scenario classes (410) associated with different driving scenarios.

6. The system according to claim 5, wherein: The indicator generator module (62) is arranged to determine whether the stored metadata (M) comprises at least a predetermined number of categories by determining whether a predetermined number of metadata data packets have respective categories that satisfy a predetermined condition of at least one of the following: said attributes (300) of said driving environment in case said metadata (M) comprises a classification of said attributes (300); and In case the metadata (M) comprises a classification of the driving scene (400), the driving scene (400).

7. The system according to claim 6, wherein: In response to determining that the predetermined number of metadata data packets have corresponding classifications that satisfy the predetermined condition, the indicator generator module (62) is configured to generate a first instruction (I) as the indicator (T) instructing the sensor data recorder (30) to stop recording sensor data (S1), The remote metadata processing device (60) is configured to send the first instruction (I) to the vehicle (5), and The sensor data recorder (30) is configured to respond to the first instruction (I) by stopping recording sensor data (S1).

8. The system according to any one of claims 5 to 7, wherein: The indicator generator module (62) is arranged to determine whether a metadata data packet stored in the data storage (61) satisfies a predetermined condition for at least one of the following: In case said metadata (M) comprises a classification of said attribute (300), said attribute (300); and In case the metadata (M) comprises a classification of the driving scene (400), the driving scene, In response to determining that the metadata data packet does not satisfy the predetermined condition, generating a second instruction as the indicator (T) instructing the sensor data recorder (30) to delete the stored sensor data measurement associated with the metadata data packet, The remote metadata processing device (60) is configured to send the second instruction to the vehicle (5), and The sensor data recorder (30) is arranged to respond to the second instruction by deleting the stored sensor data measurements associated with the metadata data packet.

9. The system according to claim 1, wherein: The sensor module is further configured to acquire vehicle state data comprising measurement results of one or more of the following items during acquisition of the sensor data (S1) by the sensor module: a speed of the vehicle (5); a yaw rate of the vehicle (5); a throttle setting of the vehicle (5); and position information of the vehicle (5), The data processing device (42) is arranged to generate the metadata (M) in such a way that the metadata also includes the vehicle status data, The remote metadata processing device (60) further comprises an anomaly detection module (66) configured to search for anomalies in the vehicle status data using one or more predetermined criteria, and The indicator generator module (62) is further configured to generate a second indicator (A) for use in the data logging activity, the second indicator (A) indicating that an anomaly has been detected in the vehicle status data.

10. The system according to claim 5, wherein The remote metadata processing device (60) further comprises a data set selection module (68) configured to select a subset of sensor data measurements from the plurality of sensor data measurements based on the metadata (M) and predetermined requirements, wherein the predetermined requirements specify at least one of the following: In case said metadata (M) comprises a classification of an attribute (300), a required number of sensor data measurements having corresponding metadata data packets comprising a classification into a predetermined class (310) of said attribute (300), and In case the metadata (M) comprises a classification of the driving scene (400), a required number of sensor data measurements having corresponding metadata data packets comprising a classification into a predetermined class of the driving scene (400).

11. The system according to claim 10, wherein: The data set selection module (68) is configured to select the subset of sensor data measurements based on the metadata (M) and the predetermined requirements by solving a system of linear equations subject to mathematical constraints defined according to the predetermined requirements using integer programming, each linear equation in the system of linear equations being formed based on a classification included in a respective set of metadata data groups of a plurality of respective metadata data groups, wherein, in the case where the metadata (M) includes a classification of the attribute (300), at least one variable of each linear equation corresponds to a predetermined class of the attribute (300), and a coefficient of the at least one variable of each linear equation corresponds to the number of classifications in the predetermined class of the attribute (300) indicated by the metadata data grouping set, and wherein, in the case where the metadata (M) comprises a classification of the driving scene (400), at least one variable of each linear equation corresponds to a predetermined driving scene class, and a coefficient of the at least one variable of each linear equation corresponds to the number of classifications in the predetermined driving scene class indicated by the metadata data grouping set.

12. The system according to claim 5, wherein: The properties (300) of the driving environment of the vehicle (5) include at least one of the following: a type of one or more objects in the driving environment of the vehicle (5), wherein the data processing device (42) is configured to process the data acquired by the sensor module to detect one or more objects in the driving environment of the vehicle (5) and classify the detected one or more objects according to object type; a number of objects of one or more predetermined types in the driving environment of the vehicle (5), wherein the data processing device (42) is configured to process the data acquired by the sensor module to detect one or more objects of the predetermined one or more types, and classify the detected one or more objects according to the number of the detected one or more objects; an environment type of the driving environment of the vehicle, wherein the data processing device (42) is configured to obtain information about the driving environment of the vehicle and classify the driving environment according to an environment type; time of day of the driving environment, wherein the data processing device (42) is configured to obtain a local time of the vehicle and classify the driving environment according to the local time; and A weather type of the driving environment of the vehicle (5), wherein the data processing device (42) is configured to obtain information about the weather conditions of the driving environment of the vehicle (5), and to classify the driving environment according to weather types using the obtained information about the weather conditions.

13. The system of claim 1, wherein: The indicator generator module (62) is further configured to display the indicator (T) on a display.

14. A metadata generating device (40) for use in a system for evaluating the progress of a data recording activity performed while a vehicle is driven to collect sensor data (S1) recorded by a sensor data recorder (30) mounted on the vehicle (5), the sensor data (S1) being used in testing an advanced driver assistance system (ADAS) (10) of the vehicle (5), wherein: The ADAS (10) is configured to provide driving assistance by processing sensor data (S1) acquired by a sensor module installed on the vehicle (5) while the vehicle (5) is being driven, the system comprising a remote metadata processing device (60), wherein the metadata generating device (40) is configured to generate metadata (M) when installed on the vehicle (5) and used for the data recording activity, the metadata generating device (40) comprising: A data processing device (42) is configured to process the data acquired by the sensor module to generate metadata (M) of the sensor data (S1), the metadata (M) including classification of the following items: Classifying driving scenarios (400) involving the vehicle (5) occurring during the acquisition of the sensor data (S1) into respective classes in a set of predetermined driving scenario classes (410) associated with different driving scenarios, wherein each driving scenario is associated with a respective driving maneuver performed by the vehicle (5) or a driving maneuver performed by a second vehicle relative to the vehicle (5), and wherein the set of predetermined driving scenario classes (410) associated with different driving scenarios comprises at least one of: an overtaking class associated with an overtaking maneuver performed by said vehicle (5); an overtaken vehicle class associated with the vehicle (5) being overtaken by the second vehicle; an emergency braking class associated with emergency braking performed by the vehicle (5); A second vehicle to the left or right of the vehicle (5) moves to a merging class associated with the same lane as the vehicle (5); a yaw rate related maneuvering class associated with a yaw rate of the vehicle that achieves a predetermined condition; and a speed-related maneuvering class associated with the speed of the vehicle to achieve a predetermined condition; A communication device (44) operable to: sending the metadata (M) to the remote metadata processing device (60); and An indicator for use in the data recording activity is received from the remote metadata processing device (60), the indicator indicating a determination result made by the remote metadata processing device (60) as to whether the metadata (M) received thereby includes at least a predetermined number of classifications of the following classes: A predetermined driving scenario class in the set of predetermined driving scenario classes (410) associated with different driving scenarios.

15. A remote metadata processing device (60) for use in a system for evaluating the progress of a data recording activity performed while a vehicle is being driven to collect sensor data (S1) recorded by a sensor data recorder (30) mounted on the vehicle (5), the sensor data (S1) being used in testing an advanced driver assistance system (ADAS) (10) of the vehicle (5), wherein: The ADAS (10) is configured to provide driving assistance by processing sensor data (S1) acquired by a sensor module installed on the vehicle (5) when the vehicle (5) is driven, and the remote metadata processing device (60) includes: A data storage unit (61) operable to store metadata (M) received from a metadata generating device (40), the metadata (M) being based on data acquired by the sensor module and including classification of the following items: Classifying driving scenarios (400) involving the vehicle (5) occurring during the acquisition of the sensor data (S1) into respective classes in a set of predetermined driving scenario classes (410) associated with different driving scenarios, wherein each driving scenario is associated with a respective driving maneuver performed by the vehicle (5) or a driving maneuver performed by a second vehicle relative to the vehicle (5), and wherein the set of predetermined driving scenario classes (410) associated with different driving scenarios comprises at least one of: an overtaking class associated with an overtaking maneuver performed by said vehicle (5); an overtaken vehicle class associated with the vehicle (5) being overtaken by the second vehicle; an emergency braking class associated with emergency braking performed by the vehicle (5); A second vehicle to the left or right of the vehicle (5) moves to a merging class associated with the same lane as the vehicle (5); a yaw rate related maneuvering class associated with a yaw rate of the vehicle that achieves a predetermined condition; and a speed-related maneuvering class associated with the speed of the vehicle to achieve a predetermined condition; An indicator generator module (62) is arranged to determine whether the received metadata (M) comprises at least a predetermined number of classifications according to the following categories: in case the metadata (M) comprises a classification of the driving scene (400), a predetermined driving scene class in the set of predetermined driving scene classes (410) associated with different driving scenes, Therein, the indicator generator module (62) is arranged to generate an indicator (T) for use in the data logging activity based on the determination.

Images