Storing sensor data

Abstract

A computer-implemented method comprising obtaining sensor data from one or more sensors of an autonomous vehicle 201 extracting, from the sensor data, driving scene data 202 and performing a similarit

Description

FIELD AND BACKGROUND

[0001] The present techniques relate to the field of autonomous driving. More particularly, but not exclusively, the present techniques relate to data collection associated with autonomous driving.

[0002] In recent times, the use of autonomous vehicles (also known as self-driving vehicles, SDVs) has become more widespread. Autonomous vehicles are vehicles capable of sensing their own environment and operating without human involvement. The degree of autonomy may vary from fully autonomous requiring no human involvement (sometimes known as Level 5 - Full Driving Automation), to lower levels of autonomy requiring some degree of human involvement (Levels 1 to 4). As used herein, an autonomous vehicle refers to a vehicle or robot having at least some capability of operating without human involvement (i.e. any vehicle from Level 1 to Level 5).

[0003] Autonomous vehicles may collect data associated with their vehicle state, surroundings or environment using one or more sensors. This collected data may then be used by on-board systems of the vehicle and / or off-loaded for use off-vehicle. SUMMARY

[0004] Particular aspects and embodiments are set out in the appended claims.

[0005] Viewed from a first aspect, there is provided a computer-implemented method comprising: obtaining sensor data from one or more sensors of an autonomous vehicle; extracting, from the sensor data, driving scene data; and performing a similarity process and / or a dissimilarity process, wherein the similarity process comprises determining whether a similarity condition associated with a similarity between the driving scene data and reference driving scene data is satisfied; and in response to determining that the similarity condition is satisfied, storing the sensor data in a data store, and wherein the dissimilarity process comprises: determining whether a difference condition associated with a difference between the driving scene data and second reference driving data is satisfied; and in response to determining that the difference condition is satisfied, storing the sensor data in the data store.

[0006] Hence, the present approach provides an improved trigger mechanism for collecting sensor data obtained from one or more sensors of an autonomous vehicle. The present inventors have identified that, with advancements in sensor technology and the vast amount of data that an autonomous vehicle may potentially collect (such as vehicle state information, environment information, etc.), existing techniques for data collection are not suitable. Indeed, in some examples, it may not be practical to store all sensor data collected by the autonomous vehicle in a given time period, for example due to storage, processing, time and / or resource limitations. The present technique therefore provides a selective triggering 1 mechanism for storing sensor data based on whether a similarity condition associated with a similarity between driving scene data extracted from the sensor data and reference driving scene data is satisfied and / or whether a difference condition associated with a difference between the driving scene data and second reference driving data is satisfied.

[0007] As a result, sensor data associated with driving scenes that are similar to reference driving scenes and / or that are different to second reference driving scenes may be selectively stored. This allows, for example, for the collection of sensor data associated with driving scenes that may be particularly significant or useful. This selective triggering provides a more configurable and efficient approach to sensor data collection obtained from one or more sensors of autonomous vehicles.

[0008] For example, in some cases, it may be advantageous to be able to selectively collect driving data that relates to a certain driving scenario, such as the navigation of a certain road layout like an intersection / junction, or a certain driving scenario involving certain other road users. The reference driving scene data thus may specify such a driving scenario and the autonomous vehicle may therefore selectively store driving data that is determined to be similar to the reference driving scene data.

[0009] Further, in some cases, it may be advantageous to be able to selectively collect driving data that relates to a reference driving scenario that corresponds to a road accident or possible safety event or common failure mode of the autonomous vehicle. Large trucks, parked vehicles, and pedestrian crossings are examples where autonomous vehicles may perform poorly and so being able to selectively request data associated with these road scenarios is advantageous for improving autonomous vehicle performance and safety.

[0010] Thus, with the present technique, the autonomous vehicle may be able to selectively store data that relates to requested driving scenarios. This driving data may then be later used to improve the performance of the autonomous vehicle or other autonomous vehicles in such driving scenarios. For example, in some cases, data collected by autonomous vehicles may be used to train machine learning models such as machine learning motion planners or driving policies. It may therefore be advantageous to be able to selectively collect data that relates to a driving scene that may be particularly useful for training purposes.

[0011] It is noted that various uses of the stored sensor data are envisioned. For example, if a road accident is reported, the reference driving scene data may be used to provide a description of a driving scene of the road accident. In effect, the autonomous vehicle may then 'search' for similar driving scenes as the vehicle traverses a roadway and store data relating to a similar driving scene if encountered. This allows an effective reporting mechanism to be deployed on an autonomous vehicle. If deployed across a fleet of autonomous vehicles, this may be enhanced further. For example, additional data relating to the 2 scene of an accident may be selectively collected using the present techniques. In some examples, in response to storing the sensor data in the data store, the sensor data may be sent to a data store server over a network connection.

[0012] By selectively storing sensor data that is sufficiently similar to the reference driving data, a more configurable triggering mechanism is realised. Further, the efficiency of data collection is increased because data not meeting the minimum similarity threshold (i.e. driving data less relevant to the reference driving scene data) may not be stored. As such, possible limitations with existing data collection techniques are overcome as only data of particular interest is stored. Further, the efficiency of this triggering mechanism and the similarity determination allows for fast triggering decisions, which allows a data buffer storing the sensor data for a given time window to be reduced in size, thereby optimising storage and processing requirements of the data-processing apparatus implementing the method.

[0013] When the similarity condition is satisfied, the sensor data may be stored in a data store designated for storing sensor data that causes the similarity condition to be satisfied. Hence, the sensor data may be stored in a dedicated data store or region of memory designated for storing sensor data that corresponds to a driving scene considered similar to the reference driving scene data. It will be appreciated that the data store used for storing the sensor data in response to the similarity condition being satisfied may be different from a buffer that may initially store the sensor data before or while the similarity determination is being performed.

[0014] In some examples, the sensor data comprises sensor data obtained from other autonomous vehicles. For example, an autonomous vehicle implementing the technique described herein may receive, from a further autonomous vehicle, sensor data obtained by the further autonomous vehicle, and the driving scene data may be extracted from the sensor obtained by the further autonomous vehicle in addition to the sensor data obtained by the autonomous vehicle. Thus, in some examples, the sensing capabilities of a given autonomous vehicle may be enhanced or augmented with the sensing capabilities of other autonomous vehicles. In some examples, the other autonomous vehicles are encountered by the autonomous vehicle as the autonomous vehicle traverses a roadway. The sensor data may be transferred between vehicles using vehicle-to-vehicle communication, using one or more wireless communication protocols.

[0015] In some examples, the method further comprises performing the dissimilarity process, wherein the dissimilarity process comprises: determining whether a difference condition associated with a difference between the driving scene data and second reference driving data is satisfied; in response to determining that the difference condition is satisfied, storing the sensor data in the data store.

[0016] Thus, not only may sensor data that is similar to a reference driving scene (i.e. one that may be of particular significance or interest) be collected and stored, but sensor data that differs sufficiently from existing data may also be stored. This counter-intuitive approach of saving data similar to that which is requested (via the reference driving scene) but also data which is significantly different to data that is requested (via the second reference driving scene data) ensures that rare occurrences (for example infrequently occurring events experienced by the autonomous vehicle) are still collected. These rare occurrences may be useful for various reasons and so their collection may be particularly advantageous. For example, a rare occurrence may relate to a rare safety event or a rare driving scenario. Accordingly, it can be advantageous not to discard data relating to a rare safety event just because it is not similar to the reference driving scene data as this data may be used to improve autonomous vehicle driving performance and safety. Hence, an improved triggering and sensor data storing technique is provided.

[0017] Indeed, in some cases, collected driving data (e.g. sensor data) may be used for training machine learning models associated with autonomous vehicles. For example, to ensure machine learning model inference performance satisfies certain thresholds, it may be necessary to train the machine learning model on a diverse range of input data which includes a diverse range of driving scenarios, including rare driving scenarios, to ensure that the machine learning model performs appropriately when confronted with a rare scenario once trained. Indeed, by storing both sufficiently similar and sufficiently dissimilar data to that requested an improved sensor data collection trigger mechanism is provided that collects data optimised for subsequent training of machine learning models, and supports improved training of machine learning driving policies.

[0018] When the difference condition is satisfied, as discussed above, the sensor data may be stored in the same data store as when the similarity condition is satisfied, or a different data store designated for storing sensor data that results in the difference condition being satisfied. In some examples, the sensor data may be stored in a data store designated for storing sensor data that causes the difference condition to be satisfied. Hence, the sensor data may be stored in a dedicated data store or region of memory designated for storing sensor data that corresponds to a driving scene considered different to the second reference driving scene data. It will be appreciated that the data store used for storing the sensor data in response to the difference condition being satisfied may be different from a buffer that may initially store the sensor data before or while the similarity and / or difference determination is being performed.

[0019] In some examples, the dissimilarity process may be performed in response to determining that the similarity threshold is not satisfied. That is to say, in some examples, the similarity process may be performed and then the dissimilarity process may be formed dependent on the outcome of the similarity process. In this way, a more configurable and efficient approach may be realised, that invokes the dissimilarity processing once the similarity processing has returned a result that the similarity threshold is not satisfied. Thereby, processing resources may be conserved.

[0020] It will be appreciated that in some examples either the similarity processing or the dissimilarity processing may be performed, and this may depend on the specific use-case of the implementation. Indeed, in some examples, both the similarity processing and dissimilarity processing may be performed. In some examples, the dissimilarity processing is performed in response to determining that the similarity threshold is not satisfied, and in other examples the dissimilarity processing is performed irrespective of whether the similarity threshold is satisfied or not. In some examples, the similarity processing may be performed in response to determining that the difference condition is not satisfied. Thus, a configurable approach to efficient sensor data storage may be realised.

[0021] In some examples, determining whetherthe similarity condition is satisfied comprises: comparing the driving scene data and reference driving scene data; and determining, based on evaluating one or more predefined rules using the comparison, whether the similarity condition is satisfied. As a result, an efficient similarity comparison is performed in a timely and performant manner. In some implementations, the autonomous vehicle may need to make fast decisions associated with whether to store data or not store (discard, delete, or otherwise allow overwrite for) data. For example, the buffer for storing data may be limited and so only sensor data recorded during a limited window of time may be stored. Thus, the triggering process may determine whether to store the data quickly to ensure that the decision is made before the sensor data is overwritten, discarded or otherwise no longer present in the data buffer.

[0022] In some examples, storing the sensor data in a data store comprises transferring the sensor data from a sensor data buffer to a data store designated for storing sensor data that results in the similarity or difference condition being satisfied.

[0023] In some examples, determining whether the similarity condition associated with the similarity between the driving scene data and reference driving scene data is satisfied is based on using a machine learning technique. This increases efficiency of the similarity comparison, and also allows more granular and complex factors associated with the similarity to be taken into consideration for the comparison of the driving scene data and the reference driving scene data. In some implementations, the reference driving scene data may be associated with a complex driving scene (for example, having a plurality of vehicles or road users present, a certain type of road layout, and a certain type of weather condition. For example, a multi-lane roadway with at least one other vehicle and a cyclist during a storm), and thus greater detail may be considered in the comparison by using a machine learning technique.

[0024] In some examples, determining whether the similarity condition is satisfied comprises: generating, using a neural network trained to transform driving scene data into feature embeddings, a feature embedding of the driving scene data; comparing the feature embedding of the driving scene data with a feature embedding of the reference driving scene data; and determining, based on the comparison, whether the similarity condition is satisfied.

[0025] Thus, advantageously, the comparison may be based on feature embeddings of the driving scene data extracted from the sensor data and the reference driving scene data. This allows a fast comparison to be made. For example, the comparison of the feature vectors may be a comparison between two multidimensional vectors which can be calculated in an efficient manner. As discussed herein, the use of a neural network to transform the driving scene data extracted from the sensor data into a feature embedding allows a 'summary' of the driving scene to be represented in a relatively lower dimensional manner to provide an efficient comparison. Indeed, complex aspects of the driving scene (such as other road users, road layouts, weather conditions etc.) may be taken into consideration and converted into a feature vector that represents the scene. The feature vector of the reference driving scene data may be received or otherwise input to the apparatus performing the method. In some examples, the feature embedding may comprise a multi-dimensional matrix that represents the driving scene data.

[0026] In some examples, determining, based on the comparison, whether the similarity condition is satisfied comprises determining whether a distance in feature embedding space between the feature embedding of the driving scene data and the feature embedding of the reference driving scene data is less than a predetermined maximum similarity distance. Accordingly, an efficient comparison may be performed which is suitable for possible constraints of the implementation (such as storage and computing resources). A fast determination of similarity may thus be achieved. In some examples, the distance between feature embeddings refers to a difference between two multi-dimensional matrixes.

[0027] In some examples, determining whether the difference condition associated with the difference between the driving scene data and second reference driving data is satisfied is based on using a machine learning technique. Hence, the dissimilarity determination (i.e. determining whether the difference condition is satisfied) may be performed using a machine learning technique, resulting in an efficient comparison and allowing for increased detail in the comparison to be considered, in a similar way to that described above.

[0028] In some examples, determining whether the difference condition associated with the difference between the driving scene data and second reference driving data is satisfied comprises: generating, using a neural network trained to transform driving scene data into feature embeddings, a feature embedding of the driving scene data; comparing the feature embedding of the driving scene data with a feature embedding of the second reference driving scene data; and determining, based on the comparison, whether the difference condition is satisfied.

[0029] Hence, advantageously, the comparison may be based on feature embeddings of the driving scene data extracted from the sensor data and the reference driving scene data. This allows a fast comparison to be made. For example, the comparison of the feature embeddings may be a comparison between two multi-dimensional vectors or matrixes which can be calculated in an efficient manner. As discussed herein, the use of a neural network to transform the driving scene data extracted from the sensor data into a feature embedding allows a 'summary' of the driving scene to be represented in a relatively lower dimensional manner to provide an efficient comparison. For example, complex aspects of the driving scene (such as other road users, road layouts, weather conditions etc.) may be taken into consideration and converted into a feature embedding that represents the scene.

[0030] In some examples, determining, based on the comparison, whether the difference condition is satisfied comprises determining whether a distance in feature embedding space between the feature embedding of the driving scene data and the feature embedding of the second reference driving scene data is greater than a predetermined minimum novelty distance. Accordingly, an efficient comparison may be performed which is suitable for possible constraints of the implementation (such as storage and computing resources). A fast determination of dissimilarity may thus be achieved. Further, the predetermined minimum novelty distance may be configurable based on user input, and so the similarity threshold may be set in a flexible and configurable manner depending on implementation.

[0031] In some examples, the distance may be a cosine distance. This provides a computationally efficient approach for determining the similarity / dissimilarity in a feature embedding space.

[0032] In some examples, the reference driving scene data and the second reference driving scene data are different. Thus, the similarity and dissimilarity comparisons may use different driving scenes thereby providing a more flexible and configurable approach to selectively triggering on and storing sensor data.

[0033] It will be appreciated that while the sensor data may be stored in the data store in response to the similarity condition being satisfied or the difference condition being satisfied, data indicative of the sensor data may be stored instead. In some examples, the driving scene data or the feature embedding of the driving scene data is stored in the data store instead of the sensor data.

[0034] In some examples, the method further comprises, in response to determining that the difference condition is not satisfied, not storing the sensor data in the data store. In some examples, the method further comprises, in response to determining that the difference condition is not satisfied, allowing the sensor data to be overwritten or deleting the sensor data.

[0035] In some examples, the reference driving scene data is data associated with a requested driving scene and the second reference driving scene data is data associated with clustered predetermined driving scenes. Accordingly, the similarity comparison may be performed using a configurable driving scene, and the dissimilarity comparison may be performed using a cluster of predetermined driving scenes. In other words, for the dissimilarity comparison, it may be determined whether the driving scene data extracted from the sensor data is sufficiently different from an existing corpus of driving scene data. In some examples, for the dissimilarity comparison, it may be determined whether the driving scene data extracted from the sensor data is sufficiently different from all driving scene data in an existing corpus of driving scene data. In this example, determining whether the difference condition is satisfied may comprise determining whether a distance in feature embedding space between the feature embedding of the driving scene data and the feature embeddings of each driving scene in the clustered predetermined driving scenes is greater than ta predetermined minimum novelty distance. This allows for an effective determination of dissimilarity. For example, driving scene data may be triggered on if the driving scene data is sufficiently different from a large pre-existing collection of driving scene data. By using data associated with clustered predetermined driving data in the dissimilarity determination rather than the requested driving scene data (as used in the similarity comparison), it may be less likely that driving scene data which is different from the reference driving scene data but that is otherwise relatively commonly occurring (with reference to a greater range / amount of driving scene data) is stored. This advantageously improves the triggering mechanism to reduce the unnecessary storing of sensor data.

[0036] In some examples, the reference driving scene data is data associated with one or more requested driving scenes, and determining whether the similarity condition is satisfied comprises comparing the feature embedding of the driving scene data with one or more feature embeddings of the one or more requested driving scenes. In this example, determining, based on the comparison, whether the similarity condition is satisfied may comprise determining that a distance in feature embedding space between the feature embedding of the driving scene and any one of the feature embeddings of the one or more requested driving scenes is less than a predetermined maximum similarity distance. In this way, it can be possible to use more than one requested / reference driving scene for the similarity comparison, to increase efficiency of the search.

[0037] In some examples, the method may comprise receiving the reference driving scene data or a feature embedding of the reference driving scene data, and receiving the second reference driving scene data. Thus, the reference driving scene data (or feature embeddings thereof) may be configurable based on a user input.

[0038] In some examples, the second reference driving scene data comprises coordinates in feature embedding space of a centre or point associated with clustered predetermined driving scene data. Thus, the historical corpus of driving scene data may be efficiently summarised. Further, by using second reference driving scene data that comprises coordinates in a feature embedding space of a centre or point associated with clustered predetermined driving scene data, the efficiency of the dissimilarity determination is increased. For example, the autonomous vehicle may use a multi-dimensional feature embedding (e.g. a vector or matrix) representing the centre point of the cluster for the determination. This avoids comparisons with large numbers of other driving scenes. Further, the feature embedding representing the centre point of the cluster may be received and stored efficiently, reducing the storage and processing resources required for the dissimilarity comparison.

[0039] In some examples, obtaining sensor data from one or more sensors of an autonomous vehicle comprises obtaining sensor data from one or more sensors of the autonomous vehicle during a predetermined time period, and extracting, from the sensor data, driving scene data comprises aggregating the sensor data over the predetermined time period and extracting driving scene data from the aggregated sensor data. Hence, a predetermined time period, or collection window, may be utilised for collecting sensor data over a 'snap-shot' of time. This data may be aggregated and then used in the driving scene extraction. This allows a more detailed view of a driving scene to be collected as an increased amount of data from the sensors may be used in the driving scene extraction.

[0040] In some examples, one or more feature perception models may be used in the extraction of driving scene data from the aggregated sensor data to determine features present in the aggregated sensor data. The perception models may be one or more machine learning techniques trained to extract features from input sensor data, for example one or more neural networks trained to extract features from input sensor data. Each respective sensor data type (camera, RADAR, LiDAR, etc) may have an associated perception model for extracting features from the respective sensor data type.

[0041] In some examples, the sensor data is associated with a perceived environment of the autonomous vehicle and / or a driving or vehicle state of the autonomous vehicle. Thus, the obtained data may relate to the autonomous vehicle and its surroundings as the autonomous vehicle navigates a trajectory. This sensor data may provide a level of detail optimised for the driving scene data extraction. As discussed above, the sensor data may comprise sensor data obtained from one or more other autonomous vehicles.

[0042] In some examples, the sensor data is data captured by one or more of a camera, radar, and / or LiDAR sensor. The camera, radar, and / or LiDAR sensor may be an on-board sensor on-board the autonomous vehicle. Thus, the autonomous vehicle itself may collect the sensor data using on-board sensing systems. In addition or alternatively, the sensor data may be captured by one or more sensors (camera, RADAR, LiDAR, etc.) of one or more other autonomous vehicles and obtained by the autonomous vehicle. Thus, in these examples, the sensor data may be augmented with sensor data from other vehicles.

[0043] In some examples, storing the sensor data comprises storing the sensor data in on-vehicle memory storage and / or sending the sensor data over a network connection to a data store server. Thus, depending on the implementation, the data may be stored on the vehicle. This may allow for data collection when a network connection is not practical, for example, due to network connectivity requirements or being in a remote location. The sensor data may then be offloaded at a later time, for example using a network connection or by physically off-loading the data storage or transferring the data off-vehicle using a wired or wireless connection, for example.

[0044] Alternatively, the sensor data may be sent over a network connection to a data store server. In some examples, these techniques may be combined. For example, the sensor data may be stored on-vehicle for a first time period, and then sent over a network connection to a data store server in a subsequent time period. This allows flexibility in the approach depending on the storage, processing, and network connection limitations of the autonomous vehicle at a given time or for a given implementation.

[0045] In some examples, the similarity threshold and / or novelty threshold is configurable based on a sensor data collection time. This advantageously increases configurability and also collection efficiency depending on the live data being collected. For example, if the autonomous vehicle is experiencing a relatively large number of occurrences of sensor data relating to driving scenes similar to the reference driving scene (for example above a predetermined threshold number per time period), the threshold for similarity may be increased to reduce the number of scenes considered similar and thus reduce the amount of collected data considered similar. Further, if the number of driving scenes relating to the obtained sensor data considered dissimilar is below a predetermined threshold (for example a predetermined threshold number per time period), the threshold for considering a given scene dissimilar may be reduced. In this way, the data collection thresholds may be dynamically configured based on live scenes and conditions that the autonomous vehicle is experiencing. Hence, an improved (more flexible, more configurable, more efficient) triggering technique may be realised.

[0046] In some examples, the autonomous vehicle performs the steps of the computer-implemented method. For example, the autonomous vehicle performs the obtaining, extracting, determining, and storing steps. Thus, the autonomous vehicle may be provided with a data processing apparatus or circuitry configured to perform the steps of the method described herein. Accordingly, an autonomous vehicle is able to itself trigger data collection for potentially useful or significant driving scenes that the autonomous vehicle may experience while driving.

[0047] Viewed from a second aspect, there is provided an apparatus comprising: sensor data obtaining circuitry to obtain sensor data from one or more sensors of an autonomous vehicle; driving scene extracting circuitry to extract, from the sensor data, driving scene data; similarity process performing circuitry and / or dissimilarity process performing circuitry, wherein the similarity process performing circuitry is to determine whether a similarity condition associated with a similarity between the driving scene data and reference driving scene data is satisfied; and store the sensor data in a data store in response to determining that the similarity condition is satisfied, and wherein the dissimilarity process performing circuitry is to determine whether a difference condition associated with a difference between the driving scene data and second reference driving data is satisfied, and store the sensor data in the data store in response to determining that the difference condition is satisfied.

[0048] Viewed from a third aspect, there is provided a computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method as described above and herein.

[0049] Viewed from a fourth aspect, there is provided an autonomous vehicle comprising the apparatus described above and herein.

[0050] Other aspects will also become apparent upon review of the present disclosure, in particular upon review of the Brief Description of the Drawings, Detailed Description and Claims sections. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Examples of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:

[0052] Figure la: schematically illustrates an autonomous vehicle which may implement the present techniques.

[0053] Figure lb: schematically illustrates an apparatus according to the present techniques.

[0054] Figure 2: schematically illustrates steps for triggering the storing of sensor data as described herein.

[0055] Figure 3: schematically illustrates a driving scenario as described herein.

[0056] Figure 4: schematically illustrates driving scene data as described herein.

[0057] Figure 5: schematically illustrates steps for triggering the storing of sensor data as described herein.

[0058] Figure 6: schematically illustrates steps for triggering the storing of sensor data as described herein.

[0059] Figure 7: schematically illustrates steps for generating a feature embedding from driving scene data as described herein.

[0060] Figure 8: schematically illustrates a computing device that may implement the techniques described herein.

[0061] While the disclosure is susceptible to various modifications and alternative forms, specific example approaches are shown by way of example in the drawings and are herein described in detail. It should be understood however that the drawings and detailed description attached hereto are not intended to limit the disclosure to the particular form disclosed but rather the disclosure is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the claimed invention.

[0062] It will be recognised that the features of the above-described examples of the disclosure can conveniently and interchangeably be used in any suitable combination. DETAILED DESCRIPTION

[0063] Figure la shows an illustration of an autonomous vehicle 100 that may implement the techniques described herein. Vehicle 100 is depicted as comprising a first sensor 110, a second sensor 120, further sensors 130 and a computing device 140. It will be understood that, in some examples, sensors may be image sensors and that the second sensor 120 and further sensors 130 are optional and that a single sensor may be provided. It will be appreciated that the present techniques may be applied to any vehicle (unmanned, autonomous or otherwise) provided with one or more sensors as described herein.

[0064] In some examples, one or more of the first sensor 110, the second sensor 120, and the further sensors 130 is a camera. The camera uses visible and / or invisible light. Thereby performance may be enhanced in certain environmental conditions, for example, when rain or fog reduces visibility at certain electromagnetic wavelengths. In some examples, one or more of the image sensors may be a LIDAR system, a RADAR system, a SONAR system and / or a LASER scanning system. Thereby, higher spatial accuracy may be achieved and performance may be enhanced in certain environmental conditions, for example, when rain or fog reduces visibility at certain electromagnetic wavelengths. In the example of figure la, first sensor 110 is a camera sensor, second sensor 120 is a LiDAR sensor and further sensors 130 are RADAR sensors. Vehicle 100 may comprise further sensors (not shown) for sensing information relating to the state of the vehicle 100. For example, vehicle 100 may comprise one or more sensors for sensing information associated with / indicative of one or more of a speed, position, jerk, curvature, acceleration, wheel-turning amount, engine power, fuel or battery level, etc.

[0065] In the present example, computing device 140 is a general-purpose computer, for example of the form depicted in figure 8. In other examples, the computing device 140 may comprise a specialist computing component such as an ASIC or FPGA. Thereby power efficiency, speed and / or latency may be enhanced.

[0066] Computing device 140 may comprise one or more processors for executing and evaluating a driving policy (also referred to as a motion planner), and data storage for storing a trained driving policy and program instructions for evaluating the driving policy. It will be appreciated that computing device 140 may be loaded with a trained driving policy, and computing device 140 may be configured to control the vehicle 100 based on the output of the driving policy.

[0067] Indeed, vehicle 100 may collect sensor data from the sensors 110, 120, 130 and / or one or more sensors for sensing the state of the vehicle 100 and input the sensor data to the computing device 140. Computing device 140 may then execute a driving policy / motion planner based on the sensor input, and control the autonomous vehicle based on the output of the executed driving policy. For example, the driving policy may be a trained machine learning driving policy which takes as input sensor data (for example relating to the vehicle environment and vehicle state) and outputs data indicative of a trajectory for the autonomous vehicle to follow during a subsequent time period. The autonomous vehicle 100 may then be controlled to follow the determined trajectory, for example through control of one or more of an engine system or motor system of the vehicle, a turning system of the vehicle, and / or a braking system of the vehicle, etc.

[0068] It will be understood that autonomous vehicle 100 may comprise additional components and computing circuitry not shown in figure 1. It will be appreciated that the term "autonomous vehicle" may refer to: a self-driving vehicle, such as a car, a van, a lorry or other vehicle; or an aerial vehicle, for example a manned aerial vehicle, an unmanned aerial vehicle or a drone; or a robot, for example an industrial robot or a domestic robot.

[0069] Autonomous vehicle 100 may also be provided with a network communication apparatus (not shown) for communicating over one or more communication networks. Further, autonomous vehicle 100 may communicate with one or more other vehicles using one or more vehicle-to-vehicle communication protocols. Autonomous vehicle 100 may receive sensor data from other autonomous vehicles using the network communication apparatus, and may send sensor data to other autonomous vehicles in a similar way.

[0070] Figure lb shows an apparatus 1 that may implement the techniques as described herein. Apparatus 1, for example data processing apparatus 1, may form part of the computing device 140 of figure la or may otherwise be a separate computing device of autonomous vehicle 100 of figure la. Further, apparatus 1 may be of the form depicted in figure 8.

[0071] Apparatus 1 includes sensor data obtaining circuitry 2 to obtain sensor data from one or more sensors of an autonomous vehicle, for example sensors 110, 120, 130 of vehicle 100 and sensors for sensing a vehicle state of vehicle 100. Sensor data obtaining circuitry 2 may also obtain sensor data from one or more other autonomous vehicles. Apparatus 1 also includes driving scene data extracting circuitry 3 to extract, from the sensor data, driving scene data. Apparatus 1 also includes similarity process performing circuitry 4 to determine whether a similarity condition associated with a similarity between the driving scene data and reference driving scene data is satisfied and to store the sensor data in a data store (not shown) in response to determining that the similarity condition is satisfied. Apparatus 1 also includes dissimilarity process performing circuitry 5 to determine whether a difference condition associated with a difference between the driving scene data and second reference driving data is satisfied and to store the store sensor data in the data store (not shown) in response to determining that the difference condition is satisfied. It will be appreciated that apparatus 1 may be provided with either or both of the similarity process performing circuitry 4 and dissimilarity process performing circuitry 5.

[0072] Apparatus 1 may include additional circuitry not shown in figure lb. Apparatus 1 may include one or more processors. Further, apparatus 1 may include data storage or be connected to data storage for storing sensor data.

[0073] Figure 2 shows an example method 200 according to the present techniques. Method 200 may be performed by apparatus 1 of figure lb.

[0074] At step 201, sensor data from one or more sensors of an autonomous vehicle may be obtained. The one or more sensors may include one or more camera, radar and / or LiDAR sensors and so the sensor data may comprise camera, radar and / or LiDAR sensor data. The one or more sensors may comprise one or more sensors for sensing an environment of the vehicle, and / or a vehicle state of the autonomous vehicle as described herein. Thus, the sensor data may comprise data indicative of the vehicle state, such as jerk, curvature, position, speed, acceleration, braking, turning, etc. Further, in some examples, the step 201 may comprise receiving or determining the sensor data rather than directly obtaining the sensor data from the one or more sensors. The sensor data may be data directly sensed by the one or more sensors or the sensor data may be data indicative of sensor data sensed by the one or more sensors, i.e. the raw sensor data may be processed before being obtained at step 201. Further, the sensor data may comprise sensor data obtained by one or more other autonomous vehicles.

[0075] The sensor data may be collected continuously during operation of the autonomous vehicle, and the sensor data may be obtained at periodic time intervals. The sensor data, once obtained, may be stored in a buffer while the determination of whether to store the sensor data in a data store is made. The frequency of sensor data collection is not overly limited.

[0076] At step 202, driving scene data may be extracted from the sensor data. In some examples, a rulebased model may be used to extract driving scene data from the sensor data. For example, a speed of the vehicle may be compared against a predetermined threshold, or the presence of another road user in the sensor data may be determined in order to extract driving scene data. In other examples, one or more perception models may be used to extract features from the sensor data. The driving scene data may correspond to the features extracted from the sensor data by one or more perception models. In some examples, a different perception model may be used for different types of sensor data. For example, a camera feature extraction model may be used to extract features from camera data, a LiDAR feature extraction model may be used to extract features from LiDAR data, and / or a RADAR feature extraction model may be used to extract features from RADAR data.

[0077] In examples where obtaining sensor data comprises obtaining sensor data from one or more sensors during a time period or window, the extracting of step 202 may comprise aggregating the sensor data over the time period of the window and extracting the driving scene data from the aggregated sensor data. In some examples, sensor data values of respective sensor data may be averaged over the time period of the window. In other examples, one or more predetermined weightings factors may be applied to the sensor data to aggregate the sensor data. In other examples, the sensor data may be associated with a given point in time and so aggregation of the sensor data may not be performed.

[0078] At step 203, it may be determined whether a similarity condition associated with a similarity between the driving scene data and reference driving scene data is satisfied. In some examples, one or more predefined rules may be used for the determination. For example, the driving scene data may comprise data indicating a speed of the autonomous vehicle of 60 miles per hour, and the reference driving scene data may comprise data indicating a speed of 55 miles per hour. In this example, the similarity condition may be that the speed indicated by the driving scene data is greater than the speed indicated by the reference driving scene data, and so the similarity condition would be satisfied. Thus, the sensor data in this example may be stored.

[0079] In other examples, the determination of step 203 may be based on using a machine learning technique. For example, at step 203, a neural network trained to transform driving scene data into feature embeddings may be used to generate a feature embedding of the driving scene data from step 202. This feature embedding may then be compared to a feature embedding of the reference driving scene data and, based on this comparison, it may be determined whether the similarity condition is satisfied.

[0080] In some examples, a distance between the feature embeddings in a feature embedding space may be compared to a predetermined maximum similarity distance. Thus, in this example, the similarity condition may be that the distance is less than a maximum similarity distance. The distance may be a cosine distance between the feature embeddings in the feature embedding space. Thus, in this example, if the distance between the feature embeddings is less than the predetermined maximum similarity distance, the driving scene data may be determined to be similar to the reference driving scene data. The sensor data associated with the driving scene data may then be stored in the data store.

[0081] On the other hand, if the distance between the feature embeddings is greater than the predetermined maximum similarity distance, the driving scene data may be determined to not be similar to the reference driving scene data. The subsequent processing that may be performed in this case is described in more detail with reference to figure 5.

[0082] The predetermined maximum similarity distance may be configurable based on user input. Further, the value may change during operation of the autonomous vehicle, for example based on the frequency at which the similarity condition is satisfied during a given time period.

[0083] At step 204, the sensor data (or data indicative of the sensor data) is stored in a data store in response to determining that the similarity condition is satisfied. Thus, in some examples, when it is determined that one or more rules are satisfied (for example a speed-based rule) or that the distance in feature embedding space between the feature embeddings of the driving scene data and the reference driving scene data is below a predetermined maximum similarity distance, the sensor data may be stored. For example, the sensor data may be stored in long term storage, for example cloud storage.

[0084] Figure 3 shows an example driving scenario. Example driving scenario includes an autonomous vehicle or SDV 6 driving along a roadway 7. SDV 6 may correspond to the autonomous vehicle 100 of figure la, and may comprise the apparatus 1 of figure lb. As such, SDV 6 comprises sensor / perception circuitry (i.e. one or more sensors) for sensing data related to an environment of the SDV 6 and sensing data related to a state of the SDV 6. In this example, SDV 6 includes on-board camera, radar and LiDAR systems and on-board vehicle sensor systems for sensing the vehicle state, like those described in relation to figure la.

[0085] As shown in figure 3, also present in the driving scenario and in the opposite and oncoming lane of the roadway 7, is another vehicle 8. Vehicle 8 may be an SDV or a human-operated vehicle (or indeed any other road user such as a pedestrian, cyclist, motorcyclist etc.). SDV 6 may sense vehicle 8 using its on-board camera, radar and / or LiDAR systems. SDV 6 may sense other features of its environment as the SDV 6 traverses the roadway 7 along trajectory 11, for example a building 9 and a tree 10.

[0086] The sensor data collected by the sensor systems of the SDV 6 may be processed by one or more on-board processors, such as computing device 140 of figure la and / or apparatus 1 of figure lb. The sensor data may then be obtained by apparatus 1 of figure lb according to the present techniques.

[0087] From the sensor data, which comprises data indicative of the environment of the vehicle, driving scene data may be extracted. The driving scene data may comprise data associated with the other road users sensed by the vehicle, as well as the vehicle sensing the data. For example, driving scene data extracted from sensor data collected while the SDV 6 traverses the roadway 7 according to the example of figure 3 may comprise data indicating that the vehicle is traversing a single lane road and a car is in the oncoming lane. This data may also indicate the presence of building 9 and tree 10, for example.

[0088] In other words, in some examples, the driving scene data includes a representation of actors (i.e. cars and other road users) in a given driving scenario at a given point. Other driving scene data may for example include a number of road users, a type of road users, a relative location of the other road users etc. It will be appreciated that the driving scene data may take various forms. In some cases, the driving scene data may correspond to a relatively high-level description of a driving scenario, such as that shown in figure 3. In other cases, the driving scene data may include data of the exact positions of road users in a driving scenario.

[0089] Figure 4 shows an example representation of driving scene data. In this example, driving scene data 12 includes data relating to one or more vehicles 13 present on a roadway 14. The driving scene data may comprise data indicative of a graphical representation of a driving scene as shown in this figure, or may be a description of such a scene. In some examples, the driving scene data comprises data identifying one or more road users and their trajectories. Further, driving scene data may correspond to a snapshot in time of a driving scenario, thus including one or more road users, their relative positions, and a road layout. Driving scene data may also be represented by coordinates (i.e. numbers) in a driving scene space, and may also include associated timestamps. In some examples, driving scene data comprises sensor data (recorded by a vehicle), vehicle GPS data (of the vehicle), and vehicle controlling signals (of the vehicle), each associated with timestamps indicative of when the data was generated. In this way, the driving scene data may be used to reconstruct the driving scene that the vehicle (that recorded the driving scene data) experienced.

[0090] A technique for selectively triggering and storing sensor data will now be described with reference to figure 5. The method of figure 5 may be performed by apparatus 1 of figure lb, for example. It will be appreciated that while steps 503, 504, 505, 506 (the similarity processing) are shown as being performed before steps 507, 508, 509, 510 (the dissimilarity processing), the order may vary. Indeed, in some examples, steps 507, 508, 509, 510 may be performed before steps 503, 504, 505, 506 or may be performed instead of steps 503, 504, 505, 506, and vice versa.

[0091] At 501, sensor data is obtained. The sensor data may be sensor data collected by one or more sensor systems of the autonomous vehicle (or other autonomous vehicles) as described herein. The sensor data may then be processed by one or more perception models to extract one or more features (i.e. the driving scene data). These features may correspond to one or more road users identified by the sensor data, or other features identified by the sensor data, for example road layout, buildings, street furniture, traffic lights, road signs, and / or weather conditions. Thus, from one perspective, the perception models create a representation of the environment the vehicle is traversing using the various sensor data collected by the sensors as the vehicle traverses the environment.

[0092] At 502, the driving scene data / features extracted in 501 are input to a neural network trained to transform driving scene data into feature embeddings. At 502, feature embedding of driving scene data d is generated. The neural network used to generate the feature embeddings may be an autoencoder, and is described in greater detail with reference to figure 7.

[0093] At 503, the feature embedding of the driving scene data d is compared with a feature embedding of reference driving scene data, r, which may have been received or otherwise obtained (for example by user input). Reference driving scene data r may relate to a driving scene of particular interest or significance, and may be selected by a user to form the basis of the similarity trigger as described herein.

[0094] As shown in figure 5, a distance x in feature embedding space between the feature embedding of the driving scene data d and the feature embedding of the reference driving scene data r is determined. This distance may be measured using Euclidean distance, cosine or dot product, depending on implementation.

[0095] At 504, the distance x is compared to a predetermined maximum similarity distance. The predetermined maximum similarity distance may be configurable based on user input, and in some examples may be configurable based on the frequency of driving scene data being considered similar to the reference driving scene data in a given time period.

[0096] At 505, if the distance x is less than the maximum similarity distance, the similarity condition is determined to be satisfied and so the driving scene data is determined to be similar to the reference driving scene data. Thus, the sensor data (and / or driving scene data and / or data indicative of the sensor data) is stored in a data store.

[0097] At 506, if the distance x is not less than the maximum similarity distance, the similarity condition is not satisfied and so the driving scene data is determined not to be similar to the reference driving scene data. In some examples, the sensor data is not stored in the data store. In some examples, responsive to this determination, the sensor data may be overwritten in the sensor data buffer or actively deleted, depending on implementation, without the sensor data being stored in the data store. In some cases, active deletion and active overwriting is not performed, and instead the sensor data is overwritten in the buffer as a result of the buffer implementation.

[0098] In other examples, at step 506, the process instead continues to step 507. At 507, the feature embedding of the driving scene data d is compared to a feature embedding of second reference driving scene data s. In some examples as described herein, second reference driving scene data corresponds to a centre point of a cluster of driving scene data. This cluster of driving scene data may be maintained externally from the apparatus 1 and may comprise a corpus of historical driving scene data, from which clustering may be performed and statistical techniques may be applied to determine an average or centre point of a cluster. This cluster may represent a historical 'average' driving scene, from which outlier driving scene data may be determined. This outlier driving scene data may correspond to rare driving events which may advantageously be stored even though the driving scene data is not considered similar to the reference driving scene data. It will be appreciated that in this example, the feature embedding of the driving scene data d is compared against a single point associated with the feature embedding of the second reference driving scene data. However, in other examples, d may be compared against a plurality of feature embeddings of second reference driving scene data. For example, the second reference driving scene data may comprise a plurality of clusters, and so d may be compared against a plurality of points (i.e. feature embeddings) each associated with a different cluster.

[0099] As part of step 507, a distance y is determined between the feature embedding of the driving scene data d is and the feature embedding of second reference driving scene data s. As for step 503, this distance may be measured using Euclidean distance, cosine or dot product, depending on implementation.

[00100] At 508, the distance y is compared to a predetermined minimum novelty distance. The predetermined minimum novelty distance may be configurable based on user input, and in some examples may be configurable based on the frequency of driving scene data being considered dissimilar to the second reference driving scene data in a given time period.

[00101] At 509, if the distance y is greater than the minimum novelty distance, the difference condition is determined to be satisfied and so the driving scene data is determined to be significantly different to the second reference driving scene data. In other words, not only is the driving scene data different from the reference driving scene data, but the driving scene data is different from historical driving scene data too, and thus considered worth storing. Thus, the sensor data (and / or driving scene data and / or data indicative of the sensor data) is stored in a data store. This may include transferring the sensor data from a sensor data to a dedicated data store.

[00102] At 510, if the distance y is not greater than the minimum novelty distance, the difference condition is determined not to be satisfied and so the driving scene data is determined to not be significantly different to the second reference driving scene data. Thus, the sensor data is not stored in the data store. Accordingly, the sensor data may be overwritten in the sensor data buffer or actively deleted, depending on implementation, without the sensor data being stored in the data store.

[00103] Thus, not only may sensor data that is similar to a reference driving scene (i.e. one that may be of particular significance or interest) be collected and stored, but sensor data that differs sufficiently from existing data is also stored. This counter-intuitive approach of saving data similar to that which is requested (via the reference driving scene) but also data which is significantly different to data that is requested (via the second reference driving scene data) ensures that rare occurrences (for example infrequently occurring events experience by the autonomous vehicle) are still collected. Hence, the triggering technique is more configurable, flexible, and provides an improved sensor data storing technique.

[00104] A further example method will now be described with reference to figure 6. Steps 1 to 4 of this method may be performed by a computing device of the vehicle, for example by apparatus 1 of figure lb.

[00105] As shown in step 1, camera, LiDAR, and RADAR data is collected by sensors of the vehicle.

[00106] At step 2, features are extracted using one or more perception models. As shown, a perception model is used to extract features from a given type of sensor data. Temporal feature aggregation is then performed on these features. This includes aggregating the features over the sensor data collection time window. In some cases, the feature extraction and temporal feature aggregation is not performed sequentially, rather temporal features may be extracted directly from the sensor data.

[00107] At step 3, the feature embedding of the data segment (i.e. the result of the temporal feature aggregation) is computed. This may be performed using a neural network as described herein. The distance to the input example embedding is then computed. This input example may correspond to the reference driving scene data as described herein. The distance is then compared to a predetermined threshold, such as the maximum similarity threshold described herein, and if less than the threshold, the feature embedding is considered similar to the input example embedding and the process continues to step 5.

[00108] At step 5, the sensor data is stored in the data store designated as storing sensor data similar to the input example embedding.

[00109] If the distance is greater than the predetermined threshold, the feature embedding is considered not similar to the input example embedding and the process continues to step 4.

[00110] At step 4, the distance between the feature embedding and the existing cluster centres is computed, i.e. the second reference driving scene data. If the distance is greater than a predetermined threshold, i.e. the minimum novelty threshold, then the sensor data is stored in the data store. The data store may be designated for storing new sensor data, or may be the same data store as the similar sensor data.

[00111] At step 6, the data stored in the data store(s) may be uploaded to cloud storage, for example one or more network connected storage servers.

[00112] The artificial neural network for transforming driving scene data into feature embeddingswill now be discussed in more detail with reference to figure 7.

[00113] The neural network may be an encoder configured or trained to convert driving scene data into a feature vector such that similar driving scene data is encoded to numerically close feature vectors and driving scene data that is dissimilar are encoded to numerically far apart feature vectors. The input for the neural network may be the driving scene data, comprising data indicative of road users in a certain road layout for example. An example of driving scene data that may be used to train such an artificial neural network is shown in figure 4.

[00114] The neural network may be trained to minimise a reconstruction loss as shown in figure 7. Driving scene data is input to an encoder to generate a feature embedding in a feature embedding space. A decoder may then decode the feature embedding to reconstruct the original driving scene data. The encoder may therefore be trained to minimise a reconstruction loss. Another example is a combination of two PerceiverlO architectures to embed inputs of a variable input size into a single embedding. The resulting neural embeddings are multi-dimensional vectors. It will be appreciated that the neural network may be an artificial neural network implemented as hardware or software, or indeed emulated in software and thus considered hardware.

[00115] An example computing apparatus will now be described with reference to figure 8. Figure 8 schematically illustrates an example of a computing device 800 which can be used to implement teachings described above, for example methods of figures 2, 5, 6 and 7. In some examples, the computing device 800 may correspond to a computing device provided at the autonomous vehicle or robot (or SDV) described herein, which may be loaded with the trained driving policy as described herein. In addition, computing device 800 can, in some examples, correspond to computing device 140 of figure la and / or apparatus 1 of figure lb.

[00116] The computing device 800 has processing circuitry 810 for performing data processing in response to program instructions and data storage 820 for storing data and instructions to be processed by the processing circuitry 810. In some examples, the processing circuitry 810 includes one or more caches for caching recent data or instructions. It will be appreciated that Figure 8 is merely an example of possible hardware that may be provided in the computing device and other components may also be provided. For example, the device may include a dedicated sensor interface 830 for communicating with sensors. As another example, for some devices for which user interaction is expected, the device may be provided with one or more user input / output device(s) 840 to receive input from a user or to output information to a user. The computing device 800 may additionally or alternatively have a communications interface 850 for communicating with external devices. For example, communications interface 850 could use any of a range of different communication protocols, such as Ethernet, WiFi®, Bluetooth®, ZigBee®, etc. In some examples, the communications interface 850 can be used to retrieve information for the disclosed technique including the calibration value(s) and / or mapping information of the operating environment.

[00117] The techniques discussed above may be implemented within a data processing apparatus which has hardware circuitry as discussed in relation to figure lb. The techniques discussed above may be performed under control of a computer program executing on a computing device. Hence a computer program may comprise instructions for controlling a computing device to perform any of the methods discussed above. The program can be stored on a computer-readable medium. A computer readable medium may include non-transitory type media such as physical storage media including storage discs and solid state devices. A computer readable medium may additionally or alternatively include transient media such as carrier signals and transmission media. A computer-readable storage medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[00118] In the present application, the words "configured to..." are used to mean that an element of an apparatus has a configuration able to carry out the defined operation. In this context, a "configuration" means an arrangement or manner of interconnection of hardware or software. For example, the apparatus may have dedicated hardware which provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not imply that the apparatus element needs to be changed in any way in order to provide the defined operation.

[00119] The various embodiments described herein are presented only to assist in understanding and teaching the claimed features. These embodiments are provided as a representative sample of embodiments only, and are not exhaustive and / or exclusive. It is to be understood that advantages, embodiments, examples, functions, features, structures, and / or other aspects described herein are not to be considered limitations on the disclosure scope defined by the claims or limitations on equivalents to the claims, and that other embodiments may be utilised and modifications may be made without departing from the scope of the invention as defined by the claims.

Claims

20 01 251. A computer-implemented method comprising:obtaining sensor data from one or more sensors of an autonomous vehicle;extracting, from the sensor data, driving scene data;performing a similarity process, wherein the similarity process comprises:determining whether a similarity condition associated with a similarity between the driving scene data and reference driving scene data is satisfied;in response to determining that the similarity condition is satisfied, storing the sensor data in a data store; andin response to determining that the similarity condition is not satisfied, performing a dissimilarity process, wherein the dissimilarity process comprises:determining whether a difference condition associated with a difference between the driving scene data and second reference driving data is satisfied; andin response to determining that the difference condition is satisfied, storing the sensor data in the data store.

2. The computer-implemented method of any preceding claim, wherein determining whether the similarity condition is satisfied comprises:comparing the driving scene data and reference driving scene data; anddetermining, based on evaluating one or more predefined rules using the comparison, whether the similarity condition is satisfied.

3. The computer-implemented method of any preceding claim, wherein determining whether the similarity condition associated with the similarity between the driving scene data and reference driving scene data is satisfied is based on using a machine learning technique.

4. The computer-implemented method of any preceding claim, wherein determining whether the similarity condition is satisfied comprises:generating, using a neural network trained to transform driving scene data into feature embeddings, a feature embedding of the driving scene data;comparing the feature embedding of the driving scene data with a feature embedding of the reference driving scene data; anddetermining, based on the comparison, whether the similarity condition is satisfied.

5. The computer-implemented method of claim 4, wherein determining, based on the comparison, whether the similarity condition is satisfied comprises determining whether a distance in feature embedding space between the feature embedding of the driving scene data and the feature embedding of the reference driving scene data is less than a predetermined maximum similarity distance.

6. The computer-implemented method of any preceding claim, wherein determining whether the difference condition associated with the difference between the driving scene data and second reference driving data is satisfied is based on using a machine learning technique.20 01 257. The computer-implemented method of any preceding claim, wherein determining whether thedifference condition associated with the difference between the driving scene data and second reference driving data is satisfied comprises:generating, using a neural network trained to transform driving scene data into feature embeddings, a feature embedding of the driving scene data;comparing the feature embedding of the driving scene data with a feature embedding of the second reference driving scene data; anddetermining, based on the comparison, whether the difference condition is satisfied.

8. The computer-implemented method of claim 7, wherein determining, based on the comparison, whether the difference condition is satisfied comprises determining whether a distance in feature embedding space between the feature embedding of the driving scene data and the feature embedding of the second reference driving scene data is greater than a predetermined minimum novelty distance.

9. The computer-implemented method of claims 5 to 8, wherein the distance is a cosine distance.

10. The computer-implemented method of any preceding claim, wherein the reference driving scene data and the second reference driving scene data are different.

11. The computer-implemented method of any preceding claim, wherein the reference driving scene data is data associated with a requested driving scene and the second reference driving scene data is data associated with clustered predetermined driving scenes.

12. The computer-implemented method of any preceding claim, wherein obtaining sensor data from one or more sensors of an autonomous vehicle comprises obtaining sensor data from one or more sensors of the autonomous vehicle during a predetermined time period, and extracting, from the sensor data, driving scene data comprises aggregating the sensor data over the predetermined time period and extracting driving scene data from the aggregated sensor data.

13. The computer-implemented method of any preceding claim, wherein the sensor data is associated with a driving or vehicle state of the autonomous vehicle and a perceived environment of the autonomous vehicle.

14. The computer-implemented method of any preceding claim, wherein the sensor data is data captured by one or more of a camera, radar, and / or LiDAR sensor.20 01 2515. The computer-implemented method of any preceding claim, wherein storing the sensor data comprises storing the sensor data in on-vehicle memory storage and / or sending the sensor data over a network connection to a data store server.

16. The computer-implemented method of any preceding claim, wherein the similarity threshold and / or novelty threshold is configurable based on a sensor data collection time.

17. The computer-implemented method of any preceding claim wherein the autonomous vehicle performs the obtaining, extracting, determining, and storing steps.

18. An apparatus comprising:sensor data obtaining circuitry to obtain sensor data from one or more sensors of an autonomous vehicle;driving scene extracting circuitry to extract, from the sensor data, driving scene data;similarity process and dissimilarity process performing circuitry, wherein the similarity process performing circuitry is to:determine whether a similarity condition associated with a similarity between the driving scene data and reference driving scene data is satisfied; andstore the sensor data in a data store in response to determining that the similarity condition is satisfied, andwherein the dissimilarity process performing circuitry is to, in response to the similarity process performing circuitry determining that the similarity condition is not satisfied:determine whether a difference condition associated with a difference between the driving scene data and second reference driving data is satisfied; andstore the sensor data in the data store in response to determining that the difference condition is satisfied.

19. A computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method of claims 1 to 17.

20. An autonomous vehicle comprising the apparatus of claim 18.20 01 25

Images