Method for collecting data for subsequent training of object detection model

Abstract

The invention relates to a method for collecting data for subsequent training of an object detection model, a computer-implemented method (100) performed by a vehicle comprising: in response to detecting an evasive action by the vehicle or other road user, acquiring (S102) sensor data relating to a scene in which the evasive action is detected; determining (S104), by an object detection model, whether an object is detected in the scene based on the sensor data; and in response to not detecting the object, storing (S106) the sensor data for subsequent training of the object detection model. The invention further relates to a method performed in a server.

Description

Technical Field

[0001] The inventive concept relates to the field of autonomous vehicles. In particular, the invention relates to a method and apparatus for collecting data for subsequent training of an object detection model. Background Art

[0002] In recent years, with the development of technology, image capture and processing technology has been widely used in different technical fields. In particular, vehicles produced today are usually equipped with some form of vision or perception system to realize new functions. In addition, more and more modern vehicles have advanced driver assistance systems (ADAS) to improve vehicle safety and more general road safety. ADAS (for example, it can be represented by adaptive cruise control (ACC), collision avoidance system, forward collision warning, lane support system, etc.) is an electronic system that assists the driver of the vehicle. Today, research and development are being carried out in many technical fields associated with the field of ADAS and the field of autonomous driving (AD). ADAS and AD can also be mentioned under the general term automated driving system (ADS), which corresponds to all different automation levels defined by SAE J3016 driving automation levels (0-5), for example.

[0003] An important task for ADAS and AD systems is to detect obstacles on the road, which today is usually done by performing object detection in images captured by the vehicle based on trained machine learning models. One of the key challenges in developing systems for object detection is collecting large and representative datasets.

[0004] With the machine learning algorithms available today, people can be relatively confident that good performance can be achieved in detecting those types of objects that have good support in the data set. Examples such as other vehicles, pedestrians, cyclists, traffic cones, etc. are easy to obtain because these types of objects are often encountered in driving scenarios. However, for less common objects with little or no data, this is not the case. Examples of such objects are waste or other types of unexpected obstacles (such as road damage) on the road. The basic characteristic of such objects is that they can appear in any way, shape or form, and the number of occurrences is extremely irregular. Due to the wide variety and rarity of waste or other unexpected obstacles, training object detection models to detect all these objects well is a challenging and time-consuming task at the moment. For example, a person may drive hundreds or even thousands of kilometers without even encountering any waste.

[0005] Therefore, improvements are needed when it comes to the development of object detection models for identifying waste or other unexpected obstructions, especially when it comes to collecting training data for such objects. Summary of the invention

[0006] The technology disclosed herein seeks to mitigate, alleviate or eliminate one or more of the above-mentioned deficiencies and shortcomings in the prior art to address various problems associated with collecting data on rare or unexpected objects to develop autonomous or semi-autonomous vehicles.

[0007] In order to improve the performance of autonomous driving systems, and in particular the performance of object detection models of said systems, as much data as possible should be collected and labeled for use in the training of object detection models. As highlighted above, the fundamental characteristics of waste or other types of unexpected obstacles, namely their diversity and rarity, make this a very challenging task.

[0008] The inventors have realized a new and improved way of collecting training data, particularly training data for such objects. The technology of the present disclosure provides a way of collecting data that can then be used for subsequent training of an object detection model. The inventive concept is based at least in part on the recognition that unexpected evasive maneuvers performed by a vehicle can be used as a predictive signal that an object is present. This is the first metric that the technology is built on. In addition, if the object detection model fails to detect an object in a scene where an evasive maneuver has occurred, it can be used as a predictive signal that there is some type of waste or unexpected obstruction at that location, which the object detection model has never encountered before in the current training data set. This is the second metric that the technology is built on.

[0009] Various aspects and embodiments of the disclosed invention are defined below and in the accompanying independent and dependent claims.

[0010] According to a first aspect, a computer-implemented method performed by a vehicle is provided for collecting data for subsequent training of an object detection model of an autonomous driving system. The method includes: in response to detecting an evasive maneuver of a vehicle or other road user, acquiring sensor data related to a scene in which the evasive maneuver was detected. The method further includes: determining, by the object detection model, whether an object is detected in the scene based on the sensor data. The method further includes: in response to not detecting the object, storing the sensor data for subsequent training of the object detection model.

[0011] A possible related advantage of the disclosed technology is that it provides an improved way of collecting training data for object detection models, which in turn facilitates safer and more capable object detection models because these models can be trained on a wider variety of objects. In particular, the present inventive concept provides an improved way of mining data for new or rare objects that may appear on the road, such as waste or other types of obstructing objects. The method can do this in an automated form, which allows such data to be extracted from large data streams. More specifically, the present technology can collect this data from real-world driving scenarios using a fleet of vehicles. Storing and manually browsing all of this data is not a viable alternative.

[0012] The techniques of the present disclosure can further increase the speed of collection of relevant data and / or annotated data. This can be achieved by the fact that the evasive action was detected, together with the fact that the object detection model failed to detect the object, can be used as a pre-annotation of the sensor data acquired for the scene. The pre-annotation can be an indication of the presence of new or rare objects in the scene that the object detection model has not yet been trained on. Therefore, this pre-annotation can also be used as an indication of the potential relevance of the data, in a sense that if the object detection model is trained on the data, its performance can be improved.

[0013] Furthermore, some embodiments of the disclosed technology provide improved accuracy and efficiency when processing potential training samples because it can rely on multiple detections of evasive maneuvers before taking action on potential training samples.

[0014] According to a second aspect, a computer program product comprising instructions is provided, which, when executed by a computing device, causes the computing device to perform a method according to any embodiment of the first aspect. According to an alternative embodiment of the second aspect, a (non-transitory) computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a processing system, the one or more programs comprising instructions for performing a method according to any embodiment of the first aspect. When applicable, the above-mentioned features of the first aspect also apply to the second aspect. In order to avoid unnecessary repetition, reference may be made to the above content.

[0015] According to a third aspect, a computer-implemented method performed by a server is provided for collecting data for subsequent training of an object detection model of an autonomous driving system. The method comprises: receiving data from a vehicle indicating that an evasive action has been detected although the object detection model did not detect an object in a scene where an evasive action of a vehicle or other road user was detected. The method further comprises: receiving sensor data related to the scene where the evasive action was detected. The method further comprises: storing the sensor data for subsequent training of the object detection model. When applicable, the above-mentioned features of the first and second aspects also apply to the third aspect. In order to avoid unnecessary repetition, reference may be made to the above content.

[0016] According to a fourth aspect, a computer program product comprising instructions is provided, and when the program is executed by a computing device, these instructions cause the computing device to perform a method according to any embodiment of the third aspect. According to an alternative embodiment of the fourth aspect, a (non-transitory) computer-readable storage medium is provided. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a processing system, and the one or more programs include instructions for performing the method according to any embodiment of the third aspect. When applicable, the above-mentioned features of the first aspect, the second aspect and the third aspect also apply to the fourth aspect. In order to avoid unnecessary repetition, reference may be made to the above content.

[0017] According to a fifth aspect, a vehicle is provided that is capable of collecting data for subsequent training of an object detection model configured to detect objects in the vehicle's surroundings. The vehicle includes one or more sensors. The vehicle further includes: a control circuit configured to acquire sensor data related to a scene in which an evasive action is detected in response to detecting an evasive action of the vehicle or other road user. The control circuit is further configured to determine whether an object is detected in a scene based on the sensor data by the object detection model. The control circuit is further configured to store the sensor data for subsequent training of the object detection model in response to no object being detected. When applicable, the above-mentioned features of the first to fourth aspects also apply to the fifth aspect. In order to avoid unnecessary repetition, reference may be made to the above content.

[0018] According to a sixth aspect, a server is provided for collecting data for subsequent training of an object detection model of an autonomous driving system. The server includes: a control circuit configured to receive the following data from a vehicle, the data being used to indicate that although the object detection model did not detect an object in a scene where an evasive action performed by a vehicle or other road user was detected, the evasive action has been detected. The control circuit is further configured to: receive sensor data related to the scene where the evasive action was detected. The control circuit is further configured to: store the sensor data for subsequent training of the object detection model. When applicable, the above-mentioned features of the first aspect to the fifth aspect also apply to the sixth aspect. In order to avoid unnecessary repetition, reference may be made to the above content.

[0019] According to the seventh aspect, a method for training an object detection model is provided. The method comprises: obtaining a training data set, the training data set comprising data for training an object detection model collected by a method according to any embodiment of the first aspect or the third aspect. The method further comprises: training an object detection model on the acquired training data set. The step of training the object detection model is performed in a server or in a vehicle. When applicable, the above-mentioned features of the first aspect to the sixth aspect also apply to the seventh aspect. In order to avoid unnecessary repetition, reference may be made to the above content.

[0020] As used herein, the term "non-transitory" is intended to describe computer-readable storage media (or "memory"), excluding propagating electromagnetic signals, but is not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrase "computer-readable medium or memory." For example, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible, computer-accessible storage medium may be further transmitted via a transmission medium or signals such as electronic, electromagnetic, or digital signals, which may be communicated over a communication medium such as a network and / or wireless link. Thus, as used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, rather than a signal), rather than a limitation on the persistence of data storage (e.g., RAM versus ROM).

[0021] The various aspects and preferred embodiments disclosed may be appropriately combined with each other in any manner obvious to a person skilled in the art, such that one or more features or embodiments disclosed with respect to one aspect may also be considered disclosed with respect to another aspect or embodiment of another aspect.

[0022] Further embodiments are defined in the dependent claims. It should be emphasized that when used in this specification, the term "comprises / comprising" is used to indicate the presence of the stated features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0023] These and other features and advantages of the disclosed technology will be further elucidated below with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The foregoing aspects, features, and advantages of the disclosed technology will be more fully understood by reference to the following illustrative and non-limiting detailed description of example embodiments of the present disclosure when used in conjunction with the accompanying drawings, in which:

[0025] Figure 1 is a schematic flow chart representation of a method performed by a vehicle for collecting data for subsequent training of an object detection model for an autonomous driving system according to some embodiments.

[0026] Figure 2 is a schematic flow chart representation of a method performed by a server for collecting data for subsequent training of an object detection model for an autonomous driving system according to some embodiments.

[0027] Figure 3 is a schematic flow chart representation of a method for training an object detection model according to some embodiments.

[0028] Figure 4 is a schematic illustration of a vehicle according to some embodiments.

[0029] Figure 5 is a schematic illustration of a server according to some embodiments.

[0030] Figure 6 is a schematic illustration of a distributed system for collecting data for subsequent training of an object detection model according to some embodiments.

[0031] Fig. 7A and Figure 7B By way of example, a first driving scenario and a second driving scenario in which an evasive maneuver is detected are schematically illustrated. DETAILED DESCRIPTION

[0032] The present disclosure will now be described in detail with reference to the accompanying drawings, in which some example embodiments of the disclosed technology are shown. However, the disclosed technology may be embodied in other forms and should not be construed as being limited to the disclosed example embodiments. The disclosed example embodiments are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will appreciate that the steps, services, and functions explained herein may be implemented using separate hardware circuits, using software that works in conjunction with a programmed microprocessor or general-purpose computer, using one or more application specific integrated circuits (ASICs), using one or more field programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0033] It will also be understood that when the present disclosure is described in the form of a method, the present disclosure may also be embodied in an apparatus including one or more processors, one or more memories coupled to the one or more processors, in which a computer code is loaded to implement the method. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the apparatus to perform the steps, services, and functions disclosed herein.

[0034] It should also be understood that the terms used herein are only used for the purpose of describing specific embodiments, and are not intended to be limiting. It should be noted that, as used in this specification and the appended claims, the terms "one", "an", "the" and "said" are intended to indicate the presence of one or more elements, unless the context clearly specifies otherwise. Therefore, for example, in certain contexts, references to "a unit" or "the unit" may refer to more than one unit. In addition, the words "comprise", "include", "contain" do not exclude other elements or steps. It should be emphasized that, when used in this specification, the term "comprises / contains" is used to specify the presence of the features, wholes, steps or parts. It does not exclude the presence or addition of one or more other features, wholes, steps, parts or their groups. The term "and / or" should be interpreted as "both" and each as the meaning of an alternative.

[0035] It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements or features, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the embodiments. The first element and the second element are both elements, but they are not the same element.

[0036] As used herein, the term "one or more" in a group of elements (e.g., "one or more of A, B, and C" or "at least one of A, B, and C") should be interpreted as conjunction or disjunction. In other words, it can refer to all elements, one element in a group of elements, or a combination of two or more elements. For example, the term "one or more of A, B, and C" can be interpreted as A or B or C, A and B and C, A and B, B and C, or A and C.

[0037] Current solutions for object detection in autonomous (or semi-autonomous) vehicles typically utilize some kind of deep learning algorithm. Throughout this disclosure, reference is made to an object detection model, which refers to a machine learning model based on such an algorithm (which may also be referred to as a machine learning algorithm and a neural network, etc.). More specifically, the "object detection model" herein refers to a computing system or algorithm designed to identify and locate instances of specific objects in digital images, video frames, laser radar (LIDAR) data, or other types of data. The model can use a combination of advanced techniques from computer vision, machine learning, and pattern recognition to analyze visual data and output a bounding box or region of interest around an object of interest present in an input image. The object detection model can be further configured to classify the type of object detected. The object detection model can cover different architectures, including but not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, and other existing or future alternatives.

[0038] The deployment of object detection models typically involves a training phase, in which the model learns from labeled training data to achieve accurate object localization and / or classification in the subsequent reasoning or detection phase. The training data (and input data during reasoning) can be, for example, images or image sequences, LIDAR data (i.e., point clouds), radar data, etc. In addition, the training / input data can include a combination or fusion of one or more different data types. The training / input data can include, for example, an image depicting an annotated object and a corresponding LIDAR point cloud containing the same annotated object.

[0039] In some embodiments, the machine learning model may be implemented using publicly available appropriate software development machine learning code elements, such as, for example, those available in Pytorch, TensorFlow, and Keras, or in any other appropriate software development platform, in any manner known to a person of ordinary skill in the art to be appropriate.

[0040] Figure 1is a schematic flow chart representation of a method 100 for collecting data for subsequent training of an object detection model for an automated driving system (ADS). Thus, the collected data may be used to train the object detection model. Hereinafter, for ease of reference, the term "data for subsequent training" will be referred to as "training data". Furthermore, although "data for subsequent training" is referred to as "training data", it should be understood that it may cover labeled (also referred to as annotated) data and / or data that has not yet been labeled / annotated. In the latter case, such data may be labeled / annotated in a subsequent step before being used for training the object detection model.

[0041] In particular, the method 100 can be used to detect training data of rare or unexpected objects in a road environment (such as waste or other types of obstacles or obstructions). The method 100 can be performed by a vehicle. In other words, the method 100 can be performed by a computing device provided locally in the vehicle (such as by an autonomous driving system of the vehicle). Figure 4 As used herein, a "vehicle" is any form of motor vehicle. For example, a vehicle can be any road vehicle, such as a car (e.g. Figure 4 as well as FIG. 7A to FIG. 7B ), motorcycles, (freight) trucks, buses, etc.

[0042] Below, the different steps of method 100 are described in more detail. Even though illustrated in a particular order, the steps of method 100 may be performed in any suitable order and may be performed multiple times. Figure 1 The specific order of method steps can be shown, but the order of these steps can be different from that depicted. In addition, two or more steps can be performed simultaneously or partially simultaneously. For example, the steps represented as S106 to S116 can be performed in any order or at any time point based on a specific implementation. This variant will depend on the selected software and hardware system and on the designer's choice. All these variants are within the scope of the present invention. Similarly, software implementation can be completed with standard programming techniques with rule-based logic and other logics to complete various steps. Other variants of method 100 will become apparent according to the present disclosure. The above-mentioned and described embodiments are only given as examples and should not be limited to the present invention. Other solutions, uses, purposes and functions within the scope of the present invention as claimed in the patent claims described below should be apparent to those skilled in the art.

[0043] The method 100 includes, in response to detecting an evasive maneuver by a vehicle or other road user, acquiring ( S102 ) sensor data related to a scenario in which the evasive maneuver was detected.

[0044] The scene can be understood as the general area where the evasive maneuver was performed. In other words, the scene can be considered the physical environment where the evasive maneuver was detected. Therefore, the scene includes any potential objects that may cause evasive maneuvers, that is, any objects that the vehicle (or driver) tried to avoid by performing evasive maneuvers.

[0045] Sensor data is related to the scene because the sensor data reflects one or more attributes of the scene. In other words, the sensor data includes information related to the scene. The sensor data related to the scene can be any type of data related to the scene. For example, the sensor data may include one or more of image data, LIDAR data, radar data, and ultrasonic data. In this article, image data should be understood as one or more images used to depict the scene where the evasive action is performed. Therefore, the image can depict any object present in the scene. One or more images may constitute an image sequence. In other words, each image in the image sequence may depict the scene at different time points and / or from different perspectives. Therefore, the image sequence may be understood as a frame sequence of a video of the scene. Alternatively, one or more images may constitute a plurality of images captured by at least two different cameras. Therefore, a plurality of images may depict the scene from different perspectives. The sensor data may further include information related to the evasive steering action, such as inertial measurement data or the path of the evasive action.

[0046] The term "evasive action" as used in the context of this patent application refers to the movement action taken by a vehicle to avoid a collision, potential obstacle, obstruction or dangerous situation, etc. An evasive action can be, for example, a sudden deviation from the current direction of travel. An evasive action may involve dynamic adjustment of the vehicle's speed (i.e., by increasing speed or braking) and / or dynamic adjustment of its trajectory. In some embodiments, the evasive action is an evasive steering action. In other words, the evasive action can be achieved by adjusting the trajectory of the vehicle.

[0047] An evasive action may be an action that deviates from the expected trajectory of the vehicle or other road users. In other words, the deviation of the vehicle from the expected trajectory of the vehicle or other road users may be detected as an evasive action. The expected trajectory herein refers to the predicted or planned path or movement that the vehicle is expected to follow in a given environment or driving scenario. The expected trajectory may be determined based on the machine learning-based trajectory prediction of the self-vehicle. The expected trajectory may be formed based on various factors of the vehicle, including but not limited to sensor data, map data or position data, vehicle dynamics, planned routes, behavior predictions, and real-time decision algorithms. Sensor data may provide information about the vehicle's surroundings, i.e., how the vehicle perceives its surroundings. The vehicle's surroundings should be understood as a general area around the vehicle, in which objects (such as other vehicles, landmarks, obstacles, etc.) may be detected and identified by vehicle sensors (RADAR, LIDAR, cameras, etc.), i.e., within the sensor range of the vehicle. Therefore, sensor data may provide information about the location of other vehicles, pedestrians, obstacles, road conditions, etc. It may further provide information about the lane direction or road direction in which the vehicle is traveling. Map data or location data can provide additional information about how the vehicle can be expected to navigate. Vehicle dynamics can provide information about how the vehicle is driven, including speed, acceleration, etc. The planned route can be based on the vehicle's destination and provide information about the route and / or lane along which the vehicle is expected to drive. Behavior predictions can be based on driving scenarios, such as driving along a highway, for example. In this case, it can be expected that the vehicle will travel at a certain speed and along certain lanes, so any deviation from these can indicate the presence of evasive action. Behavior predictions can also be related to the driver's driving characteristics. Real-time decision-making refers to how the vehicle (driver or autonomous driving system) reacts to sudden changes in the driving scenario, which may result in changes in the vehicle's expected trajectory, such as a planned lane change. Some examples of how to determine the expected trajectory and / or how to detect evasive actions are described above. In the following, some further examples will be given.

[0048] The expected trajectory may be based on the vehicle's current lane. More specifically, the expected trajectory may be for the vehicle to continue traveling in the current lane. Thus, if the vehicle deviates from the center of the lane, crosses a lane marker, or temporarily changes lanes (e.g., switches to an adjacent lane and then returns to the previous lane), an evasive maneuver may be detected. In another example, the expected trajectory may be a planned lane change, such as indicated by the driver using a turn signal or indicated by a planned path of an automated driving system. If the planned lane change is terminated, an evasive maneuver may be detected.

[0049] In summary, the vehicle's expected trajectory can be viewed as a dynamic and calculated path that takes into account several factors related to the vehicle.

[0050] In some embodiments, the expected trajectory is formed based on a lane estimation model. A lane estimation model herein refers to a computational algorithm or system configured to detect and track the location and boundaries of lanes on a road. The lane estimation model can be part of the perception module of an autonomous vehicle as it enables the vehicle to understand its position within the lane and make appropriate decisions for navigation and control. Thus, if the vehicle deviates from the lane as predicted by the lane estimation model, an evasive action can be detected.

[0051] In another example, a sudden or abrupt change in lateral movement of the vehicle may be detected as an evasive action. Thus, detecting an evasive action may be based on lateral movement of the vehicle or other road user. The lateral movement of the vehicle or other road user may be, for example, lateral acceleration and / or lateral yaw rate of the vehicle. Thus, a peak or spike in lateral acceleration and / or lateral yaw rate may be detected as an evasive action.

[0052] As will be readily appreciated by those skilled in the art, other ways of forming an expected trajectory or detecting the evasive action of a vehicle other than those explained above are also possible. The technology disclosed herein is not limited to the examples given above, but also includes such other ways. Fig. 7A and Figure 7B Illustrative examples are further given.

[0053] The evasive action may be caused by a driver input. In other words, the driver of the vehicle may perform an evasive action by adjusting a steering input, a gas input, and / or a brake input. Alternatively, the evasive action may be caused by a control system of the vehicle. For example, the ADS of the vehicle may perform an evasive action by adjusting a steering input, a gas input, and / or a brake input according to a determined control strategy. Subsequently, different subsystems of the ADS may detect the evasive action. Alternatively or in combination, the evasive action may be braking of the vehicle. The evasive action may be initiated by the vehicle driver or by an autonomous driving system.

[0054] As described above, the vehicle itself (also referred to as the ego vehicle) can detect the evasive maneuver. In this case, the evasive maneuver can be detected based on the vehicle's onboard sensors. The onboard sensors can collect sensor data about the vehicle itself, such as vehicle dynamics (e.g., direction, speed, acceleration, or other inertial measurement data). The onboard sensors can further collect data about sensors (such as cameras, LIDAR, radar, and / or ultrasonic sensors) about the vehicle's surroundings.

[0055] Alternatively, the ego vehicle may detect an evasive maneuver by another road user (such as another vehicle) observed by the ego vehicle. In this case, the ego vehicle detects the evasive maneuver based on sensors (such as cameras, LIDAR, radar, and / or ultrasonic sensors) configured to collect sensor data about its surroundings.

[0056] Continuing, the method 100 further includes determining (S104) whether an object is detected in the scene based on the sensor data by an object detection model. In other words, the acquired sensor data can be input into the object detection model, which determines whether an object exists in the scene. In this article, the existence of an object in the scene should be understood as the existence of an object at the location of the evasive action. In other words, the step represented by S104 can be understood as determining whether an object is detected that may cause an evasive action. In other words, it can be determined (S104) whether an object exists at a location that the vehicle (or driver) avoids by performing an evasive action. The object in this article refers to any type of obstacle that may exist on the road. In particular, the object can be any object that may pose a potential danger to the driver or other road users because it may hinder traffic, cause an accident or damage the vehicle. The object can be, for example, road debris. Examples of road debris include items such as garbage, fallen branches, rocks, tire fragments, fallen vehicle parts, road traffic accidents, construction materials, and other objects that are not usually part of the road infrastructure. In addition, the object can be damage to the road surface, such as potholes, cracks or sharp edges. Furthermore, objects may be puddles, snowbanks, or ice patches, etc., which the driver may want to avoid.

[0057] The method 100 further includes, in response to not detecting the object, storing (S106) the sensor data as training data for the object detection model. In other words, in the event that the object detection model fails to detect the object at the scene of the evasive action, the sensor data is stored as training data for the object detection model. The method may further include storing free space estimation data associated with the scene. The free space estimation data may provide further indications as to whether an object may be present at the scene of the evasive action. Therefore, in the event that the free space estimation data indicates that there is an obstruction at the scene of the evasive action, this may further increase the likelihood that the object actually exists, despite the object detection model failing to detect such an object. Therefore, this may be used to enhance the signal of sensor data used to depict rare or new objects that the object detection model has never seen before.

[0058] Free space estimation data in this article refers to the output of a so-called free space estimation algorithm, which is configured to detect whether there is an obstruction in front of the vehicle. In other words, the free space estimation algorithm can provide information about what is occupied space and what is unoccupied space around the vehicle.

[0059] The method may further include storing a timestamp associated with the detected evasive action. The timestamp may indicate the point in time when the evasive action was detected. The server may later use the timestamp to make decisions based on how long ago or recently the detected evasive action occurred.

[0060] The method 100 may further include acquiring (S108) additional sensor data related to the scene in which the evasive action is detected. The method may further include storing (S110) the additional sensor data as additional training data for the object detection model. The additional sensor data may be sensor data captured by the same vehicle but at different points in time. The sensor data and the additional sensor data may be separated in time according to a time frame or according to a longer time interval. Alternatively or in combination, the additional sensor data may be sensor data captured by the same vehicle but from different perspectives. Therefore, the additional sensor data may be related to scenes from different perspectives and / or at different points in time. Even further, the additional sensor data may be sensor data of a different type from previously acquired sensor data. Even further, the additional sensor data may be sensor data collected by different vehicles. Acquiring (S108) and storing (S110) the additional sensor data makes it possible to collect more training data for the same instance. In particular, by collecting training data for a scene at different points in time and / or from different viewpoints, improved training of the object detection model is provided, as it can learn to recognize objects in several different scenes.

[0061] Acquiring (S108) the additional sensor data and subsequently storing (S110) the additional sensor data as additional training data may be performed in response to, for example, receiving a request for such data from a server. Figure 5 Examples of such servers are described further below.

[0062] The method 100 may further include transmitting (S116) to the server sensor data related to the scene in which the evasive maneuver was detected. In other words, the sensor data stored as training data may be transmitted (S116) to the server. Transmitting (S116) the sensor data to the server may further include transmitting additional sensor data to the server. The vehicle may further transmit free space estimation data and / or a timestamp associated with the detected evasive maneuver to the server. The sensor data may be transmitted as raw data. Alternatively, the sensor data may be transmitted as fused data, i.e., after being processed in some manner.

[0063] Transmitting (S116) the sensor data may be performed in response to receiving a request from the server. Figure 2 Further describing the request sent from the server element. This can be advantageous because it provides more efficient data processing / transmission.

[0064] The method 100 may further include storing (S112) the geographic location of the scene where the evasive action was detected. In other words, the geographic location where the evasive action was performed may be stored (S112). The geographic location herein refers to any form of geographic location data, such as a location in a global coordinate system (e.g., GPS coordinates) or a location in a local coordinate system (e.g., a location on a map).

[0065] The method 100 may further include transmitting (S114) the geographic data to a server. Figure 6 As further explained, the transmitted geographic location may trigger the server to send a request for sensor data related to the scene in which the evasive action was detected. Subsequently, the vehicle may transmit the sensor data in response to receiving the request.

[0066] Optionally, executable instructions for performing these functions are included in a non-transitory computer-readable storage medium, or in other computer program products configured to be executed by one or more processors.

[0067] In general, a computer-accessible medium may include any tangible or non-transitory storage medium or memory medium, such as electronic, magnetic, or optical media (e.g., a disk or CD / DVD-ROM coupled to a computer system via a bus). As used herein, the terms "tangible" and "non-transitory" are intended to describe computer-readable storage media (or "memory"), excluding propagating electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the phrase "computer-readable medium or memory." For example, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may be further transmitted via a transmission medium or signals such as electronic, electromagnetic, or digital signals, which may be communicated via a communication medium such as a network and / or wireless link.

[0068] Figure 2 is a schematic flow chart representation of a method 200 performed by a server for collecting training data for an object detection model of an autonomous driving system according to some embodiments. The server may be communicatively connected to one or more vehicles that are capable of performing the above-described Figure 1 The method 100 described below is combined with Figure 5 Such a server is further described.

[0069] Below, the different steps of method 200 are described in more detail. Even though illustrated in a particular order, the steps of method 200 may be performed in any suitable order and may be performed multiple times. Figure 2 A specific order of method steps can be shown, but the order of these steps may be different from that depicted. In addition, two or more steps may be performed simultaneously or partially simultaneously. This variant will depend on the selected software and hardware system and on the designer's choice. All these variants are within the scope of the present invention. Similarly, software implementation can be completed with standard programming techniques with rule-based logic and other logics to complete various steps. The embodiments mentioned and described above are only given as examples and should not be limited to the present invention. Other solutions, uses, purposes and functions within the scope of the present invention as claimed in the patent claims described below should be apparent to those skilled in the art.

[0070] The method includes receiving (S202) data from the vehicle indicating that an evasive maneuver performed by the vehicle or other road user has been detected, although the object detection model has not detected an object in the scene where the evasive maneuver is detected. The received data may be, for example, an identification given by the vehicle indicating that an evasive maneuver has been detected and that the object detection model of the vehicle's autonomous driving system has not detected an object in the scene. In some embodiments, the data indicating that an evasive maneuver by the vehicle or other road user has been detected further includes a geographic location at which the evasive maneuver is detected. The data indicating that an evasive maneuver by the vehicle or other road user has been detected may further include a timestamp. The timestamp may indicate when the evasive maneuver was detected.

[0071] The method further includes receiving (S206) sensor data related to the scene in which the evasive action was detected. The sensor data may be received (S206) from the vehicle. In some embodiments, the sensor data is image data. In other words, the sensor data may be one or more images used to depict the scene. However, as previously described with reference to Figure 1 As explained, the sensor data may include one or more types of data, including, but not limited to, LIDAR data, radar data, and ultrasonic data. In response to transmitting (S204) a request for sensor data to the vehicle, sensor data related to the scene in which the evasive maneuver was detected may be received (S206). The method 200 may further include transmitting a request for additional sensor data related to the scene in which the evasive maneuver was detected. Thus, more potential training data for the scene may be collected.

[0072] The method further includes storing ( S208 ) the sensor data as training data for the object detection model.

[0073] In some embodiments, in response to the number of evasive actions detected at a geographical location within a defined geographical area exceeding a threshold, a request for transmitting (S204) sensor data is performed. In other words, the server can collect instances of evasive actions detected from one or more vehicles. The geographical locations of these instances can be stored by the server. Once the number of evasive actions associated with a geographical location falling within the defined geographical area exceeds a threshold, the server can transmit (S204) a request for sensor data to the vehicle. In the case where more than one evasive action occurs at approximately the same area by different road users, this can be used as a signal to indicate that there is an increased probability of any type of object or obstacle at the location. Therefore, by adopting a threshold value for the number of evasive actions detected before requesting sensor data, the reliability of the method (i.e., in the sense of collecting relevant training data) can be improved. In other words, the threshold value can be used to enhance the signal of an object present at the location. The threshold value can be a user-defined or predefined threshold value. In addition, the timestamp associated with each detected evasive action can be used to ignore any irrelevant detection of the evasive action, for example, an evasive action that is too early caused by the same obstructing object.

[0074] A request for sensor data related to the scene of the detected evasive maneuver can be transmitted (S204) to the vehicle that detected the evasive maneuver. In other words, the request can be transmitted (S204) to the vehicle that caused the threshold to be exceeded. Alternatively or additionally, the request can be transmitted (S204) to a subsequent vehicle that is about to pass through the geographical area. Therefore, the subsequent vehicle can be instructed to collect sensor data of the scene at the geographical area and transmit the collected sensor data to the server. Even further, before the threshold is exceeded, the server can transmit (S204) a request to one or more vehicles that detected the evasive maneuver at the geographical area. Subsequently, any vehicle that stores sensor data related to the scene can transmit these data to the server.

[0075] A geographical area may be understood as an area comprising a plurality of geographical locations. A geographical area may, for example, be limited to a specific area. In other words, a threshold may be exceeded when a certain number of evasive maneuvers are detected at geographical locations within a certain distance of each other. By using a geographical area, any inaccuracies in the geographical locations of evasive maneuvers may be appropriately handled. For example, in a real-world scenario, it is unlikely that two geographical locations of evasive maneuvers detected are exactly the same.

[0076] The method 200 may further include assigning (S210) annotation data to the sensor data. The annotation data may be data for specifying whether an object exists in the scene to which the sensor data belongs. The annotation data may further specify the type of the object. The annotation data may be, for example, a label for declaring the presence / absence of an object, or a label for declaring what type of object it is. The annotation data may further include bounding box data or segmentation data for specifying where the object exists in the sensor data (e.g., where in the image). The annotation data may be assigned based on user input. Thus, the annotation data may be manually generated data. In some embodiments, the annotation data may be generated based on an annotation algorithm different from an object detection model trained using training data.

[0077] The sensor data may be stored (S208) as training data for an object detection model in response to the annotation data indicating the presence of an object in the scene. In this case, even though the object detection model fails to detect the object, the annotation data may be used to verify the presence of the object at the scene. Furthermore, during training of the object detection model, the annotation data may be used as ground truth for the sensor data.

[0078] In response to the annotation data indicating that no object is present in the scene, method 200 may further include storing (S212) the sensor data as training data for a lane estimation model of the autonomous driving system. As explained above, evasive action may be detected based on the lane estimation model. Therefore, in the case where the annotation data indicates that no object is present, it may be assumed that the object detection model has correctly detected that no object is present at the scene. This situation may indicate that the evasive action was incorrectly detected, for example, due to a scenario in which the lane estimation model failed to correctly estimate the lane. Therefore, the sensor data related to the scene may be stored as training data for the lane estimation model, rather than training data for the object detection model. Therefore, method 100 may be described as method 100 for collecting training data for an object detection model or a lane estimation model of an autonomous driving system.

[0079] Optionally, executable instructions for performing these functions are included in a non-transitory computer-readable storage medium, or in other computer program products configured to be executed by one or more processors.

[0080] Figure 3 is a schematic flow chart representation of a method for training an object detection model according to some embodiments. In the following, the different steps will be described in more detail. As described above, throughout this disclosure, an object detection model should be understood as a machine learning model for detecting and / or classifying objects.

[0081] The term "training" in this article refers to the process of teaching a machine learning model to recognize patterns and relationships in a dataset, or more specifically, to detect / classify / recognize objects in images and / or other types of data. The training process can refer to training a new machine learning model from scratch. However, the process can also refer to retraining an existing machine learning model or fine-tuning an existing model.

[0082] Retraining a machine learning model refers to the process of updating an existing model with a new or updated dataset. Retraining a model can include training the model from scratch, where the training data includes new data (in this case, training data collected using the techniques described above). Alternatively, retraining a model can include training an already trained model only on new data. The purpose of retraining can be to improve the accuracy and generalization of the model for new, never-before-seen data or for rare data for which the model is not commonly seen.

[0083] Retraining may also be needed when the original training data is no longer representative of real-world data, or when the performance of the model has degraded due to changes in the data distribution. Retraining may also be needed when the model needs to be adapted to a new task or application.

[0084] The retraining process consists of several steps, similar to the initial training process. New data is added to the existing dataset, and the model is retrained using the updated data. Depending on how much the data has changed, the model may need to be modified or reconfigured before retraining.

[0085] Fine-tuning a machine learning model refers to the process of training a pre-trained model that has been trained for a different task or different data on new data. Fine-tuning can include training a portion of the pre-trained model (e.g., the last few layers), the entire model, or replacing or adding new layers and training only the new layers. The pre-trained model can be, for example, a general machine learning model that is fine-tuned to be suitable for a new task in a new domain.

[0086] According to the seventh aspect, a method for training an object detection model is provided. The method comprises: obtaining a training data set, the training data set comprising training data collected by the method according to any embodiment of the first aspect or the third aspect. The method further comprises: training an object detection model on the acquired training data set. The step of training the object detection model is performed in a server or in a vehicle. When applicable, the features of the first to sixth aspects mentioned above also apply to the seventh aspect. In order to avoid unnecessary repetition, reference may be made to the above content.

[0087] The method 300 includes obtaining (S302) a training data set, the training data set including the above combined Figure 1 The method 100 described above and / or the combination of Figure 2 The method 200 is described for collecting training data.

[0088] The term "obtain" is to be interpreted broadly throughout this disclosure and encompasses receiving, retrieving, collecting, and obtaining, etc., directly and / or indirectly, between two entities configured to communicate with each other or further with other external entities. However, in some embodiments, the term "obtain" should be interpreted as determining, deriving, forming, calculating, etc. In other words, obtaining a training data set may include collecting, determining, calculating, or generating a training data set, for example, by performing a combination of Figure 1 The method 100 described and / or in combination Figure 2 The method 200 described herein. Alternatively, the training data set may be received, for example, from another entity, or from a memory or database storing the training data set. Thus, as used herein, "obtaining" may indicate receiving a parameter at a first entity / unit from a second entity / unit or determining a parameter at a first entity or unit, for example based on data received from another entity / unit.

[0089] The method 300 further includes: training (S304) an object detection model on the acquired training data set. Training (S304) the object detection model can be done according to any suitable process that is readily understood by those skilled in the art. For example, the object detection model can be trained (S304) using supervised learning, where each instance of the training data set has an associated label that declares that an object exists and / or what type of object it is.

[0090] The step of training (3204) the object detection model may be performed on a server (such as a server that combines Figure 5 Alternatively, the step of training (S304) the object detection model may be performed in a vehicle (such as a server in combination with Figure 4 The following will be combined with Figure 6 Further describe the training of the object detection model.

[0091] The method 300 may further include transmitting the trained object detection model to the server if the trained object detection model is trained in the vehicle. Alternatively, the method 300 may further include transmitting the trained object detection model to the vehicle if the trained object detection model is trained in the server.

[0092] Optionally, executable instructions for performing these functions are included in a non-transitory computer-readable storage medium, or in other computer program products configured to be executed by one or more processors.

[0093] Figure 4 4 is a schematic illustration of an example of a vehicle 400 according to some embodiments. The vehicle 400 is equipped with an automated driving system (ADS) 410. As used herein, a "vehicle" is any form of motor vehicle. For example, the vehicle 400 may be any road vehicle, such as a car (as illustrated herein), a motorcycle, a (freight) truck, a bus, a smart bike, etc.

[0094] Vehicle 400 includes a number of elements that may be common in autonomous or semi-autonomous vehicles. It will be appreciated that vehicle 400 may have Figure 4 In addition, the vehicle 400 may include any combination of the various elements shown in FIG. Figure 4 Although various elements are shown herein as being located inside the vehicle 400, one or more elements may also be located outside the vehicle 400. In addition, even though various elements are depicted herein in a particular arrangement, various elements may also be implemented in different arrangements as will be readily appreciated by those skilled in the art. It should be further noted that various elements may be communicatively connected to each other in any suitable manner. Figure 4 The vehicle 400 should be considered as an illustrative example only, as the various elements of the vehicle 400 may be implemented in a number of different ways.

[0095] The vehicle 400 includes a control system 402. The control system 402 is configured to perform overall control of various functions and operations of the vehicle 400. The control system 402 includes a control circuit 404 and a memory 406. The control circuit 402 may physically include a single circuit device. Alternatively, the control circuit 402 may be distributed across several circuit devices. As an example, the control circuit 402 may share its control circuit 404 with other parts of the vehicle. The control circuit 402 may include one or more processors, such as a central processing unit (CPU), a microcontroller, or a microprocessor. One or more processors may be configured to execute program codes stored in the memory 406 to perform various functions and operations of the vehicle 400. The processor may be or may include any number of hardware components for performing data or signal processing or for executing computer codes stored in the memory 406. In some embodiments, the control circuit 404 or some of its functions may be implemented on one or more so-called systems on chips (SoCs). As an example, the ADS 410 may be implemented on the SoC. Memory 406 optionally includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices; and optionally includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 406 may include database components, object code components, script components, or any other type of information structure used to support various activities in this description.

[0096] In the illustrated example, the memory 406 further stores map data 408. The map data 408 may be used, for example, by the ADS 410 of the vehicle 400 in order to perform autonomous functions of the vehicle 400. The map data 408 may include high-definition (HD) map data. It is conceivable that the memory 408, even if illustrated as an element separated from the ADS 410, may be provided as an integral element of the ADS 410. In other words, according to some embodiments, any distributed memory device or local memory device may be used to implement the concept of the present invention. Similarly, the control circuit 404 may be distributed, for example, so that one or more processors of the control circuit 404 are provided as an integrated element of the ADS 410 or any other system of the vehicle 400. In other words, according to exemplary embodiments, any distributed control circuit device or local control circuit device may be used to implement the concept of the present invention.

[0097] The vehicle 400 further includes a sensor system 420. The sensor system 420 is configured to acquire sensory data related to the vehicle itself or its surroundings. The sensor system 420 may, for example, include a global navigation satellite system (GNSS) module 422 (e.g., GPS) configured to collect geographic location data of the vehicle 400. The sensor system 420 may further include one or more sensors 424. The one or more sensors 424 may be any type of on-board sensor, such as a camera, LIDAR and RADAR, an ultrasonic sensor, a gyroscope, an accelerometer, an odometer, etc. It should be understood that the sensor system 420 may also provide the possibility of acquiring sensory data directly or via a dedicated sensor control circuit in the vehicle 400.

[0098] The vehicle 400 further includes a communication system 426. The communication system 426 is configured to communicate with external units, such as other vehicles (i.e., via a vehicle-to-vehicle (V2V) communication protocol), remote servers (e.g., cloud servers, as described below in conjunction with Figure 5 Further explained), databases or other external devices (i.e., vehicle-to-infrastructure (V2I) or vehicle-to-everything (V2X) communication protocols). The communication system 426 can use one or more communication technologies to communicate. The communication system 426 may include one or more antennas. Cellular communication technology can be used for long-distance communication, such as to a remote server or cloud computing system. In addition, if the cellular communication technology used has low latency, it can also be used for V2V, V2I or V2X communication. Examples of cellular radio technologies include Global System for Mobile Communications (GSM), General Radio Packet Service (GPRS), Enhanced Data Rate for GSM Evolution (EDGE), Long Term Evolution (LTE), Fifth Generation Mobile Communication Technology (5G), 5G NR, etc., and also include future cellular solutions. However, in some solutions, medium- and short-range communication technologies, such as wireless local area networks (LANs), such as solutions based on IEEE 802.11, can be used to communicate with other vehicles near the vehicle 400 or with local infrastructure elements. The European Telecommunications Standards Institute (ETSI), which is developing cellular standards for vehicle communications, for example, considers 5G to be an appropriate solution due to its low latency and efficient handling of high bandwidth and communication channels.

[0099] The communication system 426 may further provide the possibility of sending the output to a remote location (e.g., a remote operator or a control center) via one or more antennas. In addition, the communication system 426 may be further configured to allow various elements of the vehicle 400 to communicate with each other. As an example, the communication system may provide a local network setup such as CAN bus, I2C, Ethernet, and fiber optics, etc. The local communication within the vehicle may also be of wireless type with protocols such as Wireless Fidelity (WiFi), Long Range Radio (LoRa), Zigbee, Bluetooth, or similar medium / short range technologies.

[0100] The vehicle 400 further includes a steering system 428. The steering system 428 is configured to control the steering of the vehicle 400. The steering system 428 includes a steering module 430 configured to control the driving direction of the vehicle 400. The steering system 428 further includes a throttle module 432 configured to control the actuation of the throttle of the vehicle 400. The steering system 428 further includes a brake module 434 configured to control the actuation of the brakes of the vehicle 400. The various modules of the steering system 428 can receive manual inputs from the driver of the vehicle 400 (i.e., from the steering wheel, the accelerator pedal, and the brake pedal, respectively). However, the steering system 428 can be communicatively connected to the ADS 410 of the vehicle to receive instructions related to how the various modules should act. Therefore, the ADS 410 can control the steering of the vehicle 400.

[0101] As described above, the vehicle 400 includes the ADS 410. The ADS 410 may be part of the vehicle's control system 402. The ADS 410 is configured to perform functions and operations of autonomous functions of the vehicle 400. The ADS 410 may include a plurality of modules, each of which is assigned a different function of the ADS 410.

[0102] The ADS 410 may include a positioning module 412 or positioning block / system. The positioning module 412 is configured to determine and / or monitor the geographic location and direction of travel of the vehicle 400 and may utilize data from a sensor system 420, such as data from a GNSS module 422. Alternatively or in combination, the positioning module 412 may utilize data from one or more sensors 424. However, the positioning system may alternatively be implemented as a real-time kinematic (RTK) GPS to improve accuracy.

[0103] The ADS 410 may further include a perception module 414 or a perception block / system. The perception module 414 may refer to any known module and / or function, such as included in one or more electronic control modules and / or nodes of the vehicle 400, adapted and / or configured to interpret sensory data related to driving of the vehicle 400 to identify, for example, obstacles, lanes, relevant landmarks, appropriate navigation paths, etc. Thus, the perception module 414 may be adapted to rely on and obtain input from multiple data sources, such as automotive imaging, image processing, computer vision, and / or in-vehicle networking, etc., and combined with sensory data from, for example, the sensor system 420.

[0104] The object detection model for detecting and / or classifying objects in the surrounding environment of the vehicle 400 may be part of the ADS 410, or more specifically part of the perception module 414. The vehicle 400 is configured to perform the functions of the method 100 for collecting training data for the object detection model. These functions may be implemented in a separate computing device provided in the vehicle. The computing device may include control circuits configured to perform the above-mentioned combined Figure 1 The steps of the method 100 are described. Alternatively, as will be readily appreciated by those skilled in the art, these functions may be distributed across one or more modules, systems, or components of the vehicle 400. For example, the control circuit 404 of the control system 402 may be configured to perform the steps of the method 100.

[0105] The positioning module 412 and / or the perception module 414 may be communicatively connected to the sensor system 420 to receive the sensing data from the sensor system 420. The positioning module 412 and / or the perception module 414 may further transmit control instructions to the sensor system 420.

[0106] The ADS may further include a path planning module 416. The path planning module 416 is configured to determine a planned path for the vehicle 400 based on the perception and position of the vehicle determined by the perception module 414 and the positioning module 412, respectively. The planned path determined by the path planning module 416 may be sent to the manipulation system 428 for execution. As explained above, the planned path may constitute an expected trajectory.

[0107] The ADS may further include a decision and control module 418. The decision and control module 418 is configured to execute control of the ADS 410 and make decisions. For example, the decision and control module 418 may determine whether the planned path determined by the path planning module 416 should be executed. The decision and control module 418 may be further configured to detect any evasive actions of the vehicle, such as deviations from the planned path or the expected trajectory of the path planning module 416. This includes evasive actions performed by the ADS 410 and by the vehicle driver. In addition, the perception module 414 may be configured to detect evasive actions of another vehicle or road user.

[0108] It should be understood that parts of the described solutions may be implemented in the vehicle 400, in a system located external to the vehicle, or in a combination of the inside and outside of the vehicle; for example, in a server in communication with the vehicle, a so-called cloud solution. The different features and steps of the various embodiments may be combined in other combinations than those described. In addition, the elements (i.e., systems and modules) of the vehicle 400 may be implemented in combinations different from those described herein.

[0109] Figure 5 is a schematic diagram of a server 500 for collecting training data for an object detection model of an autonomous driving system according to some embodiments. The server 500 may be configured to perform the following steps in conjunction with Figure 2 Method 200 is described.

[0110] For the purpose of this patent application, the server 500 as described herein (which may also be referred to as a remote server, cloud server, central server, back-end server, fleet server, or back-end server) refers to a computer system or networked device configured to provide various computing services, data storage, processing capabilities, or resources to clients or users through a communication network. In this case, the term "client" refers to a networked vehicle in a fleet (such as the vehicle 400 described above).

[0111] Even though the server 500 is illustrated herein as one device, the server 500 may be a distributed computing system formed by a plurality of different computing devices.

[0112] Server 500 includes control circuitry 502. Control circuitry 502 may physically include one single circuit device. Alternatively, control circuitry 502 may be distributed across several circuit devices.

[0113] like Figure 5As shown in the example of , the server 500 may further include a transceiver 506 and a memory 508. The control circuit 502 is communicatively connected to the transceiver 506 and the memory 508. The control circuit 502 may include a data bus, and the control circuit 502 may communicate with the transceiver 506 and / or the memory 508 via the data bus.

[0114] The control circuit 502 may be configured to perform overall control of the functions and operations of the server 500. The control circuit 502 may include a processor 504, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 504 may be configured to execute program code stored in the memory 508 to perform the functions and operations of the server 500. The control circuit 502 is configured to perform the above-mentioned combination Figure 2 The steps of method 200 are described. These steps may be implemented in one or more functions stored in memory 508.

[0115] The transceiver 506 is configured to enable the server 500 to communicate with other entities, such as vehicles or other servers. The transceiver 506 can transmit data to the server 500 and can receive data from the server 500.

[0116] The memory 508 may be a non-transitory computer-readable storage medium. The memory 508 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile storage, a random access memory (RAM), or another suitable device. In a typical arrangement, the memory 508 may include a non-volatile memory for long-term data storage and a volatile memory used as a system memory of the server 500. The memory 508 may exchange data with the circuit 502 via a data bus. There may also be accompanying control lines and an address bus between the memory 508 and the circuit 502.

[0117] The functions and operations of the server 500 may be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) that are stored on a non-transitory computer-readable recording medium (e.g., memory 508) of the server 500 and executed by the circuit 502 (e.g., using the processor 504). In other words, when the circuit 502 is described as being configured to perform a particular function, the processor 504 of the circuit 502 may be configured to execute a portion of the program code stored on the memory 508, where the stored portion of the program code corresponds to the particular function. In addition, the functions and operations of the device 502 may be independent software applications, or may form part of a software application for performing additional tasks associated with the device 502. The functions and operations described may be viewed as (e.g., in conjunction with the above) Figure 2The method 200 discussed above is described above, and the corresponding device is configured to perform the method. In addition, although the functions and operations described can be implemented in software, such functions can also be implemented via dedicated hardware or firmware or some combination of one or more of hardware, firmware and software. In the following, the functions and operations of the server 500 are described.

[0118] The control circuit 502 is configured to receive data from the vehicle indicating that an evasive maneuver performed by the vehicle or other road user has been detected despite the object detection model not detecting an object in the scene in which the evasive maneuver was detected.

[0119] The control circuit 502 is further configured to receive sensor data related to the scenario in which the evasive maneuver was detected.

[0120] The control circuit 502 is further configured to store the sensor data as training data for the object detection model.

[0121] It should be noted that, if the above is combined Figure 2 The principles, features, aspects and advantages of the described method 200 are also applicable to the server 500 described herein. To avoid unnecessary repetition, reference may be made to the above content.

[0122] The server 500 may be further configured to perform the steps of the method 300 for training an object detection model, such as the above combination Figure 3 Alternatively, other servers for executing the method 300 for training an object detection model may be provided according to the concept of the present invention. Other servers may have the same configuration as the server 500 described above.

[0123] Other servers (or the server 500 described above) may be configured to obtain a training data set, which includes the training data set obtained by combining the above Figure 1 The method described above is combined with 100 or more Figure 2 The training data collected by the described method 200. Other servers (or the server 500 described above) may be further configured to train an object detection model on the acquired training data set.

[0124] Figure 6 A distributed system 600 for collecting training data for an object detection model is illustrated by way of example. The system 600 may additionally or alternatively be a system 600 for training an object detection model. The system 600 may be viewed as an implementation of a non-limiting example of various aspects of the inventive concept disclosed herein. For example, the system 600 is configured to perform the above-described Figure 1The system 600 may further be configured to perform the above-described method 100. Figure 2 The system 600 may further be configured to perform the above combined Figure 3 The method 300 described herein. Therefore, unless otherwise stated, the above combined Figures 1 to 5 Any features or principles described are also applicable to the system 600 described herein.

[0125] System 600 includes server 602 (or remote server, cloud server, central server, backend server, fleet server or backend server), hereinafter referred to as remote server 602 or just server 602. Server 602 can be a combination of the above Figure 5 Therefore, the server 602 can be configured to perform the above combined Figure 2 The method 200 described above. The server 602 may be further configured to perform the above combined Figure 3 Described method 300. As illustrated, server 602 may be provided in the cloud (ie, as a cloud-implemented server).

[0126] The system 600 further includes one or more vehicles 604a-604c, also referred to as a fleet 604a-604c. The one or more vehicles 604a-604c may be a combination of the above Figure 4 Thus, one or more vehicles 604a-604c may be configured to perform the above combined Figure 1 The vehicle 604a-604c may further be configured to perform the following steps in conjunction with the method 100. Figure 3 Method 300 is described.

[0127] One or more vehicles 604a-604c are communicatively connected to a remote server 602 for transmitting and / or receiving data 606 between the vehicle and the server. One or more vehicles 604a-604c may be further communicatively connected to each other. Data 606 may be any type of data, such as a communication signal or sensor data. Communication may be implemented by any appropriate wireless communication protocol. The wireless communication protocol may be, for example, a remote communication protocol, such as a cellular communication technology (e.g., GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc.), or a short to medium range communication protocol, such as a solution based on a wireless local area network (LAN) (e.g., IEEE 802.11). The server 602 includes appropriate memory and control circuitry, such as one or more processors or processing circuitry and one or more other components (such as a data interface and a transceiver). The server 602 may also include a software module or other component so that the control circuitry may be configured to execute machine-readable instructions loaded from the memory to implement the steps of the method to be performed.

[0128] By way of example, Figure 6 The fleet illustrated in FIG. 6 includes three vehicles (first, second, and third vehicles 604a-604c). However, the system 600 may include any number of vehicles 604a-604c. Hereinafter, the system 600 will be described primarily with reference to the first vehicle 604a. It should be understood that these principles apply to any vehicle in the fleet.

[0129] In the following, an example of how the system 600 according to some embodiments performs the collection of training data for an object detection model will be described. For further details of the different steps, refer to the above Figure 1 and Figure 2 , to avoid excessive repetition.

[0130] In response to vehicle 604a detecting an evasive maneuver performed by vehicle 604a itself or by another road user (e.g., another vehicle in the convoy or a vehicle not belonging to the convoy), vehicle 604a acquires (i.e., corresponding to step S102) sensor data related to the scene in which the evasive maneuver was detected.

[0131] Subsequently, the vehicle 604a determines (ie, corresponding to step S104) whether an object is detected in the scene by the object detection model of the vehicle 604a. This can be done by inputting the sensor data into the object detection model.

[0132] If no object is detected, the vehicle 604a then at least temporarily stores (i.e., corresponding to step S106) the sensor data. The sensor data may be stored, for example, until a request for the sensor data is received or the storage reaches a certain amount of time. In addition, the vehicle 604 transmits (i.e., corresponding to step S114) data indicating that an evasive maneuver has been detected to the server. The data further includes the geographic location of the scene where the evasive maneuver was detected.

[0133] A server 602 in communication with a fleet of vehicles may receive (i.e., corresponding to step S202) such data indicating detected evasive maneuvers from one or more vehicles of the fleet. Thus, the server 602 may aggregate the received evasive maneuver detections to find out whether a certain number of evasive maneuvers have occurred in the same general area (i.e., in approximately the same geographic location). More specifically, in response to receiving the data indicating that an evasive maneuver has been detected, the server 602 may determine whether the number of evasive maneuvers detected at geographic locations within a particular geographic area exceeds a threshold. In the event that the threshold has been exceeded, the server 602 transmits (i.e., corresponding to step S204) a request for sensor data related to the scene in which the evasive maneuver has been detected.

[0134] As explained previously, the request to obtain sensor data may be transmitted to vehicle 604a or any other vehicle in the fleet that may have stored sensor data related to the scene or that may be able to obtain such sensor data.

[0135] In some embodiments, the server 602 may further receive a trajectory of the detected evasive maneuver. Subsequently, the server 602 may be further configured to determine an expected geographic location of the object and / or a zone in which the object may be present. Information about the geographic location and / or zone of the object may be used as annotation data for the object.

[0136] When the vehicle 604 a receives the request for sensor data, the vehicle 604 a transmits the sensor data to the server 602 (ie, corresponding to step S116 ).

[0137] The server 602 in turn stores any received sensor data (ie, corresponding to step S208 ) as training data for the object detection model.

[0138] The server 602 may further assign (i.e., corresponding to step S210) annotation data to the sensor data. In the case where the annotation data indicates that there is no object, the server may store (i.e., corresponding to step S212) the sensor data as training data for a lane estimation model of the autonomous driving system, rather than storing it as training data for an object detection model.

[0139] The above-described process of system 600 should be understood as a non-limiting example of the disclosed technology for improving understanding. Further variations are obvious in light of the present disclosure and are easily implemented by those skilled in the art. For example, in some embodiments, the data transmitted from vehicle 604a to server 602 for indicating that an evasive action has been detected may further include the acquired sensor data. Subsequently, server 602 may decide whether to store the sensor data as training data or discard the sensor data based on further evaluation (e.g., checking whether the number of evasive actions detected in the same geographic area exceeds a threshold).

[0140] The illustrated system as described above provides an efficient way to collect training data for, for example, new or rare objects that appear on the road (such as waste or other obstacles). The system is efficient in terms of time because a fleet of multiple vehicles is used to collect and transmit potential training samples. This can increase the number of training samples that can be collected in a particular time. System 600 may be more efficient in data management (e.g., transmission and storage) because it only needs to be transmitted from the vehicle to the server when the server explicitly requests sensor data (in response to finding sensor data of interest). A further advantage of system 600 is that the automatic collection of data can be made more reliable when multiple detected evasive actions are used in the same area as the signals of related objects present in the scene.

[0141] As described above, the system 600 may be further configured to perform the method 300 for training an object detection model. The training of the object detection model may be performed by so-called federated learning.

[0142] Federated learning is a scheme that integrates a "global" or "central" model from a "local" model trained at the edge. This is common in, for example, "voice assistants" and "typing models" on smartphones. In this context, the vehicle constitutes an edge device or node, and a remote server (or multiple remote servers) may constitute a central server that is responsible for aggregating or merging model updates from each of the edge devices. This will be referred to as a centralized federated learning scheme. However, in some embodiments, the need for a central entity may be omitted, and edge devices (e.g., a fleet of vehicles) may be configured to coordinate themselves to obtain a global model, a so-called decentralized federated learning scheme. In the following, a description is given mainly with reference to a centralized federated learning scheme. However, it is assumed that those skilled in the art will easily understand how to implement the teachings herein as a decentralized federated learning scheme, and that the scope conveyed by the invention disclosed herein covers such an implementation. It should be noted that in this context, the term "global" does not necessarily mean "full range", but should be interpreted as certain transactions being "shared" between multiple "local" entities.

[0143] In this example, training (or retraining) S304 an object detection model on a training data set may be performed in vehicle 604a, the training data set comprising training data collected by method 100 performed by the vehicle or method 200 performed by the server. The training data set may be obtained from a remote server 602 and collected by sensor data obtained by the vehicle itself or other vehicles in the fleet. Optionally, vehicle 604a transmits the trained (or retrained) object detection model to remote server 602. Remote server 602 may in turn distribute the trained object detection model to other vehicles in the system. Alternatively, remote server 602 may form an aggregated object detection model from two or more object detection models and then distribute it.

[0144] Alternatively, the training of the object detection model (S304) can be performed by the remote server 602. As described above, the remote server 602 can collect training data from the fleet, as explained above. Subsequently, the server 602 can distribute the trained object detection model to one or more vehicles 604a-604c of the system 600.

[0145] Fig. 7A A first driving scenario 700 in which an evasive maneuver is detected is schematically illustrated by way of example. More specifically, Fig. 7A The figure shows a scenario in which a vehicle 702 detects its own evasive action. The evasive action may be caused, for example, by the driver of the vehicle or by the vehicle's autonomous driving system.

[0146] The vehicle 702 is illustrated herein as traveling along a first lane 706a defined by a first lane marker 708a and a second lane marker 708b. An adjacent or second lane 706b is also illustrated and is defined by a second lane marker 708b and a third lane marker 708c. The vehicle 702 may be the vehicle 400 as described above. As explained above, the vehicle 702 includes one or more sensors, represented herein by a camera 704. In addition, the vehicle's expected trajectory 714 is illustrated as a dashed arrow along the center of the lane of the first lane 706a.

[0147] like Fig. 7A As further illustrated in FIG. 7 , object 710 is located in front of vehicle 702 in the direction of travel. Object 710 herein represents some kind of waste or other forms of obstacles on the road.

[0148] Herein, the position of the vehicle 702 at a later point in time is shown by the vehicle 702' in a dashed line diagram. In addition, the actual trajectory is shown by the dashed line diagram represented by 714'. The actual trajectory 714' represents the trajectory along which the vehicle 702' has actually traveled. The actual trajectory 714' herein deviates from the expected trajectory 714 of the vehicle 702. The deviation may be caused, for example, by the vehicle driver trying to avoid the object 710. Therefore, the evasive action of the vehicle can be detected based on the deviation of the vehicle 702 from the expected trajectory.

[0149] Fig. 7A 712a-712g of a plurality of previously detected evasive actions are further illustrated in FIG. These trajectories 712a-712g may be stored by a server. As explained above, a server may request sensor data related to a scene of a detected evasive action in response to the number of evasive actions detected at a geographic location within a geographic area exceeding a threshold. In this article, a geographic area is represented by a circle with a dashed line and represented by 716. However, it should be noted that the shape and size of a geographic area are not limited to the illustrated example. In the illustrated example, seven evasive actions are currently detected within a geographic area 716, namely, trajectories 712a-712g of previously detected evasive actions and an actual trajectory 714' of a vehicle 702. In the case of a threshold of 6, a vehicle 702 that performs (and detects) the seventh evasive action at a geographic area 716 may trigger a server request for sensor data of a scene in which an evasive action has occurred.

[0150] Figure 7B A second driving scenario 700' is schematically illustrated by way of example in which an evasive action is detected. More specifically, Figure 7BThe diagram illustrates a scenario in which the vehicle 702 detects an evasive maneuver by another road user. As with the first scenario 700, the second scenario 700' illustrates the vehicle 702 traveling along a first lane 706a. The other road user (illustrated herein as another vehicle 720) is illustrated as traveling along a second lane 706b. The expected trajectory 718 of the other vehicle 720 is illustrated by a dashed arrow. The expected trajectory 718 may be determined by the vehicle 702 as the trajectory along which the other vehicle 720 is expected to travel.

[0151] In the second scenario 700', an object 710 is present on the road ahead of another vehicle 720. The vehicle illustrated in dashed lines (represented herein as 720') represents the other vehicle 720 at a later point in time, at which the other vehicle 720' encounters the object 710. The actual trajectory 718' of the other vehicle 720', represented by dashed lines, is further illustrated.

[0152] In the case where the other road user 720 is within the sensor range of the vehicle 702, the vehicle 702 may detect the trajectory of the other vehicle 720 as an evasive action. Subsequently, the vehicle 702 may acquire sensor data of the scene where the evasive action was detected and continue to perform the method 100 as described above.

[0153] The present invention has been presented above with reference to specific embodiments. However, other embodiments in addition to the embodiments described above are also feasible and within the scope of the present invention. Within the scope of the present invention, method steps different from the method steps described above may be provided, and the method may be performed by hardware or software. Therefore, according to an exemplary embodiment, a non-transitory computer-readable storage medium is provided, which stores one or more programs, which are configured to be executed by one or more processors of a vehicle control system, and the one or more programs include instructions for performing a method according to any of the embodiments discussed above. Alternatively, according to another exemplary embodiment, a cloud computing system may be configured to perform any method presented herein. The cloud computing system may include distributed cloud computing resources that jointly perform the method presented herein under the control of one or more computer program products.

[0154] It should be noted that any reference signs do not limit the scope of the claims, that the present invention may be implemented at least partially by means of hardware and software, and that the same hardware item may represent multiple "means" or "units".

Claims

1. A method (100) for collecting data for subsequent training of an object detection model of an autonomous driving system, the method (100) being executable by a vehicle, comprising the following steps: In response to detecting an evasive action by the vehicle or other road user, acquiring ( S102 ) sensor data related to a scene in which the evasive action was detected; determining ( S104 ) by the object detection model based on the sensor data whether an object is detected in the scene; as well as In response to not detecting the object, storing ( S106 ) the sensor data for subsequent training of the object detection model.

2. The method (100) according to claim 1, further comprising: acquiring (S108) additional sensor data related to the scene in which the evasive action was detected; as well as The additional sensor data is stored ( S110 ) as additional data for subsequent training of the object detection model.

3. The method (100) according to claim 1 or 2, wherein: The evasive maneuver is a maneuver that deviates from the intended trajectory of the vehicle or the other road user.

4. The method (100) according to claim 3, wherein: The expected trajectory is formed based on a lane estimation model.

5. The method (100) of claim 1, wherein: Detecting the evasive maneuver is based on lateral movement of the vehicle or the other road user.

6. The method (100) of claim 1, further comprising: The sensor data related to the scene where the evasive action is detected is transmitted (S116) to a server.

7. The method (100) of claim 1, further comprising: The geographical location of the scene where the evasive action is detected is stored (S112).

8. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method (100) according to claim 1.

9. A method (200) for collecting data for subsequent training of an object detection model of an autonomous driving system, the method (200) being executable by a server, comprising the following steps: receiving ( S202 ) data from a vehicle indicating that an evasive action has been detected despite the object detection model not detecting an object in a scenario where an evasive action of the vehicle or other road user is detected; receiving ( S206 ) sensor data related to the scene in which the evasive action was detected; The sensor data is stored (S208) for subsequent training of the object detection model.

10. The method (200) according to claim 9, wherein: In response to transmitting (S204) a request for the sensor data to the vehicle, the sensor data related to the scenario in which the evasive maneuver was detected is received (S206).

11. The method (200) of claim 10, wherein: the data indicating that an evasive maneuver by the vehicle or the other road user has been detected further comprises a geographical location at which the evasive maneuver was detected, In response to a number of evasive actions detected at a geographical location within the defined geographical area exceeding a threshold, transmitting ( S204 ) a request for the sensor data is performed.

12. The method (S200) according to claim 9, further comprising: assigning ( S210 ) annotation data to the sensor data; wherein in response to the annotation data indicating the presence of an object in the scene, storing (S208) the sensor data for subsequent training of the object detection model is performed; and In response to the annotation data indicating that no object is present in the scene, the method (200) further comprises storing (S212) the sensor data for subsequent training of a lane estimation model of the autonomous driving system.

13. A computer program product comprising instructions which, when executed by a computing device, cause the computing device to perform the method (200) according to any one of claims 9 to 12.

14. A vehicle (400) capable of collecting data for subsequent training of an object detection model, the object detection model being configured to detect objects in the vehicle's surroundings, the vehicle comprising: one or more sensors (424); as well as The control circuit (404) is configured to: In response to detecting an evasive maneuver by the vehicle or other road user, acquiring sensor data related to a scene in which the evasive maneuver was detected; determining, by the object detection model based on the sensor data, whether an object is detected in the scene; as well as In response to not detecting the object, the sensor data is stored for subsequent training of the object detection model.

15. A server (500) for collecting data for subsequent training of an object detection model of an autonomous driving system, the server (500) comprising a control circuit (502), the control circuit (502) being configured to: receiving data from a vehicle indicating that an evasive maneuver has been detected despite the object detection model not detecting an object in a scenario in which an evasive maneuver by the vehicle or other road user was detected; receiving sensor data related to the scene in which the evasive maneuver was detected; as well as The sensor data is stored for subsequent training of the object detection model.

Images