Computer implementation method for continuously and adaptively detecting environmental features in autonomous and assisted driving of a self-driving vehicle

Abstract

The present invention relates to a method for continuously and adaptively detecting environmental features in autonomous and assisted driving of a self-vehicle connected to a server and a plurality of other vehicles. The self-vehicle and the other vehicles are provided with their respective machine learning models that are similar to one another. The method includes a data collection step by the self-vehicle, wherein sensor data acquired based on the self-vehicle's machine learning model, among other things, images, generate a specific scenario interpretation. When feedback is received that the scenario interpretation is incorrect, the corresponding image of the self-vehicle is encoded and hashed, and then transmitted to the server along with a request to the other vehicles to broadcast the corresponding image to the plurality of other vehicles and to transmit the specific similar scenario interpretation to the server. Then, the other vehicles identify a specific combination of environmental features corresponding to the incorrect scenario interpretation, acquire their own sensor data, and process it, including extracting, encoding, and hashing the image. The other vehicles compare the similarity between the image hashes received from the self-vehicle and each of the other vehicles (Vi) by a similarity score that exceeds a given threshold, and a high similarity score means that it corresponds to data that is similar in terms of structure and content. For image hashes of other vehicles that exceed a given threshold, the other vehicle identifies a specific similar scenario interpretation and transmits it to a server. The server aggregates sensor data from other vehicles corresponding to image hashes of the other vehicle and its own vehicle, progressively retrains the machine learning model of its own vehicle, and transmits the progressively retrained model together to its own vehicle. The present invention also relates to a trained machine learning model, a system for performing the method, and a plurality of non-transient computer-readable media.

Description

Technology Field

[0001] The present invention relates to a computer-implemented method used for continuously and adaptively detecting environmental features in autonomous and assisted driving of an ego-vehicle, comprising using a machine training model; a trained machine learning model; a system for performing the method; and a plurality of non-transient computer-readable media. Background Technology

[0002] As increasingly autonomous and intelligent modes of transportation become more common, there is a growing need for systems that enable vehicles to understand their surroundings better, more powerfully, and more comprehensively. The cornerstone of such systems is represented by Advanced Driver Assistance Systems (ADAS) sensors capable of reliably detecting, measuring, and classifying specific features of the vehicle environment (e.g., road elements).

[0003] Vehicle's Advanced Driver Assistance Systems (ADAS) sensors acquire various types of data used for the vehicle's advanced driver assistance functions, including images.

[0004] In this context, the latest developments in hardware and software have led to the widespread deployment of artificial intelligence, or machine learning technology, in advanced driving assistance sensor functions.

[0005] Training machine learning models used for advanced driving assistance sensor functions is generally based on generating multiple scenarios with the help of machine learning models.

[0006] From multiple scenarios, each specific scenario relates to a specific combination of environmental features. Unlimited examples of scenarios include straight roads with trees, straight roads without trees, tunnel entrances, tunnel exits, the interior of a tunnel when the road is straight, parking lots, curves with no change in elevation, and curves with change in elevation. By using a machine learning model, it is expected that the vehicle will be able to identify combinations of environmental features similar to the combinations of scenarios, based on a limited number of scenarios corresponding to the training data.

[0007] The output data of a machine learning model is an interpretation of the data obtained from the server that generates the scenario interpretation.

[0008] Each scenario then includes relevant instructions for the vehicle (V1) for the purpose of taking specific measures for driving safety. These instructions generally include messages for the driver when a person is in the vehicle, or messages for driving instructions when the vehicle is in an autonomous driving state. These instructions are outside the scope of the present invention.

[0009] Disadvantages of conventional technology

[0010] Although very powerful, one of the major challenges associated with machine learning models is training. During training, a series of examples is provided to the machine learning model, allowing it to automatically learn specific details of the features it is intended to extract (e.g., texture, shape, color, etc.). The most important aspect is that sufficient examples with high variability must be provided during training for the machine learning model to truly "understand" the relationship between specific features (e.g., the shape and wheels of a vehicle, the silhouette of a pedestrian, etc.) and how these specific features are identified. If not properly trained using sufficient and diverse examples, the machine learning model may extract attributes that fit the provided examples but are unrelated to the objects and phenomena and do not generalize well. This is called overfitting, and it is one of the most significant drawbacks affecting the quality and robustness of machine learning models.

[0011] For detection, classification, segmentation, and localization applications, training machine learning models requires manually marked data, resulting in enormous human effort to provide sufficient training examples. This is particularly true for deep neural networks, where complex and powerful architectures require training hundreds of thousands or millions of parameters. To train these networks, tens of thousands of training examples are needed to prevent overfitting and build robust machine learning models. This is impractical due to the limited availability of data and the human effort required for marking.

[0012] A much more efficient solution is to detect situations where machine learning models are likely to fail, such as those resulting in false positives, false negatives, or misinterpretation of scenarios due to incorrect classification, and to perform additional training using these samples or similar cases, commonly referred to as corner cases. This significantly improves the precision of the machine learning model using a relatively small amount of additional data.

[0013] As mentioned earlier, it is important to acquire these corner cases where the machine learning model is likely to fail, and to use them as feedback to retrain the model and significantly improve performance. These cases are determined not only by the limitations of the trained model but, perhaps more frequently, by environmental conditions. A few examples are as follows:

[0014] Difficult lighting conditions, such as those found in tunnels and indoor parking lots, can reduce the accuracy of camera-based machine learning models tasked with detecting and classifying lane markings, traffic signs, and traffic lights;

[0015] Bridges, guardrails, and other metal structures can negatively impact the performance of radar-based machine learning models when detecting and locating traffic participants and static environmental elements;

[0016] In addition, complex scenarios such as intersections and roundabouts can present challenges to machine learning models.

[0017] Currently, the retraining of machine learning models is being carried out incrementally at the vehicle level. Modifications to scenarios are performed manually by server operators with the processing power to do so, after which the correct data is transmitted to the vehicle and used to retrain its respective machine learning model.

[0018] The disadvantages of this approach are as follows:

[0019] - There are limitations to improving one's own vehicle's machine learning model after retraining using a single example:

[0020] - Improvements are slow because additional time is required for modifications, which are performed manually:

[0021] - In some cases, if there is no information available for the server operator to modify, the operator must take other measures that entail additional costs to find this information.

[0022] Problems to be solved by the present invention

[0023] For this reason, the technical challenge to be solved is not only to detect situations where such machine learning models fail, but also to extract sensor data corresponding to similar scenarios where the machine learning models fail, which can be analyzed later and used as additional training data for the vehicle's machine learning models.

[0024] Accordingly, the objective of the present invention is to provide a method for continuously and adaptively detecting environmental features in autonomous and assisted driving of a self-vehicle, so that when the self-vehicle detects such corner instances, it can automatically search for similar instances from other vehicles in order to resolve the shortcomings of the prior art, detect the correct scenario, and update the machine learning model of the self-vehicle.

[0025] The terms "retraining a machine learning model" and "updating a machine learning model" are used interchangeably in this invention.

[0026] Overview of the present invention

[0027] The subject of the present invention is, in a first aspect, a computer-implemented method for continuously and adaptively detecting environmental features in autonomous and assisted driving of a self-vehicle,

[0028] The vehicle is connected to a server, the server is connected to a plurality of other vehicles, the vehicle, other vehicles, and the server receive a machine learning model, and

[0029] The method is,

[0030] As a data collection step (I) performed by the data collection and processing unit of the vehicle,

[0031] - Acquire sensor data related to the above environmental characteristics from the surroundings of the vehicle using the vehicle's ADAS sensors;

[0032] - Interpret sensor data based on machine learning models to generate specific scenario interpretations, and

[0033] - Transmits related commands to the vehicle's execution electronic control unit based on the interpretation of a specific scenario, and

[0034] - If a specific scenario is misinterpreted and results in a misinterpreted specific scenario, receive feedback from the execution electronic control unit, and

[0035] - Extract camera images from acquired sensor data corresponding to incorrect interpretation of specific scenarios, and

[0036] - Encode and hash camera images corresponding to specific misinterpreted scenarios and obtain the self-vehicle image hash;

[0037] - A data collection step (I) of transmitting to a server the acquired sensor data and a self-vehicle image hash corresponding to a specific misinterpreted scenario, along with a request for other vehicles to identify a specific similar scenario by the vehicle;

[0038] As a data processing step (II) performed by the server,

[0039] - Identify specific combinations of environmental features corresponding to incorrect interpretation of specific scenarios by the vehicle, and

[0040] - Each of the multiple other vehicles,

[0041] - A specific similar scenario by the said vehicle based on the data image hash received from the vehicle, and

[0042] - Sensor data acquired regarding other vehicles in a similar scenario

[0043] Data processing step (II), which broadcasts a hash of one's own vehicle image to multiple other vehicles along with a server request to transmit it to the server;

[0044] As a data processing step (III) performed by the respective data collection and processing unit of other vehicles,

[0045] - Receive the self-vehicle image hash broadcast from the server and the server request;

[0046] - Upon a server request, sensor data is acquired from the surroundings of each of the other vehicles by the ADAS sensors of each of the other vehicles, and

[0047] - Processing sensor data acquired by other vehicles, including extracting, encoding, and hashing other vehicle camera images from acquired sensor data, and acquiring each other vehicle image hash, and

[0048] - The similarity between the self-vehicle and the image hashes received from each other vehicle is compared based on a similarity score exceeding a given threshold, and a high similarity score indicates that it corresponds to an image similar in terms of structure and content, and

[0049] - Based on the similarity score,

[0050] - Search for sensor data acquired by the vehicle and specific similar scenarios corresponding to image hashes where the similarity score exceeds a given threshold, and

[0051] - The following items on the server, namely

[0052] - A specific similar scenario by the said vehicle based on the data image hash received from the vehicle, and

[0053] - Sensor data acquired regarding other vehicles in similar scenarios

[0054] Data processing step (III) that transmits;

[0055] As a data processing step (IV) performed by the server,

[0056] - The following items, namely

[0057] - Magnetic vehicle sensor data corresponding to specific misinterpreted scenarios, and

[0058] - Sensor data acquired in relation to other vehicles for similar scenarios

[0059] Compare,

[0060] - Collects acquired sensor data received from one's own vehicle and other vehicles, and uses the collected sensor data to progressively retrain a machine learning model,

[0061] - Data processing step (IV) that transmits the progressively retrained model to the vehicle

[0062] It is a computer implementation method that includes

[0063] In a second aspect of the present invention, a training method for continuously and adaptively detecting environmental features in autonomous and assisted driving of a self-vehicle, performed by a server according to step (4) of any embodiment based on additional training data, wherein the additional training data is sensor data acquired in relation to each other vehicle corresponding to the incorrect scenario interpretation of the self-vehicle.

[0064] In a third aspect of the present invention, a trained machine learning model is presented, trained according to the method of any embodiment.

[0065] In the fourth aspect of the present invention, as a data processing system,

[0066] - My own vehicle,

[0067] - Server,

[0068] - Multiple other vehicles, and

[0069] - A communication network that connects one's own vehicle and other vehicles to the server.

[0070] Includes,

[0071] A data processing system is presented, comprising means configured to perform a step of a method of any embodiment.

[0072] Finally, in a fifth aspect of the present invention, a plurality of non-transient computer-readable media for storing instructions are provided, wherein the instructions cause each of one or more server processors to perform the method of any embodiment when the instructions are executed by one or more respective processors of a server, a self-vehicle, and another vehicle.

[0073] Additional advantageous embodiments are the subject of dependent claims.

[0074] Advantages of the present invention

[0075] The main advantages of using the method according to the present invention are as follows:

[0076] - The degree of improvement in each vehicle's machine learning model through retraining is higher than current technology because sensor data containing a subset of images collected from multiple vehicles facing similar scenarios exists, resulting in much higher training data variability;

[0077] - On the one hand, the improvement time is significantly reduced due to the automation and time saving of the data acquisition process;

[0078] Financial resources are saved because, in addition to the measures described in the method of the present invention, there is no need to perform any other measures to obtain information for retraining the model;

[0079] - Efficiently allocates additional server computing resources and significantly improves the accuracy of machine learning models using a relatively small amount of additional data;

[0080] - This method advantageously enables the continuous improvement of machine learning-based detection and classification of static environmental elements and traffic participants, such as lanes, traffic signs, traffic lights, and road markings.

[0081] Further special features and advantages of the present invention may be obtained from the following description and the attached drawings. Brief explanation of the drawing

[0082] FIG. 1 is a schematic block diagram of a method according to the present invention. Figure 2a is a schematic diagram of a scenario in which a vehicle faces a corner case. Figure 2b is a schematic diagram of a scenario similar to the scenario in Figure 2a identified by one of the other vehicles (Vi). Specific details for implementing the invention

[0083] Now, referring to FIG. 1, an exemplary embodiment of the present invention is illustrated.

[0084] The method of the present invention is,

[0085] - Own vehicle (V1),

[0086] - Server,

[0087] - Multiple other vehicles (Vi) (where i=2,...n), and

[0088] - A communication network that connects each of the vehicle (V1) and other vehicles (Vi) to the server.

[0089] It is performed in a system that includes.

[0090] The vehicle (V1) and other vehicles (Vi) are equipped with Advanced Driver Assistance Systems (ADAS) sensors.

[0091] The server may be an all-in-one hardware server or a multi-component hardware server that is physically close or communicates within a communication network. The configuration of the server follows the prior art.

[0092] The own vehicle (V1), other vehicles (Vi), and the server are provided with the same machine learning model. If a person skilled in the art believes that minor adaptations to the machine learning model are necessary for each vehicle and / or server, such minor adaptations should be understood as falling within the scope of the "same machine learning model."

[0093] To facilitate understanding of the present invention, specific examples are considered with reference to FIGS. 2a and 2b. In the actual landscape of FIG. 2a, there is a tunnel entrance. For example, due to the characteristics of radar sensors, the vehicle recognition system may interpret radar reflection from the tunnel wall as the presence of another vehicle in its lane, which can lead to unwanted maneuvers such as sudden lane changes. The driver of the vehicle is then required to correct the actions of the autonomous driving system.

[0094] FIG. 2b illustrates a scenario similar to the one faced by vehicle (V1) that can be detected by other vehicles by the present invention, despite differences unrelated to the presence of bob-cats, vegetation, etc.

[0095] The actual landscape is considered dynamic, that is, the landscape changes continuously as the position changes while the vehicle is driving. Figure 2a shows the situation at a specific point in time for example. Based on this reality, it is necessary to provide a method for detecting environmental features continuously and adaptively, and for this, it is necessary to continuously and adaptively enrich the scenarios. Adaptation of scenarios is performed by using machine learning models on the server, the own vehicle (V1), and other vehicles (Vi).

[0096] The method according to the present invention has five steps.

[0097] The first step of the method is a data collection step (I) performed by the data collection and processing unit of the self-vehicle (V1).

[0098] The vehicle (V1) uses ADAS sensor data by ADAS sensors that understand cameras, lidar, radar, etc. The ADAS sensor data includes, but is not limited to, camera images acquired by a front camera, surround view camera system, etc., to capture environmental features around the vehicle (V1).

[0099] These sensor data are processed based on a machine learning model of the vehicle (V1) that can detect and classify traffic participants and environmental characteristics, namely environmental features such as lanes, traffic signs, traffic lights, road markings, etc.

[0100] The data collection and processing unit of the vehicle (V1) uses a machine learning model to interpret sensor data and generate a specific scenario interpretation.

[0101] In the examples of FIGS. 2a and 2b, the data collection and processing unit of the self-vehicle (V1) is expected to use a machine learning model to interpret sensor data acquired from the landscape of FIG. 2a according to the scenario shown in FIG. 2b, and then transmit relevant commands to the execution control unit of the self-vehicle (V1) according to the interpretation of a specific scenario.

[0102] The relevant command corresponds to a specific expected behavior of the vehicle (V1).

[0103] As an example of a related command, slowing down and turning on the light corresponds to specific expected behaviors of slowing down and turning on the light, respectively.

[0104] The integrity of sensor data processing (detection, classification, etc.) is continuously monitored and verified through the calculation of reliability and validity scores or system deactivation due to driver intervention. The verification itself, performed by interpreting driver actions, is not the purpose of the present invention.

[0105] When the execution control unit detects a corner case, that is, when it detects any data inconsistency such as machine learning failure, consistency check, high uncertainty, or driver departure, the execution control unit sends feedback to the data collection and processing unit of its vehicle (V1) indicating that the interpretation of a specific scenario is incorrect, which means that there is a specific scenario that has been misinterpreted.

[0106] Whenever a discrepancy occurs between the expected behavior and the actual behavior of the vehicle (V1), feedback is transmitted to the data collection and processing unit of the vehicle (V1).

[0107] After receiving feedback, the data collection and processing unit of the vehicle (V1) extracts a camera image from the sensor data, preferably from the front camera, and this camera image corresponds to a specific scenario that was misinterpreted.

[0108] Then, the extracted image is encoded and hashed. The corresponding image hash of the self-vehicle (V1) is transmitted to the server along with the acquired sensor data corresponding to the incorrect interpretation of a specific scenario, and a request made by the data collection and processing unit of the self-vehicle (V1) to the server for each other vehicle (Vi) to identify a specific similar scenario.

[0109] Taking the example of FIGS. 2a and FIGS. 2b, the image extracted from FIGS. 2a is encoded and hashed, and then transmitted to a server along with all other sensor data corresponding to the image. In this case, the request to the server is to "request" another vehicle (Vi) to identify a specific similar scenario that matches the image extracted from FIGS. 2a.

[0110] Each other vehicle (Vi) is provided with its own data collection and processing unit similar to the data collection and processing unit of its own vehicle (V1). Similarity between data collection and processing units is considered when the functions of their respective data collection and processing units are performed identically.

[0111] In the data processing step (II) performed by the server, a specific combination of environmental features corresponding to a specific scenario misinterpreted by the vehicle (V1) is first identified.

[0112] Then, the server [allows] each of the multiple other vehicles (Vi)

[0113] - A specific similar scenario by the vehicle (Vi) based on the data image hash received from the vehicle (V1),

[0114] - and

[0115] - Sensor data acquired in relation to other vehicles (Vi) in similar scenarios

[0116] Broadcasts the image hash received from its own vehicle (V1) along with a server request to send it to the server.

[0117] In the example of FIG. 2a, the server request is about the task of identifying a tunnel entrance in a specific scenario, and the other vehicle (Vi) must transmit sensor data related to a tunnel entrance with similar characteristics, i.e., an image hash of the other vehicle (Vi), along with all acquired sensor data corresponding to an image hash of the other vehicle (Vi) related to a tunnel entrance with similar environmental characteristics, i.e., a tunnel entrance with characteristics similar to those faced by the own vehicle (V1).

[0118] The data processing step (III) is performed by the respective data collection and processing units of other vehicles (Vi).

[0119] First, the data collection and processing unit of each other vehicle (Vi) receives the broadcasted hash image of its own vehicle (V1) and a server request from the server.

[0120] Then, the ADAS sensors of other vehicles (Vi) acquire sensor data related to a specific combination of environmental features from the surroundings of each of the other vehicles (Vi) in response to a request from the server. In this example, the sensor data of the other vehicles (Vi) acquires information about a tunnel entrance having characteristics similar to those of the request from the server. The duration of this step is pre-set and is, for example, one day or one week, depending on the number of available other vehicles, the content of the scenery, etc.

[0121] Then, the respective data collection and processing unit of the other vehicle (Vi) processes the acquired sensor data in real time, and this processing includes extracting a camera image from the sensor data, hashing the extracted image, and obtaining the image hash of each other vehicle (Vi).

[0122] Then, the respective data collection and processing unit of the other vehicle (Vi) compares the similarity between the image hashes received from its own vehicle (V1) with its own image hash based on a similarity score that exceeds a given threshold, and a high similarity score means that it corresponds to a scenario similar in terms of structure and content.

[0123] Broadcasts from the server are received by each of the other vehicles (Vi), but, for example, some of the multiple other vehicles (Vi) are driving in areas without such similar environments, for example, areas without tunnels, so not all of the multiple other vehicles (Vi) may be able to acquire images from their respective surroundings in response to the server request. Instead, each of the other vehicles (Vi) that have acquired images from their respective surrounding environments outputs their own image hash and applies a similarity score to the comparison between their own image hash and the image hash received from their own vehicle (V1).

[0124] Then, based on the similarity score, the respective data collection and processing unit of the other vehicle (Vi) searches for a specific similar scenario with the scenario acquired by said vehicle (Vi) corresponding to an image hash where the similarity score exceeds a given threshold.

[0125] The present invention applies when at least one of the other vehicles (Vi) can transmit the aforementioned acquired sensor data to a server.

[0126] At the end of step (III), the respective data collection and processing units of the other vehicles (Vi) are,

[0127] - A specific similar scenario by the vehicle (Vi) based on the data image hash received from the vehicle (V1), and

[0128] - Sensor data acquired in relation to other vehicles (Vi) for similar scenarios

[0129] Send to the server.

[0130] The data processing step (IV) is performed by the server.

[0131] At this stage, the server,

[0132] - Self-vehicle (V1) sensor data corresponding to the misinterpreted scenario of V1, and

[0133] - Sensor data acquired in relation to other vehicles (Vi) for similar scenarios

[0134] Compares.

[0135] Given the definition of the similarity score and the fact that the machine learning models of other vehicles (Vi) are identical, it is reasonable to expect that the specific similarity scenarios by the majority of the aforementioned other vehicles (Vi) in the previous step will be identical. In this example, this corresponds to the image shown in FIG. 2b.

[0136] When the server receives sensor data acquired by its own vehicle (V1) and another vehicle (Vi), the server aggregates the acquired sensor data received from the other vehicle (Vi) and its own vehicle (V1). This is performed using the server's machine learning model.

[0137] This is why, in step (II), the server requested the other vehicle (Vi) to transmit the sensor data acquired in relation to the other vehicle (Vi) image hash. In fact, the other vehicle (Vi) data hash is used to identify corner cases, whereas sensor data provides more information about environmental features than images alone; therefore, sensor data for corner cases is used to aggregate to progressively retrain a machine learning model.

[0138] The aggregation of acquired sensor data is performed using aggregation techniques suitable for the sensor data.

[0139] The collected sensor data is then used by the server to progressively retrain the machine learning model.

[0140] At the end of step (IV), the server transmits the progressively retrained model to its vehicle (V1).

[0141] Transmitting a progressively retrained machine learning model by a server is performed as a general method for transmitting updates to a self-vehicle (V1), including but not limited to over-the-air updates.

[0142] Then, in the data processing step (V) performed by the data collection and processing unit of the vehicle (V1), the machine learning model of the vehicle (V1) is updated by replacing the existing machine learning model with a progressively retrained machine learning model received from the server.

[0143] Again, for example, by retraining the machine learning model of the vehicle (V1), when it encounters the landscape of FIG. 2a again, the data collection and processing unit of the vehicle (V1) will correctly interpret the acquired image according to a specific scenario of FIG. 2b, and this specific scenario is now enriched with these new environmental features.

[0144] In a preferred embodiment, the retraining of a machine learning model performed by a server is transmitted to multiple other vehicles (Vi) to update each of the other vehicles' respective machine learning models.

[0145] The transmission of a progressively retrained machine learning model by a server is performed in a general manner for transmitting updates, including but not limited to wireless updates, to multiple other vehicles (Vi).

[0146] This has the advantage that each vehicle (Vi) can act as its own vehicle (V1) at a specific point in time, and all other vehicles (Vi) can utilize the progressively retrained machine learning model of its own vehicle (V1).

[0147] In another preferred embodiment, an example of a machine learning model used to detect and classify environmental features is a Convolutional Neural Network (CNN).

[0148] In another preferred embodiment, image encoding and hashing are performed using a representation called a bag-of-words (BoW). According to this technique, visual information is obtained by extracting image features from a raw image. Image features correspond to image regions with high discriminant values ​​that represent the image content. The image features are then represented using feature descriptors that describe the corresponding regions in a concise form, generally taking into account the gradient distribution within the regions. The classical approach uses Scale-Invariant Feature Transformation (SIFT) and Speed-Up Robust Feature (SURF) techniques for feature extraction and description.

[0149] Using visual word bag (BoW) representations for image encoding and hashing is advantageous because this technique improves object classification for objects in real-world landscapes.

[0150] In other preferred embodiments using a visual bag of words (BoW) representation, binary technology is used. Non-limiting examples of binary technology include Binary Robust Independent Elementary Feature (BRIEF) or Oriented and Rotated Binary Robust Independent Elementary Feature (ORB), which provide a more concise representation by reducing the computational cost associated with processing, storing, and transmitting image features.

[0151] The advantage of using binary technology is that feature point description is efficient, which improves the ability of binary technology to describe the essence of the actual landscape of an image hash.

[0152] Visual word bag processing technology generally includes the following three steps:

[0153] 1. Training Phase: In this phase, image features are extracted from a series of training images. Then, visually similar image features are grouped together to obtain a so-called visual vocabulary, which represents a set of generalized image features called visual words. Feature grouping is performed using clustering techniques such as k-means and aggregation.

[0154] 2. Hashing (Indexing) Step: This allows for a concise representation of the image. Here, features are extracted from the image. These features are then associated with words in the vocabulary generated during the training step using visual similarity criteria. The result is a histogram of visual word occurrence frequencies for a given image representing the image hash.

[0155] 3. Calculation of visual similarity: To calculate the visual similarity between two images, the Euclidean distance or cosine similarity of the hashes is calculated.

[0156] In a second aspect of the present invention, a training method for continuously and adaptively detecting environmental features in autonomous and assisted driving of a self-vehicle, performed by a server according to step (4) of any embodiment based on additional training data, wherein the additional training data is sensor data acquired in relation to each other vehicle (Vi) corresponding to an incorrect scenario interpretation of the self-vehicle (V1).

[0157] The machine learning model is already trained by a server using initial training data before the method of the present invention is initiated. The present invention deals with the continuous and adaptive detection of environmental features, including updates to the machine learning method. The present invention does not deal with the initial training of the machine learning model before the method is initiated. This is why the training method of the present invention uses additional training data received from each other vehicle (Vi).

[0158] In a third aspect of the present invention, a trained machine learning model is presented, trained according to the method of any embodiment.

[0159] A trained machine learning model of the third embodiment is generated from each update of the machine learning method, and this update is the result of the method of the present invention in any embodiment.

[0160] In the fourth aspect of the present invention, as a data processing system,

[0161] - Own vehicle (V1),

[0162] - Server,

[0163] - Multiple other vehicles (Vi), and

[0164] - A communication network that connects one's own vehicle and other vehicles to the server.

[0165] A data processing system including is presented.

[0166] The system of the present invention includes means configured to perform the steps of the method of any embodiment described in the section relating to the method.

[0167] Finally, in a fifth aspect of the present invention, a plurality of non-transient computer-readable media are provided for storing instructions, wherein the instructions cause each of one or more server processors to perform the method of any embodiment when the instructions are executed by one or more processors of a server, a self-vehicle, and another vehicle.

[0168] When the method of the present invention is performed in a distributed system, each system component, namely the server, its own vehicle (V1), and each other vehicle (Vi), has its own non-transient computer-readable medium for storing instructions and has one or more processors. According to the steps of the method, each of the aforementioned system components performs a specific step of the method described in the section related to the method.

[0169] Industrial applicability

[0170] The present invention is used whenever there is a system comprising a server and a vehicle in an industrial field, wherein ADAS sensors are provided to the vehicle and machine learning models are provided to both the server and the vehicle, and whenever it is necessary to transmit progressively retrained models to all vehicles.

[0171] One specific application is the automotive industry, where the vehicles are road vehicles. A cornering scenario encountered by one of the road vehicles initiates the incremental retraining of a machine learning model, after which this machine learning model is transmitted to all vehicles.

[0172] Another specific application is in the construction industry, where vehicles such as cranes, civil engineering equipment, and tractors are equipped with ADAS sensors. When a corner case is encountered by one of the cranes, earthmoving equipment, or tractors, it initiates the incremental retraining of a machine learning model, after which this model is transmitted to all vehicles.

[0173] Another specific application is in the field of robotics, where vehicles are robots equipped with ADAS sensors. When a corner case encountered by one of the robots initiates the incremental retraining of a machine learning model, and this machine learning model is then transmitted to all vehicles.

[0174] The only difference between the three non-limiting examples of specific uses described above stems from the content of the actual landscape and the content of the corresponding scenario. However, the content itself is not the subject of the present invention.

[0175] Although specific embodiments of the present invention have been described in detail, those skilled in the art related to the present invention will recognize various alternative designs and embodiments for carrying out the present invention as defined by the following claims.

Claims

Claim 1 A computer-implemented method for continuously and adaptively detecting environmental features in autonomous and assisted driving of a self-vehicle (V1), wherein the self-vehicle (V1) is connected to a server, the server is connected to a plurality of other vehicles (Vi) (i=2, ... n), the self-vehicle (V1), the other vehicles (Vi), and the server receive a machine learning model, and the method comprises a data collection step (I) performed by a data collection and processing unit of the self-vehicle (V1), wherein sensor data is acquired from the surroundings of the self-vehicle (V1) by means of an ADAS sensor of the self-vehicle (V1), the sensor data is interpreted based on the machine learning model to generate a specific scenario interpretation, and a related command is transmitted to an execution electronic control unit of the self-vehicle (V1) according to the specific scenario interpretation, and if the specific scenario interpretation is incorrect, the execution electronic control unit provides feedback indicating that the specific scenario interpretation is incorrect, which implies that any data discrepancy, such as machine learning failure, driver departure, or a discrepancy between the expected behavior and actual behavior of the self-vehicle (V1), is detected based on the incorrectly interpreted scenario. A data collection step (I) of transmitting to a data collection and processing unit of the self vehicle (V1), receiving feedback from the execution electronic control unit regarding the specific incorrect scenario interpretation, extracting a camera image from the acquired sensor data corresponding to the specific incorrect scenario interpretation, encoding and hashing the camera image corresponding to the specific incorrectly interpreted scenario and obtaining the self vehicle (V1) image hash, and transmitting the self vehicle (V1) image hash corresponding to the specific incorrectly interpreted scenario to the server along with a request to identify the specific similar scenario by the other vehicle (Vi) using the acquired sensor data and the vehicle (Vi);A data processing step (II) performed by the server, comprising identifying a specific combination of environmental features corresponding to an incorrect specific scenario interpretation by the self vehicle (V1), and broadcasting the image hash of the self vehicle (V1) to the plurality of other vehicles (Vi) along with a server request for each of the plurality of other vehicles (Vi) to transmit to the server a specific similar scenario by the vehicle (Vi) according to the data image hash received from the self vehicle (V1), and sensor data acquired in relation to the other vehicle (Vi) in the similar scenario; and a data processing step (III) performed by each of the other vehicles (Vi)'s respective data collection and processing units, comprising receiving the image hash of the self vehicle (V1) broadcast from the server and the server request, acquiring sensor data from the surroundings of each other vehicle (Vi) by the ADAS sensor of each other vehicle (Vi) according to the server request, extracting, encoding, and hashing the camera image of the other vehicle (Vi) from the acquired sensor data, and acquiring the image hash of each other vehicle (Vi). A data processing step (III) for processing sensor data acquired by the vehicle (Vi), comparing the similarity between the self vehicle (V1) and the image hash received from each other vehicle (Vi) based on a similarity score exceeding a given threshold, wherein a high similarity score corresponds to an image similar in terms of structure and content, and based on the similarity score, searching for a specific similar scenario with the sensor data acquired by the vehicle (Vi) corresponding to the image hash where the similarity score exceeds the given threshold, and transmitting to the server the specific similar scenario by the vehicle (Vi) according to the data image hash received from the self vehicle (V1), and the sensor data acquired in relation to the other vehicle (Vi) in the similar scenario.A method comprising: a data processing step (IV) performed by the server, wherein sensor data of a self-vehicle (V1) corresponding to a specific misinterpreted scenario is compared with sensor data acquired in relation to the other vehicle (Vi) for the similar scenario, aggregates the acquired sensor data received from the self-vehicle (V1) and the other vehicle (Vi), progressively retrains the machine learning model using the aggregated sensor data, and transmits the progressively retrained model to the self-vehicle (V1); and a data processing step (V) performed by a data collection and processing unit of the self-vehicle (V1), wherein the machine learning model of the self-vehicle (V1) is updated by replacing the existing machine learning model with the progressively retrained machine learning model received from the server. Claim 2 A method according to claim 1, wherein the progressively retrained machine learning model is transmitted to the plurality of other vehicles (Vi) by the server to update the machine learning model of the other vehicle (Vi). Claim 3 In claim 1, the machine learning is a convolutional neural network, method. Claim 4 A method according to claim 1, wherein the image encoding and hashing are performed by a Bag-of-Words (BoW) technique. Claim 5 In paragraph 4, the preferred visual word bag technique is a binary technique such as a Binary Robust Independent Elementary Feature (BRIEF) or an Oriented and Rotated Binary Robust Independent Elementary Feature (ORB). Claim 6 delete Claim 7 A data processing device comprising: - a self-vehicle (V1), - a server, - a plurality of other vehicles (Vi) (wherein i=2, ... n), and - a communication network connecting each of the self-vehicle (V1) and other vehicles (Vi) to the server, wherein the device comprises means configured to perform the steps of the method of any one of claims 1 to 5. Claim 8 A plurality of non-transient computer-readable media for storing instructions, wherein the instructions, when executed by one or more respective processors of a server, a self-vehicle (V1), and another vehicle (Vi), cause each of one or more server processors to perform the method of any one of claims 1 to 5. Claim 9 delete

Images