Computer-implemented method for continuously, adaptively detecting environmental features in automated and assisted driving of a vehicle

Abstract

The invention relates to a method for continuously self-adapting detection of environmental features in automated and assisted driving of a host vehicle connected to a server and to a plurality of other vehicles. The host vehicle and these other vehicles are provided with respective machine learning models similar to each other. The method comprises a data collection step of the host vehicle, wherein a specific scene interpretation is generated based on the machine learning model of the host vehicle, acquired sensor data, including images. When receiving feedback that this scene interpretation is wrong, corresponding images of the host vehicle are encoded, hashed and sent to the server and a request is made to broadcast these images to the plurality of other vehicles to request these other vehicles to transmit specific similar scene interpretations to the server. These other vehicles then identify a specific combination of environmental features corresponding to this wrong scene interpretation, acquire and process their own sensor data, including extracting, encoding and hashing images. The other vehicles compare the similarity between the image hashes received from the host vehicle and each of these other vehicles Vi by a similarity score that exceeds a given threshold, which means that in terms of structure and content, a high similarity score corresponds to similar data. For the image hashes of these other vehicles that exceed the given threshold, these other vehicles identify specific similar scene interpretations and send them to the server. The server aggregates the sensor data of these other vehicles corresponding to the image hashes of these other vehicles and the host vehicle, incrementally re-trains the machine learning model of the host vehicle and sends this incrementally re-trained model to the host vehicle. The invention further relates to a trained machine learning model, a system for performing the method and a plurality of non-transitory computer readable media.

Description

Technical Field

[0001] This invention relates to a computer-implemented method for continuously and adaptively detecting environmental features in the autonomous and assisted driving of a vehicle, the method comprising using a machine-trained model; the invention relates to a trained machine learning model, a system for performing the method, and a plurality of non-transitory computer-readable media. Background Technology

[0002] As vehicles become increasingly automated and intelligent, there is a need for systems that enable vehicles to better, more robustly, and more comprehensively understand their surroundings. The foundation of such systems is Advanced Driver Assistance Systems (ADAS) sensors, which reliably detect, measure, and classify certain features in the vehicle's environment, such as road elements.

[0003] Advanced driver assistance systems (ADAS) sensors in a vehicle acquire various types of data, including images, for the vehicle's advanced driver assistance functions.

[0004] In this context, recent advancements in both hardware and software have led to the widespread deployment of artificial intelligence (i.e., machine learning techniques in advanced driver assistance sensor functions).

[0005] The training of machine learning models used in advanced driver assistance sensor functions is typically based on creating multiple scenarios with the help of machine learning models.

[0006] Each specific scenario within a larger set of scenarios involves a specific combination of environmental features. Examples of unrestricted scenarios include: a straight highway with trees, a straight highway without trees, a tunnel entrance, a tunnel exit, a straight road inside a tunnel, a parking lot, a curve where the elevation remains constant, a curve where the elevation changes, etc. By using machine learning models, the expectation is that vehicles can identify combinations of environmental features similar to those derived from a limited number of scenarios corresponding to the training data.

[0007] The output data of the machine learning model is data interpretation obtained from the server that generates the scene interpretation.

[0008] Each scenario then includes associated instructions for the vehicle V1, intended to take some action for driving safety. These instructions typically include messages sent to the driver when the vehicle is manually operated or messages sent to the driver when the vehicle is autonomously driven. These instructions are beyond the scope of this invention.

[0009] Disadvantages of existing technology

[0010] Despite their immense power, one of the main challenges associated with machine learning models is training them. During training, machine learning models are provided with multiple sets of examples, allowing them to automatically learn certain details (such as texture, shape, color, etc.) of the features they are designed to extract. Crucially, a sufficient number of highly variable examples need to be provided during training so that the model truly “understands” the relevance and discriminativeness of a particular feature (e.g., the shape and wheels of a vehicle, the silhouette of a pedestrian, etc.). If not trained correctly with enough diverse examples, the model will extract irrelevant properties of objects and phenomena that conform to the provided examples but do not generalize well. This is known as overfitting and is one of the most significant drawbacks affecting the quality and robustness of machine learning models.

[0011] In applications such as detection, classification, segmentation, and localization, training machine learning models requires manually labeled data, leading to significant manpower demands for providing sufficient training examples. This is especially true for deep neural networks, where complex, robust architectures require training hundreds of thousands or even millions of parameters. Training such networks necessitates tens of thousands of training examples to avoid overfitting and to build powerful machine learning models. This is impractical due to limited data availability and the manpower required for labeling.

[0012] A more efficient solution is to detect situations where the machine learning model might fail (such as false positives and false negatives, misclassification, leading to incorrect interpretations of the scene, etc.) and perform additional training using these samples or similar cases (often called edge cases). This significantly improves the accuracy of the machine learning model using relatively little additional data.

[0013] As mentioned above, it is important to identify these edge cases where the machine learning model might fail and feed them back into the model for retraining, thereby significantly improving its performance. These edge cases are determined not only by the limitations of the trained model but also, perhaps more often, by environmental conditions. Here are a few examples:

[0014] - Challenging lighting conditions, such as those found in tunnels and indoor parking lots, can reduce the accuracy of camera-based machine learning models responsible for detecting and classifying lane markings, traffic signs, traffic lights, etc.

[0015] Bridges, guardrails, and other metallic structures may negatively impact the performance of radar-based machine learning models in detecting and locating traffic participants and static environmental elements.

[0016] - In addition, complex scenarios (such as intersections and roundabouts) may prove challenging for machine learning models.

[0017] Currently, the retraining of the machine learning model is performed incrementally at the vehicle level. Scene correction is manually performed by an operator of a server with the processing capabilities to do so, and then the correct data is sent to the vehicle and used to retrain the corresponding machine learning model for that vehicle.

[0018] The disadvantages of this method are as follows:

[0019] - The improvement of the corresponding machine learning model for this vehicle after retraining using a single example is negligible;

[0020] - Because the calibration is performed manually, it takes extra time, resulting in slower improvements;

[0021] In some cases, when the server operator does not have the information available for correction, he has to take other actions to find that information, which incurs additional costs.

[0022] The problem solved by this invention

[0023] For these reasons, the technical problem to be solved is not only to detect when such a machine learning model fails, but also to be able to extract sensor data corresponding to similar scenarios where the machine learning model has failed, which can be analyzed later and used as additional training data for the machine learning model of this vehicle.

[0024] Therefore, the purpose of this invention is to overcome the deficiencies of the prior art and to provide a method for continuously adaptively detecting environmental features under the autonomous and assisted driving of the vehicle, such that when such edge cases are detected by the vehicle, similar cases of other vehicles are automatically retrieved in order to detect the correct scene and update the machine learning model of the vehicle.

[0025] The terms "retrain the machine learning model" and "update the machine learning model" are used interchangeably in this invention. Summary of the Invention

[0026] In a first aspect, the subject of the invention is a computer-implemented method for continuously adaptively detecting environmental features in automated and assisted driving of a vehicle connected to a server, the server being connected to multiple other vehicles, the vehicle, the other vehicles, and the server being equipped with machine learning models, the method comprising:

[0027] Data collection step I, which is performed by the vehicle's data collection and processing unit:

[0028] - Sensor data from the vehicle's surrounding environment is acquired by the vehicle's ADAS sensors, and the sensor data relates to the environmental characteristics;

[0029] - This machine learning model is used to interpret the sensor data and generate a specific scene explanation.

[0030] - Based on the interpretation of this specific scenario, the associated instructions are sent to the vehicle's execution electronic control unit.

[0031] - If the specific scenario is misinterpreted, resulting in an incorrectly interpreted specific scenario, feedback is received from the execution electronic control unit.

[0032] - Extract camera images from the acquired sensor data corresponding to the specific scenario explanation of the error.

[0033] - Encode and hash these camera images corresponding to the specific scene of the misinterpretation to obtain the image hash of this vehicle;

[0034] - The image hash of the vehicle corresponding to the specific scenario of the misinterpretation, along with the acquired sensor data and requests from these other vehicles to identify the specific similar scenario of the vehicle, is sent to the server;

[0035] Data processing step II, which is performed by the server:

[0036] - Identify specific combinations of these environmental features that correspond to the specific scenario interpretation of the error in this vehicle.

[0037] - Broadcast the image hash of this vehicle and a server request to the plurality of other vehicles, the server request requesting each of the plurality of other vehicles to transmit the following data to the server:

[0038] -A specific similar scene obtained by the vehicle based on the hash of the data image received from the vehicle itself.

[0039] as well as

[0040] - These other vehicles acquired associated sensor data from other vehicles in similar scenarios;

[0041] Data processing step III, which is performed by the corresponding data collection and processing units of these other vehicles:

[0042] - Receive the image hash of this vehicle and the server requests from this server;

[0043] - Based on the server's request, the ADAS sensors of each of these other vehicles acquire sensor data from the surrounding environment of each of the other vehicles.

[0044] - Sensor data acquired by these other vehicles, including: extracting, encoding, and hashing camera images from other vehicles from the acquired sensor data to obtain corresponding image hashes of the other vehicles.

[0045] - A similarity score exceeding a given threshold is used to compare the similarity between the hashes of these images received from this vehicle and these corresponding other vehicles. This means that, in terms of structure and content, a high similarity score corresponds to similar images, and

[0046] -Based on this similarity score,

[0047] - Sensor data obtained from vehicle retrieval corresponding to the hashes of these images whose similarity scores exceed the given threshold, and these specific similar scenes.

[0048] - Send the following items to the server:

[0049] -A specific similar scene obtained by the vehicle based on the hash of the data image received from the vehicle itself.

[0050] as well as

[0051] -Related sensor data of other vehicles acquired for similar scenarios of these other vehicles;

[0052] Data processing step IV, which is performed by the server:

[0053] Compare the following two items:

[0054] - Sensor data of this vehicle corresponding to the specific scenario in which this error is interpreted.

[0055] and

[0056] -Related sensor data from other vehicles acquired for these similar scenarios,

[0057] - The sensor data received from this vehicle and these other vehicles will be aggregated, and the aggregated sensor data will be used to incrementally retrain the machine learning model.

[0058] - Send the incrementally retrained model to the vehicle.

[0059] In a second aspect of the invention, a training method for continuously adaptively detecting environmental features in the autonomous and assisted driving of the vehicle is proposed. This training method is performed by a server based on additional training data according to step 4 of any embodiment, wherein the additional training data is associated sensor data of other vehicles corresponding to the error scenario interpretation of the vehicle.

[0060] In a third aspect of the invention, a trained machine learning model trained according to the method of any embodiment is proposed.

[0061] In a fourth aspect of the invention, a data processing system is provided, the data processing system comprising:

[0062] -This vehicle,

[0063] -server,

[0064] - Several other vehicles,

[0065] A telecommunications network that connects this vehicle and every other vehicle to the server.

[0066] The system includes means configured to perform the steps of the method of any embodiment. Finally, in a fifth aspect of the invention, a plurality of non-transitory computer-readable media storing instructions that, when executed by one or more corresponding processors of a server, the present vehicle, and other vehicles, cause the corresponding one or more server processors to perform the method of any embodiment.

[0067] Further advantageous embodiments are the subject of the dependent claims.

[0068] Advantages of the present invention

[0069] The main advantages of using the method according to the present invention are as follows:

[0070] - The improvement achieved by retraining the corresponding machine learning model for this vehicle is greater than that of the existing technology because of the presence of sensor data (including a subset of images collected by multiple vehicles encountering similar scenarios), which results in a much higher degree of variability in the training data.

[0071] - The improvement time was significantly shortened, partly due to the automation of the data acquisition process and partly due to time savings;

[0072] - Saves financial resources because no other actions are required to obtain information to retrain the model, apart from the actions described in the method of this invention.

[0073] - It efficiently allocates additional computing resources to the server and greatly improves the accuracy of machine learning models using a relatively small amount of additional data.

[0074] This approach advantageously allows for continuous improvement in machine learning-based detection and classification of traffic participants and static environmental elements such as lanes, traffic signs, traffic lights, and road markings.

[0075] Other special features and advantages of the present invention can be seen from the following description and drawings. Attached Figure Description

[0076] Figure 1 This is a schematic block diagram of the method according to the present invention.

[0077] Figure 2a This is an illustrative representation of a scenario where the vehicle encounters a borderline case.

[0078] Figure 2b It is recognized by one of the other vehicles Vi. Figure 2a A schematic representation of a scene similar to the one in question. Detailed Implementation

[0079] Now for reference Figure 1 This illustrates an exemplary embodiment of the present invention.

[0080] The method of the present invention is executed in a system comprising:

[0081] -This vehicle is V1.

[0082] -server,

[0083] - Multiple other vehicles Vi, where i = 2, ..., n,

[0084] - A telecommunications network that connects this vehicle V1 and each of the other vehicles Vi to the server.

[0085] This vehicle (V1) and other vehicles (Vi) are equipped with advanced driver assistance (ADAS) sensors.

[0086] A server can be a single hardware server or multiple hardware servers, and it can be physically located near a telecommunications network or communicate within the telecommunications network. The server configuration is based on existing technology.

[0087] Vehicle V1, other vehicles Vi, and the server are configured with the same machine learning model. It should be understood that if an expert in the art deems it necessary to make minor adjustments to the machine learning model for each vehicle and / or server, such minor adjustments fall within the scope of "same machine learning model".

[0088] For ease of understanding of the present invention, please refer to Figure 2a and Figure 2b Specific examples were considered. Figure 2a In real-world scenarios, tunnel entrances exist. For example, due to the nature of radar sensors, the vehicle's perception system might interpret the presence of other vehicles in its lane as radar reflections from the tunnel walls, leading to undesirable actions such as sudden lane changes. The vehicle's driver will then correct the actions of the autonomous driving system.

[0089] Figure 2b This describes a scenario that is similar to the scenario encountered by vehicle V1 and can be detected by other vehicles using this invention, but with the following unrelated differences: the presence of a bobtail cat, vegetation, etc.

[0090] Real-world landscapes are considered dynamic, meaning they change continuously as the vehicle's position changes. For illustrative purposes, Figure 2a The scenario at a specific moment is illustrated. Based on this reality, a method for continuously adaptively detecting environmental features is needed, which in turn requires that these scenarios be continuously adaptive and rich. Scene adaptation is performed using a server, machine learning models of the current vehicle V1, and other vehicles Vi.

[0091] The method according to the present invention has five steps.

[0092] The first step of this method is data collection step I, which is performed by the data collection and processing unit of vehicle V1.

[0093] By understanding ADAS sensor data (including but not limited to camera images acquired by forward-looking cameras, surround-view camera systems, etc.) from cameras, lidar, radar, etc., this vehicle V1 uses ADAS sensor data to capture environmental features of the environment surrounding this vehicle V1.

[0094] This sensor data is processed based on the machine learning model of the vehicle V1, which is capable of detecting and classifying environmental features such as traffic participants and environmental characteristics (i.e., lanes, traffic signs, traffic lights, road markings, etc.).

[0095] By using a machine learning model, the data collection and processing unit of vehicle V1 interprets sensor data to generate scenario-specific interpretations.

[0096] From Figure 2a and Figure 2b In the example, it is expected that the data collection and processing unit of vehicle V1 will use a machine learning model based on Figure 2b To explain from the scenario shown Figure 2aThe system acquires sensor data about the scenery and then, based on the specific scenario, sends associated instructions to the execution control unit of vehicle V1.

[0097] The associated instructions correspond to specific expected behaviors from this vehicle V1.

[0098] Examples of related instructions are: slow down and turn on the lights, corresponding to the specific expected behaviors of slowing down and turning on the lights, respectively.

[0099] The rationality of sensor data processing (detection, classification, etc.) is continuously monitored and verified by calculating confidence levels, credibility scores, or by monitoring system disengagement due to driver intervention. Verification performed by interpreting driver actions is not the purpose of this invention.

[0100] When the execution control unit detects an edge case, i.e. any data inconsistency (such as machine learning failure, consistency check, high uncertainty, driver disconnection, etc.), the execution control unit sends feedback on the specific scenario interpretation error to the vehicle's V1 data collection and processing unit, which means that there is a specific scenario with misinterpretation.

[0101] Whenever a discrepancy occurs between the expected behavior and the actual behavior of vehicle V1, feedback is sent to the data collection and processing unit of vehicle V1.

[0102] Upon receiving feedback, the data collection and processing unit of vehicle V1 extracts camera images from sensor data, preferably images from the forward-facing camera, which correspond to the specific scenario of the misinterpretation.

[0103] The extracted images are then encoded and hashed. The corresponding image hash of vehicle V1 is sent to the server, along with the acquired sensor data corresponding to the incorrect interpretation of a specific scene, and a request from the data collection and processing unit of vehicle V1 to the server for other vehicles Vi to identify specific similar scenes for each of the other vehicles Vi.

[0104] by Figure 2a and Figure 2b For example, from Figure 2a The extracted image is encoded, hashed, and sent to the server along with all other sensor data corresponding to the image. In this case, the request to the server is a "request" for other vehicles, Vi, to identify and access the image. Figure 2a The extracted image matches a specific similar scene.

[0105] Each of the other vehicles V1 is equipped with a corresponding data collection and processing unit similar to that of vehicle V1. If the functions of the corresponding data collection and processing units are performed identically, then the data collection and processing units are considered to have similarity.

[0106] In data processing step II performed by the server, a specific combination of environmental features corresponding to a specific scenario misinterpreted by vehicle V1 is first identified.

[0107] The server then broadcasts the image hash received from its own vehicle V1 and a server request that requests each of several other vehicles Vi to transmit the following data to the server:

[0108] -A specific similar scene obtained by vehicle Vi based on the hash of the data image received from vehicle V1.

[0109] -as well as

[0110] - These other vehicle Vis respectively acquire associated sensor data of other vehicle Vis in similar scenarios.

[0111] exist Figure 2a In the example, the server request refers to the following facts: there is a problem in identifying the tunnel entrance (specific scenario); and other vehicles Vi must send sensor data of tunnel entrances with similar characteristics (image hashes of other vehicles Vi), as well as all acquired sensor data corresponding to image hashes of other vehicles Vi (tunnel entrances with similar environmental characteristics): entering a tunnel with similar characteristics to the tunnel encountered by this vehicle V1.

[0112] Data processing step III is performed by the corresponding data collection and processing units of these other vehicles Vi.

[0113] First, each of the data collection and processing units of other vehicles Vi receives an image of its own vehicle V1 hash and a server request broadcast from the server.

[0114] Then, the ADAS sensors of other vehicles Vi acquire sensor data from the surrounding environment of each other vehicle Vi, based on a specific combination of environmental characteristics requested by the server. In this example, the sensor data acquired by the other vehicles Vi provides information about entering a tunnel with similar characteristics to the tunnel requested from the server. The duration of this step is preset: for example, one day or one week, depending on the number of other vehicles available, the content of the scenery, etc.

[0115] Then, the corresponding data collection and processing units of other vehicles Vi process the acquired sensor data in real time: the processing includes extracting camera images from the sensor data, hashing the extracted images, and obtaining the corresponding image hashes of other vehicles Vi.

[0116] Then, the corresponding data collection and processing units of other vehicles Vi compare the similarity between the image hash received from this vehicle V1 and the corresponding image hash of these other vehicles themselves by a similarity score that exceeds a given threshold. This means that, in terms of structure and content, a high similarity score corresponds to a similar scene.

[0117] Although the broadcast from the server is received by each of the other vehicles Vi, it is possible that not all of the other vehicles Vi from multiple other vehicles Vi are able to acquire images from their respective surroundings upon the server's request, for example, because some of the other vehicles Vi from multiple other vehicles Vi are traveling in areas where such similar surroundings do not exist (e.g., there are no tunnels). Instead, each other vehicle Vi that has already acquired images from its respective surroundings will output the corresponding image hash and apply a similarity score to the comparison between its own image hash and the image hash received from its own vehicle V1.

[0118] Then, based on the similarity score, the corresponding data collection and processing units of other vehicles Vi retrieve the acquired sensor data and the specific similar scenes of the vehicle Vi corresponding to the image hash of the similarity score exceeding a given threshold.

[0119] The invention is effective if at least one of the other vehicles Vi is able to send the sensor data acquired as described above to the server.

[0120] At the end of step III, the corresponding data collection and processing units of other vehicles Vi will send the following items to the server:

[0121] -A specific similar scene obtained by vehicle Vi based on the hash of the data image received from vehicle V1.

[0122] - Associated sensor data for other vehicles Vi acquired for these similar scenarios.

[0123] Data processing step IV is performed by this server.

[0124] In this step, the server compares the following two items:

[0125] - Sensor data for this vehicle's V1 system corresponding to the scenario in which V1 is misinterpreted.

[0126] and

[0127] -Related sensor data of other vehicles Vi acquired for these similar scenarios.

[0128] Given that the definition of similarity score and the machine learning models for other vehicles Vi are the same, it is reasonable to expect that the vast majority of the specific similar scenarios for other vehicles Vi mentioned in the preceding steps are the same. In the example, this corresponds to Figure 2b The image depicted in the text.

[0129] Once the server has received sensor data from vehicle V1 and other vehicles Vi, it will aggregate the sensor data received from other vehicles Vi and vehicle V1. This is done using the server's machine learning model.

[0130] This is why, in step II, the server requests the other vehicles Vi to send associated sensor data obtained from the image hashes of those other vehicles Vi. In effect, the data hashes of the other vehicles Vi are used to identify edge cases, while the sensor data for edge cases is used for aggregation, for the purpose of incrementally retraining the machine learning model, since the sensor data provides more information about environmental features than individual images.

[0131] The aggregation of acquired sensor data is performed using aggregation techniques suitable for sensor data.

[0132] The aggregated sensor data is then used by the server to incrementally retrain the machine learning model.

[0133] At the end of step IV, the server sends the incrementally retrained model to vehicle V1.

[0134] The server sends incrementally retrained machine learning models in the usual manner of sending updates to the vehicle V1, such as, but not limited to, over-the-air updates.

[0135] Then, in the data processing step V performed by the data collection and processing unit of this vehicle V1, the machine learning model on this vehicle V1 is updated by replacing the existing machine learning model with a machine learning model that is incrementally retrained from the server.

[0136] To give another example, by retraining the machine learning model of vehicle V1, if it faces the same situation again... Figure 2a The scenery, the data collection and processing unit of this vehicle V1 will, according to Figure 2b The acquired images are correctly interpreted for specific scenarios, which are now enriched with these new environmental features.

[0137] In a preferred embodiment, the retraining of the machine learning model performed by the server is sent to multiple other vehicles Vi to update each of the corresponding machine learning models on the other vehicles Vi.

[0138] The server sends incrementally retrained machine learning models in a manner commonly used to send updates to multiple other vehicles Vi, such as, but not limited to, over-the-air updates.

[0139] This has the following advantages: each of the vehicles Vi can play the role of its own vehicle V1 at any given time, and all other vehicles Vi can utilize the machine learning model that is incrementally retrained by its own vehicle V1.

[0140] In another preferred embodiment, an example of a machine learning model for detecting and classifying environmental features is a convolutional neural network (CNN).

[0141] In another preferred embodiment, image encoding and hashing are performed using a bag-of-words (BoW) representation. According to this technique, visual information is obtained from the original image by extracting image features. Image features correspond to regions in the image that have high discriminative values, thus making them representative of the image content. The image features are then represented using feature descriptors that describe the corresponding regions in a compact form, typically considering the gradient distribution within these regions. Classical methods use Scale Invariant Feature Transform (SIFT) and Speeded Robust Feature Transform (SURF) techniques for feature extraction and description.

[0142] For image encoding and hashing, using the bag-of-words (BoW) representation is advantageous because this technique enhances the object classification of objects in real-world landscapes.

[0143] In another preferred embodiment using the bag-of-words (BoW) representation, binary techniques are employed. Non-limiting examples of binary techniques include binary robust independent basic features (BRIEF) or oriented and rotated binary robust independent basic features (ORB), which produce a more compact representation and reduce the computational costs associated with the processing, storage, and transmission of image features.

[0144] The advantage of using binary techniques is that they are efficient feature point descriptions, which leads to an improved ability to describe the nature of real-world landscapes in image hashes.

[0145] Visual bag-of-words processing typically involves three stages:

[0146] 1. Training Phase: In this phase, image features are extracted from a series of training images. Visually similar image features are then grouped together to obtain a so-called visual vocabulary representing a generalized set of image features (called visual words). Feature grouping is performed using clustering techniques such as k-means clustering and hierarchical clustering.

[0147] 2. Hashing (Indexing) Stage: This stage enables a compact representation of the image. Features are extracted from the image here. These features are then associated with words in the vocabulary generated during the training stage using a visual similarity criterion. The result is a histogram of the visual word frequencies for a given image, representing the image hash.

[0148] 3. Visual similarity calculation: In order to calculate the visual similarity between two images, the Euclidean distance or cosine similarity of their hashes is calculated.

[0149] In a second aspect of the invention, a training method for continuously adaptively detecting environmental features in the autonomous and assisted driving of the vehicle is proposed. This training method is performed by a server based on additional training data according to step 4 of any embodiment, wherein the additional training data is sensor data of a corresponding other vehicle Vi associated with the error scenario interpretation of the vehicle V1.

[0150] Prior to the commencement of the method of this invention, the server has already trained the machine learning model using initial training data. This invention relates to continuously adaptive detection of environmental features, including updates to the machine learning method. Prior to the commencement of this method, the invention does not involve the initial training of the machine learning model. This is why the training method of this invention uses additional training data, such as data received from other corresponding vehicles Vi.

[0151] In a third aspect of the invention, a trained machine learning model trained according to the method of any embodiment is proposed.

[0152] The trained machine learning model in the third aspect is generated from each update of the machine learning method, which is the result of the method of the present invention in any embodiment.

[0153] In a fourth aspect of the invention, a data processing system is provided, the data processing system comprising:

[0154] -This vehicle is V1.

[0155] -server,

[0156] -Multiple other vehicle VIs

[0157] - A telecommunications network that connects this vehicle and every other vehicle to the server.

[0158] The system of the present invention includes means configured to perform the steps of the method in any of the embodiments, as disclosed above in the section concerning the method.

[0159] Finally, in a fifth aspect of the invention, a plurality of non-transitory computer-readable media storing instructions that, when executed by one or more corresponding processors of a server, the present vehicle, and other vehicles, cause the corresponding one or more server processors to perform the method of any embodiment.

[0160] Because the method of the present invention is executed in a distributed system, each of the system elements—namely, the server, the vehicle V1, and each other vehicle Vi—has its corresponding non-transitory computer-readable medium storing instructions and has one or more processors. According to the steps of the method, each of the aforementioned system elements performs certain steps of the method, as disclosed above in the section concerning the method.

[0161] Industrial applications

[0162] This invention is used in industrial applications whenever a system comprising servers and vehicles exists, wherein the vehicles are equipped with ADAS sensors, and wherein both the servers and vehicles are equipped with machine learning models, and whenever it is necessary to send incrementally retrained models to all vehicles.

[0163] One specific application is in the automotive industry, where the vehicles are road vehicles. An edge case encountered by one of the road vehicles initiates the incremental retraining of a machine learning model, which is then sent to all vehicles.

[0164] Another specific application is in the construction industry, where vehicles such as cranes, bulldozers, and tractors are all equipped with ADAS sensors. An edge case encountered by one of the cranes, bulldozers, or tractors triggers an incremental retraining of the machine learning model, which is then sent to all vehicles.

[0165] Another specific application is in the field of robotics, where the vehicles are robots equipped with ADAS sensors. An edge case encountered by one of the robots initiates incremental retraining of the machine learning model, which is then sent to all vehicles.

[0166] The only difference between the three non-limiting examples of specific uses shown above lies in the content of the real-world scenery and the content of the corresponding scenes. However, the content itself is not the purpose of this invention.

[0167] Although certain embodiments of the invention have been described in detail, those skilled in the art related to the invention will recognize various alternative designs and embodiments for practicing the invention as defined in the following claims.

Claims

1. A computer-implemented method for continuously and adaptively detecting environmental features in autonomous and assisted driving of a vehicle V1, wherein the vehicle V1 is connected to a server, the server being connected to multiple other vehicles Vi, i = 2, ... n, and the vehicle V1, the other vehicles Vi, and the server are configured with machine learning models. The method includes: Data collection step I, which is performed by the data collection and processing unit of vehicle V1: The ADAS sensors of vehicle V1 acquire sensor data from the surrounding environment of vehicle V1. This machine learning model is used to interpret the sensor data and generate a specific scene explanation. Based on the interpretation of this specific scenario, the associated instructions are sent to the execution electronic control unit of vehicle V1. When the interpretation of a specific scenario is incorrect, meaning that any data inconsistency is detected based on the incorrect interpretation, the execution electronic control unit sends feedback of the incorrect interpretation of the specific scenario to the data collection and processing unit of the vehicle V1. The electronic control unit receives feedback regarding the interpretation of specific error scenarios. Extract camera images from the acquired sensor data corresponding to the explanation of this specific error scenario. These camera images corresponding to the specific scenes of misinterpretation are encoded and hashed to obtain the image hash of vehicle V1. The image hash of the vehicle V1 corresponding to the specific scenario of the misinterpretation, along with the acquired sensor data and the request from these other vehicles Vi to identify the specific similar scenario of the vehicle Vi, is sent to the server. Data processing step II, which is executed by the server: Identify specific combinations of environmental features corresponding to the specific error scenario explanation of this vehicle V1. The system broadcasts an image hash of the vehicle V1 and a server request to the plurality of other vehicles V1. The server request requests each of the plurality of other vehicles V1 to transmit the following data to the server: a specific similar scene obtained by the vehicle V1 based on the image hash of the data received from the vehicle V1, and associated sensor data of other vehicles V1 obtained by each of the other vehicles V1 in the similar scene. Data processing step III, which is performed by the corresponding data collection and processing units of these other vehicles Vi: The image hash of this vehicle V1 and the server requests from this server are received by receiving these broadcasts. Based on the server's request, the ADAS sensors of each of these other vehicles Vi acquire sensor data from the surrounding environment of each other vehicle Vi. The sensor data acquired by these other vehicles Vi includes: extracting, encoding, and hashing camera images from the acquired sensor data of other vehicles Vi to obtain the corresponding image hashes of the other vehicles Vi. The similarity score is used to compare the similarity between the image hashes received from this vehicle V1 and these corresponding other vehicles Vi, based on a similarity score exceeding a given threshold. This means that, in terms of structure and content, a high similarity score corresponds to similar images, and Based on this similarity score Sensor data retrieved from the vehicle Vi corresponding to the hashes of these images whose similarity scores exceed a given threshold, and these specific similar scenes. The following items are sent to the server: a specific similar scene obtained by vehicle Vi based on the hash of the data image received from vehicle V1, and associated sensor data of other vehicles Vi obtained for similar scenes of these other vehicles Vi respectively. Data processing step IV, which is executed by the server: Compare the following two items: sensor data of vehicle V1 corresponding to the specific scenario of this misinterpretation, and associated sensor data of other vehicles Vi obtained for these similar scenarios. The acquired sensor data received from this vehicle V1 and these other vehicles Vi is aggregated, and the aggregated sensor data is used to incrementally retrain the machine learning model. The incrementally retrained model is then sent to vehicle V1. Data processing step V, which is performed by the data collection and processing unit of the vehicle V1: The machine learning model on vehicle V1 is updated by replacing the existing machine learning model with the incrementally retrained machine learning model received from the server.

2. The method according to claim 1, wherein, The incrementally retrained machine learning model is sent by the server to the multiple other vehicles Vi to update the machine learning models on those other vehicles Vi.

3. The method according to claim 1 or 2, wherein, This machine learning method is a convolutional neural network.

4. The method according to claim 1 or 2, wherein, Image encoding and hashing are performed using the bag-of-words (BoW) technique.

5. The method according to claim 4, wherein, Visual bag-of-words technology is a binary technology, including binary robust independent basic features (BRIEF) or orientation and rotation binary robust independent basic features (ORB).

6. A trained machine learning model trained according to the method of any one of claims 1 to 5.

7. A data processing system, comprising: -This vehicle is V1. -server, - Multiple other vehicles Vi, where i = 2, ..., n, - A telecommunications network that connects this vehicle V1 and each of the other vehicles Vi to the server. The system includes means configured to perform the steps of the method according to any one of claims 1 to 5.

8. A plurality of non-transitory computer-readable media storing instructions which, when executed by one or more corresponding processors of a server, the present vehicle V1, and other vehicles Vi, cause the corresponding one or more server processors to perform the method as described in any one of claims 1 to 5.

Images