Platform for perception system development for autonomous driving systems

Abstract

The present invention relates to a platform for perception system development for autonomous driving systems. Furthermore, the present invention by utilizing a fleet of vehicles to develop new perception features related to new sensor hardware as well as develop new algorithms for existing sensors using federated learning. To achieve this, the fleet of vehicles' own world observations are post-processed and used as a reference to compare the output of the software (SW) or hardware (HW) under development. In cases where there is a large discrepancy between the baseline world observations and the world observations perceived by the module under test, the data is weakly annotated by the baseline world observations. This weakly annotated data can then be used to update the software parameters of the "perception model" in the module under test in each individual vehicle, or transmitted to a "back office" for off-vehicle processing or more accurate annotation.

Description

[0001] Cross-reference to related applications

[0002] This patent application claims priority to European Patent Application No. 21158016, filed on February 19, 2021, entitled "Platform for the Development of Perception Systems for Automated Driving Systems," which is assigned to its assignee and is expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates to methods and systems for weakly annotating data to train a perception development module for a vehicle equipped with an Automated Driving System (ADS). Background Technology

[0004] Over the past few years, research and development activities related to autonomous vehicles have surged, exploring many different approaches. An increasing number of modern vehicles are equipped with Advanced Driver Assistance Systems (ADAS) to improve vehicle safety and, more generally, road safety. ADAS (which can be represented, for example, by adaptive cruise control, ACC, collision avoidance systems, forward collision warning, etc.) are electronic systems that assist the driver while driving. Today, research and development are underway in many technical fields related to both ADAS and Automated Driving (AD). In this paper, ADAS and AD refer to the general term Automated Driving Systems (ADS), corresponding to all the different levels of automation (and particularly levels 4 and 5) defined, for example, by SAE J3016 levels (0-5) of driving automation.

[0005] In the near future, ADS solutions are expected to be incorporated into most new cars on the market. ADS can be interpreted as a complex combination of components, defined as a system where perception, decision-making, and vehicle operation are performed electronically and mechanically rather than by a human driver, and can be defined as the introduction of automation into road traffic. This includes the processing of information about the vehicle, destination, and surroundings. While the automated system can control the vehicle, it allows the human operator to leave all or at least some responsibility to the system. ADS typically combines multiple sensors such as radar, lidar, sonar, cameras, navigation systems (e.g., GPS), odometers, and / or inertial measurement units (IMUs) to perceive the vehicle's surroundings. Advanced control systems can then interpret this sensor information to identify appropriate navigation paths and obstacles, free space areas, and / or relevant landmarks.

[0006] Much of the current effort in ADS development revolves around safely bringing the first system to market. However, once this goal is achieved, it is crucial to improve the system in a safe and efficient manner, achieving both cost reduction and performance improvement. Typically, there are high costs associated with the development and validation of ADS's perception algorithms. Perception algorithms (e.g., "vision" algorithms) rely on massive datasets of hand-labeled (annotated) data used to train machine learning models. Creating such hand-labeled datasets is not only very expensive but also extremely time-consuming, especially when domain expertise is required.

[0007] Therefore, there is a need in the art for new solutions to facilitate the development and validation of ADS, enabling the continuous delivery of safer and higher-performance systems. In particular, there is a need for new and effective solutions for identifying high-quality data suitable for training machine learning algorithms used in perception algorithms employed in ADS. Furthermore, such high-quality data can be advantageously used for the validation and verification of ADS features and for facilitating the development of new ADS features. As always, improvements are preferably made without significantly impacting the size, power consumption, and cost of the in-vehicle system or platform. Summary of the Invention

[0008] Therefore, one object of the present invention is to provide a method for weakly annotating data to train a vehicle's perception development module, executed by a computing system, a computer-readable storage medium, a system, and a vehicle including systems that mitigate all or at least some of the aforementioned disadvantages of currently known systems and methods.

[0009] This objective is achieved by providing a method for weakly annotating data to train a vehicle's perception development module, performed by a computing system as defined in the appended claims, a computer-readable storage medium, a system, and a vehicle. In the current context, the term "exemplary" should be understood as used as an instance, example, or illustration.

[0010] According to a first aspect of the invention, a method is provided for training a perception development module of a vehicle by weakly annotating data, performed by a computing system, wherein the vehicle is equipped with an Adaptive Detection and Analysis (ADS). The method includes: storing sensor data obtained from a set of onboard sensors configured to monitor the vehicle's surrounding environment during a time period. The method further includes: storing a first set of perception data obtained from the vehicle's perception module during the time period. The perception module is configured to generate the first set of perception data based on sensor data obtained from one or more of the set of onboard sensors during the time period. Further, the method includes: forming a baseline worldview from the obtained first set of perception data by post-processing the first set of perception data. The baseline worldview indicates the vehicle's surrounding environment during the time period. The method further includes: storing a second set of perception data from the perception development module during the time period. The perception development module is configured to generate the second set of perception data based on a perception model and sensor data obtained from one or more of the set of onboard sensors during the time period. The second set of perception data indicates perception parameters of the vehicle's surrounding environment during the time period. Furthermore, the method includes: evaluating a second set of sensing data relative to the formed baseline world observations to identify matches between sensing parameters in the second set of sensing data and corresponding sensing parameters in the baseline world observations. Then, if a match is identified between the sensing parameters in the second set of sensing data and corresponding sensing parameters in the baseline world observations, the method further includes: determining an estimation error of the matched sensing parameters in the second set of sensing data with reference to the corresponding sensing parameters in the baseline world observations. Furthermore, if no match is identified between the sensing parameters in the second set of sensing data and corresponding sensing parameters in the baseline world observations, or if the estimation error exceeds a threshold, the method further includes: updating one or more parameters of the sensing model of the sensing development module using a weakly supervised learning algorithm based on sensor data for the second set of sensing data and by using the baseline world observations as weakly annotated data to form a supervisory signal, or transmitting the stored sensor data, the stored second set of sensing data, and the baseline world observations as weakly annotated data to a remote entity for off-vehicle processing.

[0011] According to a second aspect of the invention, a (non-transitory) computer-readable storage medium is provided for storing one or more programs configured to be executed by one or more processors of an in-vehicle processing system, the one or more programs including instructions for performing methods according to any of the embodiments disclosed herein. With respect to this aspect of the invention, there are similar advantages and preferred features as to the first aspect of the invention previously discussed.

[0012] As used herein, the term "non-transitory" is intended to describe computer-readable storage media (or "memory") that do not propagate electromagnetic signals, but is not intended to otherwise limit the types of physical computer-readable storage devices included in the term computer-readable media or memory. For example, the terms "non-transitory computer-readable media" or "tangible memory" are intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium can be further transmitted via a transmission medium or via signals (such as electrical, electromagnetic, or digital signals) that can be transmitted via communication media such as networks and / or wireless links. Thus, as used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, not signaling), rather than a limitation on the persistence of data storage (e.g., RAM and ROM).

[0013] According to a third aspect of the invention, a system is provided for weakly annotating data to train a perception development module of a vehicle, wherein the vehicle is equipped with an Adaptive Data System (ADS). The system includes: control circuitry configured to store sensor data obtained from one or more onboard sensors configured to monitor the vehicle's surrounding environment during a time period. The control circuitry is further configured to store a first set of perception data obtained from a perception module of the vehicle ("acquisition platform") during the time period. The perception module is configured to generate the first set of perception data based on the sensor data obtained from the one or more onboard sensors during the time period. Furthermore, the control circuitry is configured to form a baseline world observation indicating the scene in the vehicle's surrounding environment during the time period by post-processing the first set of perception data. The control circuitry is further configured to obtain a second set of perception data from the perception development module, wherein the perception development module is configured to generate the second set of perception data based on a perception model and the sensor data obtained from the one or more onboard sensors during the time period. The second set of perception data indicates perception parameters of the vehicle's surrounding environment during the time period. Furthermore, the control circuit is configured to evaluate the acquired second set of sensing data relative to the formed baseline world observation to identify a match between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observation. Then, if a match is identified between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observation, the control circuit is further configured to determine the estimation error of the matched sensing parameters in the second set of sensing data by referring to the corresponding sensing parameters in the baseline world observation. Further still, if no match is identified between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observation, or if the estimation error exceeds a threshold, the control circuit is further configured to perform at least one of the following:

[0014] • Update one or more parameters of the perception model of the perception development module by using a weakly supervised learning algorithm based on sensor data used for the second set of perception data and by using baseline world observations as weakly annotated data to form a supervision signal.

[0015] • The stored sensor data, the stored second set of sensing data, and the baseline world observations are transmitted as weakly annotated data to a remote entity for off-vehicle processing.

[0016] This aspect of the invention has similar advantages and preferred features to the first aspect of the invention previously discussed.

[0017] According to a fourth aspect of the invention, a vehicle is provided, including a set of onboard sensors configured to monitor the vehicle's surrounding environment. The vehicle further includes: an automated driving system (ADS) with a perception module configured to generate a first set of perception data based on sensor data obtained from one or more of the set of onboard sensors. The vehicle further includes: a perception development module configured to generate a second set of perception data based on a perception model and sensor data obtained from one or more of the set of onboard sensors; and a system for weakly annotating the data to train the perception development module according to any of the embodiments disclosed herein. Similar advantages and preferred features exist for this aspect of the invention as for the first aspect of the invention previously discussed.

[0018] The proposed method, computer-readable storage medium, system, and vehicle provide a learning platform that leverages the acquisition system and sensors of the ADS (Autonomous Detection and Analysis System) to jointly learn new software and hardware versions of various perception features for autonomous driving applications. More specifically, the proposed learning mechanism relies on the post-processing of the acquired ADS's "world observations" (i.e., perception outputs), the performance of which has been validated and verified, thus constructing a "benchmark" against which the output of the software / hardware under development is compared to generate weakly annotated training data. This readily provides advantages in the following aspects:

[0019] • Implement a cost-effective data annotation engine that enables the efficient development of new perceptual features.

[0020] • An automated solution for finding data suitable for annotation (partial data that, if annotated, can improve the performance of the software under development). This reduces the annotation cost of developing new perception software for autonomous driving applications.

[0021] • This enables the use of weakly supervised algorithms to locally update the perception software under development in vehicles, which significantly reduces annotation costs by reducing the need for manual data annotation.

[0022] • Effective utilization of available data collection resources in data collection vehicles to facilitate the development of new perception features in autonomous driving applications.

[0023] • An automated solution for collecting high-quality data on mismatches between two perception outputs, which can then be used for human analysis and / or annotation to develop new perception features for autonomous driving applications. Furthermore, the same approach can be applied to “inspecting” the acquisition system to identify potential flaws, thereby improving the reliability of the acquisition system itself.

[0024] Other embodiments of the invention are defined in the dependent claims. It should be emphasized that, when used in this specification, the term "comprising" is used to specify the presence of the stated features, integrals, steps, or components. It does not exclude the presence or addition of one or more other features, integrals, steps, parts, or groups thereof.

[0025] These and other features and advantages of the invention will be further illustrated below with reference to the embodiments described herein. Attached Figure Description

[0026] Other objects, features, and advantages of embodiments of the present invention will become apparent from the following detailed description with reference to the accompanying drawings, in which:

[0027] Figure 1 This is a schematic flowchart representation of a method performed by a computing system according to an embodiment of the present invention for weakly annotating data to train a perception development module for a vehicle, wherein the vehicle is equipped with an automated driving system (ADS).

[0028] Figure 2 This is a schematic top view of a post-processing method according to an embodiment of the present invention, in the form of a series of scenes depicting the time progression of a vehicle approaching an object.

[0029] Figure 3 This is a schematic block diagram representation of a system for weakly annotating data to train a vehicle's perception development module according to an embodiment of the present invention.

[0030] Figure 4 This is a schematic perspective view of an evaluation process for identifying a match between baseline world observations and outputs from a perception development module, according to an embodiment of the present invention.

[0031] Figure 5This is a schematic top view of an evaluation process for identifying a match between baseline world observations and outputs from a perception development module, according to an embodiment of the present invention.

[0032] Figure 6 This is a schematic top view of an evaluation process for identifying a match between baseline world observations and outputs from a perception development module, according to an embodiment of the present invention.

[0033] Figure 7 This is a schematic side view of a vehicle including a system for weakly annotating data to train a vehicle's perception development module, according to an embodiment of the present invention. Detailed Implementation

[0034] Those skilled in the art will understand that the steps, services, and functions described herein can be implemented using separate hardware circuitry, software that works in conjunction with a programmed microprocessor or general-purpose computer, one or more application-specific integrated circuits (ASICs), and / or one or more digital signal processors (DSPs). It will also be understood that, when the invention is described in terms of method, it can also be implemented in one or more processors and one or more memories coupled to the one or more processors, wherein the one or more memories store one or more programs that, when executed by the one or more processors, perform the steps, services, and functions disclosed herein.

[0035] In the following description of exemplary embodiments, the same reference numerals denote the same or similar components.

[0036] Figure 1 This is a schematic flowchart representing a performance evaluation of a perception development module for a vehicle performed by a computing system according to an embodiment of the present invention, and a method 100 for weakly annotating data to train / update the perception development module for the vehicle. Method 100 can also be understood as a method 100 for weakly supervised learning of a perception development module performed by an onboard computing system. The vehicle is further equipped with an automated driving system (ADS). The term "perception development module" can be understood as "module under test" (MUT), meaning it is "new" (under development) software and / or hardware perception features. Perception features can be, for example, object detection features, object classification features, object state estimation features, road reference estimation features, free space estimation features, road friction estimation features, object trajectory estimation features, target / object tracking features, and / or drivable area estimation features. In other words, in the current context, a perception development module can be understood as software and / or hardware configured to generate perception outputs, wherein the module is currently "under development" and has not yet been "put into acquisition" (e.g., unverified / verified).

[0037] Method 100 includes storing sensor data acquired from one or more vehicle-mounted sensors during a time period. The vehicle-mounted sensors can be represented by any sensor suitable for sensing and / or perceiving the vehicle's environment and / or location, and can, for example, refer to one or more combinations of radar, lidar, sonar, cameras, navigation systems (e.g., GPS), odometers, and / or inertial measurement units. The term "acquired" will be interpreted broadly herein, meaning and including receiving, retrieving, collecting, acquiring, etc.

[0038] Method 100 further includes: storing 102 a first set of perception data obtained from the vehicle's perception module (also referred to as a perception system) during the time period. The perception module is configured to generate the first set of perception data based on sensor data obtained from one or more onboard sensors during the time period. In the current context, the perception module can be understood as a system responsible for acquiring raw sensor data from onboard sensors such as cameras, lidar, radar, ultrasonic sensors, navigation systems (e.g., GPS), odometers, and / or inertial measurement units and converting this raw data into scene understanding.

[0039] The term "store" sensing / sensor data can refer to "stored in one or more memories," "stored on the vehicle," "stored on the vehicle in one or more memories," and / or digitally and / or electronically storing a set of sensing / sensor data, and can further refer to "collecting" and / or "acquiring" a set of sensing data. On the other hand, the term "set" of sensing / sensor data can refer to the "range," "quantity," "series," "continuous and / or intermittent stream and / or collection" of sensing / sensor data, while "sensing data" can refer to sensing data collected continuously and / or intermittently. Furthermore, the term "perceived data" can refer to "environmental assessment" data, "spatial perception" data, "processed sensing" data, and / or "time-dependent" data, while "perceived data" can refer to "perceived information" and / or "estimation." On the other hand, the term "obtained from" the perception module can refer to "derived" from the perception model and / or "based on output data from the perception module," while the perception module configured to "generate the set of perception data" can refer to a module suitable for and / or configured to "estimate the environment of the vehicle," "estimate at least a portion of the environment of the vehicle," "determine the environment of the vehicle," "interpret sensing information related to the automatic control of the vehicle," and / or "estimate the environment of the vehicle and make model predictions about the future state of the vehicle's environment."

[0040] The phrase "stored during a period of time" can refer to storing something that "lasts" for a period of time, while the "range" of the time period from a first time point to a second time point can refer to the time period that "extends" and / or "occurs" from the first time point to the second time point. "Time period" can refer to a "predictable time period" and / or a "predetermined time period." On the other hand, "time point" can refer to a "moment," and "from the first time point to the second time point" can refer to "from the first time point to a subsequent second time point."

[0041] Further, method 100 includes: forming a baseline world observation 102 by post-processing the first set of perception data stored in storage 103. The baseline world observation includes information indicating the scenario in the environment surrounding the vehicle during the time period. The scenario can be an instantaneous scene at a point in time during the time period, including the location of detected objects, object category / type, location of lane markings, extension of lane markings, free space detection, and / or the trajectory of detected objects. It should be noted that this list is only for illustrative purposes of the parameters included in the “scenario” and may include other parameters that are readily understood by those skilled in the art and can be detected by the vehicle’s perception module.

[0042] Now refer to Figure 2 To further illustrate post-processing step 103, Figure 2 A series of schematic top views (a)-(d) are depicted, illustrating a vehicle 1 moving along a section of the road toward an external object 24. Each illustration is associated with a time point within a time period 21 from a first time moment T1 to a second time moment T2.

[0043] In the first illustration (a), vehicle 1 (also referred to as vehicle 1) is moving toward an external object (here, in the form of truck 24 traveling in the same direction in an adjacent lane of the road section). However, due to the distance from truck 24, the vehicle's perception system / module may not be able to determine the position of the external object with sufficient accuracy and classify it as a truck. This is indicated by box 22a surrounding truck 24, which schematically indicates the "uncertainty" in detection and classification.

[0044] At the subsequent moment, that is Figure 2 As shown in Figure (b), vehicle 1 is closer to the external object, and as indicated by the reduced size of box 22b compared to the first box 22a, the uncertainty regarding the position and category / type of the external object 24 is reduced.

[0045] At another point in the future, namely Figure 2As shown in Figure (c), the perception system / module of vehicle 1 is able to accurately determine the position of external object 2 and classify it as truck 2. More specifically, compared to when vehicle 1 was located further away from the truck, vehicle 1 is now close enough to truck 2 to classify it and estimate the truck's position on the road with greater accuracy.

[0046] Then, as Figure 2 The bottom illustration in the middle (i.e., Figure 2 As indicated in the illustration (d), by employing appropriate filtering techniques and based on the temporal development of the "scenario," a "baseline world observation" can be established at the intermediate time point 23 between T1 and T2. More specifically, the filtering can be based, for example, on the temporal development of the trajectory, position, etc., of a predefined model (e.g., a motion model) combining vehicle 1 and external object 2. This established baseline world observation can then be used as "foundational facts" to train and / or validate various sensory outputs, and particularly for training and / or validating outputs obtained from the perception development module. Thus, in some embodiments, the baseline world observation constitutes the foundational facts of a second set of sensory data.

[0047] According to some embodiments, the step of forming a baseline world observation 102 includes: determining a baseline world observation indicating the vehicle's surrounding environment based on post-processing of a portion of a first set of sensing data back from a second time point to an intermediate time range between a first time point T1 and a second time point T2. The baseline world observation is accordingly conditioned on this portion of the first set of sensing data. Furthermore, according to some embodiments, the post-processing of this portion of the first set of sensing data includes passing this portion of the first set of sensing data through a backward filter. Here, the backward filter is configured to align a set of sensing parameters of the first set of sensing data at the intermediate time point based on the development of the state of the set of sensing parameters from the intermediate time point to the second time point T2.

[0048] In other words, as the understanding of the vehicle 1 environment increases as time flows from the intermediate time point 23 to the second time point T2, and by analyzing the data in the reverse time direction, one can determine the "state" (i.e., category, location, trajectory, etc.) of objects in the vehicle 1 environment with greater accuracy at the intermediate time point compared to what can be done "during runtime". More specifically, post-processing may, for example, include running a first set of perceived data through a reverse filter configured to align, for example, the current state of an object and its predicted future state with what happens in the future—i.e., from the intermediate time point to the second time point T2. Post-processing may include further processing steps than running post-processing through a reverse filter. More specifically, post-processing may include fusing data from multiple sensors and applying backward and forward filtering to the fused information. Suitable filters for this purpose may, for example, be particle filters or different types of Kalman filters (e.g., extended Kalman filters).

[0049] Back Figure 1 In some embodiments, method 100 may further include checking the reliability of the baseline world observations formed 103. The reliability check may be a byproduct of post-processing the outputs of all sensors in the vehicle, and the reliability check may be performed in a manner similar to the post-processing 103 of the outputs from the perception module of the acquisition platform. If the baseline world observations fail the reliability check, the entire associated scenario can be stored as a scenario for developing the perception module of the acquisition platform by transmitting the sensor data stored in storage 101 and the set of perception data stored in storage 102 to a remote entity (where it can be manually analyzed).

[0050] However, in some embodiments, post-processing 103 can be configured to inherently generate an indication of the reliability of the baseline world observations. More specifically, as... Figure 2 As described, the post-processing module generates baseline world observations by processing data generated by the acquisition sensing system over a time interval, for example, [T1, T2]. Figure 2 An example of object detection. During this time period [T1, T2], the perception algorithm (used by the platform's perception system) can also output confidence in its detections. For example, the object detection algorithm could indicate that the object is a truck with an 85% probability and make an uncertainty estimate of the truck's location at each timestamp. This information can then be used to estimate how much people can trust baseline world observations. For example, if Figure 2If an object in front of vehicle 1 is classified as a truck with 50% confidence during the time interval T1 to T2, then post-processing its trajectory will not produce a reliable baseline world observation because the confidence in its object category is too low throughout the time interval (e.g., below a certain threshold). However, if it is classified as a truck with a 90% probability, then post-processing this information will generate a more “reliable” baseline world observation. The same reasoning can be applied to the uncertainty estimation of the detected object trajectory, as well as other perceptual parameters illustrated in this paper.

[0051] Another example of how to measure the reliability (or accuracy) of baseline world observations is to measure, for example, the deviation between the post-processed trajectory of a detected object and its instantaneous detection before post-processing. A larger deviation therefore means that the baseline world observation is less reliable.

[0052] This provides an additional technical advantage because any discrepancies will capture scenarios where the underlying model (of the acquisition platform) struggles to make specific predictions / estimates / reads. Therefore, cases of high uncertainty (uncertainty of the external environment) are essentially captured. Since these are (likely) challenging environmental scenarios, they are useful for training future perception models. Thus, such scenarios can be considered so-called "extreme cases" for perception development, and by storing these scenarios, the development of future high-performance systems can be accelerated. This reliability check of the baseline world observations can advantageously be performed before any evaluation 105 of the second set of perception data to avoid propagating errors from the perception module of the acquisition platform to the perception development module. Therefore, in some embodiments, the evaluation 105 of the second set of perception data referencing the baseline world observations is performed only if the baseline world observations pass the reliability check.

[0053] Method 100 further includes storing 104 a second set of perception data from a perception development module. The perception development module is configured to generate the second set of perception data based on a perception model and sensor data obtained from one or more onboard sensors during the time period. The second set of perception data indicates at least one perception parameter of the vehicle's surrounding environment at at least one moment during the time period. For example, the second set of perception data may include a set of bounding boxes indicating the location and classification of a group of external objects (e.g., one or more vehicles) in the vehicle's surrounding environment.

[0054] In the current context, the term "perception model" will be understood as a software algorithm configured to receive input in the form of sensor data (raw or with some level of preprocessing) and thus generate an output that includes a representation of at least a portion of the vehicle's surrounding environment. The perception model can, for example, be in the form of a neural network, and the model parameters can accordingly be in the form of network weights.

[0055] Sensor data, the first set of sensing data, and the second set of sensing data can be stored or saved, for example, in one or more data buffers. As already mentioned, the first set of sensing data and the second set of sensing data can be, for example, the detected object or the state of the object and / or the vehicle location, and / or statistical and physical model predictions of the future state derived continuously and / or intermittently from a first time point T1 to a second time point T2. This time period (and the corresponding length of the optional data buffer) can be any size deemed feasible, for example, taking into account data capacity limitations and / or the characteristics of ADS, and the range can be, for example, from less than a second to several minutes or more preferably from several seconds to less than a minute.

[0056] Further, method 100 includes: evaluating 105 a second set of sensory data against the baseline world observations formed 103 to identify matches between sensory parameters in the second set of sensory data and corresponding sensory parameters in the baseline world observations. In other words, evaluation 105 can be interpreted as a comparison step between the second set of sensory data and the formed baseline world observations to identify the degree of match between one or more sensory parameters (e.g., object position) in the second set of sensory data and corresponding sensory parameters in the baseline world observations. Therefore, the term "evaluating the second set of sensory data relative to the formed baseline world observations" as used herein can be understood as "processing the second set of sensory data and the formed baseline world observations" to identify matches between sensory parameters in the second set of sensory data and corresponding sensory parameters in the baseline world observations.

[0057] In other words, the purpose of evaluation 105 is to match objects detected based on the second set of perception data with an object from baseline world observations, both of which correspond to a unique object in the real world. After matching the corresponding objects from the two sets of perception data together, the next step is to compare their perception parameters (e.g., position, velocity, heading, etc.) and evaluate the performance of the MUT.

[0058] Based on the illustrative example, evaluating 105 may include (for example, such as...) Figure 4 (As illustrated in the diagram) The positions of the identified vehicles in the second set of perception data are compared with the corresponding vehicle positions in the baseline world observations. (Referencing...) Figure 4-6 Further examples will be provided to illustrate and clarify the matching process for 104.

[0059] Furthermore, if a match is identified between the perceptual parameters of the second set of perceptual data and the corresponding perceptual parameters in the baseline world observation, method 100 further includes: determining, 106, the estimation error (e) of the matched perceptual parameters of the second set of perceptual data with reference to the corresponding perceptual parameters in the baseline world observation. The estimation error (e) can be interpreted as a parameter indicating how well the "matched" perceptual parameters correlate. In other words, in some embodiments, the estimation error (e) can be interpreted as the "degree of matching" between the two perceptual parameters.

[0060] However, if no match is found between the sensing parameters of the second set of sensing data and the corresponding sensing parameters in the baseline world observations, or if the estimation error of the determined 106 exceeds a threshold (e.g., zero), then method 100 further includes at least one of the following steps.

[0061] • Update one or more parameters of the perception model of the 107 perception development module by using a weakly supervised learning algorithm based on sensor data used for the second set of perception data and by using baseline world observations to weakly annotate the second set of perception data.

[0062] • The stored sensor data, the stored second set of sensing data, and the baseline world observations are transmitted as weakly annotated data 108 to a remote entity for off-vehicle processing.

[0063] It should be noted that "mismatch" can also be implicitly represented by an estimation error that tends toward infinity.

[0064] The threshold for estimation error can be defined differently depending on the type of perception parameter being evaluated. For example, the perception parameter could be an object position estimate or an object occupancy estimate. The threshold could then be in the form of the "maximum lateral and longitudinal offset of the nearest point" between the bounding box representation of the object in the second set of perception data and the bounding box representation of the corresponding object in the baseline world observation. The term "nearest point" can be understood as the point where the detected object is closest to the vehicle. However, the threshold could also be in the form of the "maximum lateral and longitudinal offset" of any point between the two bounding box representations (e.g., the lower left corner, upper right corner, etc.).

[0065] Similarly, if the perception parameters are free-space estimates, the threshold can be in the form of the “maximum size” (i.e., the number of region units) of the non-overlapping free-space region between the free-space region estimate of the second set of perception data and the baseline world observations. In set theory terms, this can be understood as a symmetric difference between the set of free spaces defined by the free-space estimates of the perception development module and the set of free spaces defined by the baseline world observations. Furthermore, in some embodiments, there are multiple different thresholds associated with the free-space estimates, where the threshold depends on the position of the “erroneous” portion relative to the vehicle. More specifically, one can employ a “weight matrix” or “weight graph” where the threshold for the estimation error is lower for certain portions of the scene surrounding the vehicle. For example, accuracy closer to the front of the vehicle is more important than accuracy further away or to the side of the vehicle. Therefore, in the perception model update process 107, the penalty associated with the error estimation in certain portions of the scene surrounding the vehicle is higher.

[0066] Therefore, based on the examples above, it should be clear to readers in this field how to define the estimation error thresholds for the various perception parameters illustrated herein. The value of the threshold can be set based on the type of the parameter and / or the maturity level of the perception development module. In some embodiments, the estimation error can be set to zero or any value higher than zero, meaning that a weak annotation step is triggered if there is any difference between the output of the perception development module and the baseline world observations.

[0067] Furthermore, if the estimation error of 106 is determined to be below (or equal to) a threshold, i.e. approximately zero, then method 100 proceeds to the next iteration to process the buffered data.

[0068] Further, in some embodiments, method 100 may include determining a cost function 109 based on the determined estimation error (e) 106, wherein the cost function indicates the performance of the perception development module. In other words, once a match is identified / established between the perception parameters of the second set of perception data and the corresponding perception parameters in the baseline world observations, a cost function indicative of how well the "perception development module" is performing can be calculated. Additionally, in some embodiments, the cost function (also referred to as a loss function) is determined based on the type of perception parameters. In other words, if the perception model of the perception development module is an object detection algorithm, one cost function can be formed / defined, while if it is a lane tracking / tracing algorithm, another cost function can be formed / defined.

[0069] Furthermore, once the cost function 109 has been determined / computed, one or more parameters of the perception development module 110 can be updated by an optimization algorithm (e.g., backpropagation of a neural network) configured to minimize the computed cost function.

[0070] However, in some embodiments, the determination of the cost function 109 is performed only if the estimation error (e) between the sensing parameters of the second set of sensing data and the corresponding sensing parameters in the baseline world observation is less than or within a threshold range (e.g., [0, x1]). Therefore, the risk of attempting to determine or compute the cost function can be mitigated in situations where it is uncertain whether the sensing parameters of the second set of sensing data have been incorrectly compared to “incorrect” sensing parameters in the baseline world observation.

[0071] Furthermore, in some embodiments, the evaluation 105 of the second set of perceived data and the resulting baseline world observations, representing the region of interest in the vehicle's surrounding environment, is performed only on the second set of perceived data. For example, one can limit the "area" within which the evaluation process 105 is performed, such that one only compares two datasets representing areas within a specific detection range or at a certain distance from the vehicle. Thus, objects farther from the vehicle are more likely to be associated with higher uncertainty and are therefore more difficult to "match," and may therefore be ignored. Similarly, one can ignore "less relevant" areas, such as relative lanes on a controlled highway, during the evaluation 105. Furthermore, the region of interest can be dynamically set based on one or more vehicle parameters, such as, for example, the vehicle's speed. For example, if the vehicle is traveling at a low speed (e.g., <20 km / h), the region of interest can be set to include a smaller area around the vehicle compared to the corresponding case when the vehicle is traveling at a high speed (e.g., >100 km / h). Therefore, this may result in a reduction in the processing power requirements of the evaluation process 105.

[0072] In some embodiments, method 100 further includes: transmitting 111 one or more updated parameters of the perception model of the perception development module to a remote entity. The remote entity may be, for example, a central system or cloud system capable of receiving data and transmitting it to a fleet of vehicles equipped with ADS. Furthermore, in some embodiments, method 100 includes: receiving 112 a set of globally updated parameters of the perception model of the perception development module from the remote entity. Accordingly, this set of globally updated parameters is based on information obtained from multiple vehicles including the perception development module. Once 112 the set of globally updated parameters is received, method 100 may further include: updating 113 the perception model of the perception development module based on the received set of globally updated parameters.

[0073] In other words, the remote entity (also referred to as the "back office") integrates the parameters "learned" by each vehicle in the entire fleet and pushes updates to the perception development module to each vehicle in the fleet. Therefore, one can utilize the entire fleet of acquisition vehicles (rather than a few dedicated "test vehicles") to develop and validate various perception features associated with the ADS. Furthermore, because the data transmitted between the vehicles and the remote entity is in the form of "model parameters" (e.g., network weights), rather than, for example, raw or processed sensor data, advantages in terms of bandwidth requirements, data storage, data security, data privacy, and compliance with data regulations are readily realized. In the current context, the "back office" can be understood as the central entity responsible for managing and maintaining the ADS deployed in the vehicle fleet.

[0074] Executable instructions for performing these functions may optionally be included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0075] Therefore, this invention provides a time- and cost-effective solution for the continuous release of perception features for developing autonomous vehicles. More specifically, the inventors recognized that once the “first” ADS is released and deployed in a fleet of acquisition vehicles, the significant effort (in terms of time and financial investment) associated with that first version can be leveraged to develop successive versions of various perception features. More specifically, this paper proposes using vehicles “far in traffic” (i.e., acquisition vehicles) to develop, test, and validate new software and hardware. The solution proposed herein can be understood as a scheme that effectively utilizes learning at the edge (in each acquisition vehicle) and then potentially integrates the learned parameters / weights into a central / cloud unit through joint learning. In other words, this invention provides a solution for developing new perception features associated with new sensor hardware and developing new algorithms for existing sensors by leveraging acquisition vehicles through joint learning.

[0076] As mentioned in the preceding background section, perception algorithms for autonomous driving (such as, for example, "vision" algorithms) rely on massive datasets of hand-labeled (annotated) data used to train machine learning models. Creating such hand-labeled datasets is not only extremely expensive but also very time-consuming, especially when domain expertise is required. This invention aims to alleviate these problems by automatically selecting the most informative parts of the data for annotation and enabling automatic weak annotation, thereby reducing the need for massive datasets of hand-labeled data.

[0077] Therefore, this paper proposes using a perception platform deployed in a fleet of data acquisition vehicles to identify and weakly annotate datasets that can significantly improve the performance of next-generation perception features associated with new sensor hardware and new algorithms for existing sensors. To achieve this, the paper proposes using the vehicle's own world observation supply as a benchmark to compare with the output of the software (SW) or hardware (HW) under development. In cases where there is a significant difference between the baseline world observation and the world observation perceived by the module under test, the data is weakly (i.e., “inaccurately”) annotated with the baseline world observation. This weakly annotated data can then be used to update the software parameters of the “perception model” in each individual vehicle's module under test, or it can be transferred to the “background” for off-vehicle processing or more accurate annotation. The proposed method and system effectively achieve cost-effective data annotation for “next-generation” perception modules, thereby reducing the cost and time required to develop future products / systems.

[0078] In other words, the inventors recognized that once an ADS is deployed in a data acquisition vehicle, it can be assumed that it has proven to be sufficiently safe, at least under certain conditions (including perception and its performance). Therefore, this paper proposes a learning mechanism in which world observations from the ADS are post-processed to construct a “benchmark” against which the output of the software / hardware under development can be compared. Through this comparison, a system can be built that automatically annotates world observations of the module under development and uses this weakly annotated data for local weakly supervised learning in each data acquisition vehicle, or for sending the annotated data to a “back-end” for offline processing or, possibly, more accurate annotation.

[0079] Figure 3 This is a schematic block diagram representation of system 10, used for weakly annotating data to train a vehicle's perception development module, where the vehicle is equipped with an automated driving system (ADS). Generally speaking, Figure 3 The process describes the flow of information from an event exposed in the vehicle to transmission and subsequent integration in the "background" 2. System 10 includes control circuitry configured to perform functions of the methods disclosed herein, wherein these functions may be included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by the control circuitry. However, for the sake of better illustrating the invention, the control circuitry is described in... Figure 3 The term is represented as multiple “modules”, each of which is associated with one or more specific functions.

[0080] It should be noted that the "perceptual outputs used for evaluation" compared in the "evaluation engine" 35 can be any type of perceptual output, such as objects, their states, free space (no objects), lanes, road locations, drivable areas, and combinations thereof. In the current context, the free space area can be understood as an area in the vehicle's surrounding environment that is free of objects (e.g., other vehicles, pedestrians, obstacles, animals, bicycles, static objects, etc.).

[0081] System 10 includes: control circuitry configured to store sensor data 31 acquired from a set of onboard sensors configured to monitor the vehicle's surrounding environment during a time period. Sensor data 31 may be the output of one or more of, for example, radar, lidar, cameras, and ultrasonic sensors. The control circuitry is further configured to store a first set of perception data acquired from a perception module 32 of the vehicle. Perception module 32 is configured to generate the first set of perception data based on the sensor data 31 acquired from one or more onboard sensors during the time period. In the current context, the "perception module" (i.e., the perception system of the acquisition platform) will be understood as a system responsible for acquiring raw sensor data from onboard sensors (e.g., cameras, lidar and radar, ultrasonic sensors) and converting this raw data into scene understanding (i.e., "world observation," including state estimation and its prediction).

[0082] As already mentioned, the data sets can be stored or preserved, for example, in a data buffer (not shown), wherein the first set of sensing data and the second set of sensing data can be understood as data indicative of the vehicle's environment. For example, this could be the detected objects or their states and / or the vehicle's location, and / or statistical and physical model predictions of future states derived continuously and / or intermittently from a first time point T1 to a second time point T2. This time period (and the corresponding length of the optional data buffer) can be any size deemed feasible, taking into account data capacity limitations and / or the characteristics of the ADS, and can range, for example, from less than a second to several minutes, or more preferably from several seconds to less than a minute.

[0083] Furthermore, system 10 includes a post-processing module 33 for forming a baseline world observation indicating the scene in the vehicle's surrounding environment during that time period. The baseline world observation is formed based on perception data generated by the perception module 32 of the acquired ADS. It should be noted that the term "formed by post-processing the first set of perception data" does not necessarily mean that all stored data is post-processed, but should be interpreted as at least a portion or at least a series of stored perception data being post-processed.

[0084] Furthermore, a second set of perception data is generated by the perception development module (i.e., the module under test, MUT) 34. This second set of data is acquired or received by the "evaluation engine" 35 of system 10. The perception development module 34 is configured to generate the second set of perception data based on a perception model and sensor data acquired from one or more onboard sensors during the time period. The second set of perception data can be understood as the "perception output" from MUT 34, indicating perception parameters of the vehicle's surrounding environment. For example, MUT 34 may include lane tracing / tracking algorithms or road geometry estimation algorithms. Accordingly, the output from the MUT may be a road reference feature estimate (such as a representation of the road or lane geometry in the vehicle's surrounding environment during the time period).

[0085] Furthermore, the input to the perception development module 34 can be any combination of the following: existing acquisition sensors, their raw outputs, raw detections, or any more or less processing state in the chain. It can also be any combination of newly installed sensors and their outputs (i.e., sensor outputs not used to generate baseline world observations). For example, if one wants to test and evaluate object detection algorithms based on different types of input data from novel sensors compared to the object detection algorithms of the "acquisition platform".

[0086] Evaluation engine 35 is configured to evaluate the second set of sensor data with reference to the formed baseline world observations to identify matches between one or more sensor parameters in the second set of sensor data and corresponding sensor parameters in the baseline world observations. In other words, evaluation engine 35 is configured to attempt to match the "sensory output used for evaluation" with corresponding features in the baseline world observations. If a match is found, evaluation engine 35 is further configured to derive an estimation error.

[0087] It should be noted that the outputs compared in the "Evaluation Engine" 35 can be any type of perception output, such as objects, their state or free space (no objects), lanes, road locations and drivable areas or combinations thereof.

[0088] Accordingly, if a match is identified between the perceptual parameters of the second set of perceptual data and the corresponding perceptual parameters in the baseline world observation, the estimation error of the matched perceptual parameters of the second set of perceptual data is determined by referring to the corresponding perceptual parameters in the baseline world observation. However, if no match is identified between the perceptual parameters of the second set of perceptual data and the corresponding perceptual parameters in the baseline world observation, or if the estimation error exceeds a threshold, the evaluation engine may transmit a signal to the learning engine 36.

[0089] The transmitted signal may include instructions to update one or more parameters of the perception model of the perception development module 34 by means of a weakly supervised learning algorithm based on sensor data used for the second set of perception data and by weakly annotating the second set of perception data using baseline world observations. Alternatively or additionally, the system may be configured to transmit the weakly annotated data to the background 2 for offline processing (e.g., manual annotation or analysis). The data transmitted to the background may also be advantageously used to improve the perception module 32 of the acquisition platform.

[0090] Furthermore, in some embodiments, the evaluation engine 35 is configured to evaluate the acquired second set of sensing data against a baseline world observation to determine a cost function. More specifically, the cost function is calculated based on the determined estimation error and indicates the performance of the sensing development module.

[0091] The determined cost function can then be transmitted to the learning engine 36, which is configured to update one or more parameters of the perception model of the perception development module 34 by means of an optimization algorithm (e.g., backpropagation of a neural network) configured to minimize the computed cost function.

[0092] Furthermore, the learning engine 36 is further configured to transmit one or more locally updated parameters to remote entity 2, where they are integrated with updates received from other vehicles in the convoy, and then transmitted back to the learning engine 36 as a set of globally updated parameters. The learning engine 36 can then update the perception model of the perception development module 34 based on the received set of globally updated parameters.

[0093] The learning platform proposed in this paper exposes the visual / perceptual benchmark locally within the platform, allowing the comparison / cost function for evaluation or learning of the perception development module 34 to be customized according to current needs. Furthermore, the platform includes redundant computational capabilities required to implement the comparison and learning steps.

[0094] In summary, System 10 provides a means for training a vehicle's perception development module by weakly annotating data through comparison of the outputs from the "test module" 34 with baseline world observations (used as basic facts). In some embodiments, this comparison can be evaluated using a customizable cost function. The baseline world observations can be used as weak annotations to the world observations of the test module 34 if the value of the cost function exceeds a set threshold, or if a mismatch is found between the two outputs (e.g., if parts of the world observations cannot be matched to obtain a reliable value for calculating the cost function). This is referred to as a weak annotation of the world observations compared to manually labeled world observations due to the potential uncertainties and inaccuracies in the baseline world observations. The weakly annotated data can then be used in two different ways:

[0095] • Weakly annotated data can be fed into weakly supervised learning algorithms in each individual vehicle. This allows for local updates of perception model parameters, which are then transmitted to the "backend" for integration.

[0096] Weakly annotated data can be sent to the backend for more accurate annotation or analysis, which can also be used to improve the data collection platform.

[0097] For example, the choice between these two options can depend on the confidence level of the baseline world observations, which can be obtained as a byproduct of post-processing of the outputs of all sensors in the vehicle. If the difference between the baseline world observations and the post-processed sensor world observations lies in areas where the baseline world observations have inherently low confidence, or if the perception development module 34 is very immature, it can benefit greatly from manually annotated data in the background. However, this is in contrast to the traditional data annotation process that uses massive datasets and spends considerable time and effort reviewing and analyzing them to find a “suitable” dataset for supervised learning. The solution proposed in this paper can also be used to automatically provide high-quality datasets for offline processing (e.g., manually labeled), which significantly reduces the time and effort required to develop new perception models for autonomous driving applications.

[0098] From a more comprehensive perspective, the methods and systems disclosed in this paper can be used to develop and launch new versions of sensing software or hardware according to the following phases: First, a learning phase or period, and an integration phase, in which new modules are developed based on... Figure 1 and Figure 3 The process described in the document is updated. Once sufficient maturity is achieved, the integrated model parameters are pushed to a fleet of vehicles for validation. During this validation, this step (the "learning part" of the process, i.e., the reference step) is not performed. Figure 3 The learning engine 36 is "short-circuited." Therefore, only the evaluation part of the process is used. Then, if the new module passes the validation phase in every vehicle in the fleet (i.e., the failure rate is low enough), the final software can be launched. However, if validation fails, one can return to the starting point and begin a new iteration of the process.

[0099] Figure 4This is a schematic perspective view of the matching process used to match a second set of perceived data with the resulting baseline world observations. In the example shown, the second set of perceived data includes object detection estimates, and an example of how to propagate the match between the two datasets beyond a threshold to the evaluation engine 35 and the learning engine 36 to update the model parameters of the perception development module of vehicle 1, or to transmit weakly annotated data to the "background" 2. The determined estimation error (as indicated by the double arrows) could be, for example, the positional difference between the bounding box around the detected object 41 indicated in the second set of perceived data and the bounding box around the corresponding object 42 in the baseline world observation.

[0100] also, Figure 4 This also serves to clarify the matching process described above. Here, for the two vehicles 41a to 41b detected by the perception development module, a match is identified between the perception parameters (objects) in the second set of perception data and the corresponding perception parameters (objects) in the baseline world observations, while for the third vehicle 41c, no match is identified. For the identified match, the estimation error is determined and compared with a threshold. In the case where a match cannot be identified / established (i.e., Figure 4 (The object within the dashed circle) can be defined as "infinity" or a certain "maximum value".

[0101] For example, if the second set of perception data indicates blue and red vehicles ahead of the vehicle, the matching process ensures that the red vehicle matches its corresponding red vehicle in the baseline world observation, and the blue vehicle matches its corresponding blue vehicle in the baseline world observation. Otherwise, the perception model of the perception development module may be trained with incorrect data (based on incorrect conclusions), which will degrade the performance of the perception development module. Therefore, as... Figure 4 As illustrated in the diagram, the third vehicle 41c detected by the perception development module was not close enough to any vehicle in the baseline world observation, so the evaluation engine could not identify any match.

[0102] Since neither of the objects in the two perception outputs resulted in a perfect match, it can be concluded that the perception model of the perception development module generated an inaccurate representation of the surrounding environment. Therefore, to update one or more parameters of the perception model, a weakly supervised learning algorithm can be used, where the sensor data used to generate the second set of perception data, together with baseline world observations, forms a "training dataset" or "training examples." More specifically, the sensor data used for the second set of perception data forms the "input objects," and the baseline world observations are used as weakly annotated data to form the "supervision signal" (i.e., the "desired output").

[0103] Figure 5 and Figure 6Lane geometry estimation algorithm ( Figure 5 ) and free space detection algorithm ( Figure 6 The corresponding evaluation process for the perception model is illustrated in the form of (). More detailed, Figure 5 This is a schematic top view of a vehicle traveling on a road section according to an embodiment of the present invention. The road section has two lanes and curves to the left in front of the vehicle 1. “Real” lane markings are indicated by reference numeral 53, lane geometry estimates from baseline world observations are indicated by reference numeral 52, and lane geometry estimates from the perception development module are indicated by reference numeral 51. Similar to before, mismatches between the lane trajectory output by the perception development module and the baseline world observations are identified, and the baseline world observations are used as weakly annotated data, which is used to update one or more parameters of the perception model of the perception development module via a weakly supervised learning algorithm or for transmission to the background. Here, the estimation error 55 is indicated as the difference between the position / geometry of the lane geometry estimates 51 and 52. More specifically, the lane geometry estimate 51 of the perception development module fails to generate a representation of the lane markings in some areas (indicated by reference numeral 55). Furthermore, in some embodiments, the evaluation is performed at the pixel level, in the image plane. Therefore, camera software can be evaluated and trained directly at the pixel level.

[0104] Furthermore, Figure 6 This is a schematic top view of a vehicle traveling on a road section according to an embodiment of the invention. The road section has two lanes and three external objects 63 in the form of two cars and a tractor located in front of the vehicle. The free space estimate performed by the acquisition platform (i.e., the free space estimate of the baseline world observation) is indicated by reference numeral 62, while the free space estimate of the perception development module is indicated by reference numeral 61. In the example shown, the estimation errors 65', 65' are simply derived from measurements or metrics indicating the non-overlapping portions of the two free space estimates 61, 62. In set theory terms, this can be understood as a symmetrical difference between the free space set defined by the free space estimate 61 of the perception development module and the free space estimate 62 defined by the baseline world observation.

[0105] References above Figure 4 and Figure 5 The example implementations discussed are similar; if the mismatch or estimation error 65', 65" is higher than the threshold, the stored data is used to train the perception model of the perception development module or is transmitted to the backend 2 for further analysis. However, as Figure 6As indicated, the free space estimate 62 of the perception development module is associated with two different estimation errors 65' and 65'", wherein a smaller deviation from the "ground fact" 61 is indicated by reference numeral 65', while a larger deviation is indicated by reference numeral 65'. In this example, the larger deviation or estimation error 65' is considered to be above a certain "maximum error threshold", and the data is transmitted to the backend 2 for manual annotation / further analysis. On the other hand, the smaller deviation or estimation error 65' is considered to be below the "maximum threshold" and can therefore be used to locally update the perception model according to the foregoing embodiments.

[0106] As already mentioned, the free space region can be understood as the area in the vehicle's surrounding environment where there are no objects (e.g., other vehicles, pedestrians, obstacles, animals, bicycles, static objects, etc.). Therefore, the location of the free space region can be understood as an estimate of the area where no external objects (static and dynamic objects) exist. One can consider the estimation of the "drivable area" in a similar way, which includes not only the estimation of the area where no objects exist (as in the case of free space) but also the estimation of the presence of the road surface.

[0107] Figure 7 A schematic side view of a vehicle 1 is depicted, including a system 10 for weakly annotating data to train an ADS-equipped vehicle according to an embodiment of the invention. The vehicle 1 further includes a perception module / system 6 (i.e., the perception system of the acquisition platform) and a positioning system 5. In the current context, the perception system 6 will be defined as the system responsible for acquiring raw sensor data from airborne sensors 6a, 6b, 6c (e.g., cameras, lidar and radar, ultrasonic sensors) and converting that raw data into contextual understanding. The positioning system 5 is configured to monitor the vehicle's geographic location and heading, and may be in the form of a Global Navigation Satellite System (GNSS) such as GPS. However, alternatively, the positioning system may be implemented as Real-Time Kinematic (RTK) GPS to improve accuracy.

[0108] More specifically, the perception module / system 6 can refer to any well-known system and / or function, such as that included in one or more electronic control modules and / or nodes of vehicle 1, adapted and / or configured to interpret driving-related sensing information of vehicle 1 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. Thus, the exemplary perception system 6 can be adapted to rely on and obtain input from combined sensing information from multiple data sources, such as vehicle imaging, image processing, computer vision, and / or in-vehicle networks. This exemplary sensing information can be obtained, for example, from one or more optional ambient detection sensors 6 included in and / or provided on vehicle 1. The ambient detection sensors 6 can be represented by any sensor adapted to sense and / or perceive the environment and / or location of vehicle 1, and can refer to, for example, one or a combination of radar, lidar, sonar, cameras, navigation systems (e.g., GPS), odometers, and / or inertial measurement units.

[0109] System 10 includes one or more processors 11, memory 12, sensor interface 13, and communication interface 14. Processor 11 may also be referred to as control circuitry 11 or control circuitry system 11. Control circuitry 11 is configured to execute instructions stored in memory 12 to perform a method for weakly annotating data to train a perception development module of vehicle 1 according to any of the embodiments disclosed herein. In other words, memory 12 of control device 10 may include one or more (non-transitory) computer-readable storage media for storing computer-executable instructions that, when executed by one or more computer processors 11, may, for example, cause computer processors 11 to perform the techniques described herein. Optionally, memory 12 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state storage devices; and, optionally, includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0110] Control circuit 11 is configured to store sensor data acquired from one or more onboard sensors configured to monitor the vehicle's surrounding environment during a time period. Control circuit 11 is further configured to store a first set of perception data acquired from the vehicle's ("acquisition platform") perception module 6 during this time period. Perception module 6 is configured to generate the first set of perception data based on the sensor data acquired from the one or more onboard sensors 6 during this time period. Furthermore, control circuit 11 is configured to form a baseline world observation indicating the scene in the vehicle's surrounding environment during this time period by post-processing the first set of perception data.

[0111] Control circuit 11 is further configured to receive signals from the sensing development module (not in...) Figure 7 (As shown in the diagram) A second set of perception data is obtained, wherein the perception development module is configured to generate the second set of perception data based on the perception model and sensor data obtained from one or more onboard sensors 6 during the time period. The second set of perception data indicates the perception parameters of the vehicle's surrounding environment during the time period. It should be noted that the sensor data used to generate the first set of perception data does not need to be the same as the sensor data used to generate the second set of perception data. In some embodiments, the first set of perception data is generated based on the first set of sensor data, and the second set of perception data is generated based on the second set of sensor data. In some embodiments, the first set of sensor data and the second set of sensor data are the same, while in other embodiments, the second set of sensor data is a subset of the first set of sensor data. Alternatively, the first set of sensor data may be different from the second set of sensor data.

[0112] Furthermore, control circuit 11 is configured to evaluate the acquired second set of sensing data relative to the formed baseline world observation to identify a match between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observation. Then, if a match is identified between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observation, the control circuit is further configured to determine the estimation error of the matched sensing parameters in the second set of sensing data by referring to the corresponding sensing parameters in the baseline world observation.

[0113] Furthermore, if no match is identified between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observations, or if the estimation error exceeds a threshold, the control circuit is further configured to perform at least one of the following:

[0114] • Update one or more parameters of the perception model of the perception development module by using a weakly supervised learning algorithm based on sensor data used for the second set of perception data and by using baseline world observations as underlying facts to form weakly annotated data.

[0115] • The stored sensor data, the stored second set of sensing data, and the baseline world observations are transmitted as weakly annotated data to a remote entity for off-vehicle processing.

[0116] Furthermore, vehicle 1 can connect to external network 20 via, for example, a wireless link (e.g., for transmitting and receiving model parameters). The same or some other wireless links can be used to communicate with other vehicles in the vicinity or with local infrastructure components. Cellular communication technology can be used for, for example, long-range communication to external networks, and if the cellular communication technology used has low latency, it can also be used for vehicle-to-vehicle, vehicle-to-vehicle (V2V), and / or vehicle-to-infrastructure (V2X) communication. Examples of cellular radio technologies are GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc., and also include future cellular solutions. However, in some solutions, short- to medium-range communication technologies such as wireless local area networks (LANs) based on, for example, IEEE 802.11 are used. ETSI is developing cellular standards for vehicle communication, and 5G is considered a suitable solution, for example, due to its low latency and efficient handling of high bandwidth and communication channels.

[0117] The present invention has been presented above with reference to specific embodiments. However, other embodiments besides those described above are also possible and are within the scope of the invention. Within the scope of the invention, method steps different from those described above, performed by hardware or software, can be provided. Therefore, according to exemplary embodiments, a (non-transitory) computer-readable storage medium is provided for storing one or more programs configured to be executed by one or more processors of a vehicle control system, the one or more programs including instructions for performing the method according to any of the above embodiments.

[0118] Generally, computer-accessible media can include any tangible or non-transitory storage medium or memory medium, such as electronic, magnetic, or optical media (e.g., a disk or CD / DVD-ROM coupled to a computer system via a bus). As used herein, the terms “tangible” and “non-transitory” are intended to describe computer-readable storage media (or “memory”) that do not transmit electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices included in the term computer-readable media or memory. For example, the terms “non-transitory computer-readable medium” or “tangible memory” are intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium can be further transmitted via a transmission medium or via signals (such as electrical, electromagnetic, or digital signals) that can be transmitted via communication media such as networks and / or wireless links.

[0119] Processor 11 (associated with system 10) may be or include any number of hardware components for performing data or signal processing or for executing computer code stored in memory 12. Device 10 has an associated memory 12, and memory 12 may be one or more means for storing data and / or computer code for performing or facilitating the various methods described herein. Memory may include volatile or non-volatile memory. Memory 12 may include database components, object code components, script components, or any other type of information structure for supporting the various activities described herein. According to exemplary embodiments, any distributed or local storage device may be used with the systems and methods of this specification. According to exemplary embodiments, memory 12 (e.g., via circuitry or any other wired, wireless, or network connection) may be communicatively connected to processor 11 and includes computer code for performing one or more of the processes described herein.

[0120] It should be understood that sensor interface 13 can also provide the possibility of acquiring sensor data directly or via dedicated sensor control circuitry 6 in the vehicle. Communication / antenna interface 14 can further provide the possibility of transmitting output to a remote location (e.g., a remote operator or control center) via antenna 8. Furthermore, some sensors in the vehicle can communicate with system 10 using a local network setup such as CAN bus, I2C, Ethernet, fiber optics, etc. Communication interface 14 can be arranged to communicate with other control functions of the vehicle and can therefore also be considered a control interface; however, a separate control interface (not shown) can be provided. Local communication within the vehicle can also be wireless, using protocols such as WiFi, LoRa, Zigbee, Bluetooth, or similar medium / short-range technologies.

[0121] It should be noted that the word "comprising" does not exclude the presence of other elements or steps besides those listed, and the word "a" or "the" preceding an element does not exclude the presence of a plurality of such elements. It should also be noted that any reference numerals do not limit the scope of the claims, the invention can be implemented at least in part by both hardware and software, and multiple "devices" or "units" can be represented by the same item of hardware.

[0122] Although the accompanying drawings may show a specific order of method steps, the order of steps may differ from the depicted order. Furthermore, two or more steps may be performed simultaneously or partially concurrently. For example, the steps of forming baseline world observations and acquiring a second set of sensing data may be interchanged based on the specific implementation. This variation will depend on the chosen software and hardware system and the designer's choices. All these variations are within the scope of the invention. Similarly, the software implementation can be accomplished using standard programming techniques with rule-based logic and other logic to perform various connection steps, processing steps, comparison steps, and decision steps. The embodiments mentioned and described above are given by way of example only and should not be construed as limiting the invention. Other solutions, uses, objectives, and functions within the scope of the invention claimed in the patent embodiments described above should be apparent to those skilled in the art.

Claims

1. A method (100) performed by a computing system for weakly annotating data to train a vehicle's perception development module, wherein, The vehicle is equipped with an automated driving system (ADS), and the method includes: Storing (101) sensor data obtained from a set of onboard sensors configured to monitor the surrounding environment of the vehicle during a period of time; During the time period, (102) a first set of perception data obtained from the vehicle's perception module is stored, the perception module being configured to generate the first set of perception data based on sensor data obtained from one or more of the set of on-board sensors during the time period. By post-processing (103) the first set of sensing data, a baseline world observation is formed from the obtained first set of sensing data, the baseline world observation indicating the surrounding environment of the vehicle during the time period; During the time period, a second set of perception data from the perception development module is stored (104), wherein the perception development module is configured to generate the second set of perception data based on a perception model and sensor data obtained from one or more of the set of on-board sensors during the time period; The second set of perception data indicates the perception parameters of the vehicle's surrounding environment during the time period. The second set of sensing data is evaluated relative to the formed baseline world observations to identify the match between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observations; If a match is identified between the sensing parameter in the second set of sensing data and the corresponding sensing parameter in the baseline world observation, the method further includes: determining (106) the estimation error of the matching sensing parameter in the second set of sensing data by referring to the corresponding sensing parameter in the baseline world observation; If no match is identified between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observations, or if the estimation error exceeds a threshold, the method further includes: One or more parameters of the perception model in the perception development module are updated by using a weakly supervised learning algorithm with the sensor data used as input data for the second set of perception data and by using the baseline world observations that form basic facts as weakly annotated data; or The stored sensor data, the stored second set of sensing data, and the baseline world observations are transmitted as weakly annotated data (108) to a remote entity for non-vehicle processing.

2. The method (100) according to claim 1, wherein, The sensing parameters include at least one of the following parameters: Object detection estimation; Object classification estimation; Object state estimation; Road reference feature estimation; Free space estimation; Road friction estimation; Object trajectory estimation; and Driving area estimation.

3. The method (100) according to any one of claims 1 to 2, wherein, The time period is from a first time point to a second time point, and the steps for forming the baseline world observation include: The baseline world observation, which indicates the vehicle's surrounding environment, is determined by post-processing a portion of the first set of sensing data that extends from the second time point back to an intermediate time point between the first and second time points.

4. The method (100) according to claim 3, wherein, Post-processing (103) of said portion of the first set of perceived data includes: The portion of the first set of sensing data is run by a backward filter configured to align the set of sensing parameters of the first set of sensing data at the intermediate time point based on the evolution of the state of the set of sensing parameters from the intermediate time point to the second time point.

5. The method (100) according to claim 1, further comprising: The cost function (109) indicating the performance of the perception development module is determined based on the determined estimation error; and The one or more parameters of the perception model of the perception development module (110) are updated by an optimization algorithm configured to minimize the calculated cost function.

6. The method (100) according to claim 5, wherein, The cost function is determined based on the type of the perceived parameters.

7. The method (100) according to claim 1, further comprising: Transmit (111) one or more updated parameters of the perception model of the perception development module to the remote entity; (112) Receive from the entity a set of globally updated parameters of the perception model of the perception development module, wherein the set of globally updated parameters is based on information obtained from multiple vehicles including the perception development module; The perception model of the perception development module (113) is updated based on a set of globally updated parameters received.

8. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an in-vehicle processing system, the one or more programs including instructions for performing the method (100) according to claim 1.

9. A system (10) for weakly annotating data to train a perception development module for a vehicle (1), wherein, The vehicle is equipped with an automated driving system (ADS), the system including a control circuit (11), the control circuit (11) being configured to: Sensor data (31) obtained from a set of onboard sensors (6a, 6b, 6c) configured to monitor the surrounding environment of the vehicle (1) is stored during a period of time. During the time period, a first set of perception data obtained from the perception module (6, 32) of the vehicle (1) is stored, the perception module being configured to generate the first set of perception data based on sensor data obtained from one or more of the set of on-board sensors during the time period; By post-processing the first set of sensing data, a baseline world observation is formed from the obtained first set of sensing data, the baseline world observation indicating the development of the scenario in the surrounding environment of the vehicle during the time period; The second set of perception data is stored from the perception development module (34), wherein the perception development module is configured to generate the second set of perception data based on the perception model and sensor data obtained from one or more of the set of vehicle sensors during the time period; The second set of perception data indicates the perception parameters of the vehicle's surrounding environment during the time period. The second set of sensing data is evaluated relative to the established baseline world observations to identify matches between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observations; If a match is identified between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observation, the control circuit is further configured to: determine the estimation error of the matched sensing parameters in the second set of sensing data by referring to the corresponding sensing parameters in the baseline world observation; and If no match is identified between the sensing parameters in the second set of sensing data and the corresponding sensing parameters in the baseline world observation, or if the estimation error exceeds a threshold, the control circuit is further configured to: One or more parameters of the perception model in the perception development module are updated by using a weakly supervised learning algorithm with sensor data as input data for the second set of perception data and by using baseline world observations that form basic facts as weakly annotated data; or The stored sensor data, the stored second set of sensing data, and the baseline world observations are transmitted as weakly annotated data to a remote entity for non-vehicle processing.

10. The system (10) according to claim 9, wherein, The time period ranges from a first time point to a second time point, and the control circuit is configured to form the baseline world observation through the following steps: The baseline world observation, which indicates the vehicle's surrounding environment, is determined by post-processing a portion of the first set of sensing data that extends from the second time point back to an intermediate time point between the first and second time points.

11. The system (12) according to claim 10, wherein, Post-processing of the portion of the first set of sensed data includes: The portion of the first set of sensing data is run by a backward filter configured to align the set of sensing parameters of the first set of sensing data at the intermediate time point based on the evolution of the state of the set of sensing parameters from the intermediate time point to the second time point.

12. A vehicle (1), comprising: A set of onboard sensors (6a, 6b, 6c) is configured to monitor the vehicle’s surrounding environment; An automated driving system (ADS) with a perception module, wherein the perception module is configured to generate a first set of perception data based on sensor data obtained from one or more of the set of onboard sensors; The perception development module (34) is configured to generate a second set of perception data based on the perception model and sensor data obtained from one or more of the set of on-board sensors. and The system (10) according to any one of claims 9-11.

Images