End-to-end perceptual disturbance modeling system for vehicle

Abstract

An end-to-end perceptual disturbance modeling system for a vehicle includes one or more controllers that store probe models in a memory. The detection model includes a plurality of detection model maps, each of which represents a probability value of detecting an object in the vehicle surroundings based on a current weather condition and a distance measured between the vehicle and the object detected in the vehicle surroundings. The one or more controllers execute instructions to receive an input state of the vehicle observed during a non-severe weather condition and representing one or more vehicle states, a current weather condition, and a distance. The controller determines a disturbance state of the vehicle observed during the current weather condition.

Description

Technical Field

[0001] The present disclosure relates to an end-to-end perceptual disturbance modeling system for a vehicle, wherein the end-to-end perceptual disturbance modeling system takes into account adverse weather conditions. Background Art

[0002] Autonomous vehicles perform a variety of tasks (such as, but not limited to, perception, localization, mapping, path planning, decision making, and motion control). For example, an autonomous vehicle may include perception sensors (such as cameras for collecting image data about the vehicle's surroundings). Adverse weather conditions (such as rain, fog, snow, or low light conditions at dusk or night) can reduce the visibility of surrounding objects, thereby affecting the autonomous vehicle's ability to perceive the surrounding environment. For example, the quality of image data collected by a camera can be severely affected by precipitation and low light. Therefore, an effective method for detecting objects in clear weather conditions may not accurately detect objects in adverse weather conditions. However, current methods ignore the impact of adverse weather conditions on the perception system used for autonomous vehicles, and only consider general weather conditions or only simulate the performance of specific sensors under different weather conditions.

[0003] In addition to the challenges discussed above, it should be recognized that perception-related autonomous vehicle tasks require relatively large training datasets based on real-world conditions, both in general and for each specific adverse weather condition that the vehicle may encounter. Acquiring real-world data for creating training datasets is often very time-consuming and difficult. In some cases, certain data based on real-world conditions may not be available. To overcome this limitation, synthetic data generated based on one or more machine learning techniques may be employed instead. However, synthetic data is based on real-world data, and therefore the quality of the synthetic data is limited by the capabilities of the machine learning techniques and the real-world driving scenarios present in the existing data.

[0004] Therefore, while object detection and perception systems for autonomous vehicles achieve their intended purposes, there is a need in the art for improved methods for object detection and perception that account for adverse weather conditions in the absence of sufficient real-world data. Summary of the invention

[0005] According to several aspects, an end-to-end perceptual disturbance modeling system for a vehicle is disclosed. The end-to-end perceptual disturbance modeling system includes one or more controllers that store a detection model in a memory. The detection model includes a plurality of detection model graphs, each detection model graph representing a probability value of detecting the object in the vehicle's surrounding environment based on current weather conditions and a distance measured between the vehicle and the object detected in the vehicle's surrounding environment. One or more controllers execute instructions to receive an input state of the vehicle, wherein the input state is observed during non-severe weather conditions and represents one or more vehicle states, current weather conditions, and distances. One or more controllers determine a probability value corresponding to the current weather conditions and distances based on the detection model. One or more controllers perform a simulation based on the probability value, wherein the simulation determines the probability of detecting an object located in the environment based on the current weather conditions and the distance. In response to determining that the probability of detecting an object located in the environment indicates that the object is about to be detected, the one or more controllers determine the perturbation state of the vehicle observed during the current weather conditions.

[0006] In one aspect, current weather conditions represent adverse weather conditions.

[0007] In another aspect, the adverse weather condition is one of: dusk, night, rain, heavy rain, fog, and heavy fog.

[0008] In yet another aspect, one or more controllers store in memory a set of vehicle state disturbance models, each vehicle state disturbance model corresponding to one of the vehicle states.

[0009] In one aspect, each set of vehicle state disturbance models includes a plurality of vehicle state disturbance models, each vehicle state disturbance model representing a perception error distribution corresponding to specific current weather conditions and a specific distance measured between the vehicle and the object detected in the environment.

[0010] In another aspect, one or more controllers execute instructions to determine a perceived error for each vehicle state at a distance during current weather conditions as indicated by the input state based on the set of vehicle state disturbance models.

[0011] In yet another aspect, one or more controllers execute instructions to determine a correlation value between two perception errors corresponding to two vehicle states in the vehicle state under current weather conditions and distance indicated by an input state, wherein the correlation value represents a functional relationship between two vehicle state disturbance models in the vehicle state disturbance model.

[0012] In one aspect, one or more controllers execute instructions to compare an association value of a perceived error between two vehicle states in a vehicle state with an expected range of sensitivity values, wherein the expected range of sensitivity values ​​represents an expected difference between the perceived errors of the two vehicle states; and in response to determining that the association value is within the expected range of sensitivity values, determine that the perceived error between the two vehicle states is valid.

[0013] In another aspect, one or more controllers execute instructions to determine a disturbance state of the vehicle by adding a perception error to each raw value of the vehicle state.

[0014] In yet another aspect, the one or more vehicle states include one or more of: a speed, a heading angle of the vehicle, and a position of the vehicle.

[0015] In one aspect, a detection model is created based on synthetic data.

[0016] In another aspect, the end-to-end perturbation modeling system is integrated with a trajectory prediction system and a final motion planner that are part of one of an automated driving system (ADS) and an advanced driver assistance system (ADAS).

[0017] In one aspect, a method for determining a disturbance state of a vehicle by an end-to-end perceptual disturbance modeling system is disclosed. The method includes: receiving an input state of a vehicle by one or more controllers, wherein the input state is observed during non-severe weather conditions and represents one or more vehicle states, current weather conditions, and a distance measured between the vehicle and an object detected in the vehicle's surroundings. The method includes: determining a probability value corresponding to the current weather conditions and the distance based on a detection model stored in a memory of one or more controllers, wherein the probability value indicates that an object in the vehicle's surroundings is detected. The method includes: performing a simulation based on the probability value, wherein the simulation determines a probability of detecting an object located in the environment based on the current weather conditions and the distance. Finally, in response to determining that the probability of detecting an object located in the environment indicates that the object is about to be detected, the method includes: determining a disturbance state of the vehicle observed during the current weather conditions.

[0018] In another aspect, the method includes determining a perception error for each vehicle state at a distance during current weather conditions indicated by an input state based on a set of vehicle state disturbance models stored in a memory of one or more controllers, wherein each set of vehicle state disturbance models includes a plurality of vehicle state disturbance models, each vehicle state disturbance model representing a perception error distribution corresponding to a specific current weather condition and a specific distance measured between the vehicle and an object detected in the environment.

[0019] In yet another aspect, the method includes determining a correlation value between two perception errors corresponding to two vehicle states in the vehicle state at the current weather condition and distance indicated by the input state, wherein the correlation value represents a functional relationship between two vehicle state disturbance models in the vehicle state disturbance model.

[0020] In one aspect, the method includes: comparing an association value of two perceived errors between two vehicle states with an expected sensitivity value range, wherein the expected sensitivity value range represents an expected difference between the perceived errors of the two vehicle states; and in response to determining that the association value is within the expected sensitivity value range, determining that the perceived error between the two vehicle states is valid.

[0021] In another aspect, the method includes determining a disturbance state of the vehicle by adding a perception error to each raw value of the vehicle state.

[0022] In one aspect, an end-to-end perceptual disturbance modeling system for a vehicle is disclosed. The end-to-end perceptual disturbance modeling system includes one or more controllers storing a detection model and a set of vehicle state perturbation models in a memory, wherein the detection model is created based on synthetic data and includes a plurality of detection model graphs, each detection model graph representing a probability value of detecting an object in the vehicle's surrounding environment based on current weather conditions and a distance measured between the vehicle and the object detected in the vehicle's surrounding environment; and wherein each set of vehicle state perturbation models includes a plurality of vehicle state perturbation models, each vehicle state perturbation model representing a perceptual error distribution corresponding to a specific current weather condition and a specific distance measured between the vehicle and the object detected in the environment; and wherein one or more controllers execute instructions to receive an input state of the vehicle, wherein the input state is observed during non-severe weather conditions and represents one or more vehicle states, current weather conditions, and distances. One or more controllers determine a probability value corresponding to the current weather conditions and distances based on the detection model. One or more controllers perform simulations based on the probability values, wherein the simulations determine the probability of detecting an object located in the environment based on the current weather conditions and distances. In response to determining that the probability of detecting an object located in the environment indicates that the object will be detected, the one or more controllers determine a disturbance state of the vehicle observed during current weather conditions, wherein the current weather conditions represent adverse weather conditions.

[0023] In another aspect, one or more controllers execute instructions to determine, based on the set of vehicle state disturbance models, a perceived error for each vehicle state at a distance during current weather conditions indicated by an input state; and determine a correlation value between two perceived errors between two vehicle states in the vehicle state under the current weather conditions and distance indicated by the input state, wherein the correlation value represents a functional relationship between two vehicle state disturbance models in the vehicle state disturbance model.

[0024] In yet another aspect, one or more controllers execute instructions to compare an associated value of a perceived error between two vehicle states in a vehicle state with an expected range of sensitivity values, wherein the expected range of sensitivity values ​​represents an expected difference between the perceived errors of the two vehicle states; in response to determining that the associated value is within the expected range of sensitivity values, determine that the perceived error between the two vehicle states is valid; and determine a disturbance state of the vehicle by adding the perceived error to each raw value of the vehicle state.

[0025] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0027] Figure 1 is a schematic diagram of a vehicle including the disclosed end-to-end perceptual disturbance modeling system according to an exemplary embodiment, the end-to-end perceptual disturbance modeling system including one or more controllers; Figure 2 According to an exemplary embodiment, Figure 1 The software architecture of one or more controllers of the end-to-end disturbance-aware modeling system shown; Figure 3 An exemplary detection model including a plurality of detection model graphs according to an exemplary embodiment is shown; Figure 4 An exemplary vehicle state disturbance model according to an exemplary embodiment is shown; and Figure 5 is a diagram showing a method for utilizing Figure 1 A flow chart of a method for determining a disturbance state of a vehicle by an end-to-end perceptual disturbance modeling system is shown. DETAILED DESCRIPTION

[0028] The following description is merely exemplary in nature and is not intended to limit the present invention, application, or uses.

[0029] See also Figure 1 , an exemplary end-to-end perceptual disturbance modeling system 10 for a vehicle 12 is shown. It should be appreciated that the vehicle 12 may be any type of vehicle, such as, but not limited to, a car, a truck, a sport utility vehicle, a van, or an RV. The vehicle 12 includes an automated driving system (ADS) or an advanced driver assistance system (ADAS) for assisting a driver with steering, braking, and / or acceleration. The end-to-end perceptual disturbance modeling system 10 includes one or more controllers 20 in electronic communication with a plurality of perceptual sensors 22, the perceptual sensors 22 being configured to collect perceptual data representing the environment surrounding the vehicle 12. In the example shown in FIG. Figure 1 In the illustrated non-limiting embodiment, the plurality of perception sensors 22 include one or more cameras 30, an inertial measurement unit (IMU) 32, a global positioning system (GPS) 34, radar 36, and lidar 38. However, it should be appreciated that additional sensors may be employed.

[0030] like Figure 1 As shown, one or more controllers 20 of the end-to-end perceptual disturbance modeling system 10 include a detection module 24 and a perturbation module 26. One or more controllers 20 receive an input state X of the vehicle 12, wherein the input state X is observed by a plurality of perceptual sensors 22 under non-bad weather conditions. One or more controllers 20 determine a perturbation state X' of the vehicle 12, wherein the perturbation state X' of the vehicle 12 is observed by a plurality of perceptual sensors 22 during current weather conditions W. It should be appreciated that the current weather conditions W represent adverse weather conditions that negatively affect the visibility of objects in the surrounding environment of the vehicle 12. In addition, it should also be appreciated that adverse weather conditions include weaker lighting conditions that occur at dusk or at night. Some examples of adverse weather conditions include, but are not limited to, dusk, night, rainy days, heavy rainy days, foggy days, and heavy foggy days.

[0031] The input state X indicates one or more vehicle states, current weather conditions W, a distance d measured between the vehicle 12 and an object located in the environment surrounding the vehicle 12, and a category C of the object. In a non-limiting embodiment, the vehicle state includes a speed S, a heading angle H, and a position L of the vehicle 12, however, it should be appreciated that additional or different vehicle states (such as, for example, acceleration) may also be included. As described above, examples of current weather conditions include, but are not limited to, non-inclement weather, dusk, night, rain, heavy rain, fog, and heavy fog. Objects located in the environment surrounding the vehicle 12 represent target objects, which the end-to-end perception perturbation modeling system 10 determines to be detectable during the current weather conditions W. Some examples of the category C of the object include, but are not limited to, vehicles, pedestrians, bicycles, or animals. It should be appreciated that different object categories C produce different perturbation patterns, which are part of a probability distribution indicating whether an object is detected, wherein the probability distribution for indicating the detection of an object is described below.

[0032] The disturbance state X' represents the probability of detecting an object in the environment and one or more disturbance vehicle states, where the disturbance vehicle states are determined based on the current weather conditions W. The probability of detecting an object in the environment is expressed as a true or false value (T / F), where a true value T indicates that the object will be detected and a false value F indicates that the object will not be detected. Figure 1 In the non-limiting embodiment shown, the disturbed vehicle states include a disturbed speed S*, a disturbed heading angle H*, and a disturbed position L* of the vehicle 12 observed during the current weather condition W. However, as described above, other disturbed vehicle states may also be employed.

[0033] As described below, the detection module 24 of the one or more controllers 20 receives the input state X and determines a probability of detecting an object located in the environment based on the current weather conditions W and the distance d measured between the vehicle 12 and the object located in the environment surrounding the vehicle 12. In response to determining that the probability is true, the disturbance module 26 of the one or more controllers 20 then determines a disturbed state X' of the vehicle 12. Therefore, in the event that the vehicle 12 encounters adverse weather conditions that affect the visibility of the environment surrounding the vehicle 12, the disclosed end-to-end perceptual disturbance modeling system 10 determines the probability of detecting the object and the disturbed vehicle state X' observed during the current weather conditions W.

[0034] Figure 2 The software architecture of the one or more controllers 20 is shown. The detection module 24 of the one or more controllers 20 includes a detection model library 50, a detection model selector 52, and a probability calculation block 54. The disturbance module 26 of the one or more controllers 20 includes a disturbance model library 56, a disturbance model selector 58, a sensitivity block 60, a correlation block 62, and a disturbance calculation block 64.

[0035] The detection module 24 will now be described. The detection model library 50 of the detection module 24 stores the detection models 82 in the memory of one or more controllers 20. Figure 3 The detection model 82 includes a plurality of detection model graphs 84, each of which represents a probability value of detecting an object located in the surrounding environment of the vehicle 12 based on the current weather conditions W and a probability value of detecting an object located in the surrounding environment of the vehicle 12 based on the multiple perception sensors 22 ( Figure 1 ) The x-axis of the detection model 82 represents the distance d between the vehicle 12 and the object, which is expressed as a bin distance measured in meters. The y-axis of the detection model 82 represents the probability of detecting the object, where the probability is determined based on a metric for evaluating object detection. Figure 3 In the example shown, the metric used to evaluate object detection is a recall function, which represents the ratio of the number of true positive examples to the sum of true positive examples and false negative examples. However, it should be appreciated that other metrics, such as mean average precision (mAP), may also be used. Each detection model graph 84 represents a weather condition. For example, detection model graph 84A represents non-bad weather, detection model graph 84B represents rainy days, and the remaining detection model graphs 84C-84H represent other adverse weather conditions (such as fog, heavy fog, heavy rain, darkness, night, and dusk).

[0036] In one embodiment, the detection model 82 is created based on synthetic data. However, it should be appreciated that real-world data may also be used. The synthetic data is generated by one or more machine learning techniques based on real-world data collected under non-adverse weather conditions. Specifically, in one embodiment, a generative adversarial network (GAN) is used to create the synthetic data based on real-world data, wherein the real-world data is modified to simulate a corresponding one of the adverse weather conditions. See Figure 3 , each detection model graph 84 of the detection model 82 is determined by first determining whether an object is detected at each distance d, wherein the real world data is compared with the ground truth data to determine the ratio of true positive detections, and the result is plotted as one of the detection model graphs 84. Then, a polynomial function fitting the corresponding one of the detection model graphs 84 is determined by an optimization problem, which is expressed in Equation 1 as: Equation 1

[0037] in, represents the distance d under the i-th distance value, represents a polynomial function, Denotes the coefficients of the polynomial function. In one embodiment, the polynomial function is expressed in Equation 2 as:

[0038] Equation 2

[0039] in, , are the coefficients of the polynomial function.

[0040] See also Figure 2 and Figure 3 , the detection model selector 52 of the detection module 24 of the one or more controllers 20 determines a probability value corresponding to the current weather condition W indicated by the input state X, the distance d measured between the vehicle 12 and the object detected in the surrounding environment of the vehicle 12, and the category C of the object based on the detection model 82 stored in the detection model library 50. Figure 3 In the example shown, if the current weather condition indicates rain and the distance d is approximately 40 meters, the probability value is approximately 0.6. The probability calculation block 54 of the detection module 24 of the one or more controllers 20 then performs a simulation based on the probability value, wherein the simulation determines the probability of detecting the object located in the environment based on the current weather condition W and the distance d measured between the vehicle 12 and the object. As described above, the probability of detecting the object located in the environment is expressed as a true value or a false value (T / F), wherein a true value T indicates that the object will be detected, and a false value F indicates that the object will not be detected.

[0041] In one non-limiting embodiment, the probability calculation block 54 of the detection module 24 performs a simulation by first drawing a random number from a uniform distribution of values ​​ranging from 0 to 1 and comparing the random number to the probability value, which is represented in Equation 3 as: p = rand(); extract a random number from [0,1]p to U(0,1) Equation 3 Wherein, p represents a random number. Then, the random number is compared with the probability value. In response to determining that the random number is greater than the probability value, the probability calculation block 54 determines that the object is detected, or the probability is true. Otherwise, the probability calculation block 54 determines that the object is not detected, or the probability is false.

[0042] In response to the probability calculation block 54 of the detection module 24 of the one or more controllers 20 determining that the probability of detecting an object located in the environment includes a true value T indicating that the object will be detected, the disturbance module 26 of the one or more controllers 20 then determines one or more disturbed vehicle states observed at the distance d during the current weather condition W. The disturbance module 26 will now be described. A plurality of vehicle state disturbance models 70 are stored in the memory of the one or more controllers 20 by the disturbance model library 56. Specifically, a set of vehicle state disturbance models 70A, 70B, 70C are stored in the memory of the one or more controllers 20, each vehicle state disturbance model corresponding to one of the vehicle states of the vehicle input X, wherein each set of vehicle state disturbance models 70A, 70B, 70C includes a plurality of vehicle state disturbance models 70, each vehicle state disturbance model 70 representing a perception error distribution corresponding to a specific current weather condition W and a specific distance d. Therefore, in the case of Figure 2 In the illustrated embodiment, the one or more vehicle state disturbance models 70 include a set of speed disturbance models 70A, a set of heading disturbance models 70B, and a set of positioning disturbance models 70C. As an example, the set of speed disturbance models 70A includes a plurality of independent speed disturbance models 70 corresponding to a specific current weather condition W and a specific distance d.

[0043] Figure 4 An exemplary vehicle state disturbance model 70 is shown. Specifically, in the Figure 4 In the example shown, a speed disturbance model 70 is shown. The vehicle state disturbance model 70 includes an x-axis representing the perception error and a y-axis representing the probability. Figure 2 and Figure 4 Each vehicle state disturbance model 70 represents a probability distribution of the perception error corresponding to a specific vehicle state, a specific current weather condition W, and a specific distance d. For example, the set of speed disturbance models 70A includes a plurality of vehicle state disturbance models 70, each of which represents a speed under a specific distance d and a specific weather condition. Figure 4 In the example shown, the velocity disturbance model 70 represents a rainy day, wherein the distance d is approximately 20 meters.

[0044] The plurality of vehicle state disturbance models 70 stored in the disturbance model library 56 of the disturbance module 26 are determined based on the following method. First, a set of perception errors E are calculated for each vehicle state, each weather condition, and each distance d as part of the vehicle input X. The perception error E is expressed as E = GT-P, where GT represents the ground truth data and P represents the predicted vehicle state.

[0045] The probability distribution of the perception error E is expressed as ,in is a probability distribution, represents a vehicle state among vehicle states, Represents probability distribution Parameters of . It should be appreciated that a set of probability distributions is calculated for each set of perceptual errors E, wherein the set of probability distributions comprises a predefined number N of candidate probability distributions {P1, P2, P3 . . . PN}. Each candidate probability distribution represents a different type of probability distribution (such as, for example, a normal probability distribution, a Poisson probability distribution, a gamma probability distribution, and a hyperbolic probability distribution). For each set of perceptual errors E, the best fitting probability distribution is identified from the predetermined number N of candidate probability distributions.

[0046] First, the probability distribution is calculated based on the maximum likelihood estimation (MLE) algorithm Parameters To determine the best fitting probability distribution for each set of perceptual errors, the probability distribution is calculated Parameters It is defined in Equation 4 as:

[0047] Equation 4

[0048] The best fitting probability distribution is then found based on the Kolmogorov-Smirnov test. The Kolmogorov-Smirnov test converts the empirical distribution function It is compared with the cumulative distribution function (CDF) of each candidate probability distribution, which is expressed in Equation 5 as:

[0049] Equation 5

[0050] in, represents the best fitting probability distribution, and and Denotes the CDF of the j-th candidate probability distribution with MLE parameters. Then, the best fitting probability distribution is selected, which represents the vehicle state disturbance model 70 corresponding to the specific vehicle state under the specific weather condition and the specific distance d. It should be appreciated that the best fitting probability distribution is determined for each vehicle state as part of the vehicle input X, each weather condition and each distance d.

[0051] Determination of one or more disturbed vehicle states observed at a distance d during a current weather condition W by a disturbance module 26 of one or more controllers 20 will now be described. A disturbance model selector 58 of the disturbance module 26 determines a perceived error for each vehicle state at a distance d during the current weather condition W indicated by the input state X based on the set of vehicle state disturbance models 70A, 70B, 70C stored by the disturbance model library 56. Figure 2In the example shown, the disturbance model selector 58 of the disturbance module 26 determines the speed perception error S of one of the vehicle state disturbance models 70 from the set of speed disturbance models 70A. error , the heading perception error H from the set of heading disturbance models 70B error and the position perception error L from the set of positioning disturbance models 70C error For example, the speed perception error S errorw is 0.3, and the heading perception error H error is 0.1, and the position perception error L error is (0.4, 0.8).

[0052] It should be appreciated that the set of vehicle state disturbance models 70A, 70B, 70C stored by the disturbance model library 56 are not independent of each other. Instead, it should be appreciated that there is a functional relationship between the values ​​of each set of vehicle state disturbance models, and these values ​​are associated with each other. Specifically, for example, the set of speed disturbance models 70A is highly correlated with the positioning disturbance model 70C because the perception error associated with speed is highly correlated with the position of the vehicle 12. The sensitivity block 60 of the disturbance module 26 determines the correlation value between the two perception errors between the two vehicle states in the vehicle state under the current weather condition W indicated by the input state X and the distance d, wherein the correlation value represents the functional relationship between the two disturbance models 70.

[0053] The correlation block 62 of the disturbance module 26 compares the correlation value of the two perception errors between the two vehicle states in the vehicle state with the expected sensitivity value range, wherein the expected sensitivity value range represents the expected difference between the perception errors of the two vehicle states. In response to the correlation value being within the expected sensitivity value range, the correlation block 62 determines that the perception error between the two vehicle states is valid, and then the disturbance calculation block 64 determines the disturbance state of the vehicle 12 observed during the current weather conditions. Conversely, if the correlation block 62 determines that the perception error between the two vehicle states is invalid, the disturbance model selector 58 of the disturbance module 26 can re-determine the perception error between the two vehicle states.

[0054] The disturbance calculation block 64 of the disturbance module 26 of the one or more controllers 20 then determines a disturbed vehicle state of the disturbed state X' of the vehicle 12 by adding the perception error to each original value of the vehicle state. For example, if the vehicle state includes an initial value of speed S (21 kilometers per hour (kph)), the disturbance calculation block 64 will add a speed perception error S of 0.3 to the original value of the vehicle state. errorw Added to the initial value of the speed, the disturbance speed S* that disturbs the vehicle state will be 21.3 kph. In this example, if the heading angle H is 45 degrees, the disturbance calculation block 64 will calculate the heading perception error H of 0.1 errorAdded to the original value of the heading, the perturbed heading H* that perturbs the vehicle state will therefore be 45.1 degrees.

[0055] See also Figure 1 and Figure 2 It should be appreciated that the end-to-end perception perturbation modeling system 10 is integrated with a trajectory prediction system and final motion planner that is part of the ADS or ADAS of the vehicle 12 (which can be used as a tool to evaluate the perception of other vehicles by modeling perception under different conditions). The end-to-end perception perturbation modeling system 10 improves trajectory prediction as well as final motion planning by determining whether the vehicle 12 perceives objects during adverse weather conditions. For example, vehicles located in the environment surrounding the vehicle 12 that are visible during non-adverse weather conditions may not be perceived during adverse weather conditions due to limited visibility. In addition, the trajectory prediction system and final motion planner of the vehicle 12 know that other vehicles located in the environment may also not perceive the host vehicle (i.e., vehicle 12) during adverse weather conditions, and utilize this information during final motion planning.

[0056] Figure 5 The following is an example of a method for Figure 1 Flowchart of an exemplary method 500 for determining a disturbance state X' of a vehicle 12 using the end-to-end perceptual disturbance modeling system 10 is shown. Figures 1 to 5 , the method 500 may begin at block 502. In block 502, the detection module 24 of one or more controllers 20 receives an input state X of the vehicle 12 (e.g. Figure 1 ), where the input state X is observed during non-severe weather conditions and represents one or more vehicle states, current weather conditions W, and distance d. Then, the method 500 may proceed to block 504.

[0057] In block 504, one or more controllers 20 (e.g. Figure 2The detection model selector 52 of the detection module 24 of the controller 20 (shown in FIG. 500 ) determines the probability value corresponding to the current weather condition W and the distance d based on the detection model 82. Specifically, the block 504 includes blocks 504A to 504D for determining the probability value. In block 504A, the detection model selector 52 of the detection module 24 of one or more controllers 20 determines the probability value corresponding to the current weather condition W indicated by the input state X, the distance d measured between the vehicle 12 and the object detected in the surrounding environment of the vehicle 12, and the category C of the object based on the detection model 82 stored in the detection model library 50. Then, the block 500 may enter block 504B, where the probability calculation block 54 of the detection module 24 performs a simulation by first extracting a random number p, which is represented in the above equation 3. Then, the block 500 may enter the decision block 504C. In the decision block 504C, the probability calculation block 54 of the detection module 24 compares the random number p with the probability value. In response to determining that the random number p is less than the probability value, the probability is false, the method 500 may terminate. Otherwise, the method 500 proceeds to block 504D, where the probability calculation block 54 of the detection module 24 determines that the object is detected, then the probability is true. Then, the method 500 may proceed to block 506.

[0058] In block 506, in response to determining that the probability of detecting an object located in the environment indicates that the object will be detected, the one or more controllers 20 determine the disturbance state X' of the vehicle 12 observed during the current weather condition W. Specifically, the disturbance model selector 58 of the disturbance module 26 of the one or more controllers 20 determines the perception error of each vehicle state at the distance d during the current weather condition W indicated by the input state X based on the set of vehicle state disturbance models 70A, 70B, 70C. Block 506 includes blocks 506A and 506B that determine the probability value. In block 506A, the disturbance model selector 58 of the disturbance module 26 selects the appropriate vehicle state disturbance model 70A, 70B, 70C for each vehicle state based on the current weather condition W and the distance d. Then, the method 500 can enter block 506B. In block 506B, the disturbance model selector 58 of the disturbance module 26 determines the perceived error for each vehicle state based on the appropriate vehicle state disturbance model 70A, 70B, 70C selected in block 506A. The method 500 may then proceed to block 508.

[0059] In block 508 , the sensitivity block 60 of the disturbance module 26 of the one or more controllers 20 determines a correlation value between two perception errors between two vehicle states in the vehicle state at the current weather condition W and the distance d indicated by the input state X, wherein the correlation value represents a functional relationship between the two disturbance models. The method 500 may then proceed to block 510 .

[0060] In block 510, the correlation block 62 of the disturbance module 26 of the one or more controllers 20 compares the correlation value of the perceived error between the two vehicle states to an expected sensitivity value range. The expected sensitivity value range represents the expected difference between the perceived errors of the two vehicle states. The method 500 may then proceed to decision block 512.

[0061] In decision block 512, in response to the associated value being outside the sensitivity value range, the method returns to block 506. In response to determining that the associated value is within the expected sensitivity value range, the method 500 proceeds to block 514. In block 514, the disturbance calculation block 64 of the disturbance module 26 of the one or more controllers 20 determines the disturbance state X' of the vehicle by adding the perception error to each raw value of the vehicle state. The method 500 may then terminate.

[0062] Referring generally to the accompanying drawings, the disclosed end-to-end perceptual perturbation modeling system provides various technical effects and benefits. Specifically, the end-to-end perceptual perturbation modeling system provides a method for detecting objects during adverse weather conditions (such as rain, low light, and fog) that affect the visibility of objects in the vehicle's surroundings. It should also be recognized that collecting real-world data for training data sets is relatively time-consuming and may not be available in some cases. In contrast, by building a model based on synthetic data, the end-to-end perceptual perturbation modeling system does not require real-world data.

[0063] The controller may refer to an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared processor, dedicated processor or processor group) that executes code, or a combination of some or all of them, or a portion of them, such as in a system on a chip. In addition, the controller may be based on a microprocessor, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor may run under the control of an operating system residing in the memory. The operating system may manage computer resources so that computer program code embodied as one or more computer software applications (such as applications residing in the memory) may have instructions executed by the processor. In an optional embodiment, the processor may directly execute the application, in which case the operating system may be omitted.

[0064] The description of the present invention is merely exemplary in nature, and changes that do not depart from the gist of the present invention are included within the scope of the present invention. These changes should not be regarded as departing from the spirit and scope of the present invention.

Claims

1. An end-to-end perturbation modeling system for a vehicle, comprising: one or more controllers storing a detection model in a memory, wherein the detection model comprises a plurality of detection model maps, each of the detection model maps representing a probability value of detecting an object in the vehicle's surroundings based on current weather conditions and a distance measured between the vehicle and the detected object in the vehicle's surroundings, and wherein the one or more controllers execute instructions to: receiving an input state of the vehicle, wherein the input state is observed during a non-inclement weather condition and represents one or more of the vehicle state, the current weather condition, and the distance; determining a probability value corresponding to the current weather condition and the distance based on the detection model; performing a simulation based on the probability value, wherein the simulation determines a probability of detecting the object located in the environment based on the current weather conditions and the distance; as well as In response to determining that the probability of detecting the object located in the environment indicates that the object will be detected, a disturbance state of the vehicle observed during the current weather conditions is determined.

2. The end-to-end perturbation modeling system of claim 1, wherein the current weather condition represents an adverse weather condition.

3. The end-to-end perceptual disturbance modeling system according to claim 2, wherein the adverse weather condition is one of the following: dusk, night, rainy day, heavy rainy day, foggy day and heavy foggy day.

4. The end-to-end perceptual disturbance modeling system of claim 1, wherein the one or more controllers store a set of vehicle state disturbance models in the memory, each of the vehicle state disturbance models corresponding to one of the vehicle states.

5. The end-to-end perceptual disturbance modeling system of claim 4 , wherein each group of vehicle state disturbance models comprises a plurality of vehicle state disturbance models, each of the vehicle state disturbance models representing a perceptual error distribution corresponding to specific current weather conditions and a specific distance measured between the vehicle and the object detected in the environment.

6. The end-to-end perturbation modeling system of claim 5, wherein the one or more controllers execute instructions to: A perceived error for each of the vehicle states at a distance indicated by the input state during the current weather conditions is determined based on the set of vehicle state disturbance models.

7. The end-to-end perturbation modeling system of claim 6, wherein the one or more controllers execute instructions to: Determine a correlation value between two perception errors of two of the vehicle states corresponding to the current weather condition and the distance indicated by the input state, wherein the correlation value represents a functional relationship between two of the vehicle state disturbance models.

8. The end-to-end perturbation modeling system of claim 7, wherein the one or more controllers execute instructions to: comparing the associated value of the perceived error between the two vehicle states to an expected range of sensitivity values, wherein the expected range of sensitivity values ​​represents an expected difference between the perceived errors of the two vehicle states; and In response to determining that the correlation value is within the expected sensitivity value range, it is determined that the perceived error between the two vehicle states is valid.

9. The end-to-end perturbation modeling system of claim 8, wherein the one or more controllers execute instructions to: A disturbance state of the vehicle is determined by adding the perception error to each raw value of the vehicle state.

10. The end-to-end perceptual disturbance modeling system of claim 1, wherein the one or more vehicle states include one or more of: a speed, a heading angle of the vehicle, and a position of the vehicle.

Images