SYSTEM FOR THE CONTINUOUS MODELING OF PERCEPTIONAL DISTURBANCES IN A VEHICLE

Abstract

A system for end-to-end modeling of perceptual disturbances for a vehicle comprises one or more controllers that store a detection model in memory. The detection model includes a variety of detection model graphs, each representing a probability value that an object in the vehicle's environment will be detected, based on current weather conditions and a distance measured between the vehicle and the object being detected. The one or more controllers execute instructions to receive an input state of the vehicle observed under non-harsh weather conditions, specifying one or more vehicle states, the current weather conditions, and the distance. The controllers then determine a disturbed state of the vehicle observed during the current weather conditions.

Description

INTRODUCTION

[0001] The present disclosure relates to an end-to-end perceptual disturbance modeling system for a vehicle, wherein the end-to-end perceptual disturbance modeling system takes adverse weather conditions into account.

[0002] An autonomous vehicle performs various tasks such as perception, localization, mapping, path planning, decision making, and motion control. For example, an autonomous vehicle may be equipped with perception sensors such as a camera to collect image data about the vehicle's surroundings. Adverse weather conditions such as rain, fog, snow, or reduced lighting conditions at dusk or night reduce the visibility of objects in the environment and therefore affect how an autonomous vehicle perceives its surroundings. For example, the quality of image data captured by a camera is severely degraded by precipitation and lower lighting. A successful approach for object detection in clear weather conditions may therefore result in inaccurate detection of objects in adverse weather conditions.However, current approaches ignore the impact of adverse weather conditions on perception systems for autonomous vehicles and instead only consider average weather conditions or simply model the performance of a given sensor under different weather conditions.

[0003] In addition to the challenges mentioned above, it should be noted that autonomous vehicle perception-related tasks require relatively large training datasets based on real-world conditions, both in general and for each specific adverse weather condition the vehicle may encounter. Obtaining the real-world data used to create the training dataset is often very time-consuming and difficult. In some cases, it may not be possible to obtain some of the data under real-world conditions. To overcome this limitation, synthetic data generated based on one or more machine learning techniques can be used instead.However, the synthetic data is based on real data, and therefore the quality of the synthetic data is limited by the capabilities of the machine learning techniques as well as the real driving scenarios contained in the existing data.

[0004] Although object detection and perception systems for autonomous vehicles serve their purpose, there is a need for an improved approach to object detection and perception that accounts for adverse weather conditions in the absence of sufficient real-world data. SUMMARY

[0005] According to several aspects, an end-to-end perceptual disturbance modeling system for a vehicle is disclosed. The end-to-end perceptual disturbance modeling system includes one or more controllers having a detection model stored in memory. The detection model includes a plurality of detection model graphs, each indicating a probability value that an object is detected in a surrounding of the vehicle based on a current weather condition and a distance measured between the vehicle and the object detected in the surrounding of the vehicle. The one or more controllers execute instructions to receive an input state of the vehicle, the input state being observed during non-severe or normal weather conditions and indicative of one or more vehicle states, the current weather condition, and the distance.The one or more controllers determine a probability value based on the detection model that corresponds to the current weather conditions and the distance. The one or more controllers perform a simulation based on the probability value, wherein the simulation determines a probability that the ambient object is detected based on the current weather conditions and the distance. In response to determining the probability that the ambient object indicates that the object is detected, the one or more controllers determine a faulty condition of the vehicle observed during the current weather conditions.

[0006] In one aspect, the current weather situation indicates unfavorable weather conditions.

[0007] In another aspect, the adverse weather condition is one of the following: dusk, night, rain, heavy rain, fog and heavy mist.

[0008] In another aspect, the one or more controllers store a set of vehicle state fault models, each corresponding to one of the vehicle states in the memory.

[0009] In one aspect, each set of vehicle state disturbance models comprises a plurality of vehicle state disturbance models, each representing a perception error distribution corresponding to a particular current weather condition and a particular measured distance between the vehicle and the object detected in the environment.

[0010] In another aspect, the one or more controllers execute instructions to determine a perception error based on the set of vehicle state disturbance models for each vehicle state during the current weather condition at the distance specified by the input state.

[0011] In another aspect, the one or more controllers execute instructions to determine a correlation value between two perception errors corresponding to two of the vehicle states at the current weather condition and the distance indicated by the input condition, the correlation value representing a functional relationship between two of the vehicle state disturbance models.

[0012] In one aspect, the one or more controllers execute instructions to compare the correlation value of the perception errors between the two vehicle states to an expected sensitivity value range, wherein the expected sensitivity value range indicates an expected difference between the perception errors of the two vehicle states, and in response to determining that the correlation value falls within the expected sensitivity value range, determine that the perception error between the two vehicle states is valid.

[0013] In another aspect, the one or more controllers execute instructions to determine the faulty state of the vehicle by adding the perception error to each original value of the vehicle state.

[0014] In another aspect, the one or more vehicle states include one or more of: speed, a heading angle of the vehicle, and a location of the vehicle.

[0015] In one aspect, the detection model is created based on synthetic data.

[0016] In another aspect, the end-to-end perceptual disturbance modeling system is integrated with a trajectory prediction system and a final motion planner that are part of one of the following systems: an automated driving system (ADS) and an advanced driver assistance system (ADAS).

[0017] In one aspect, a method for determining a faulty state of a vehicle by an end-to-end perceptual fault modeling system is disclosed. The method includes receiving, by one or more controllers, an input state of the vehicle, the input state observed during non-harsh weather conditions and indicative of one or more vehicle states, a current weather condition, and a measured distance between the vehicle and an object detected in an environment surrounding the vehicle. The method includes determining a probability value corresponding to the current weather condition and the distance based on a detection model stored in memory of the one or more controllers, the probability value indicating that the object is detected in the environment of the vehicle.The method includes performing a simulation based on the probability value, wherein the simulation determines a probability that the surrounding object will be detected based on the current weather conditions and the distance. Finally, in response to determining the probability that the surrounding object indicates that the object will be detected, the method includes determining a faulty condition of the vehicle observed during the current weather conditions.

[0018] In another aspect, the method comprises determining a perception error for each vehicle state during the current weather condition at the distance specified by the input condition based on a set of vehicle state disturbance models stored in the memory of the one or more controllers, wherein each set of vehicle state disturbance models comprises a plurality of vehicle state disturbance models each representing a perception error distribution corresponding to a specific current weather condition and a specific distance measured between the vehicle and the object detected in the environment.

[0019] In another aspect, the method comprises determining a correlation value between two perception errors corresponding to two of the vehicle states at the current weather condition and the distance indicated by the input state, wherein the correlation value represents a functional relationship between two of the vehicle state disturbance models.

[0020] In one aspect, the method comprises comparing the correlation value of the two perception errors between two of the vehicle states to an expected sensitivity value range, wherein the expected sensitivity value range indicates an expected difference between the perception errors of the two vehicle states, and in response to determining the correlation value falling within the expected sensitivity value range, determining that the perception error between the two vehicle states is valid.

[0021] In another aspect, the method includes determining the disturbed state of the vehicle by adding the perception error to each original value of the vehicle state.

[0022] In one aspect, a system for end-to-end perceptual disturbance modeling for a vehicle is disclosed. The end-to-end perceptual disturbance modeling system includes one or more controllers storing a detection model and a set of vehicle state disturbance models in a memory, wherein the detection model is created based on synthetic data and includes a plurality of detection model graphs, each indicating a probability value that an object is detected in an environment of the vehicle based on a current weather condition and a distance measured between the vehicle and the object detected in the environment of the vehicle, and wherein each set of vehicle state disturbance models includes a plurality of vehicle state disturbance models, each representing a perceptual error distribution,corresponding to a specific current weather condition and a specific distance measured between the vehicle and the object detected in the environment, and wherein the one or more controllers execute instructions to receive an input state of the vehicle, the input state being observed during non-harsh weather conditions and indicative of one or more vehicle states, the current weather condition, and the distance. The one or more controllers determine, based on the detection model, a probability value corresponding to the current weather condition and the distance. The one or more controllers execute a simulation based on the probability value, the simulation determining a probability that the object in the environment is detected based on the current weather conditions and the distance. In response to determining the probability,that the object located in the environment indicates that the object will be detected, the one or more controllers determine a disturbed condition of the vehicle observed during the current weather condition, wherein the current weather condition indicates an adverse weather condition.

[0023] In another aspect, the one or more controllers execute instructions to determine, for each vehicle state during the current weather condition at the distance specified by the input state, a perception error based on the set of vehicle state disturbance models; and determine a correlation value between two perception errors between two of the vehicle states at the current weather condition and the distance specified by the input state, wherein the correlation value represents a functional relationship between two of the vehicle state disturbance models.

[0024] In another aspect, the one or more controllers execute instructions to compare the correlation value of the two perception errors between two of the vehicle states to an expected sensitivity value range, wherein the expected sensitivity value range indicates an expected difference between the perception errors of the two vehicle states, and in response to determining that the correlation value falls within the expected sensitivity value range, determine that the perception error between the two vehicle states is valid and determine the faulty state of the vehicle by adding the perception error to each original value of the vehicle state.

[0025] Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. Fig. 1 is a schematic diagram of a vehicle incorporating the disclosed system for end-to-end perceptual disturbance modeling according to an exemplary embodiment with one or more controllers; Fig. 2 shows the software architecture for one or more controllers of the Fig. 1 shown system for the continuous modeling of perceptual disorders according to an exemplary embodiment; Fig. 3 shows an exemplary detection model with a plurality of detection model diagrams according to an exemplary embodiment; Fig. 4 shows an exemplary vehicle state disturbance model according to an exemplary embodiment; and Fig. 5 is a process flow diagram illustrating a method for determining a faulty condition of the vehicle using the Fig. 1 for the continuous modeling of perceptual disorders according to an exemplary embodiment. DETAILED DESCRIPTION

[0027] The following description is merely illustrative in nature and is not intended to limit the present disclosure, its application, or uses.

[0028] In Fig. 1, an exemplary system 10 for end-to-end perceptual disturbance modeling for a vehicle 12 is shown. The vehicle 12 may be any type of vehicle, such as a sedan, a truck, a sport utility vehicle (SUV), a van, or a motor home. The vehicle 12 includes either an automated driving system (ADS) or an advanced driver assistance system (ADAS) that assists the driver with steering, braking, and / or acceleration. The end-to-end perceptual disturbance modeling system 10 includes one or more controllers 20 that are in electronic communication with a plurality of perceptual sensors 22 that are configured to collect perceptual data related to the environment of the vehicle 12. The Fig. The perception sensors 22 shown in Figure 1 include one or more cameras 30, an inertial measurement unit (IMU) 32, a global positioning system (GPS) 34, radar 36, and LiDAR 38, but other sensors may also be used.

[0029] As in Fig. 1, the one or more controllers 20 of the system 10 for modeling perceptual disturbances include a sensing module 24 and a disturbance module 26. The one or more controllers 20 receive an input state X of the vehicle 12, wherein the input state X is observed by the plurality of perceptual sensors 22 during non-harsh weather conditions. The one or more controllers 20 determine a disturbance state X' of the vehicle 12, wherein the disturbance state X' of the vehicle 12 is observed by the plurality of perceptual sensors 22 during a current weather condition W. It is understood that the current weather condition W indicates an adverse weather condition that negatively impacts the visibility of objects in the vicinity of the vehicle 12. The adverse weather conditions also include reduced lighting conditions that occur at dusk or at night.Some examples of adverse weather conditions include dusk, night, rain, heavy rain, fog, and heavy mist.

[0030] The input state X indicates one or more vehicle states, the current weather condition W, a measured distance d between the vehicle 12 and an object in the vicinity of the vehicle 12, and a class C of the object. In one non-limiting embodiment, the vehicle state includes the speed S, the heading angle H, and the location L of the vehicle 12; however, it should be understood that additional or different vehicle states, such as acceleration, may be included. As previously mentioned, examples of the current weather condition W include non-severe weather, twilight, night, rain, heavy rain, fog, and heavy fog. The object in the vicinity of the vehicle 12 represents a target object that the end-to-end perceptual disturbance modeling system 10 determines is detectable in the current weather condition W.Some examples of object class C include vehicle, pedestrian, bicycle, or animal. Note that different object classes C produce different noise patterns, which are part of a probability distribution indicating whether an object has been detected. The probability distribution indicating that an object has been detected is described below.

[0031] The faulty state X' indicates a probability that the object in the surroundings is detected, as well as one or more faulty vehicle states, where the faulty vehicle states are determined based on the current weather situation W. The probability that the object in the surroundings is detected is expressed either as a true value or as a false value (T / F), where the true value T means that the object is detected, and the false value F means that the object is not detected. In the Fig. 1, the perturbed vehicle states include a perturbed speed S*, a perturbed heading angle H*, and a perturbed location L* of the vehicle 12 observed during the current weather condition W. However, as previously mentioned, other perturbed vehicle states may also be used.

[0032] As explained further below, the detection module 24 of the one or more controllers 20 receives the input condition X and determines the probability that the ambient object is detected based on the current weather condition W and the measured distance d between the vehicle 12 and the ambient object of the vehicle 12. In response to determining that the probability is the true value T, the disturbance module 26 of the one or more controllers 20 then determines the disturbed condition X' of the vehicle 12. In the event that the vehicle 12 encounters an adverse weather condition that impairs the view of the ambient environment of the vehicle 12, the disclosed end-to-end perceptual disturbance modeling system 10 determines the probability that an object is detected and the disturbed vehicle condition X' observed during the current weather conditions W.

[0033] Fig. 2 illustrates the software architecture for the one or more controllers 20. The acquisition module 24 of the one or more controllers 20 includes an acquisition model bank 50, an acquisition model selector 52, and a probability calculation block 54. The disturbance module 26 of the one or more controllers 20 includes a disturbance model bank 56, a disturbance model selector 58, a sensitivity block 60, a correlation block 62, and a disturbance calculation block 64.

[0034] The acquisition module 24 will now be described. The acquisition model bank 50 of the acquisition module 24 stores a acquisition model 82 in the memory of the one or more controllers 20. Referring to Fig. 3, the detection model 82 includes a plurality of detection model diagrams 84, each indicating a probability value that the object located in the surroundings of the vehicle 12 is detected based on the current weather condition W and the distance d based on perception data received from the plurality of perception sensors 22 ( Fig. 1) are detected. An x-axis of the detection model 82 represents the distance d between the vehicle 12 and the object, which is specified as an interval distance (bin distance) measured in meters. The y-axis of the detection model 82 represents the probability that the object is detected, wherein the probability is determined based on a metric for evaluating object detection. In the Fig. In the example shown in Figure 3, the metric for evaluating object detection is the recall function, which represents the ratio between the number of true positives and the sum of the true positives and false negatives. However, it should be understood that other metrics may also be used, such as the arithmetic mean precision (mAP). Each detection model graph 84 represents a weather condition. For example, detection model graph 84A represents non-severe weather, detection model graph 84B represents rain, and the remaining detection model graphs 84C-84H represent other adverse weather conditions such as fog, heavy fog, heavy rain, darkness, night, and twilight.

[0035] In one embodiment, the detection model 82 is created based on synthetic data, but real-world data may also be used. The synthetic data is generated by one or more machine learning techniques based on real data collected during non-harsh or normal weather conditions. Specifically, in one embodiment, the synthetic data is created based on real data using a generative adversarial network (GAN), where the real data is modified to simulate a corresponding adverse weather condition. Referring to Fig. 3, each detection model graph 84 of the detection model 82 is determined by first determining whether the object is detected at each of the distances d, comparing the real-world data with the ground data to determine a ratio of true positive detections, and plotting the results as one of the detection model graphs 84. A polynomial function fitting a corresponding one of the detection model graphs 84 is then determined by an optimization problem expressed in Equation 1 as follows: minimize∑i[Recall(W,Di)−F(Di,θ)]2 where D i for the distance d at an i-th distance value, F(D i , θ) represents the polynomial function and θ represents the coefficients of the polynomial function. In one embodiment, the polynomial function in Equation 2 is expressed as follows: F(Di,θ)=a∗Di3+b∗Di2+c∗Di+d where θ = (a, b, c, d) and a, b, c, d are the coefficients of the polynomial function.

[0036] With reference to the Fig. 2 and Fig. 3, the detection model selector 52 of the detection module 24 of the one or more controllers 20 determines a probability value corresponding to the current weather condition W, the measured distance d between the vehicle 12 and the object detected in the vicinity of the vehicle 12, as indicated by the input state X, and the class C of the object, based on the detection model 82 stored in the detection model bank 50. In the Fig. 3, the probability is 0.6 when the current weather condition indicates rain and the distance d is approximately 40 meters. The probability calculation block 54 of the detection module 24 of the one or more controllers 20 then performs a simulation based on the probability value, wherein the simulation determines the probability that the object in the environment is detected based on the current weather condition W and the distance d measured between the vehicle 12 and the object. As previously mentioned, the probability that an object in the environment is detected is expressed either as a true value or as a false value (T / F), where the true value T means that the object is detected and the false value F means that the object is not detected.

[0037] In one non-limiting embodiment, the probability calculation block 54 of the detection module 24 performs the simulation by first drawing a random number from a uniform distribution of values ​​in the range of 0 to 1 and comparing the random number to the probability value, which is expressed in Equation 3 as follows: p=rand(); draw a random number from [0,1]p∼U(0,1) where p represents the random number. The random number is then compared to the probability value. In response to determining that the random number is greater than the probability value, the probability calculation block 54 determines that the object is detected or the probability is true. Otherwise, the probability calculation block 54 determines that the object is not detected or the probability is false.

[0038] In response to the probability calculation block 54 of the detection module 24 of the one or more controllers 20 determining that the probability of detecting the ambient object includes the true value T, indicating that the object is detected, the disturbance module 26 of the one or more controllers 20 then determines the one or more disturbed vehicle states observed at distance d during the current weather condition W. The disturbance module 26 will now be described. A plurality of vehicle state disturbance models 70 are stored in the disturbance model bank 56 in the memory of the one or more controllers 20.In particular, a set of vehicle state disturbance models 70A, 70B, 70C, each corresponding to one of the vehicle states of the vehicle input X, is stored in the memory of the one or more controllers 20, wherein each set of vehicle state disturbance models 70A, 70B, 70C comprises a plurality of vehicle state disturbance models 70, each representing a perception error distribution corresponding to a particular current weather condition W and a particular distance d. In the example shown in . Fig. 2, the one or more vehicle state disturbance models 70 therefore comprise a set of speed disturbance models 70A, a set of heading disturbance models 70B, and a set of localization disturbance models 70C. For example, the set of speed disturbance models 70A comprises individual speed disturbance models 70 corresponding to a particular current weather condition W and a particular distance d.

[0039] Fig. 4 shows an exemplary vehicle state disturbance model 70. In the Fig. 4, a speed disturbance model 70 is specifically shown. The vehicle state disturbance model 70 includes an x-axis indicating the perception error and a y-axis indicating a probability. As shown in Fig. 2 and Fig. As shown in Figure 4, each vehicle state disturbance model 70 represents a probability distribution for the perception error corresponding to a specific vehicle state, a specific current weather condition W, and a specific distance d. For example, the set of speed disturbance models 70A includes a plurality of vehicle state disturbance models 70, each representing the speed at a specific weather condition and a specific distance d. In the Fig. In the example shown in Figure 4, the velocity disturbance model 70 represents rain, where the distance d is approximately 20 meters.

[0040] The plurality of vehicle state disturbance models 70 stored in the disturbance model bank 56 of the disturbance module 26 are determined based on the following approach. First, a set of perception errors E is calculated for each vehicle state that is part of the vehicle input X, each weather condition, and each distance d. The perception errors E are expressed as E = GT - P, where GT (ground truth) represents the ground data and P represents a predicted vehicle state.

[0041] A probability distribution of the perceptual error E is called P i (x i , θ i ), where P i the probability distribution is, x i represents one of the vehicle states and θ i the parameters of the probability distribution Pi It should be understood that for each set of perceptual errors E, a set of probability distributions is calculated, where the set of probability distributions includes a predefined number N of candidate probability distributions. Each candidate probability distribution represents a different type of probability distribution, such as Normal, Poisson, Gamma, and Hyperbolic. For each set of perceptual errors E, the best-fitting probability distribution is determined from a predefined number N of candidate probability distributions.

[0042] The best fitting probability distribution for each set of perceptual errors is determined by first calculating the parameters θ ij the probability distribution P icalculated based on the maximum likelihood estimation (MLE) algorithm; it is defined in equation 4 as follows: θijMLE=arg maxθij∑log(Pij(xi;θij))

[0043] The best-fitting probability distribution is then determined using the Kolmogorov-Smirnov test. The Kolmogorov-Smirnov test compares an empirical distribution function F(x i ) with a cumulative distribution function (CDF) of each candidate probability distribution and is expressed in equation 5 as follows: jbest=arg minj max|F(xi)−PijCDF(xi;θijMLE)| where j best represents the best fitting probability distribution and PijCDF and (xi;θijMLE) represent the CDF of the jth candidate probability distribution with MLE parameters. The best-fitting probability distribution is then selected and represents the vehicle state disturbance model 70 corresponding to a particular vehicle state at a particular weather condition and a particular distance d. It should be understood that the best-fitting probability distribution is determined for each vehicle state that is part of the vehicle input X, for each weather condition and each distance d.

[0044] The determination of the one or more faulty vehicle states observed during the current weather condition W at the distance d by the fault module 26 of the one or more controllers 20 will now be described. The fault model selector 58 of the fault module 26 determines a perception error for each vehicle state during the current weather condition W at the distance d specified by the input state X based on the set of vehicle state fault models 70A, 70B, 70C stored in the fault model bank 56. In the Fig. 2, the disturbance model selector 58 of the disturbance module 26 determines a speed perception error S error from one of the vehicle state disturbance models 70 from the set of speed disturbance models 70A, a course perception error H error from the set of course disturbance models 70B and a location perception error L errorfrom the set of localization disturbance models 70C. For example, the speed perception error S errorw 0.3, the course perception error H error 0.1 and the position perception error L error (0.4, 0.8).

[0045] It should be noted that the vehicle state disturbance models 70A, 70B, and 70C stored in the disturbance model bank 56 are not independent of one another. Instead, a functional relationship exists between the values ​​for each set of vehicle state disturbance models, and the values ​​are correlated with one another. For example, the speed disturbance models 70A are closely related to the location disturbance models 70C because the perception error associated with speed is strongly correlated with the location of the vehicle 12. The sensitivity block 60 of the disturbance module 26 determines a correlation value between two perception errors between two of the vehicle states at the current weather condition W and the distance d as specified by the input state X, where the correlation value represents a functional relationship between the two disturbance models 70.

[0046] The correlation block 62 of the disturbance module 26 compares the correlation value of the two perception errors between two of the vehicle states with an expected sensitivity value range, where the expected sensitivity value range indicates an expected difference between the perception errors of the two vehicle states. In response to the correlation value falling within the expected sensitivity value range, the correlation block 62 determines that the perception error between the two vehicle states is valid, and the disturbance calculation block 64 determines the disturbed state of the vehicle 12 observed during the current weather condition. Otherwise, the correlation block 62 determines that the perception error between the two vehicle states is invalid, and the disturbance model selector 58 of the disturbance module 26 can redetermine the perception error between the two vehicle states.

[0047] The disturbance calculation block 64 of the disturbance module 26 of the one or more controllers 20 then determines the disturbed vehicle state of the disturbed state X' of the vehicle 12 by adding the perception error to each original value of the vehicle state. For example, if the vehicle state includes an original value for speed S of 21 kilometers per hour (km / h), the disturbance calculation block 64 would determine the speed perception error S errorw of 0.3 to the original speed value, so that the perturbed speed S* of the perturbed vehicle state would be 21.3 km / h. In the present example, if the heading angle H is 45 degrees, the perturbation calculation block 64 would calculate the heading perception error H error of 0.1 to the original value of the heading, so that the perturbed heading H* of the perturbed vehicle state would be 45.1 degrees.

[0048] From the Fig. 1 and Fig. 2, it can be seen that the end-to-end perceptual interference modeling system 10 is integrated with a trajectory prediction system and final motion planner that are part of the ADS or ADAS of the vehicle 12 (which is useful as a tool for evaluating the perception of other vehicles by modeling perception under different conditions). The end-to-end perceptual interference modeling system 10 improves trajectory prediction and final motion planning by determining whether the vehicle 12 perceives objects during adverse weather conditions. For example, a vehicle in the vicinity of the vehicle 12 that is visible under normal weather conditions may not be perceived during adverse weather due to the reduced visibility.In addition, the trajectory prediction system and the planner for the final movement of the vehicle 12 are aware that the other vehicle in the environment may also not perceive the ego vehicle (i.e., the vehicle 12) in adverse weather conditions and use this information in the final movement planning.

[0049] Fig. 5 shows a process flow diagram illustrating an exemplary method 500 for determining the faulty state X' of the vehicle 12 by the Fig. 1 represents the system 10 for the continuous modeling of perceptual disorders. With general reference to Fig. 1-5, the method 500 may begin in block 502. In block 502, the sensing module 24 of the one or more controllers 20 receives the input state X of the vehicle 12 (illustrated in Fig. 1), where the input state X is observed during non-harsh or normal weather conditions and indicates one or more vehicle states, the current weather situation W, and the distance d. The method 500 may then proceed to block 504.

[0050] In block 504, the acquisition model selector 52 of the acquisition module 24 of the one or more controllers 20 (shown in Fig.2) the probability value corresponding to the current weather condition W and the distance d, based on the detection model 82. In particular, block 504 includes blocks 504A-504D, which determine the probability value. In block 504A, the detection model selector 52 of the detection module 24 of the one or more controllers 20 determines a probability value corresponding to the current weather condition W, the measured distance d between the vehicle 12 and the object detected in the vicinity of the vehicle 12, as indicated by the input state X, and the class C of the object, based on the detection model 82 stored in the detection model bank 50. The method 500 may then proceed to block 504B, where the probability calculation block 54 of the detection module 24 performs a simulation by first drawing a random number p expressed in Equation 3 above.The method 500 may then proceed to decision block 504C. At decision block 504C, the probability calculation block 54 of the detection module 24 compares the random number p to the probability value. In response to determining that the random number p is less than the probability value and the probability is false, the method 500 may terminate. Otherwise, the method 500 proceeds to block 504D, where the probability calculation block 54 of the detection module 24 determines that the object is detected and the probability is true. The method 500 may then proceed to block 506.

[0051] In block 506, in response to determining the probability that the ambient object indicates that the object is detected, the one or more controllers 20 determine the perturbed state X' of the vehicle 12 observed during the current weather condition W. Specifically, the perturbation model selector 58 of the perturbation module 26 of the one or more controllers 20 determines the perception error for each vehicle state during the current weather condition W at distance d, as indicated by the input state X, based on the set of vehicle state perturbation models 70A, 70B, 70C. Block 506B includes blocks 506A-506B that determine the probability value. In block 506A, the fault model selector 58 of the fault module 26 selects the appropriate vehicle state fault model 70A, 70B, 70C for each vehicle state based on the current weather condition W and the distance d.The method 500 may then proceed to block 506B. In block 506B, the fault model selector 58 of the fault module 26 determines the perception error for each vehicle state based on the appropriate vehicle state fault model 70A, 70B, 70C selected in block 506B. The method 500 may then proceed to block 508.

[0052] In block 508, the sensitivity block 60 of the disturbance module 26 of the one or more controllers 20 determines the correlation value between two perception errors between two of the vehicle states at the current weather condition W and the distance d as indicated by the input state X, where the correlation value is representative of a functional relationship between the two disturbance models. The method 500 may then proceed to block 510.

[0053] In block 510, the correlation block 62 of the fault module 26 of the one or more controllers 20 compares the correlation value of the perception errors between the two vehicle states with an expected sensitivity value range. The expected sensitivity range indicates an expected difference between the perception errors of the two vehicle states. The method 500 may then proceed to decision block 512.

[0054] At decision block 512, the method returns to block 506 if the correlation value falls outside the sensitivity value range. In response to determining that the correlation value falls within the range of expected sensitivity values, the method 500 proceeds to block 514. At block 514, the disturbance calculation block 64 of the disturbance module 26 of the one or more controllers 20 determines the disturbed state X' of the vehicle by adding the perception error to each original vehicle state value. The method 500 may then terminate.

[0055] With general reference to the figures, the disclosed end-to-end perceptual perturbation modeling system offers various technical effects and advantages. The end-to-end perceptual perturbation modeling system provides an approach for detecting objects in adverse weather conditions such as rain, poor lighting, and fog, which impair the visibility of objects in the vehicle's surroundings. It should also be understood that collecting real data for a training dataset is relatively time-consuming and, in some cases, not feasible. The end-to-end perceptual perturbation modeling system eliminates the need for real data by instead building models based on synthetic data.

[0056] The controllers may refer to or be part of an electronic circuit, a combinational logic circuit, a field-programmable gate array (FPGA), a (shared, dedicated, or clustered) processor executing code, or a combination of some or all of the above, such as in a system-on-chip. Furthermore, the controllers may be microprocessor-based, such as a computer having at least one processor, memory (RAM and / or ROM), and associated input and output buses. The processor may operate under the control of an operating system residing in memory. The operating system may manage computer resources so that computer program code embodied as one or more computer software applications, e.g., an application residing in memory, can have instructions executed by the processor.In an alternative embodiment, the processor may execute the application directly; in this case, the operating system may be omitted.

[0057] The description of the present disclosure is merely exemplary, and variations that do not depart from the gist of the present disclosure are intended to be included within the scope of the present disclosure. Such variations should not be considered a departure from the scope of the present disclosure. Figure labeling of Fig. 5: T = True F = False Y = Yes N = No

Claims

[1] System for the continuous modelling of perceptual disturbances for a vehicle, the system for the continuous modelling of perceptual disturbances comprising: one or more controllers storing a detection model in a memory, wherein the detection model includes a plurality of detection model graphs each indicating a probability value that an object is detected in an environment of the vehicle based on a current weather condition and a distance measured between the vehicle and the object detected in the environment of the vehicle, and wherein the one or more controllers execute instructions to: Receiving an input state of the vehicle, the input state being observed during non-harsh weather conditions and indicating one or more vehicle states, the current weather conditions, and the distance; Determining a probability value corresponding to the current weather condition and the distance based on the detection model; Performing a simulation based on the probability value, wherein the simulation determines a probability that the object in the environment will be detected based on the current weather conditions and the distance; and Determining a disturbed state of the vehicle observed during the current weather condition in response to the probability that the object in the environment will be detected. [2] The system for continuous modeling of perceptual disturbances according to claim 1, wherein the current weather condition indicates an unfavorable weather condition. [3] The continuous perceptual disturbance modeling system of claim 2, wherein the adverse weather condition is one of the following: dusk, night, rain, heavy rain, fog, and heavy fog. [4] The system for end-to-end perceptual disturbance modeling of claim 1, wherein the one or more controllers store in memory a set of vehicle state disturbance models each corresponding to one of the vehicle states. [5] The system for end-to-end perceptual error modeling of claim 4, wherein each set of vehicle state error models includes a plurality of vehicle state error models each representing a perceptual error distribution corresponding to a particular current weather condition and a particular measured distance between the vehicle and the object detected in the environment. [6] The system for end-to-end modeling of perceptual disorders according to claim 5, wherein the one or more controllers execute instructions to: Determining a perception error for each vehicle state during the current weather condition at the distance specified by the input state based on the set of vehicle state disturbance models. [7] The system for end-to-end modeling of perceptual disorders according to claim 6, wherein the one or more controllers execute instructions to: Determining a correlation value between two perception errors corresponding to two of the vehicle states at the current weather condition and the distance specified by the input state, the correlation value being representative of a functional relationship between two of the vehicle state disturbance models. [8] The system for end-to-end modeling of perceptual disorders according to claim 7, wherein the one or more controllers execute instructions to: Comparing the correlation value of the perception errors between the two vehicle states with an expected sensitivity value range, wherein the expected sensitivity value range indicates an expected difference between the perception errors of the two vehicle states; and Determining that the perception error between the two vehicle states is valid in response to determining that the correlation value falls within the expected sensitivity value range. [9] The system for end-to-end modeling of perceptual disorders according to claim 8, wherein the one or more controllers execute instructions to: Determine the deteriorated state of the vehicle by adding the perception error to each original value of the vehicle state. [10] The end-to-end perceptual disturbance modeling system of claim 1, wherein the one or more vehicle states comprise one or more of the following: speed, a heading angle of the vehicle, and a location of the vehicle. [11] The system for end-to-end modeling of perceptual disorders according to claim 1, wherein the detection model is created based on synthetic data. [12] The end-to-end perceptual disturbance modeling system of claim 1, wherein the end-to-end perceptual disturbance modeling system is integrated with a trajectory prediction system and a final motion planner that are part of one of the following systems: an automated driving system (ADS) and an advanced driver assistance system (ADAS). [13] A method for determining a faulty state of a vehicle by means of a system for the continuous modelling of perceptual disturbances, the method comprising: Receiving, by one or more controllers, an input state of the vehicle, the input state observed during non-harsh weather conditions and indicative of one or more vehicle states, a current weather condition, and a distance measured between the vehicle and an object detected in an environment of the vehicle; Determining a probability value corresponding to the current weather condition and the distance based on a detection model stored in the memory of the one or more controllers, the probability value indicating that the object is detected in the vicinity of the vehicle; Performing a simulation based on the probability value, wherein the simulation determines a probability that the object in the environment will be detected based on the current weather conditions and the distance; and Determining a disturbed state of the vehicle observed during the current weather condition in response to the probability that the object in the environment will be detected. [14] The method of claim 13, further comprising: Determining a perception error for each vehicle state during the current weather condition at the distance specified by the input condition based on a set of vehicle state disturbance models stored in the memory of the one or more controllers, wherein each set of vehicle state disturbance models comprises a plurality of vehicle state disturbance models each representing a perception error distribution corresponding to a specific current weather condition and a specific distance measured between the vehicle and the object detected in the environment. [15] The method of claim 14, further comprising: Determining a correlation value between two perception errors corresponding to two of the vehicle states at the current weather condition and the distance specified by the input state, the correlation value being representative of a functional relationship between two of the vehicle state disturbance models.

Images