Monitoring platform of sensor fusion system

Abstract

Relates to a monitoring platform of a sensor fusion system. Input data of the sensor fusion system is stored. Perceptual output data from the output of the sensor fusion system is obtained using the input data obtained over the time period. For each sensor or each subset of the plurality of in-vehicle sensors: comparing the obtained perceptual output data with stored input data, and determining whether any object detection indicated in the obtained perceptual output data is indicated in the field of view of the sensor or subset of sensors, to validate any object detection indicated in the obtained perceptual output data; and any free space region in the obtained perceptual output data indicates whether it is indicated in the field of view of the sensor or subset of sensors in order to validate any free space region indicated in the obtained perceptual output data. A signal indicative of a state of the sensor fusion system is output based on the validated metric of the object and the validated metric of the free space region.

Description

Technical Field The disclosed technology relates to monitoring the reliability of a sensor fusion system. In particular, but not exclusively, the disclosed technology relates to methods and systems for monitoring the reliability of an output of a sensor fusion system of a vehicle, such as sensory output data generated by the sensor fusion system. Background Art The development of autonomous vehicles has surged over the past few years, and many different solutions are being explored. More and more modern vehicles have advanced driver assistance systems (ADAS) for improving vehicle safety and more general road safety. ADAS, which can be represented by adaptive cruise control ACC, collision avoidance systems, forward collision warnings, etc., are electronic systems that can help the driver of a vehicle while driving. Today, development of both ADAS and autonomous driving (AD) is underway in many different technical fields within these fields. In this article, ADAS and AD will be mentioned under the general term automated driving system (ADS) corresponding to all different automation levels defined, for example, by the SAE J3016 levels (0-5) of driving automation. Accordingly, in the near future, ADS solutions will enter modern vehicles to a greater extent. ADS can be interpreted as a complex combination of various components, which can be defined as a system that performs perception, decision-making and operation of the vehicle by electronics and mechanics rather than human drivers and introduces automation into road traffic. This includes the handling of the vehicle, the destination, and the understanding of the surroundings. Although the automated system can control the vehicle, it allows the human operator to leave all or at least part of the responsibility to the system. ADS typically combines various sensors such as radar, LIDAR, sonar, cameras, navigation systems (e.g., GPS), odometers and / or inertial measurement units (IMUs) to perceive the surroundings of the vehicle, and the advanced control system can interpret the perception information based on these sensors to identify appropriate navigation paths as well as obstacles and / or related signs. The vehicle perception system plays an important role in ADS in order to ensure reliable and safe vehicle operation. In the current context, the perception system can be understood as a system responsible for acquiring raw sensor data from onboard sensors (e.g., cameras, lidars, and radars) and converting the raw data into a scene understanding of the vehicle. In addition, the perception system typically includes one or more sensor fusion systems (e.g., object tracking modules) configured to process multiple types of sensor outputs with the goal of providing a more complete perception output and thereby better understanding the surrounding environment. The output from the sensor fusion system is typically ingested by various ADS functions responsible for the control or operation of the vehicle. Therefore, since the quality of the information provided by the sensor fusion module significantly affects the perception capabilities of the vehicle and, by extension, the performance and safety of various ADS functions, it is important to be able to provide a sensor fusion system that can provide reliable and accurate scene understanding of the vehicle's surroundings. Accordingly, there is still a need in the art for methods and systems that can ensure the reliability of the perception system performance of an ADS and, by extension, the overall performance and security of the ADS. Summary of the invention The technology disclosed herein seeks to mitigate, alleviate or eliminate the deficiencies and shortcomings in the prior art to address various problems associated with ensuring the safe operation of various functions of an automated driving system. It is therefore an object of the technology disclosed herein to provide a computer-implemented method, a computer-readable storage medium, a computer program product, a system, and a vehicle that alleviate all or at least some of the disadvantages of currently known systems and methods. Furthermore, the technology disclosed herein is directed to a computer-implemented method, a computer-readable storage medium, a computer program product, a system, and a vehicle that provide a means for ensuring that the output of a sensor fusion system is accurate and reliable so as to ensure adequate performance of ADS functions that rely on outputs from the sensor fusion system and improve the overall road safety of vehicles equipped with an ADS. Various aspects and embodiments of the technology disclosed herein are defined in the following and accompanying independent and dependent claims. A first aspect of the disclosed technology includes a computer-implemented method for monitoring the reliability of an output of a sensor fusion system of a vehicle. The sensor fusion system is configured to receive input data and generate perception output data. The input data includes sensor data obtained from each of a plurality of onboard sensors configured to monitor the surroundings of the vehicle during a time period, and the method includes storing the input data of the sensor fusion system. The method further includes: obtaining perception output data output from the sensor fusion system using the input data obtained during the time period. The obtained perception output data includes one or more object detections in the surroundings of the vehicle and one or more free space area indications in the surroundings of the vehicle. For each sensor or each subset of sensors in the plurality of onboard sensors, the method includes: comparing the obtained perception output data with the stored input data; and determining: whether any object detection indicated in the obtained perception output data is indicated in the field of view of the sensor or the subset of sensors, so as to verify any object detection indicated in the obtained perception output data; and whether any free space area indication in the obtained perception output data is indicated in the field of view of the sensor or the subset of sensors, so as to verify any free space area indicated in the obtained perception output data. Additionally, the method includes outputting a signal indicative of a status of the sensor fusion system based on the verified measurement of the object and the verified measurement of the free space region. Accordingly, a solution for monitoring the output of a sensor fusion system of a vehicle is provided. In more detail, the method disclosed herein can verify objects and free space areas in the environment around the vehicle as indicated in the output of the sensor fusion system of the perception function / system of the vehicle. Then, the state of the sensor fusion system is indicated. By conveying an indication of the state of the sensor fusion system with a signal, the accuracy of the sensor fusion system output can be evaluated, and further actions can be taken. For example, if the sensor fusion system appears to be failing, the appropriate function of the ADS can act accordingly to autonomously perform appropriate actions (e.g., stop the vehicle or request to be handed over to the driver of the vehicle) due to the failure. Alternatively, if a secondary sensor fusion system exists, the ADS or its perception system can switch to use this redundancy. For example, if one or more of the sensors provide sensor data indicating that there is one or a group of objects and / or free space in the environment of the vehicle and does not match the output data of the sensor fusion system, the autonomous driving vehicle can take action based on the defined inconsistency. Therefore, the reliability of information about the environment in which the vehicle is traveling is improved, thereby providing overall road safety of vehicles equipped with ADS. A second aspect of the disclosed technology includes a computer program product containing instructions that, when executed by a computing device of a vehicle, cause the computing device to implement a method according to any one of the embodiments disclosed herein. For this aspect of the disclosed technology, similar advantages and preferred features exist as for the previously discussed aspects. A third aspect of the disclosed technology includes a (non-transitory) computer-readable storage medium containing instructions that, when executed by a computing device of a vehicle, cause the computing device to implement a method according to any one of the embodiments disclosed herein. For this aspect of the disclosed technology, similar advantages and preferred features exist as for the previously discussed aspects. As used herein, the term "non-transitory" is intended to describe computer-readable storage media (or "memory") that do not include propagating electromagnetic signals, but is not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the term computer-readable media or memory. For example, the term "non-transitory computer-readable medium" or "tangible memory" is intended to encompass types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may be further transmitted via a transmission medium or signals such as electrical, electromagnetic, or digital signals that may be transmitted via a communication medium such as a network and / or wireless link. Thus, as used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, not a signal), not a limitation on the persistence of data storage (e.g., RAM vs. ROM). A fourth aspect of the disclosed technology includes a system for monitoring the reliability of a sensor fusion system output of a vehicle, wherein the sensor fusion system is configured to receive input data and generate perception output data. The system includes one or more memory storage areas containing program code. The one or more memory storage areas and the program code are configured to cause the system to at least store input data of the sensor fusion system with one or more processors, wherein the input data includes sensor data obtained from each of a plurality of onboard sensors configured to monitor the surrounding environment of the vehicle during a time period, and to obtain perception output data output from the sensor fusion system using the input data obtained during the time period, the obtained perception output data including one or more object detections in the surrounding environment of the vehicle and one or more free space area indications in the surrounding environment of the vehicle. Then, the one or more memory storage areas and program code are configured to cause the system, with one or more processors, to: for each sensor or each subset of sensors in the plurality of onboard sensors, compare the acquired perception output data with the stored input data and determine: whether any object detection indicated in the acquired perception output data is indicated in the field of view of the sensor or the subset of sensors, so as to verify any object detection indicated in the acquired perception output data; and whether any free space area indication in the acquired perception output data is indicated in the field of view of the sensor or the subset of sensors, so as to verify any free space area indicated in the acquired perception output data. In addition, the one or more memory storage areas and program code are configured to cause the system, with one or more processors, to output a signal indicating a state of the sensor fusion system based on the verified object measurement and the verified free space area measurement. For this aspect of the disclosed technology, similar advantages and preferred features exist as the previously discussed aspects. A fifth aspect of the disclosed technology comprises a vehicle comprising a system according to any one of the embodiments disclosed herein. For this aspect of the disclosed technology, similar advantages and preferred features exist as for the previously discussed aspects. The disclosed aspects and preferred embodiments may be appropriately combined with one another in any manner apparent to a person skilled in the art, such that one or more features or embodiments disclosed with respect to one aspect may also be considered disclosed with respect to another aspect or embodiment of another aspect. Further embodiments are defined in the dependent claims. It should be emphasized that when used in this specification, the term "comprising" is used to specify the presence of stated features, integers, steps or components. It does not exclude the presence or addition of one or more other features, integers, steps, components or groups thereof. An advantage of certain embodiments is that any potential limitations of the sensor fusion system may be captured in a timely manner and potential negative impacts on ADS performance or safety may be mitigated. An advantage of some embodiments is that a robust solution for monitoring the reliability of outputs of a sensor fusion system of a vehicle is provided, thereby allowing preventive measures to be performed based on the status of the sensor fusion system to reduce the risk of accidents caused by erroneous outputs from the sensor fusion system. These and other features and advantages of the disclosed technology will be further elucidated below with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS The above-described aspects, features, and advantages of the disclosed technology will be more fully understood by referring to the following illustrative and non-limiting detailed description of example embodiments of the present disclosure when taken in conjunction with the accompanying drawings, in which: Figure 1 is a schematic flow chart of a method for monitoring the reliability of an output of a sensor fusion system of a vehicle, according to some embodiments. Figure 2 is a schematic block diagram representation of a system for monitoring the reliability of an output of a sensor fusion system of a vehicle, according to some embodiments. Figure 3 is a schematic block diagram of a sensor fusion status indicator according to some embodiments. Figure 4 is a schematic block diagram of a sensor fusion status indicator according to some embodiments. Figure 5 is a schematic diagram of a vehicle including a system for monitoring reliability of an output of a sensor fusion system of the vehicle, according to some embodiments. DETAILED DESCRIPTION The technology disclosed herein will now be described in detail with reference to the accompanying drawings in which some example embodiments of the disclosed technology are shown. However, the disclosed technology may be embodied in other forms and should not be construed as limited to the disclosed example embodiments. The disclosed example embodiments are provided to fully convey the scope of the disclosed technology to those skilled in the art. Those skilled in the art will understand that the steps, services, and functions described herein may be implemented using separate hardware circuits, using software that works in conjunction with a programmed microprocessor or general-purpose computer, using one or more application-specific integrated circuits (ASICs), using one or more field programmable gate arrays (FPGAs), and / or using one or more digital signal processors (DSPs).

[0029] It will also be understood that when the present disclosure is described in terms of methods, it may also be embodied in an apparatus comprising one or more processors, one or more memories coupled to the one or more processors, wherein computer code is loaded to implement the methods. For example, in some embodiments, the one or more memories may store one or more computer programs that, when executed by the one or more processors, cause the apparatus to perform the steps, services, and functions disclosed herein. It should also be understood that the terms used in this article are only used for the purpose of describing specific embodiments and are not intended to be limiting. It should be noted that as used in the specification and the appended claims, the articles "a", "an", "the" and "said" are intended to indicate the presence of one or more elements, unless the context clearly dictates otherwise. Thus, for example, in some contexts, reference to "a unit" or "the unit" may refer to more than one unit, and so on. In addition, the words "include", "comprise" do not exclude other elements or steps. The term "and / or" should be interpreted as meaning "both" and that each is optional. It will also be understood that although the terms first, second, etc. may be used to describe various elements or features in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the embodiment, the first signal may be referred to as the second signal, and similarly, the second signal may be referred to as the first signal. Both the first signal and the second signal are signals, but they are not the same signal. Figure 1 A schematic flow chart representation of a method 100 for monitoring the reliability of an output of a sensor fusion system of a vehicle is illustrated. Preferably, the method 100 is a computer-implemented method 100 performed by a processing system of a vehicle equipped with an ADS. The processing system may, for example, include one or more processors and one or more memories coupled to the one or more processors, wherein the one or more memories store one or more programs that, when executed by the one or more processors, cause the processing system to perform the steps, services, and functions of the method 100 disclosed herein. Generally, in the context of autonomous driving systems, sensor fusion techniques (i.e., combining the output of multiple sensors) can be used for a variety of purposes. For example, output data from multiple onboard sensors (GNSS, LiDAR, Radar, and cameras) can be fused to track other vehicles, detect static obstacles, and localize the vehicle relative to a road model. In some embodiments, the sensor fusion system is an object tracking system configured to track the position of objects present in the input data (i.e., sensor data) over time. Method 100 includes storing 101 input data of a sensor fusion system. The input data includes sensor data obtained from each of a plurality of vehicle-mounted sensors configured to monitor the surroundings of the vehicle during a time period. The surroundings of the vehicle can be understood as a general area around the vehicle in which objects (e.g., other vehicles, landmarks, obstacles, etc.) can be detected and identified by vehicle sensors (radars, lidars, cameras, etc.) (i.e., within the sensor range of the vehicle). The sensor data from each sensor indicates the surroundings of the vehicle or at least a portion thereof. Therefore, the sensor data can be data obtained from one or more radars, one or more lidars, one or more cameras, or any other suitable vehicle-mounted sensors known in the art. For example, the input data may be stored 101 or saved in a data buffer (not shown), wherein the input data may be understood as information indicating the surroundings of the vehicle. The input data may include processed sensor data and / or unprocessed (raw) sensor data. For example, the camera image may be post-processed, and the LiDAR output (point cloud) may be fed directly to the sensor fusion system as input. In some embodiments, the sensor data is in the form of a list of objects and their attributes (e.g., location or place in the surroundings of the vehicle, object type, etc.). In the context of the present disclosure, "object" may be understood as a physical object or obstacle present in the environment around the vehicle. In addition, sensor data may be obtained continuously and / or intermittently from a first time point T1 to a second time point T2. The length of the time period and the corresponding optional data buffer may be any size that is considered feasible, for example, taking into account data capacity limitations and / or the characteristics of the ADS, and may be, for example, from less than a second to several minutes or more preferably, from several seconds to less than a minute. In addition, method 100 includes obtaining 102 perception output data. Perception output data output from the sensor fusion system using input data obtained during a time period. This means that the perception output data is generated by processing sensor data that has been output by an on-board sensor and has been acquired and stored in a suitable data storage medium (e.g., a data buffer) during the aforementioned time period. In other words, the input data can be stored by a suitable data storage medium at the same time, while the input data is ingested by the sensor fusion system to provide perception output data. The perception output data obtained includes one or more object detections in the surrounding environment of the vehicle. The perception output data obtained further includes one or more free space area indications in the surrounding environment of the vehicle. In some embodiments, the free space area indication can be simply interpreted as an area for which no object detection has been performed, i.e., an indirect indication. However, in some embodiments, the free space area indication can be a direct indication, i.e., an area in the surrounding environment of the vehicle that has been explicitly indicated or classified as a free space area. Furthermore, the term "obtain" should be interpreted broadly herein and includes receiving, retrieving, collecting, acquiring, etc., directly and / or indirectly between two entities configured to communicate with each other or further communicate with other external entities. However, in some embodiments, the term "obtain" will be interpreted as determining, deriving, forming, calculating, etc. The term "perception output data" may be interpreted as information generated by fusing the outputs from various (at least two) sensors of a vehicle. The outputs from the various sensors include data collected from cameras, lidars, radars, and / or other sensors that help the vehicle "perceive" its surroundings and make decisions based on that information. The perception output data generated by the sensor fusion system may include the location, speed, and direction of nearby objects, the location and type of road markings, the location and type of traffic signs, the location and extent of free space areas, and other relevant information. This data may then be processed as input by the vehicle's onboard computers (and in particular, the various ADS functions) to make decisions about steering, acceleration, braking, and other actions required to safely navigate the environment. Accordingly, the term "perception output" data may refer to "environmental assessment" data, "spatial perception" data, "processed perception" data, and / or "time-dependent" data, while perception "data" may refer to perception "information" and / or "estimates." On the other hand, the term "obtaining" from the sensor fusion system may refer to "deriving" from a sensor fusion model and / or "based on output data" from a sensor fusion or system, and a perception fusion module / system configured to "generate perception output data" may refer to a perception module / system adapted for and / or configured to "estimate the environment of the vehicle", "estimate at least a portion of the environment of the vehicle", "determine the environment of the vehicle", "interpret perception information related to autonomous manipulation of the vehicle" and / or "estimate the surrounding environment of the vehicle and perform model predictions of future states of the environment of the vehicle". Therefore, the term "perception output data" should be understood as a term that encompasses various estimates such as one or more object positions, object classifications, lane markings, object trajectory predictions, or free space estimates. As mentioned, free space estimates can be understood as the determination of free space areas, i.e., estimates of areas without external objects (static and dynamic objects). The estimation of the "drivable area" can be considered in a similar way, where, in addition to the estimation of areas without objects (as in the case of free space), the estimation of the "drivable area" also includes the presence of the road surface. Continuing, the method 100 includes: for each sensor or each subset of sensors in a plurality of onboard sensors, comparing 103 the obtained perception output data with stored input data of the sensor fusion system. The comparison can be performed on a per-sensor basis, such that the input data of each of the onboard sensors is compared with the perception output data. Additionally or alternatively, the comparison can be performed on a grouped sensor basis, such that the input data of a subset of the onboard sensors is compared with the perception output data. The sensor subsets can be grouped based on sensor modality. For example, one or more radars can be a first subset of onboard sensors, while one or more lidars can be a second subset of onboard sensors. One or more cameras can be grouped into a third subset of onboard sensors. In some embodiments, the storage 101 of the input data includes storing each data set of the input data separately in a corresponding sensor-specific or subset-specific data store. Furthermore, the method 100 comprises, for each sensor or each subset of sensors among the plurality of onboard sensors, determining 104 whether any object detection indicated in the obtained perception output data is indicated in the field of view of the sensor or the subset of sensors, so as to verify any object detection indicated in the obtained perception output data. In other words, the method 100 comprises, for each sensor or each subset of sensors among the plurality of onboard sensors, determining 104 whether any object detection of the perception output data is indicated in the sensor data output by the sensor or the subset of sensors, so as to verify one or more object detections indicated in the obtained perception output data. Furthermore, for each sensor or each subset of sensors among the plurality of onboard sensors, the method comprises: determining 114 whether any free space region indication in the obtained perception output data is indicated in the field of view of the sensor or the subset of sensors, so as to verify any free space region indicated in the obtained perception output data. In other words, the method 100 comprises: for each sensor or each subset of sensors among the plurality of onboard sensors, determining 114 whether any free space region indication of the perception output data is indicated in the sensor data output by the sensor or the subset of sensors, so as to verify one or more free space regions indicated in the obtained perception output data. In some embodiments, this determination 114 can be interpreted as a "negative confirmation", i.e. checking whether any object detection indicated in the sensor data output by the sensor is located in the free space region indicated in the perception output data. In this case, the free space region indicated in the perception output data will not be a valid free space region indication. Typically, for any object detection or free space region indication in the perception output data, there is always at least one onboard sensor whose output can be used to verify the object detection or free space region indication indicated in the acquired perception output data. Accordingly, the object detection indicated in the obtained perception output data and the free space area indicated in the obtained perception output data are verified based on the sensor output data. In other words, the comparison 103 of the obtained perception output data with the stored input data can be understood as a process of checking whether the object detection and the free space indication (i.e., the area without objects) can be verified in the sensor data output by one or more of the on-board sensors. More specifically, the process can be understood as a check of all object detections and free space area indications in the perception output data in view of the stored input data. Therefore, on a per-sensor or per-sensor subset basis, all object detections and free space area indications in the perception output data are evaluated to verify each of the object detections and free space area indications. In the current context, the term "field of view" of a sensor can be understood as the volume within which a particular sensor can detect. This can also be referred to as the viewing frustum of the sensor. In addition, the object can be directly present in the stored input data, or indicated in another way in the stored input data. For example, input data in the form of a lidar output typically contains a point cloud that is not necessarily labeled as an object. However, a collection of points is considered to indicate the presence of an object. Accordingly, input data in the form of LIDAR output may indirectly indicate the presence of an object in the form of a set of point detections (i.e., a point cloud). Thus, the input data can include indirect evidence of an object. Since input data in the form of processed camera images is post-processed with, for example, an object detection algorithm, the "input data" may directly indicate the presence of an object. Thus, the input data may include direct evidence of an object. Similarly, the lack of object detection at a certain area can be interpreted as the presence of a free space region in that area. In some embodiments, the method includes: in response to detecting an object being indicated in the field of view of the sensor or the subset of sensors (i.e., indicated in the sensor data output of the sensor or the subset of sensors) (in the corresponding area / volume of the surrounding environment), verifying 105 the object detection indicated in the obtained perception output data of the sensor or the subset of sensors. Accordingly, if the object detection is indicated in the obtained perception output data, and if the stored sensor data directly or indirectly indicates the presence of the object in the same area or volume in the environment of the vehicle, then the object detection is valid. Furthermore, in some embodiments, the method comprises: in response to detecting that the presence of a free space region indication (in a corresponding area / volume of the surrounding environment) is indicated in the field of view of a sensor or a subset of sensors, verifying 115 the free space region indicated in the obtained perception output data of the sensor or the subset of sensors. Accordingly, if a free space detection is indicated in the obtained perception output data, and if stored sensor data of the sensor or the subset of sensors directly or indirectly indicates the presence of a free space region in the same area or volume in the environment of the vehicle, the free space region indication is verified. In an illustrative example, the obtained perception output data may include object detections A, B, and C, and the vehicle may include sensors X, Y, and Z. The comparison 103 of the obtained perception output data to the stored input data may accordingly include determining whether any of object detections A, B, and C are present in the stored sensor output from sensor X, whether object detections A, B, and C are present in the stored sensor output from sensor Y, and whether object detections A, B, and C are present in the stored sensor output from sensor Z. In an example scenario, object detection A may have been verified in the stored sensor outputs from sensors X and Z, while object detection B may have been verified in the stored sensor output from sensor Y, and object detection C may not have been verified in any of the stored sensor outputs. In such a scenario, object detections A and B would be verified object detections, while object detection C would be an unverified object detection. Furthermore, the method includes outputting 108 a signal indicating a state of the sensor fusion system based on the verified measurement of the object and the verified measurement of the free space area. The "measurement" of the verified object may be, for example, a binary signal (verified - yes / no), and the "measurement" of the verified free space area may be, for example, a binary signal (verified - yes / no). Figure 2 and Figure 3 Further details and examples related to signal output 108 and metrics are described. In addition, for each sensor or each subset of sensors in the plurality of onboard sensors, the verified object metric may be aggregated over time 107. Similarly, the verified free space area metric may be aggregated over time 117. As a result, a signal indicative of a state of the sensor fusion system based on the aggregated metric of the verified object for each sensor or each subset of sensors and the aggregated metric of the verified free space area for each sensor or each subset of sensors is output 108. In some embodiments, the method 100 may include aggregating the verified object and free space area metrics over time for the plurality of onboard sensors. Additionally, the measure of the aggregation of verified objects may include a percentage of verified objects for the sensor during the time period for which the sensor's input data is stored. Thus, in some embodiments, the method 100 may include, for each sensor or each subset of sensors among the plurality of onboard sensors, determining 106 a percentage of objects detected in the obtained perception output data that are verified. Similarly, the measure of the aggregation of verified free space areas may include a percentage of verified free space areas for the sensor during the time period for which the sensor's input data is stored. Thus, in some embodiments, the method 100 may include, for each sensor or each subset of sensors among the plurality of onboard sensors, determining 116 a percentage of free space areas detected in the obtained perception output data that are verified. In some embodiments, based on the outputted signal indicating the status of the sensor fusion system, the ADS function of the vehicle may be controlled 109. For example, if the outputted signal indicates that the output from the sensor fusion system is faulty or otherwise unreliable, the control 109 of the ADS function may include prohibiting the activation of the ADS function that relies on the output from the sensor fusion system. Similarly, if the outputted signal indicates that the output from the sensor fusion system is accurate or otherwise reliable, the control 109 of the ADS function may include allowing the activation of the ADS function that relies on the output from the sensor fusion system. In some embodiments, the control 109 of the ADS function may include outputting a request to hand over to the driver of the vehicle and to deactivate the ADS function. According to some embodiments, the method 100 further includes: sending 110 data to a remote server via one or more antennas of the vehicle, wherein the sent data represents a status of the sensor fusion system. An advantage of sending data to a remote server (e.g., a central fleet management system) is that it enables and facilitates development, debugging, and performance verification of the sensor fusion system. In more detail, it enables fleet-wide monitoring and verification of the deployed sensor fusion system across the fleet, thereby potentially detecting and mitigating potential prevalent problems at a faster rate. In addition, performance verification of new software versions of the sensor fusion system can be enabled and facilitated. Furthermore, executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors. Figure 2 and Figure 3is a schematic block diagram representation of a system 10 (or portion thereof) for monitoring the reliability of an output of a sensor fusion system 32 of a vehicle, according to some embodiments. The system 10 includes control circuitry (e.g., one or more processors) configured to perform the functions of the method 100 disclosed herein, wherein these functions may be included in a non-transitory computer-readable storage medium 12 or other computer program product configured to be executed by the control circuitry 11. In other words, the system 10 includes one or more memory storage areas 12 containing program code, and the one or more memory storage areas 12 and the program code are configured to cause the system 10 to perform the method 100 according to any of the embodiments disclosed herein with one or more processors 11. However, in order to better illustrate the embodiments disclosed herein, the control circuitry is represented as each of which is linked to one or more specific functions of the control circuitry. Figure 2 and Figure 3 Various "modules" or blocks 351-353, 3361-363, 300, 350, 360, 370, 380, 390 in. Accordingly, Figure 2 A system 10 for monitoring the reliability of an output of a sensor fusion system 32 of a vehicle according to some embodiments is shown. The sensor fusion system 32 receives outputs from a plurality of sensors 321, 322, 323 mounted on the host vehicle as input data. The system 10 is configured to store the input data, wherein the input data includes sensor data obtained during a time period from each of the plurality of onboard sensors 321, 322, 323 configured to monitor the surroundings of the vehicle. The system is further configured to obtain perception output data output from the sensor fusion system 32 using the input data obtained during the time period from the plurality of onboard sensors 321, 322, 323. The obtained perception output data includes one or more object detections in the surroundings of the vehicle and one or more free space area indications in the surroundings of the vehicle. For each sensor 321, 322, 323 or sensor subset, the system 10 is configured to compare the obtained perception output data with the stored input data and determine: whether any object detection indicated in the obtained perception output data is indicated in the field of view of the sensor or the sensor subset, so as to verify any object detection indicated in the obtained perception output data; and whether any free space area indication in the obtained perception output data is indicated in the field of view of the sensor or the sensor subset, so as to verify any free space area indicated in the obtained perception output data. In some embodiments, the system 10 includes two groups of modules. One group of modules is an object presence verifier block 351, 352, 353 that includes a plurality of presence verifiers 1, 2, ..., N and is configured to provide object presence verification (e.g., whether object detection is verified for each of the sensors 321, 322, 323 or each sensor subset). The other group of modules is an object absence verifier block 361, 362, 363 that includes a plurality of object absence verifiers 1, 2, ..., N and is configured to provide object absence verification (i.e., whether free space area is verified for each of the sensors 321, 322, 323 or each sensor subset). Figure 2 As depicted in , there can be as many presence verifier blocks as the number of sensors. Additionally or alternatively, the sensors can be arranged into groups or subsets such that there is a presence verifier for each group or subset of sensors. Similarly, there can be as many absence verifier blocks as the number of sensors. Additionally or alternatively, the sensors can be arranged into groups or "subsets" such that there is an absence verifier for each subset of sensors. Each presence verifier block 351, 352, 353 is configured to compare the output of the sensor fusion system 32 at a particular time period or moment with the set of sensor inputs over time, and determine whether a particular object that has been generated by the sensor fusion system 32 can be verified. That is, the presence of an object, such as an object detection, in the output of the sensor fusion system 32 is verified by checking whether there is sufficient evidence (in the sensor data) of the generated object. For example, the output data of the sensor fusion system 32 and the stored input data of the sensor 321, 322 or 323 or a subset of sensors can be compared to evaluate whether the two data indicate the presence of the same object. For example, the presence verifier_1 351 is configured to verify the object detection in the output of the sensor fusion system 32 by detecting the object detection of the corresponding area or position in the output of sensor_1 321. The output from each object presence verifier block 351, 352, 353 can be a binary signal (e.g., 1 / 0) output for each object, where 1 is output if the object is verified and 0 is output if the object cannot be verified. Similarly, if the sensor fusion system 32 outputs a free space region (i.e., it does not produce an output representing an object in a particular region), the object absence verifier blocks 361, 362, 363 are configured to check whether the free space region can be verified by the corresponding sensor output (i.e., the corresponding sensor output in the sensor data). Like object presence verification, object absence verification (i.e., verification of the free space region) can be done per sensor and / or per sensor subset. For example, the absence verifier_1 361 is configured to verify the free space region detection in the output of the sensor fusion system 32 by detecting the free space region (i.e., the region without the object) in the corresponding region or position in the output of sensor_1 321. The output from each object absence verifier block 361, 362, 363 can be a binary signal (e.g., 1 / 0) output for each object, where 1 is output if the free space region is verified and 0 is output if the free space region cannot be verified. In addition, the system 10 is further configured to output a signal indicating the state of the sensor fusion system 32 based on the measurement of the verified objects and the measurement of the verified free space area. This is indicated by the sensor fusion status indicator block 300 which is configured to output a signal indicating the state of the sensor fusion system based on the outputs from the presence verifier block 351 to the presence verifier block 353 and the absence verifier block 361 to the absence verifier block 363. Accordingly, the sensor fusion status indicator block 300 receives the output data of the presence verifier blocks 351, 352, 353 and the absence verifier blocks 361, 362, 363, and outputs a "status signal" based on the output data received from these verifier blocks. The measurement can include a binary value (i.e., verified or not verified) and / or an aggregated measurement such as a percentage (i.e., the percentage of objects or free space areas verified by a particular sensor). Refer to the following. Figure 3 Provides more details related to the aggregated metrics. In addition, the metrics may further include associated time periods or moments (ie, the time when the object / free space region was authenticated / unauthenticated or the time when the authenticated / unauthenticated object was detected). Figure 3is a schematic block diagram representation of the sensor fusion status indicator 300. In more detail, the status indicator 300 block may include an object aggregation block 350 configured to receive object presence verification from the object presence verifier module and a free space area aggregation block 360 configured to receive free space verification from the object absence verifier module. Thus, the system may be configured to aggregate the verified object metrics over time for each sensor or each sensor subset and aggregate the verified free space area metrics over time for each sensor or each sensor subset. Thus, the signal indicating the status of the sensor fusion system 32 may be based on the aggregated metrics of the verified objects for each sensor or each sensor subset and the aggregated metrics of the verified free space area for each sensor or each sensor subset. In more detail, the object aggregation block 350 may be configured to aggregate the outputs of the presence verifiers from each sensor or each subset of sensors. For example, the presence verification status of each individual sensor or each individual subset of sensors is checked in the object aggregation block 350 to output a measure of the aggregation of verified objects over time for each sensor or each subset of sensors. Figure 3 In the depicted example, for each sensor_i, the signal "aggregate presence verification sensor_i" can be a binary signal indicating whether the sensor_i has verified more than a defined percentage (e.g., more than T%) of the output objects in the field of view of the sensor_i. Here, T is a threshold that can be set according to the requirements of the sensor fusion system 32 or for the ADS function that ingests the output of the sensor fusion system 32. Accordingly, for each sensor, the signal of each output from the object aggregation block 350 may be a binary signal (i.e., 1 or 0), wherein the signal of the output of the object aggregation block 350 for sensor_i is 1 if the sensor verifies more than T% (e.g., 80%, 90%, 95%, 99%, etc.) of the sensor fusion system's object detection within the sensor's field of view. Similarly, the signal of the output from the object aggregation block 350 for sensor_i is 0 if the sensor_i verifies less than T% (e.g., 80%, 90%, 95%, 99%, etc.) of the sensor fusion system's object detection within the sensor's field of view. Accordingly, the metric (on which the sensor fusion status signal is based) may be a binary signal. However, in some embodiments, the object aggregation block is configured to output a metric of verified object aggregation over time in the form of a percentage value for each sensor or each subset of sensors. Thus, for each sensor, the output from the object aggregation block 350 may include a binary value, a percentage, or a combination of a binary value and a percentage of object detections verified by the sensor. For each sensor, the output from the object aggregation block 350 may further include an associated time period or moment (time stamp). Similarly, the free space area aggregation block 360 can be configured to aggregate the outputs of the absence verifiers from each sensor or each subset of sensors. For example, the absence verification status of each individual sensor or each individual subset of sensors is checked in the free space area aggregation block 360 to output an aggregated measure of the verified free space area over time for each sensor or each subset of sensors. Figure 3In the depicted example of , for each sensor_i, the signal "aggregation absence verification sensor_i" can be a binary signal that indicates whether the sensor_i verifies more than a defined percentage (e.g., more than S%) of the output free space area in the field of view of the sensor_i. Here, S is a threshold that can be set according to the requirements of the sensor fusion system 32 or for the ADS function that ingests the output of the sensor fusion system 32. Accordingly, for each sensor, the signal of each output from the free space area aggregation block 360 can be a binary signal (i.e., 1 or 0), where if the sensor verifies more than S% (e.g., 80%, 90%, 95%, 99%, etc.) of the free space area detection in the field of view of the sensor by the sensor fusion system, the signal from the output of the free space area aggregation block 360 of the sensor_i is 1. Similarly, if the sensor_i verifies less than S% (e.g., 80%, 90%, 95%, 99%, etc.) of the free space area detection in the field of view of the sensor by the sensor fusion system, the signal from the output of the free space area aggregation block 360 of the sensor_i is 0. Accordingly, the metric (on which the sensor fusion status signal is based) may be a binary signal. However, in some embodiments, the free space area aggregation block 360 is configured to output the aggregated metric of the verified free space area over time in the form of a percentage value for each sensor or each subset of sensors. Thus, for each sensor, the output from the free space area aggregation block 360 may include a binary value, a percentage, or a combination of a binary value and a percentage of object detection verified by that sensor. For each sensor, the output from the free space area aggregation block 360 may further include an associated time period or moment (timestamp). With reference to the aforementioned percentage thresholds T and S, for any sensor fusion system, there may be different requirements between the tolerances for false negatives and false positives, and such differences may be considered feature-related (i.e., ADS function-related). For example, for an ADS function that controls vehicle motion and ingests outputs from a sensor fusion system, a key goal of the ADS function is that the vehicle cannot collide with an object, which places strict requirements on false negatives from the sensor fusion system. On the other hand, false positives may not be a problem, since the ADS function will then drive with possible, unnecessary caution. In this case, a false negative would be that the sensor fusion system outputs a free space region where there is an object (i.e., it misses the object), while a false positive is that the sensor fusion system outputs an object where there is no object. In the current context, a false negative can be detected as an unverified free space region, while a false positive can be detected as an unverified object detection. Therefore, for an ADS function that controls vehicle motion and ingests outputs from a sensor fusion system, the aforementioned threshold S (related to free space region verification) can be set more strictly (i.e., set at a higher percentage value) than the threshold T (related to object verification). In addition, in some embodiments, the object aggregation block 350 can be configured to aggregate the metrics of verified objects over time for multiple onboard sensors. In other words, the system can be configured to aggregate the metrics of verified objects over time for multiple onboard sensors. Similarly, the free space area aggregation block 360 can be configured to aggregate the metrics of verified free space areas over time for multiple onboard sensors. Metrics of verified objects and free space areas can be aggregated for all sensors that contribute to the perception output data. Therefore, the signal of the output indicating the state of the sensor fusion system can be based on the aggregated metrics of verified objects of multiple onboard sensors and the aggregated metrics of verified free space areas of multiple onboard sensors. This means that another output that may be generated by the object aggregation block 350 and the free space region aggregation block 360 may be “aggregated presence verification - all sensors” and “aggregated non-existence verification - all sensors”, respectively. Specifically, the object aggregation block 350 may further output the percentage of sensors among all sensors that are verifying the presence of objects, and evaluate whether more than a certain percentage of sensors are verifying more than a certain percentage of objects. For example, a goal may be to indicate whether a significant percentage of the sensor fusion system output cannot be verified by any (or a large number of) sensors. Additionally, the free space region aggregation block 360 may further output “aggregated non-existence verification of all sensors” indicating whether a large number of free space regions cannot be verified as free space regions by many sensors. This would indicate that, according to most sensors, the sensor fusion system has failed to detect / report a significant number of objects. The sensor fusion state indicator 300 block may further include a state determination block 370. The state determination block 370 is configured to evaluate the state of the sensor fusion system based on the threshold of the verified aggregation metric of the object and the verified aggregation metric of the free space area and further based on the input provided by the object aggregation block 350 and the free space area aggregation block 360. In addition, the state determination block 370 may be configured to output two types of output signals, namely, status and detailed status data. The output signal indicates the state of the sensor fusion system. The state may be a system state 380 of any output that uses a signal to convey whether the output of the sensor fusion system is reliable (e.g., whether the sensor fusion system works satisfactorily). The state may be output based on a binary value. The state may also be detailed state data, including statistics per sensor and per area that may be recorded by a recording component 390 and further used, for example, for debugging purposes. Advantageously, based on the output signal indicating the state of the sensor fusion system, the ADS function of the vehicle may be controlled. As disclosed herein, the ADS includes an ADS feature (also referred to as an ADS function) that is preferably a Level 3 or higher feature according to the SAE J3016 level of driving automation for road vehicles. The ADS feature may be, for example, a traffic jam pilot, a highway pilot, or any other SAE J3016 Level 3 or higher ADS feature. For example, the control of the ADS may include disabling a sensor fusion system, and thereby inhibiting activation of any ADS function that relies on input from the sensor fusion system. Furthermore, the control of the ADS may include deactivating any ADS function that relies on the sensor fusion system. Furthermore, in some embodiments, the system 10 may be configured to transmit data to a remote server 400 via one or more antennas of the vehicle. More specifically, the transmitted data represents the status of the sensor fusion system. Accordingly, in addition to local monitoring of the reliability of the output of the sensor fusion system of a single vehicle, a status indicator of each vehicle may be transmitted from the vehicle to a remote entity 400, where the status indicator may be consolidated across the entire fleet. In the present context, a "remote entity" may be understood as a central entity responsible for managing and maintaining the ADS deployed in a fleet. Figure 4 is a schematic block diagram representation illustrating how the output of the sensor fusion status indicator 300 may be directed from several vehicles to a data center where further evaluation and validation may be performed, according to some embodiments. The output of the sensor fusion status indicator 300 may be sent to a data center for use in monitoring 410 a fleet of vehicles and detecting prevalent problems and validating 420 the performance of the fleet of vehicles, such as large-scale validation of a sensor fusion system. continue, Figure 5is a schematic diagram of an ADS-equipped vehicle 1 including a system 10 for monitoring the reliability of an output of a sensor fusion system of a vehicle according to some embodiments. As used herein, a "vehicle" is any form of motorized transportation. For example, the vehicle 1 can be any road vehicle, such as a car (as shown herein), a motorcycle, a (freight) truck, a bus, etc. The system 10 includes a control circuit 11 and a memory 12. The control circuit 11 may physically include a single circuit device. Alternatively, the control circuit 11 may be distributed over several circuit devices. As an example, the system 10 may share its control circuit 11 with other parts of the vehicle 1 (e.g., the ADS 310). In addition, the system 10 may form a part of the ADS 310, that is, the system 10 may be implemented as a module or feature of the ADS. The control circuit 11 may include one or more processors such as a central processing unit (CPU), a microcontroller, or a microprocessor. One or more processors may be configured to execute program codes stored in the memory 12 to implement various functions and operations of the vehicle 1 in addition to the methods disclosed herein. The processor may be or include any number of hardware components for performing data or signal processing or for executing computer codes stored in the memory 12. Optionally, the memory 12 includes a high-speed random access memory such as a DRAM, SRAM, DDR RAM, or other random access solid-state storage device; and, optionally, includes a non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 12 may include a database component, an object code component, a script component, or any other type of information structure for supporting the various actions of the present specification. In the illustrated example, the memory 12 further stores map data 308. The map data 308 may be used, for example, by the ADS 310 of the vehicle 1 in order to perform autonomous functions of the vehicle 1. The map data 308 may include high-definition (HD) map data. It is contemplated that even if the memory 12 is illustrated as an element separate from the ADS 310, it may be provided as an integrated element of the ADS 310. In other words, according to exemplary embodiments, in the implementation of the inventive concept, any distributed or local storage device may be utilized. Similarly, the control circuit 11 may be distributed, for example, so that one or more processors of the control circuit 11 are provided as an integrated element of the ADS 310 or any other system of the vehicle 1. In other words, according to exemplary embodiments, in the implementation of the inventive concept, any distributed or local control circuit device may be utilized. The ADS 310 is configured to implement the functions and operations of the autonomous or semi-autonomous functions of the vehicle 1. The ADS 310 may include a plurality of modules, wherein each module is responsible for a different function of the ADS 310. The vehicle 1 includes a number of elements commonly found in autonomous or semi-autonomous vehicles. It will be appreciated that the vehicle 1 may have Figure 5 In addition, the vehicle 1 may include any combination of the various elements shown in Figure 5 . Although various elements are shown herein as being located inside the vehicle 1, one or more elements may be located outside the vehicle 1. For example, map data may be stored in a remote server and accessed by various components of the vehicle 1 via the communication system 326. Furthermore, as will be readily appreciated by those skilled in the art, even though various elements are described herein in a particular arrangement, the various elements may be implemented in a different arrangement. It should be further noted that the various elements may be communicatively connected to each other in any suitable manner. Since the elements of the vehicle 1 can be implemented in several different ways, Figure 5 The vehicle 1 should be regarded as an illustrative example only. The vehicle 1 further comprises a sensor system 320. The sensor system 320 is configured to acquire sensory data about the vehicle itself or its surroundings. The sensor system 320 may, for example, comprise a global navigation satellite system (GNSS) module 322 (e.g., GPS) configured to collect geographic location data of the vehicle 1. The sensor system 320 may further comprise one or more sensors 324. The sensor 324 may be any type of on-board sensor such as a camera, a lidar and a radar, an ultrasonic sensor, a gyroscope, an accelerometer, an odometer, etc. It should be understood that the sensor system 320 may also provide the possibility of acquiring sensory data directly or via a dedicated sensor control circuit in the vehicle 1. The vehicle 1 further includes a communication system 326. The communication system 326 is configured to communicate with external units such as other vehicles (i.e., via a vehicle-to-vehicle (V2V) communication protocol), a remote server (e.g., a cloud server), a database, or other external devices (i.e., a vehicle-to-infrastructure (V2I) or a vehicle-to-everything (V2X) communication protocol). The communication system 318 can communicate using one or more communication technologies. The communication system 318 may include one or more antennas (not shown). Cellular communication technology can be used for remote communications such as to a remote server or a cloud computing system. In addition, if the cellular communication technology used has low latency, it can also be used for V2V communication, V2I communication, or V2X communication. Examples of cellular radio technologies are GSM, GPRS, EDGE, LTE, 5G, 5G NR, etc., and also include future cellular solutions. However, in some solutions, medium- and short-range communication technologies such as wireless local area networks (LANs) (e.g., solutions based on IEEE 802.11) can be used to communicate with other vehicles near the vehicle 1 or with local infrastructure elements. ETSI is developing cellular standards for vehicle communications, for example, and 5G is considered a suitable solution due to low latency and efficient handling of high bandwidth and communication channels. The communication system 326 may accordingly provide the possibility to send outputs to a remote location (e.g. a remote operator or a control center) and / or receive inputs from a remote location via one or more antennas. Furthermore, the communication system 326 may be further configured to allow various elements of the vehicle 1 to communicate with each other. For example, the communication system may provide a local network setup such as CAN bus, I2C, Ethernet, fiber optics, etc. The local communication within the vehicle may also be of wireless type with protocols such as WiFi, LoRa, Zigbee, Bluetooth or similar medium / short range technologies. The vehicle 1 further includes a steering system 320. The steering system 328 is configured to control the steering of the vehicle 1. The steering system 328 includes a steering module 330 configured to control the direction of the vehicle 1. The steering system 328 further includes a throttle module 332 configured to control the actuation of the throttle of the vehicle 1. The steering system 328 further includes a brake module 334 configured to control the actuation of the brakes of the vehicle 1. The various modules of the steering system 328 can also receive manual inputs from the driver of the vehicle 1 (i.e., from the steering wheel, the accelerator pedal, and the brake pedal, respectively). However, the steering system 328 can be communicatively connected to the ADS 310 of the vehicle to receive instructions on how the various modules of the steering system 328 should act. Therefore, the ADS 310 is able to control the steering of the vehicle 1, for example, via the decision and control module 318. The ADS 310 may include a positioning module 312 or positioning block / system. The positioning module 312 is configured to determine and / or monitor the geographic location and orientation of the vehicle 1 and may utilize data from the sensor system 320, such as data from a GNSS module 322. Alternatively or in combination, the positioning module 312 may utilize data from one or more sensors 324. Alternatively, the positioning system may be implemented as a real-time kinematic (RTK) GPS for improved accuracy. The ADS 310 may further include a perception module 314 or perception block / system 314. The perception module 314 may refer to any well-known module and / or function, such as included in one or more electronic control modules and / or nodes of the vehicle 1, adapted and / or configured to interpret perception data related to driving of the vehicle 1 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. Thus, the perception module 314 may be adapted to rely on and obtain input from a plurality of data sources, such as vehicle imaging, image processing, computer vision, and / or in-vehicle networking, in combination with, for example, sensory data from the sensor system 320. The positioning module 312 and / or the perception module 314 may be communicatively connected to the sensor system 320 to receive the perception data from the sensor system 320. The positioning module 312 and / or the perception module 314 may further send control instructions to the sensor system 320. Specifically, the vehicle 1 of the present disclosure includes a system 10 for monitoring the reliability of the output of a sensor fusion system of the vehicle, wherein the sensor fusion system is configured to receive input data and generate perception output data. The system 10 includes one or more memory storage areas containing program code, and the one or more memory storage areas and program code are configured to cause the system 10 to at least store input data of the sensor fusion system with one or more processors, wherein the input data includes sensor data obtained from each of a plurality of on-board sensors configured to monitor the surrounding environment of the vehicle within a time period. In addition, the program code is configured to cause the system 10 to use the input data obtained within the time period to obtain perception output data output from the sensor fusion system, and the obtained perception output data includes one or more object detections in the surrounding environment of the vehicle and one or more free space area indications in the surrounding environment of the vehicle. In addition, for each sensor or each subset of sensors in the plurality of onboard sensors, the system 10 compares the acquired perception output data with the stored input data and determines: whether any object detection indicated in the acquired perception output data is indicated in the field of view of the sensor or the subset of sensors, so as to verify any object detection indicated in the acquired perception output data; and whether any free space area indication in the acquired perception output data is indicated in the field of view of the sensor or the subset of sensors, so as to verify any free space area indicated in the acquired perception output data. Finally, the system 10 outputs a signal indicating the status of the sensor fusion system based on the verified object measurement and the verified free space area measurement. In addition, the one or more memory storage areas and program codes may be configured to cause the system to at least perform any of the functions of the method 100 disclosed herein. For example, the one or more memory storage areas and program codes may be configured to cause the system to at least control an ADS function of the vehicle based on a signal outputted indicating a state of the sensor fusion system. The present invention has been presented above with reference to specific embodiments. However, other embodiments than those described above are possible and within the scope of the present invention. Within the scope of the present invention, method steps different from the above method steps may be provided for performing the method by hardware or software. Therefore, according to an exemplary embodiment, a non-transitory computer-readable storage medium is provided, which stores one or more programs, the one or more programs being configured to be executed by one or more processors of a vehicle control system, the one or more programs including instructions for performing a method according to any one of the above embodiments. Alternatively, according to another exemplary embodiment, a cloud computing system may be configured to perform any of the methods presented herein. The cloud computing system may include distributed cloud computing resources that jointly perform the methods presented herein under the control of one or more computer program products. In general, computer-accessible media may include any tangible or non-transitory storage media or memory media such as electronic, magnetic, or optical media, for example, a disk or CD / DVD-ROM connected to a computer system via a bus. As used herein, the terms "tangible" and "non-transitory" are intended to describe computer-readable storage media (or "memory") that do not include propagating electromagnetic signals, but are not intended to otherwise limit the types of physical computer-readable storage devices encompassed by the term computer-readable media or memory. For example, the term "non-transitory computer-readable media" or "tangible memory" is intended to cover types of storage devices that do not necessarily store information permanently, including, for example, random access memory (RAM). Program instructions and data stored in a non-transitory form on a tangible computer-accessible storage medium may be further transmitted via a transmission medium or signals such as electrical, electromagnetic, or digital signals that may be transmitted via a communication medium such as a network and / or a wireless link. Processor 11 (associated with system 10) can be or include any number of hardware components for performing data or signal processing or for executing computer code stored in memory 12. System 10 has associated memory 12, and memory 12 can be one or more devices for storing data and / or computer code for completing or facilitating the various methods described in this specification. The memory may include volatile memory or non-volatile memory. Memory 12 may include database components, object code components, script components, or any other type of information structure for supporting various actions of this specification. According to an exemplary embodiment, any distributed or local storage device can be used with the system and method of this specification. According to an exemplary embodiment, memory 12 (e.g., via circuit or any other wired, wireless or network connection) is communicatively connected to the processor and includes computer code for performing one or more processes described herein. It should be noted that any reference signs do not limit the scope of the claims, that the embodiments disclosed herein may be implemented at least in part by both hardware and software, and that multiple "means" or "units" may be represented by the same item of hardware. Although the drawings may show a specific order of method steps, the order of the steps may be different from the order depicted. In addition, two or more steps may be performed simultaneously or partially concurrently. This variation will depend on the selected software and hardware systems and the designer's choice. All these variations are within the scope of the present invention. Similarly, software implementations may be accomplished using standard programming techniques with rule-based logic and other logic to accomplish various processing steps, comparison steps, and decision steps. The embodiments mentioned and described above are given as examples only and should not limit the present invention. Other solutions, uses, objectives, and functions within the scope of the present invention claimed in the patent claims described below should be apparent to those skilled in the art. It should be noted that the term "if" can be interpreted to mean "when" or "based on" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if determined" or "when determined" or "in an instance of..." can be interpreted to mean "upon determination" or "in response to determination" or "upon detecting and identifying the occurrence of an event" or "in response to detecting the occurrence of an event", depending on the context.

Claims

1. A computer-implemented method (100) for monitoring the reliability of an output of a sensor fusion system of a vehicle, wherein: The sensor fusion system is configured to receive input data and generate sensory output data, the method (100) comprising: storing (101) input data for the sensor fusion system, wherein the input data includes sensor data obtained over a time period from each of a plurality of onboard sensors configured to monitor a surrounding environment of the vehicle; obtaining (102) perception output data output from the sensor fusion system using the input data obtained during the time period, the obtained perception output data comprising one or more object detections in the surrounding environment of the vehicle and one or more free space region indications in the surrounding environment of the vehicle; For each sensor or each subset of sensors in the plurality of on-board sensors: The obtained sensory output data is compared with the stored input data (103), and it is determined (104, 114): whether any object detection indicated in said obtained sensory output data is indicated in a field of view of the sensor or the subset of sensors, so as to verify any object detection indicated in said obtained sensory output data; and whether any free space region indication in said obtained sensory output data is indicated in said field of view of said sensor or said subset of sensors so as to verify any free space region indicated in said obtained sensory output data; and Based on the verified object measurements and the verified free space region measurements, a signal indicative of a status of the sensor fusion system is output (108).

2. The method (100) according to claim 1, further comprising: For each sensor or each subset of sensors in the plurality of on-board sensors: determining (106) a percentage of objects detected in said obtained perception output data that are verified, and determining (116) a percentage of free space area detected in said obtained sensory output data that is verified; and wherein the verified metric of objects comprises the verified percentage of detected objects in the sensory output data, and wherein the verified metric of free space area comprises the verified percentage of detected free space area in the sensory output data.

3. The method (100) according to claim 1, wherein: The sensor fusion system is an object tracking system configured to track the positions of objects present in the input data over time.

4. The method (100) according to claim 1, wherein: The storing (101) of input data includes separately storing each data set of the input data in a corresponding sensor-specific or subset-specific data store.

5. The method (100) of claim 1, further comprising: For each sensor or each subset of sensors in the plurality of on-board sensors: In response to detecting that an object detection indicated in said obtained sensory output data is indicated in said field of view of said sensor or said subset of sensors: Object detection by the sensor or the subset of sensors indicated in the obtained sensory output data is verified (105).

6. The method (100) of claim 1, further comprising: For each sensor or each subset of sensors in the plurality of on-board sensors: In response to detecting that the presence of a free space region indication in said obtained sensory output data is indicated in said field of view of the sensor or the subset of sensors: A free space region of the sensor or the subset of sensors indicated in the obtained sensory output data is verified (115).

7. The method (100) of claim 1, further comprising: For each sensor or each subset of sensors in the plurality of on-board sensors: aggregating (107) the metrics of the authenticated objects over time, and aggregating (117) said metrics of the verified free space regions over time; wherein the output signal indicative of the state of the sensor fusion system is based on the aggregated measure of the validated objects for each sensor or each subset of sensors and the aggregated measure of the validated free space area for each sensor or each subset of sensors.

8. The method (100) of claim 1, further comprising: aggregating the metrics of the authenticated object over time for the plurality of onboard sensors, and aggregating, for said plurality of onboard sensors, said metrics of verified free space regions over time; wherein the output signal indicative of the state of the sensor fusion system is further based on the aggregated measure of validated objects of the plurality of onboard sensors and the aggregated measure of validated free space areas of the plurality of onboard sensors.

9. The method (100) of claim 1, further comprising: An automated driving system (ADS) function of the vehicle is controlled (109) based on the output signal indicative of the state of the sensor fusion system.

10. The method (100) of claim 1, further comprising: Data is transmitted (110) to a remote server via one or more antennas of the vehicle, the transmitted data being indicative of the status of the sensor fusion system.

11. A computer program product comprising instructions which, when executed by a computing device of a vehicle, cause the computing device to implement the method (100) according to any one of claims 1 to 10.

12. A non-transitory computer-readable storage medium storing instructions which, when executed by a computing device of a vehicle, cause the computing device to implement the method (100) according to any one of claims 1 to 10.

13. A system (10) for monitoring the reliability of an output of a sensor fusion system (32) of a vehicle (1), wherein: The sensor fusion system (32) is configured to receive input data and generate sensory output data, the system (10) comprising one or more memory storage areas containing program code, the one or more memory storage areas and the program code being configured to cause the system (10) to at least: storing input data of the sensor fusion system (32), wherein the input data comprises sensor data obtained from each of a plurality of onboard sensors (321, 322, 323) configured to monitor a surrounding environment of the vehicle over a time period; Obtaining sensory output data output from the sensor fusion system (32) using the input data obtained during the time period, the obtained sensory output data comprising one or more object detections in the surrounding environment of the vehicle (1) and one or more free space region indications in the surrounding environment of the vehicle (1); For each sensor or each subset of sensors in the plurality of vehicle-mounted sensors (321, 322, 323): comparing the obtained sensory output data with the stored input data and determining; whether any object detection indicated in said obtained sensory output data is indicated in a field of view of the sensor or the subset of sensors, so as to verify any object detection indicated in said obtained sensory output data; and whether any free space region indication in said obtained sensory output data is indicated in said field of view of said sensor or said subset of sensors so as to verify any free space region indicated in said obtained sensory output data; and Based on the verified object measurements and the verified free space region measurements, a signal indicative of a status of the sensor fusion system (32) is output.

14. The system (10) of claim 13, wherein: The one or more memory storage areas and the program code are further configured to, with the one or more processors (11), cause the system (10) to at least: Based on the output signal indicative of the state of the sensor fusion system (32), an ADS function of the vehicle (1) is controlled.

15. A vehicle (1) comprising a system (10) according to claim 13 or 14.

Images