Assessment of perception performance of vehicle adas or ads

Abstract

The present disclosure provides for a perception performance evaluation of a vehicle ADAS or ADS, involving a method performed by a perception comparison system of a vehicle to evaluate a perception performance of an ADAS or ADS of the vehicle. The perception comparison system establishes communication with a secondary vehicle; derives perception data from a perception system of the ADAS or ADS; receives secondary perception data from a secondary perception system of a secondary ADAS or ADS of the secondary vehicle; determines a discrepancy output based on a comparison of at least a portion of the perception data and at least a portion of the secondary perception data when the secondary vehicle is locatable in the perception data; transmits confirmation data when at least a portion of the discrepancy output satisfies a discrepancy exceeding condition and / or when the vehicle is perceivable in the secondary perception data but the secondary vehicle is not locatable in the perception data. The present disclosure further relates to a perception comparison system according to the foregoing, a vehicle comprising such a perception comparison system and a corresponding computer program product and non-transitory computer readable storage medium.

Description

Technical Field

[0001] This disclosure relates to the perception performance evaluation of advanced driver assistance systems (ADAS) or automated driving systems (ADS) for vehicles. Background Technology

[0002] More and more modern vehicles are equipped with Advanced Driver Assistance Systems (ADAS) to improve vehicle safety and broader road safety. ADAS, which can be represented by, for example, Adaptive Cruise Control (ACC), Collision Avoidance Systems, and Forward Collision Warning Systems, can be electronic systems that assist the driver while driving. To function as expected, ADAS can rely on inputs from multiple data sources, such as automatic imaging, LiDAR, radar, image processing, computer vision, and / or in-vehicle networks.

[0003] Moreover, in the near future, Autonomous or Automated Driving Systems (ADS) will find greater integration into modern vehicles. ADS is a complex combination of various components that can be defined as systems that perform vehicle perception, decision-making, and operation electronically and mechanically, rather than through a human driver, to introduce automation into road traffic. This includes the processing of vehicle, destination, and environmental awareness. When an automated system controls a vehicle, it allows the human operator to leave all responsibilities to the system. ADS typically combines various sensors (such as radar, lidar, sonar, cameras, navigation systems (e.g., GPS), odometers, and / or inertial measurement units) to perceive the vehicle's environment. Advanced control systems can interpret this sensor information to identify appropriate navigation paths and obstacles and / or relevant landmarks.

[0004] Finally, essentially fully autonomous vehicles that can sense their environment and move safely along, for example, public roads with little or no human input may emerge.

[0005] However, as described above, ADAS or ADS necessarily requires a high degree of integrity to provide a sufficiently low risk to vehicle users and other road users. Ensuring a sufficiently low risk may require a large amount of data for statistical proof, and, by example, would take approximately one hundred vehicles continuously driving for five centuries to achieve. Several methods exist for minimizing the risk of ADAS or ADS before deploying them on public roads. However, in addition, it is generally considered that ADAS or ADS should be monitored on-site to ensure it meets the required safety level. However, even relying solely on the perception system outputs (i.e., observations of the world) to monitor the performance of ADAS or ADS makes it difficult to achieve dynamic driving tasks. That is, if the perception system performs poorly on the system, it is difficult to detect near misses or even accidents before they occur. Summary of the Invention

[0006] Therefore, the purpose of the embodiments herein is to provide a method for evaluating the perception performance of a vehicle’s ADAS or ADS in an improved and / or alternative manner, such as in the field.

[0007] The above-mentioned objectives can be achieved through the subject matter disclosed herein. Embodiments are set forth in the appended claims, the following description, and the accompanying drawings.

[0008] The disclosed subject matter relates to a method for evaluating the perception performance of an Advanced Driver Assistance System (ADAS) or Autonomous Driving System (ADS) of a vehicle, performed by a perception comparison system of the vehicle. The perception comparison system establishes communication with a secondary vehicle identified and / or estimated to be within the potential range of environmental detection sensors mounted on the vehicle. The perception comparison system also derives perception data from the perception system of the ADAS or ADS adapted to estimate the vehicle's environment. Furthermore, the perception comparison system receives secondary perception data from the secondary perception system of the secondary ADAS or ADS of the secondary vehicle. When the secondary vehicle can be located in the perception data, the perception comparison system further determines a difference output based on a comparison of at least a portion of the perception data with at least a portion of the secondary perception data. Additionally, when at least a portion of the difference output satisfies a difference exceeding condition, and / or when the vehicle can be perceived in the secondary perception data but cannot be located in the perception data, the perception comparison system transmits confirmation data.

[0009] The disclosed subject matter also relates to a perception comparison system for and / or adapted to evaluate the perception performance of a vehicle's ADAS or ADS. The perception comparison system includes a communication establishment unit for and / or adapted to establish communication with a secondary vehicle identified and / or estimated to be located within the potential range of an environmental detection sensor mounted on the vehicle. The perception comparison system also includes a perception data export unit for and / or adapted to export perception data from an ADAS or ADS perception system adapted to estimate the vehicle's environment. Furthermore, the perception comparison system includes a secondary perception data receiving unit for and / or configured to receive secondary perception data from a secondary ADAS or ADS secondary perception system of the secondary vehicle. The perception comparison system also includes a difference output determination unit for and / or adapted to determine a difference output based on a comparison of at least a portion of the perception data with at least a portion of the secondary perception data when the secondary vehicle can be located in the perception data. In addition, the perception comparison system includes a confirmation transmission unit, which is configured and / or adapted to transmit confirmation data when at least a portion of the difference output satisfies a difference exceeding condition, and / or when the vehicle can be perceived in the secondary perception data but cannot be located in the secondary perception data.

[0010] Furthermore, the disclosed subject matter relates to a vehicle that includes ADAS or ADS, and also includes a perception comparison system as described herein.

[0011] Furthermore, the disclosed subject matter relates to a computer program product comprising a computer program containing computer program code means arranged to cause a computer or processor to perform the steps of the perceptual comparison system described herein, the computer program being stored on a computer-readable medium or carrier wave.

[0012] The disclosed subject matter also relates to a non-volatile computer-readable storage medium having the computer program product stored thereon.

[0013] Therefore, a method is introduced to assess, for example, the perception performance of an ADAS or ADS perception system in the field to evaluate whether it has failed and / or the risk of failure, and to take action if so. That is, because communication is established with a secondary vehicle identified and / or estimated to be within the potential range of the environmental detection sensors mounted on the vehicle, the vehicle can communicate with another vehicle considered to be within the sensor and / or perception range or the expected sensor and / or its perception range. Furthermore, because perception data is derived from an ADAS or ADS perception system adapted to estimate the vehicle's environment, a world view is provided to the vehicle, thus presenting a perception model of the vehicle's environment. Moreover, because secondary perception data is received from the secondary perception system of the secondary ADAS or ADS of the secondary vehicle, perception data of the vehicle, including a perception model of the secondary vehicle's environment, is provided, such as a world view. Accordingly, since the secondary vehicle is identified and / or estimated to be within the potential range of the vehicle's environmental detection sensors, the perception data from the secondary vehicle may potentially at least partially overlap, correspond to, and / or cover the geographic area included in the perception data from the vehicle. Furthermore, since a difference output is determined based on a comparison of at least a portion of the perception data with at least a portion of the secondary perception data to determine when the secondary vehicle can be located in the perception data, data related to the overlap and / or corresponding portions, geographic areas or regions, objects, etc., of the secondary perception data and the vehicle's perception data can be compared, and differences between them can be identified, such as differences associated with estimates of, for example, object status, location, free space, and / or drivable areas. Therefore, the difference output can represent a perception error, failure, and / or failure risk for the vehicle's perception system and / or the perception system for the potential secondary vehicle. Moreover, since confirmation data is transmitted when at least a portion of the difference output satisfies a difference exceeding condition and / or when a vehicle can be perceived in the secondary perception data but cannot be located in the perception data, a confirmation indicating a difference can be sent from the perception comparison system. Accordingly, the perception comparison system focuses on, for example, the extent or manner in which the perception data of the current vehicle deviates from the perception data of the secondary vehicle to the degree or manner in which one or more conditions defined by the difference exceeding a condition are met. Therefore, based on said one or more differences, the failure and / or failure risk of the current vehicle's perception system and / or the potential perception systems of the secondary vehicles can be assessed.

[0014] For this reason, a method is provided to evaluate the perception performance of a vehicle’s ADAS or ADS in an improved and / or alternative manner (such as in the field).

[0015] The technical features and advantages of the above methods will be discussed in more detail below. Attached Figure Description

[0016] Various aspects of the non-limiting embodiments, including specific features and advantages, will be readily understood from the following detailed description and accompanying drawings, wherein:

[0017] Figure 1 A schematic diagram illustrating an exemplary perception comparison system for a vehicle according to an embodiment of the present disclosure;

[0018] Figure 2 a)-c) show exemplary vehicle perception data and corresponding secondary perception data of the change of high-level view along a road according to an embodiment of the present disclosure.

[0019] Figures 3a-3c A schematic diagram illustrating alternative exemplary vehicle perception data and corresponding secondary perception data for changes in high-level view along a road according to an embodiment of the present disclosure;

[0020] Figure 4 This is a schematic block diagram illustrating an exemplary perceptual comparison system according to embodiments of the present disclosure; and

[0021] Figure 5 This is a flowchart illustrating an exemplary method performed by a perceptual comparison system according to an embodiment of the present disclosure. Detailed Implementation

[0022] Non-limiting embodiments of the present disclosure will now be described more fully below with reference to the accompanying drawings, in which presently preferred embodiments of the disclosure are illustrated. However, the present disclosure may be presented in many different forms and should not be construed as limited to the embodiments set forth herein. The same reference numerals always refer to the same elements. Dashed lines in some of the boxes in the figures indicate that these elements or actions are optional rather than mandatory.

[0023] Below, based on an embodiment of the perception performance evaluation of ADAS or ADS for vehicles described herein, a method will be disclosed that allows for the on-site assessment of the perception performance of an ADAS or ADS perception system to evaluate whether it has failed and / or the risk of failure, and to take action if such a failure does occur.

[0024] Now refer to the accompanying drawings, and in particular Figure 1 The illustration depicts a schematic diagram of an exemplary perception comparison system 1 for a vehicle 2 according to an embodiment of the present disclosure. The exemplary vehicle 2, which may be referred to as the vehicle itself, the main vehicle, the primary vehicle, and / or the first vehicle, includes and / or is adapted to support ADAS and / or ADS21, and here it also includes the perception comparison system 1.

[0025] Exemplary vehicle 2 can refer to any arbitrary manned or unmanned vehicle, such as an engine-driven or electric vehicle, like a car, truck, van, box van, bus, motorcycle, and / or tractor. Furthermore, the term "vehicle" can refer to "transport vehicle," "road vehicle," and / or "production vehicle," and further to "autonomous and / or at least partially autonomous vehicle," "unmanned and / or at least partially unmanned vehicle," "automatic and / or at least partially automatic vehicle," and / or "autonomous and / or at least partially automatic vehicle." Exemplary ADAS or ADS can refer to any arbitrary ADAS or ADS, such as those known in the art and / or not yet developed. The phrase "perception comparison system" can refer to "production vehicle perception comparison system," "perception failure assessment system," "perception performance assessment system," "perception performance verification system," and / or "perception evaluation system," while "of the vehicle" can mean "included in the vehicle" and / or "equipped with the vehicle." On the other hand, "for" perception performance evaluation can mean "adapted to" perception performance evaluation, while the phrase "perception performance evaluation for ADAS or ADS" can mean "perception data performance evaluation" for ADAS or ADS, "perception performance monitoring and evaluation" for ADAS or ADS, "perception failure and / or failure risk assessment" for ADAS or ADS, "perception performance rating" for ADAS or ADS, "perception performance verification" for ADAS or ADS, and / or "perception evaluation" for ADAS or ADS. According to an example, the phrase "perception performance evaluation for ADAS or ADS" can mean "perception performance evaluation and potential further evaluation and / or intervention" for ADAS or ADS. The phrase "perception performance evaluation of the vehicle's ADAS or ADS" can mean "perception performance evaluation associated with the vehicle's ADAS or ADS" and / or "perception performance evaluation of perception estimates associated with the vehicle's ADAS or ADS," while "a" ADAS or ADS can further mean "at least a first" ADAS or ADS. According to the example, the phrase "perception performance evaluation" in ADAS or ADS can refer to "perception performance evaluation of the perception system" in ADAS or ADS.

[0026] Vehicle 2 and / or ADAS or ADS 21 may include, be equipped with, and / or have a perception system 22 adapted to estimate the environment of vehicle 2 and subsequently adapted to estimate the world view of the environment, for example, with the support of a known digital map (e.g., a high-definition HD map). Perception system 22 may refer to any known system and / or function, such as those included in one or more electronic control modules and / or nodes of vehicle 2 and / or ADAS or ADS 21, adapted and / or configured to interpret sensor information relevant to driving of vehicle 2 to identify, for example, obstacles, lanes, relevant signs, appropriate navigation paths, etc. Exemplary perception system 22 may be adapted to support, for example, sensor fusion, tracking, localization, etc., and therefore may be adapted to combine sensor information to rely on and obtain input from multiple data sources (e.g., automatic imaging, image processing, computer vision, and / or in-vehicle networks, etc.). Such exemplary sensor information may, for example, be derived from one or more exemplary environmental detection sensors 23 included in and / or mounted on vehicle 2. The environmental detection sensor 23 can be represented by any arbitrary sensor adapted to sense and / or perceive the environment and / or whereabouts of the vehicle 2, and may, for example, refer to one or a combination of radar, lidar, sonar, camera, navigation or positioning system (e.g., GPS), odometer and / or inertial measurement unit.

[0027] Perception comparison system 1, for example by means of communication establishment unit 101 ( Figure 4 As shown), the vehicle is adapted and / or configured to establish communication with a identified and / or estimated secondary vehicle 3, which is within the potential range of the detection sensor 23 mounted on vehicle 2. Vehicle 2 can thus communicate with another vehicle 3 that is believed to be within or anticipated within the sensor and / or sensing range of vehicle 2. Communication between vehicle 2 and secondary vehicle 3, and its establishment, can be accomplished in any arbitrary, for example, known manner, supported by a communication system (not shown) mounted on the respective vehicles 2 and 3. The established communication may therefore involve the exchange of security certificates between vehicle 2 and secondary vehicle 3.

[0028] Whether to establish communication with the secondary vehicle 3 can be determined in any manner deemed appropriate and / or feasible, for example, at least based on a first pre-predictable trigger. The optional trigger may, for example, involve determining and / or estimating that the secondary vehicle 3 is within a pre-predictable distance and / or a pre-predictable angle of the primary vehicle 2, such as being estimated to be traveling along the same road near the primary vehicle 2. Additionally or alternatively, the trigger may, for example, involve determining that the secondary vehicle 3 has the same construction as the primary vehicle 2 and / or is equipped with the same brand, similar, or identical ADAS and / or ADS as the optional perception system 22 and / or ADAS or ADS 21 of the primary vehicle 2. Furthermore, additionally or alternatively, the trigger may, for example, involve the fulfillment of timing criteria, such as time of day, day of the week, time since a previous trigger, a new driving cycle, etc. Further, additionally or alternatively, the trigger may, for example, involve information from a perception comparison system and / or, for example, from a remote centralized system (e.g., Figure 4 The request of the remote entity 9 (as shown and described in more detail therein) to establish communication with the secondary vehicle 3, for example, because it is deemed appropriate and / or desired.

[0029] Determining and / or estimating that the secondary vehicle 3 is within the potential range of the environmental detection sensor 23 of the vehicle 2 can be accomplished in any, for example, known, manner. For instance, determining and / or estimating that the secondary vehicle 3 is within the potential range of the environmental detection sensor 23 of the vehicle 2 can be accomplished by the vehicle 2 itself with the support of a digital map (e.g., an HD map) and / or the environmental detection sensor 23. Additionally or alternatively, this can be achieved with the support of vehicle-to-vehicle communication and / or with the support of the exemplary remote centralized system discussed above, which can be adapted to maintain the real-time or near-real-time position of the tracking vehicles (e.g., the vehicle 2 and the secondary vehicle 3). The potential range of the environmental detection sensor 23 can refer to any feasible range, such as the relevant characteristics and / or capabilities of the sensor 23 or generally the environmental detection sensor, and for example, refers to a distance, such as an exemplary 10, 100, or 1000 meters from the vehicle 2.

[0030] In the example Figure 1In this example, the secondary vehicle 3 travels slightly ahead of the primary vehicle 2 along the same road in the same direction. However, it should be noted that the secondary vehicle 3 can be positioned arbitrarily relative to the primary vehicle 2, such as being behind it at another angle, or traveling at another angle (e.g., in the opposite direction). Similar to the primary vehicle 2, the exemplary secondary vehicle 3 can refer to any arbitrary manned or unmanned vehicle, such as engine-driven or electric vehicles, such as cars, trucks, vans, minivans, buses, motorcycles, and / or tractors. Furthermore, the term "secondary vehicle" can similarly refer to "secondary transport vehicle," "secondary road transport vehicle," and / or "secondary production vehicle," and further to "secondary autonomous and / or at least partially autonomous vehicle," "secondary driverless and / or at least partially driverless vehicle," "secondary automated and / or at least partially automated vehicle," and / or "secondary automated and / or at least partially automated vehicle." The secondary vehicle 3 may also have similar characteristics to the present vehicle 2 in terms of, for example, functions, features, onboard systems and / or equipment; therefore, a detailed description of such similar characteristics of the secondary vehicle 3 is omitted. Accordingly, the secondary vehicle 3 may include a corresponding secondary ADAS or ADS 31 with functions similar to those of the present vehicle 2, a corresponding secondary perception system 32, a corresponding secondary environmental detection sensor 33, and potentially also the corresponding perception comparison system described herein.

[0031] The phrase “establish communication with a secondary vehicle” can mean “establish communication with a nearby secondary vehicle,” “establish communication with a secondary vehicle involving the exchange of security certificates and / or signal exchanges,” and / or “establish encrypted communication with a secondary vehicle,” while the term “secondary” can always mean “second,” “reference,” and / or “other.” On the other hand, “a” secondary vehicle can mean “at least one” secondary vehicle. The phrase “determined and / or estimated to be within the potential range of an environmental detection sensor” can mean “believed and / or expected to be within the potential range of an environmental detection sensor,” while “located” can mean “positioned at.” According to an example, the phrase “determined and / or estimated to be within the potential range” can mean “determined and / or estimated to be within the potential range with the support of a high-definition (HD) map” and / or “determined and / or estimated to be within the potential range with the support of a remote, centralized system for vehicle-to-vehicle communication and / or for understanding the basic real-time vehicle location of the vehicle and the secondary vehicles.” On the other hand, "potential range" can refer to "expected range" and / or "estimated range," while "range" can refer to "sensor and / or sensing range." According to one example, the phrase "within the potential range of the environmental detection sensor mounted on the vehicle" can mean "near the vehicle," "within a predetermined distance and / or predetermined angle from the vehicle," or "within a predetermined distance from the vehicle that is shorter than the potential range of the environmental detection sensor mounted on the vehicle." This optional predetermined distance can be any feasible extension considered relevant to the current situation, and for example, refers to an exemplary distance of 5, 50, or 500 meters from the vehicle 2. On the other hand, the phrase "environmental detection sensor" can refer to "one or more environmental detection sensors" and / or "environmental detection sensors adapted to collect the environment of the vehicle," while "mounted on the vehicle" can mean "mounted on this vehicle."

[0032] Perceptual comparison system 1, for example by means of perceptual data derivation unit 102 ( Figure 4 As shown), it is adapted and / or configured to export perception data 4 from the perception system 22 of ADAS or ADS 21. Figure 2 , Figures 3a-3cAs shown, perception system 22 is adapted to estimate the environment of vehicle 2. Therefore, a world view 4 is established for vehicle 2, presenting a perception model of the vehicle's environment. The perception data 4 can be derived in any arbitrary, for example, known manner (e.g., electronic and / or digital). Furthermore, perception data 4 can refer to any arbitrary data associated with ADAS or ADS 21, which can also represent the perception of the environment of vehicle 2 (which includes, for example, sensor data derived from, or derived from, one or more environmental detection sensors 23), internal ADAS or ADS 21 processing for sensor fusion, tracking, etc., and / or digital maps (e.g., HD maps). It can be noted that, according to the example, perception data 4 can be derived from perception system 22 even if vehicle 2 is at least partially controlled by the vehicle driver during the said derivation and / or generation of perception data 4.

[0033] The phrase "derive perception data from the perception system" can mean "retrieve and / or obtain perception data from the perception system," and, according to examples, can also mean "determine the perception data of the perception system." On the other hand, "derive perception data" can mean "derive primary vehicle and / or this vehicle's perception data," "derive world view data," "derive world view," "derive world view with the support of the internal processing of the ADAS or ADS and / or digital map (e.g., HD map)," and / or "derive global world view and / or HD map coordinates with the support of an HD map." The phrase "from the perception system" can mean "from the primary vehicle and / or this vehicle's perception system." Furthermore, "perception system adapted to estimate the vehicle's environment" can mean "perception system adapted to estimate the vehicle's world view" and / or "perception system adapted to estimate the world view of the vehicle's environment," and further means "perception system adapted and / or configured to estimate at least a portion of the vehicle's environment," and "perception system adapted and / or configured to interpret sensor information related to the vehicle's driving." On the other hand, "the vehicle's environment" can mean "the environment of this vehicle, the primary vehicle, and / or the main vehicle." According to the example, the phrase "a vehicle's perception system adapted to estimate the vehicle's environment" can refer to "a vehicle's perception system adapted to estimate the vehicle's environment via input from said environmental detection sensors and / or with the support of a digital map (such as an HD map)." Furthermore, the term "perceived data" can refer to "environmental assessment" data and / or "sensor" data, while "perceived data" can refer to perceived "prediction," "information," and / or "estimation." On the other hand, "perceived system" can refer to a "vehicle environmental assessment and / or evaluation" system, while "perceived system" can refer to perceived "function and / or functionality."

[0034] Perceptual comparison system 1, for example by means of secondary perceptual data receiving unit 103 ( Figure 4 As shown), it is adapted and / or configured to receive secondary perception data 5 from the secondary perception system 32 of the secondary ADAS or ADS 31 of the secondary vehicle 3. Figure 2 , Figures 3a-3c (As shown). Therefore, the vehicle 2 is provided with perception data 5 generated by the secondary vehicle 3, such as a world view, which includes a perception model of the secondary vehicle's environment. Accordingly, since the secondary vehicle 3 is determined and / or estimated to be within the potential range of the environmental detection sensors 23 of the vehicle 2, the perception data 5 from the secondary vehicle 3 may potentially at least partially overlap, correspond to, and / or cover the geographic area included in the perception data 4 from the vehicle 2. The secondary perception data 5, which may represent, for example, a perception of the environment of the secondary vehicle 3 including and / or derived from sensor data from one or more secondary environmental detection sensors 33 and / or secondary digital maps (e.g., HD maps), may be received from the secondary vehicle 3 in any arbitrary, for example known manner (e.g., wireless, electronic, and / or digital). The secondary perception data 5 may include data relating to geographic areas that at least partially overlap with the geographic area covered by the vehicle's perception data 4, i.e., the perception data 4 derived from the vehicle 2. Furthermore, the secondary perception system 32 may optionally be similar to or the same as the perception system 22 of the vehicle 2, for example, having similar, corresponding, and / or identical configurations, settings, output formats, etc. Similarly, the secondary ADAS or ADS 31 may optionally be similar to or the same as the perception system 22 of the vehicle 2, for example, referring to similar, corresponding, and / or identical ADAS or ADS, and / or having similar, corresponding, and / or identical configurations, settings, etc. It can be noted that, according to the example, even if the secondary vehicle 3 is at least partially controlled by the vehicle driver during the reception period and / or during the generation of the secondary perception data 5, the secondary perception data 5 can still be received from the secondary vehicle 23.

[0035] The phrase "receive secondary perception data" can mean "acquire and / or retrieve secondary perception data," while "secondary perception data" can simply mean "perception data." Similarly, "secondary ADAS or ADS" can simply mean "ADAS or ADS." According to the example, the phrase "receive secondary perception data from a secondary perception system of a secondary ADAS or ADS of the secondary vehicle" can mean "receive secondary perception data from a secondary perception system of a secondary ADAS or ADS of the secondary vehicle, the secondary perception data including data relating to a geographic area that at least partially overlaps with the geographic area covered by the vehicle's perception data."

[0036] Perceptual comparison system 1, for example by means of difference output determination unit 104 ( Figure 4As shown), the system is adapted and / or configured to determine a difference output based on a comparison of at least a portion of perception data 4 with at least a portion of secondary perception data 5 when a secondary vehicle 3 can be located in perception data 4. Therefore, assuming the secondary vehicle 3 can be found in the perception data 4 of the vehicle 2, data related to overlaps and / or corresponding portions, geographic areas or regions, objects, etc., of the secondary perception data 5 and the vehicle perception data 4 can be compared, and differences between them can be identified, such as differences associated with, for example, object state estimation, location, free space, and / or drivable areas. Thus, the difference output can represent a perception error, failure, and / or failure risk for the vehicle perception system 22 and / or the perception system 32 of the potential secondary vehicle 3. The difference output can be determined by comparing data related to, for example, one or more selected overlaps and / or geographic areas of the corresponding vehicle perception data 4 and secondary perception data 5. Furthermore, at least a portion of perception data 4 can refer to any arbitrary dataset of vehicle perception data 4 having any arbitrary size, such as a selectable dataset considered relevant to the comparison. Similarly, at least a portion of the secondary perception data 5 can refer to any arbitrary dataset of secondary vehicle perception data 5 having any arbitrary size, such as a predetermined dataset, for example, the portion substantially corresponding to and / or overlapping with the present vehicle perception data. The difference output can refer to any arbitrary difference between the present vehicle perception data 4 and the perception data 5 from the secondary vehicle 3, which can be considered relevant to the evaluation, for example, in terms of quality and / or safety standards.

[0037] The phrase "determine the difference output based on a comparison of..." can mean "to calculate the difference output by comparison" and / or "to determine and subsequently store the difference output based on a comparison of...", where "difference output" can mean "difference output measure", "difference data", "perceived difference", and / or "perceived difference". On the other hand, "comparison of at least a portion of the perceived data with at least a portion of the secondary perceived data" can mean "analyzing at least a portion of the perceived data in view of at least a portion of the secondary perceived data", "comparison of at least a portion of the perceived data with at least the corresponding, related, and / or substantially geographically overlapping portions of the secondary perceived data", and / or "comparison of at least a portion of the vehicle's perceived data with at least a portion of the secondary perceived data". According to an example, the phrase "determine the difference output based on a comparison of at least a portion of the perceived data with at least a portion of the secondary perceived data" can mean "determine the difference output based on a comparison of at least a portion of the perceived data with at least a portion of the secondary perceived data, the difference output representing the difference between the perceived data and the secondary perceived data". Furthermore, the phrase "when the secondary vehicle can be located in the sensing data" can mean "assuming the secondary vehicle can be located in the sensing data" and / or "when the secondary vehicle, including and / or locating in the sensing data, is detected," and, according to the example, further means "when the secondary vehicle can be identified in the sensing data."

[0038] The sensing comparison system 1, for example by means of the confirmation transmission unit 105 ( Figure 4As shown), the adapter and / or configuration is to transmit confirmation data 6 when at least a portion of the difference output satisfies the difference exceeding condition and / or when vehicle 2 can be sensed in secondary perception data 5 but secondary vehicle 3 cannot be located in perception data 4. Therefore, assuming the difference criterion is met and / or assuming vehicle 2 can be sensed in perception data 5 from secondary vehicle 3 but secondary vehicle 3 is unexpectedly not found in perception data 4 of this vehicle, confirmation 6 indicating a difference is sent from perception comparison system 1. That is, the result of the comparison, i.e., the difference output, can be considered in light of at least a first criterion, and if said criterion is met, confirmation 6 indicating it can be sent from perception comparison system 1. Similarly, additionally or alternatively, if vehicle 2 can be sensed in perception data 5 from secondary vehicle 3 but secondary vehicle 3 is unexpectedly not included in perception data of this vehicle, confirmation 6 indicating it can be sent from perception comparison system 1. Accordingly, the perception comparison system 1 focuses on when the perception data 4 of the primary vehicle deviates from the perception data 5 of the secondary vehicle to a certain extent and / or in a manner that satisfies one or more conditions defined by the difference exceeding condition and / or when the primary vehicle 2 can be perceived in the secondary perception data 5 but the secondary vehicle 3 cannot be located unexpectedly in the primary vehicle perception data 4. Therefore, the failure and / or failure risk of the perception system 22 of the primary vehicle 2 and / or the perception system 32 of the potential secondary vehicle 3 can be assessed. The difference exceeding condition can be defined by any one or more arbitrary criteria considered relevant to the current implementation (e.g., given the characteristics and / or requirements related to the perception system 22 and / or ADAS or ADS 21, such as the maximum level of difference, the maximum number of times the difference level is exceeded, or the maximum number of times the difference exceeds within a predetermined time frame). Moreover, the difference exceeding condition can be variable and can also be predetermined.

[0039] "Transmitting" confirmation data can refer to "transmitting" confirmation data wirelessly and / or via wire and / or "transmitting" confirmation data at the appropriate time and / or when deemed feasible and / or safe," while "confirmation data" can refer to "one or more confirmation signals" and / or "confirmation messages," and further refers to "confirmation data that reflects and / or includes at least the difference output and / or indicates that a secondary vehicle cannot be located in the sensing data." On the other hand, the term "when" can refer to "assuming" and / or "if." The phrase "at least a portion of the difference output" can refer to "at least a predetermined and / or selected portion of the difference output," while the phrase "satisfying the difference exceeding condition" can refer to "satisfying a predetermined difference exceeding condition," "at least satisfying a first exceeding criterion," and / or "exceeding one or more difference exceeding thresholds." Furthermore, the phrase "however, the secondary vehicle cannot be located in the perception data" can mean "however, the secondary vehicle is not perceived and / or not included in the perception data," and, according to the example, further means "however, the secondary vehicle cannot be identified in the perception data" and / or "however, the secondary vehicle cannot be located and perceived in the perception data." Additionally, the phrase "however, the secondary vehicle cannot be located in the perception data" can further mean "however, the secondary vehicle is unexpectedly not located in the perception data." "Unexpectedly" not being located here can mean that the secondary vehicle 3, although expected and / or thought to be so, is determined to be unable to be located in the perception data 4. If it is considered possible to be located, for example based on being considered to be within sensor range and / or being considered to be not obstructed or blocked from the viewpoint of one or more of the environmental detection sensors 23, then the secondary vehicle 3 can be expected to be able to be located.

[0040] Figure 2 (a)-c) illustrate exemplary vehicle perception data 4 and corresponding secondary perception data 5 of the change in high-level visibility along a road according to an embodiment of the present disclosure, as shown in the diagram. Figure 2As further shown in a)-c), the perception comparison system 1 may, for example by means of the difference output determining unit 104, optionally adapt and / or configure itself to determine at least a first portion of the difference output based on a comparison of perception data 4 including object state estimation 41 of vehicle 2 and secondary perception data 5 including secondary object state estimation 51 of vehicle 2. Additionally or alternatively, the perception comparison system 1 may, for example by means of the difference output determining unit 104, adapt and / or configure itself to determine at least a first portion of the difference output based on a comparison of perception data 4 including object state estimation 42 of secondary vehicle 3 and secondary perception data 5 including secondary object state estimation 52 of secondary vehicle 3. Therefore, the difference output may be based at least in part on the result of comparing object state estimation 41 (e.g., the location of vehicle 2 from the vehicle perception data 4) with object state estimation 51 (e.g., the location of vehicle 2 included in perception data 5 generated by and subsequently received from secondary vehicle 3). Similarly, additionally or alternatively, the difference output can therefore be based at least in part on the result of comparing object state estimate 42 (e.g., the location of secondary vehicle 3 included in the vehicle's perception data 4) with object state estimate 52 (e.g., the location of secondary vehicle 3 from perception data 5 generated by and subsequently received from secondary vehicle 3). Accordingly, the difference between them can be identified and potentially assessed by comparing, for example, the internal belief 41 of the vehicle's perception system 22 for the location of the vehicle 2 with the internal belief 51 of the secondary vehicle perception system 32 for the location of the vehicle 2, and / or by comparing, for example, the internal belief 42 of the vehicle's perception system 22 for the location of the secondary vehicle 3 with the internal belief 52 of the secondary vehicle perception system 32 for the location of the secondary vehicle 3.

[0041] In the example Figure 2 In (a), there is an inconsistency, i.e., a difference, between the object state estimate 51 of the vehicle 2 in the secondary vehicle perception data 5 and the object state estimate 41 of the vehicle 2 in the primary vehicle perception data 4. Similarly, there is an inconsistency, i.e., a difference, between the object state estimate 52 of the secondary vehicle 3 in the secondary vehicle perception data 5 and the object state estimate 42 of the secondary vehicle 3 in the primary vehicle perception data 4. If any or both of the differences, i.e., the difference output, satisfy the difference exceeding condition, then the perception comparison system 1 can transmit confirmation data 6.

[0042] Comparatively, in the example Figure 2 In b), there is no difference between the object state estimate 51 of the vehicle 2 in the secondary vehicle perception data 5 and the object state estimate 41 of the vehicle 2 in the primary vehicle perception data 4, because the estimates 41 and 51 are consistent. Further comparablely, in the exemplary... Figure 2In c), there is no difference between the object state estimate 52 of the secondary vehicle 3 in the secondary vehicle perception data 5 and the object state estimate 42 of the secondary vehicle 3 in the primary vehicle perception data 4, because the estimates 42 and 52 are consistent.

[0043] Object state estimates 41, 42, 51, and 52 may include, for example, the states of the vehicle 2 and / or other vehicles 3, such as the speed, orientation, acceleration, and positioning of the vehicle 2 and / or other vehicles 3. The phrase "determine at least a first portion of the difference output based on..." can mean "determine the difference output at least partially based on...". According to an example, the phrase "comparison of perceived data including the object state estimate of the vehicle with secondary perceived data including the secondary object state estimate of the vehicle" can mean "applying a comparison function to the vector of the perceived data including the object state estimate of the vehicle and the secondary vector of the secondary perceived data including the secondary object state estimate of the vehicle." Similarly, according to an example, the phrase "comparison of perceived data including the object state estimate of the secondary vehicle with secondary perceived data including the secondary object state estimate of the secondary vehicle" can mean "applying a comparison function to the vector of the perceived data including the object state estimate of the secondary vehicle and the secondary vector of the secondary perceived data including the secondary object state estimate of the secondary vehicle."

[0044] According to the example, one or more or at least a portion of object state estimates 41, 42, 51, and 52 can be represented as a state vector including the location, as exemplarily represented as follows: If vehicle 2 is represented by a subscript of, for example, e, and secondary vehicle 3 is represented by a subscript of 1, then the performance check, which identifies one or more potential output discrepancies, can be expressed by the following formula:

[0045] 1.

[0046] 2.

[0047] Where ' can correspond to data 5, 51, 52 from secondary vehicle 3. Any comparison function f (e.g., absolute difference and / or mean square error) can be used to analyze each of the two exemplary comparisons, and the output, i.e., at least a portion of the difference output, can be compared with at least a portion of the difference exceeding the condition (here denoted as ∈).

[0048] Figures 3a-3c A schematic diagram illustrating exemplary vehicle perception data 4 and corresponding secondary perception data 5 along a road from a high-level view according to an embodiment of the present disclosure is shown, as follows: Figures 3a-3cAs further shown, when the positioning estimates 41 and 42 of vehicle 2 and secondary vehicle 3 in the perception data 4 are consistent with the positioning estimates 51 and 52 of vehicle 2 and said secondary vehicle 3 in the secondary perception data 5, the perception comparison system 1 may, for example by means of the difference output determination unit 104, optionally adapt and / or configure itself to further determine at least a second portion of the difference output.

[0049] As exemplary Figure 3a As shown, the at least second portion of the difference output may optionally be based on a comparison of perception data 4, which includes object state estimation 43 of at least a first environmental object, with secondary perception data 5, which includes secondary object state estimation 53 of the at least first environmental object. Therefore, assuming that the localization estimates 41, 42 of vehicle 2 and secondary vehicle 3 in perception data 4 are consistent with the localization estimates 51, 52 of vehicle 2 and the secondary vehicle 3 in secondary perception data 5, the difference output may be based at least in part on the result of comparing object state estimation 43 (e.g., localization of at least the first environmental object from the vehicle perception data 4) with object state estimation 53 (e.g., localization of the at least first environmental object included in perception data 5 generated by and subsequently received from the secondary vehicle 3). Accordingly, by comparing the internal belief 43 of the vehicle perception system 22 used for the localization of the at least first environmental object with the internal belief 53 of the secondary vehicle perception system 32 used for the localization of the at least first environmental object, the difference between them can be identified and potentially assessed.

[0050] In the example Figure 3a In the perception data 4, there is an inconsistency, or difference, between the object state estimate 53 of at least the first environmental object in the secondary vehicle perception data 5 and the object state estimate 43 of at least the first environmental object in the primary vehicle perception data 4. If the difference output satisfies the difference exceeding condition, confirmation data 6 can be transmitted by the perception comparison system 1. The at least first environmental object can refer to any arbitrary moving or static object, such as a vehicle, which is considered feasible and / or suitable for object state estimates 43, 53, and can have any arbitrary size. Furthermore, the at least first object can be arbitrarily located in the field of view of both the primary vehicle 2 and the secondary vehicle 3.

[0051] Additional or alternative land, such as Figure 3bAs shown, the at least second portion of the difference output may optionally be based on a comparison of perception data 4, which includes a free space estimate 44 of at least a region 45 surrounding vehicle 2, and secondary perception data 5, which includes a secondary free space estimate 54 of at least a secondary region 55 surrounding secondary vehicle 3, where region 45 at least partially overlaps with secondary region 55. Therefore, assuming that the localization estimates 41, 42 of vehicle 2 and secondary vehicle 3 in perception data 4 are consistent with the localization estimates 51, 52 of vehicle 2 and said secondary vehicle 3 in secondary perception data 5, respectively, the difference output may be based at least in part on the result of a comparison of the free space estimate 44 included in the vehicle perception data 4 in the overlapping region 4555 with the free space estimate 54 included in the perception data 5 generated by and subsequently received from the secondary vehicle 3. Accordingly, the difference between the two can be identified and potentially assessed by comparing the internal beliefs 54 of the secondary vehicle perception system 32 in the overlapping region 4555 for free space (e.g., the region where no other objects are believed to be located) with the internal beliefs 44 of the vehicle perception system 22 for free space.

[0052] In the example Figure 3b In the perception data 4, there is an inconsistency, i.e., difference 7, between the free space estimate 54 in the secondary vehicle perception data 5 and the free space estimate 44 in the primary vehicle perception data 4. If the difference 7, i.e., the difference output, satisfies the difference exceeding condition, confirmation data 6 can be transmitted by the perception comparison system 1. The free space region 45 can refer to an area of ​​any size located around the primary vehicle 2 along any arbitrary direction. Similarly, the secondary free space region 55 can refer to an area of ​​any size located around the secondary vehicle 3 along any arbitrary direction, assuming that the secondary region 55 overlaps with the primary vehicle region 45 by any size and / or size range.

[0053] Additional or alternative, as exemplary Figure 3cAs shown, the at least second portion of the difference output may optionally be based on a comparison of perception data 4, which includes a drivable area estimate 46 in at least one region 47 surrounding vehicle 2, with secondary perception data 5, which includes a secondary drivable area estimate 56 in at least one secondary region 57 surrounding secondary vehicle 3, where region 47 at least partially overlaps with secondary region 57. Therefore, assuming that the positioning estimates 41, 42 of vehicle 2 and secondary vehicle 3 in perception data 4 are consistent with the positioning estimates 51, 52 of vehicle 2 and said secondary vehicle 3 in secondary perception data 5, the difference output may be based at least in part on the result of a comparison of the drivable area estimate 46 included in the vehicle's perception data 4 in the overlapping region 4757 with the drivable area estimate 56 included in the perception data 5 generated by and subsequently received from the secondary vehicle 3. Accordingly, by comparing the internal beliefs 46 of the primary vehicle perception system 22 for drivable areas (such as areas considered as drivable roads) in the overlapping region 4757 with the internal beliefs 56 of the secondary vehicle perception system 32 for drivable areas, the differences between them can be identified and potentially assessed.

[0054] In the example Figure 3c In the perception data 4, there is an inconsistency, or difference 8, between the drivable area estimate 56 in the secondary vehicle perception data 5 and the drivable area estimate 46 in the primary vehicle perception data 4. If the difference 8, i.e., the difference output, satisfies the difference exceeding condition, confirmation data 6 can be transmitted by the perception comparison system 1. The drivable area 46 can refer to any area of ​​any size located around the primary vehicle 2 along the basic forward direction of the primary vehicle 2. Similarly, the secondary drivable area 55 can refer to any area of ​​any size located around the secondary vehicle 3 along the basic forward direction of the secondary vehicle 3, assuming that the secondary area 55 overlaps with the primary vehicle area 45 by any size and / or size range.

[0055] The phrase "when the positioning estimates of the vehicle and the secondary vehicle in the perception data are respectively consistent with the positioning estimates of the vehicle and the secondary vehicle in the secondary perception data" can mean "when the first portion of the difference output indicates that the positioning estimates of the vehicle and the secondary vehicle in the perception data are respectively consistent with the positioning estimates of the vehicle and the secondary vehicle in the secondary perception data," where "respectively consistent" can mean "respectively substantially consistent." On the other hand, "consistent" can mean "overlapping." On the other hand, "object state estimation of at least the first environmental object" can mean "object state estimation including the positioning of at least the first environmental object." According to the example, the phrase "when the positioning estimates of the vehicle and the secondary vehicle in the perception data are respectively consistent with the positioning estimates of the vehicle and the secondary vehicle in the secondary perception data" can mean "when the positioning estimates of the vehicle and the secondary vehicle in the perception data are respectively consistent with the positioning estimates of the vehicle and the secondary vehicle in the secondary perception data vector." According to another example, the phrase “comparison of sensing data” in this document can refer to “applying a comparison function to a vector of the sensing data”, while “secondary sensing data” in this document can refer to “the secondary vector of the secondary sensing data”.

[0056] The sensing comparison system 1 can, for example, be equipped with an acknowledgment transmission unit 105 ( Figure 4 (As shown), it can optionally be adapted and / or configured to wirelessly transmit confirmation data 6 to remote entity 9. The confirmation data 6 may, for example, include a difference output. Therefore, data related to, for example, the difference output can be sent to entity 9 for offline processing and / or analysis. Additionally or alternatively, the confirmation data 6 may include an indication in the secondary sensing data 5 that the vehicle 2 can be sensed but the secondary vehicle 3 cannot be located in the vehicle sensing data 4. The timing of the transmission can be arbitrarily chosen in a manner deemed feasible (e.g., promptly or at a deemed appropriate time). The latter case may, for example, involve waiting for a high-bandwidth connection to the remote entity 9, such as a Wi-Fi connection.

[0057] Entity 9 can refer to any (e.g., known) off-site data storage entity adapted and / or configured for off-site and / or offline processing and / or analysis, such as cloud and / or automotive cloud, storage adapted for cloud-based use, backend systems, and / or cloud networks of one or more servers. A “remote” entity can refer to an “off-site” entity and / or an “offline” entity, while “including the differential output” can mean “substantially including the differential output.”

[0058] Alternatively or additionally, the sensing comparison system 1 may, for example, be equipped with an acknowledgment transmission unit 105. Figure 4As shown), the system is adapted and / or configured to send confirmation data 6 to the vehicle-mounted ADAS / ADS control system 24, which is adapted to control ADAS or ADS 21 at least to some extent. The confirmation data 6 includes an indication that ADAS or ADS 21 is at least partially disabled. Therefore, ADAS or ADS 21 can be at least temporarily deactivated to prevent further activation.

[0059] ADAS / ADS control system 24 can refer to any generally known system and / or function, such as those included in one or more electronic control modules and / or nodes of the vehicle 2, adapted and / or configured to at least partially control ADAS or ADS 21. "ADAS / ADS control system" can refer to "disable system" and / or "ADAS or ADS control system," and an ADAS / ADS control system adapted to perform "control" can refer to an ADAS / ADS control system adapted to "at least partially perform control." On the other hand, "indicating" at least partially disabling can refer to "prompting" at least partially disabling and / or "including prompting" at least partially disabling a "command," while "indicating at least partially disabling" can refer to "indicating at least partially disabling when appropriate" and / or "indicating at least partially disabling when deemed feasible and / or safe." Furthermore, "disabling" ADAS or ADS can refer to "prohibiting" and / or "restricting" the functionality of ADAS or ADS.

[0060] Furthermore, the sensing comparison system 1 can also, for example, utilize the confirmation transmission unit 105 ( Figure 4(As shown), it can optionally be adapted and / or configured to transmit additional confirmation data 60 when the difference output satisfies an additional difference exceedance condition, which deviates from the additional difference exceedance condition. Therefore, different confirmation data 6, 60 can be transmitted depending on which difference exceedance condition is met. Accordingly, satisfying the difference exceedance condition (based on the transmission of confirmation data 6) can initiate a first action, while satisfying the additional difference exceedance condition (based on the transmission of additional confirmation data 60) can initiate a different second action. That is, the result of comparison and / or evaluation, i.e., the difference output, is considered in light of the additional second difference exceedance condition, and if the second difference exceedance condition is met, an additional confirmation 60 representing it is sent from the perception comparison system 1. Accordingly, the perception comparison system 1 can further note when the primary vehicle perception data 4 deviates from the secondary vehicle perception data 5 to the extent that it satisfies the condition defined by the additional difference exceedance condition, which may, for example, reflect larger and / or more frequent differences. Therefore, further evaluation can be achieved in light of the failure risk, failure and / or failure rate of ADAS or ADS, and / or intervention. Additional difference exceedance conditions may be defined by any one or more arbitrary criteria considered relevant to the current implementation (e.g., based on characteristics and / or requirements related to sensing system 22 and / or ADAS or ADS 21, such as maximum difference level, maximum number of exceedances of the difference level, maximum number of exceedances within a predetermined time frame). Moreover, additional difference exceedance conditions may be variable and may also be predetermined.

[0061] "Transmitting additional confirmation data" can refer to "transmitting" additional confirmation data "wirelessly and / or via wire" and / or "transmitting" additional confirmation data "in a timely manner and / or when deemed feasible and / or safe," while "additional confirmation data" can refer to "at least second confirmation data," "one or more additional confirmation signals," and / or "additional confirmation messages." Furthermore, "additional confirmation data" can refer to "additional confirmation data that reflects and / or includes at least the difference output and / or an indication that the vehicle can be sensed in the secondary sensing data but cannot be located in the sensing data." On the other hand, the phrase "when the difference output satisfies an additional difference exceeding condition" can refer to "if the difference output satisfies an additional difference exceeding condition" and / or "if the difference output satisfies an additional difference exceeding condition." Moreover, "an additional difference exceeding condition" can refer to "a predetermined and / or pre-defined additional difference exceeding condition," and can also refer to "a second and / or at least a second difference exceeding condition." On the other hand, the phrase “the difference exceeds the condition by deviating from the additional difference exceeds the condition” can mean “the difference exceeds the condition at least partially and / or to some extent deviates from the additional difference exceeds the condition”.

[0062] The sensing comparison system 1 can, for example, be equipped with an acknowledgment transmission unit 105 ( Figure 4 (As shown), it can optionally be adapted and / or configured to wirelessly transmit additional confirmation data 60 to remote entity 9. The additional confirmation data 60 may, for example, include a difference output. Therefore, when an additional difference exceeds a condition, data related to, for example, the difference output, can be sent to entity 9 for offline processing and / or analysis. Additionally or alternatively, the additional confirmation data 60 may include an indication in secondary sensing data 5 that the vehicle 2 can be sensed, but the secondary vehicle 3 cannot be located in the vehicle sensing data 4. The timing of the transmission can be arbitrarily chosen in a manner deemed feasible (e.g., promptly or at a deemed appropriate time). The latter case may, for example, involve waiting for a high-bandwidth connection to remote entity 9, such as a Wi-Fi connection.

[0063] Alternatively, the perception comparison system 1 may, for example by means of the confirmation transmission unit 105, optionally adapt and / or configure itself to send additional confirmation data 60 to the ADAS / ADS control system 24, the additional confirmation data 60 including an indication to at least partially disable ADAS or ADS 21. Thus, when additional discrepancies exceed the condition, ADAS or ADS 21 can be at least partially revoked to at least temporarily prevent further activation.

[0064] Figure 4 This is a schematic block diagram illustrating an exemplary perceptual comparison system 1 according to an embodiment of the present disclosure, such as... Figure 4As further shown, the perception comparison system 1 includes a communication establishment unit 101, a perception data export unit 102, a secondary perception data receiving unit 103, a difference output determination unit 104, and a confirmation transmission unit 105, all of which have been described in more detail above. Furthermore, the embodiments described herein for evaluating the perception performance of the ADAS or ADS 21 of vehicle 2 can be implemented by one or more processors (e.g., processor 106, referred to herein as a CPU) and computer program code for performing the functions and actions of the embodiments herein. The program code can also be provided as a computer program product, for example, in the form of a data carrier carrying computer program code for executing the embodiments herein when loaded into the perception comparison system 1. One such carrier can be in the form of a CD-ROM and / or a hard disk drive, but other data carriers are also feasible. Additionally, the computer program code can be provided as pure program code on a server and downloaded to the perception comparison system 1. The perception comparison system 1 may also include a memory 107 comprising one or more storage units. Memory 107 may be arranged to store, for example, information, and further, data, configuration, scheduling, and applications to execute the methods described herein when executed in the perception comparison system 1. For example, computer program code may be implemented in firmware, stored in memory 107 of embedded processor 106, and / or wirelessly downloaded, for example, from an off-site server. Furthermore, communication establishment unit 101, perception data export unit 102, secondary perception data receiving unit 103, difference output determination unit 104, confirmation transmission unit 105, optional processor 106, and / or optional memory 107 may be at least partially included in one or more nodes 108 of, for example, an ECU of vehicle 2. Those skilled in the art will also understand that the aforementioned units 101, 102, 103, 104, and 105 may refer to a combination of analog and digital circuitry and / or one or more processors configured with software and / or firmware, such as memory (e.g., memory 107), which, when executed by one or more processors (e.g., processor 106), are as described herein (e.g., in combination with...). Figure 5 The processors can be executed in a manner that allows for the integration of one or more of these processors, as well as other digital hardware, into a single application-specific integrated circuit (ASIC) or several processors. The various digital hardware components can be distributed among several separate parts, either individually packaged or assembled into a system-on-a-chip (SoC).

[0065] Figure 4Further shown are ADAS or ADS 21, perception system 22, optional one or more environmental detection sensors 23, optional ADAS / ADS control system 24, secondary vehicle 3, secondary perception data 5, confirmation data 6, optional additional confirmation data 60, and optional remote entity 9, all of which have been discussed in detail.

[0066] Figure 5 This is a flowchart illustrating an exemplary method performed by a perception comparison system 1 according to an embodiment of the present disclosure. The method is used to evaluate the perception performance of an ADAS or ADS 21 of a vehicle 2. Exemplary methods that can be repeated continuously include... Figures 1-4 With the support of [the relevant authority], one or more of the following actions are discussed. Furthermore, where applicable, actions may be taken in any suitable order and / or one or more actions may be performed simultaneously and / or in an alternating order. For example, actions 1002 and 1003 may be performed in an alternating order and / or simultaneously.

[0067] Action 1001

[0068] In action 1001, the performance comparison system 1, for example with the support of the communication establishment unit 101, establishes communication with the secondary vehicle 3, which is identified and / or estimated to be within the potential range of the environmental detection sensor 23 mounted on the vehicle 2.

[0069] Action 1002

[0070] In action 1002, the performance comparison system 1, for example with the support of the perception data export unit 102, exports perception data 4 from the perception system 22 of ADAS or ADS 21, which is adapted to estimate the environment of vehicle 2.

[0071] Action 1003

[0072] In action 1003, performance comparison system 1, for example with the support of secondary perception data receiving unit 103, receives secondary perception data 5 from secondary perception system 32 of secondary ADAS or ADS 31 of secondary vehicle 3.

[0073] Action 1004

[0074] In action 1004, the performance comparison system 1, for example with the support of the difference output determination unit 104, determines the difference output based on a comparison of at least a portion of the perception data 4 with at least a portion of the secondary perception data 5 when the secondary vehicle 3 can be located in the perception data 4.

[0075] Optionally, the action 1004 of determining the difference output may include determining at least a first portion of the difference output based on a comparison of perception data 4 of object state estimation 41 including vehicle 2 and secondary perception data 5 of secondary object state estimation 51 including vehicle 2.

[0076] Additionally or alternatively, optionally, the action 1004 of determining the difference output may include determining at least a first portion of the difference output based on a comparison of perception data 4 including object state estimation 42 of secondary vehicle 3 and secondary perception data 5 including secondary object state estimation 52 of secondary vehicle 3.

[0077] Further optionally, when the positioning estimates 41, 42 of vehicle 2 and secondary vehicle 3 in perception data 4 are consistent with the positioning estimates 51, 52 of vehicle 2 and secondary vehicle 3 in secondary perception data 5, respectively, the action 1004 of determining the difference output further includes at least a second portion of the difference output. The second portion of the difference output may then be based on a comparison of perception data 4 including an object state estimate 43 of at least a first environmental object with secondary perception data 5 including a secondary object state estimate 53 of said at least the first environmental object. Additionally or alternatively, the second portion of the difference output may then be based on a comparison of perception data 4 including a free space estimate 44 of at least a region 45 around vehicle 2 with secondary perception data 5 including a secondary free space estimate 54 of at least a secondary region 55 around secondary vehicle 3, wherein region 45 at least partially overlaps with secondary region 55. Additionally or alternatively, the second portion of the difference output may then be based on a comparison of perception data 4 of a drivable area estimate 46 including at least a region 47 around vehicle 2 and secondary perception data 5 of a secondary drivable area estimate 56 including at least a secondary region 57 around secondary vehicle 3, wherein region 47 at least partially overlaps with secondary region 57 4757.

[0078] Action 1005

[0079] In action 1005, when at least a portion of the difference output satisfies the difference exceeding condition and / or when vehicle 2 can be sensed in secondary perception data 5 but secondary vehicle 3 cannot be located in perception data 4, performance comparison system 1 transmits confirmation data 6, for example, with the support of confirmation transmission unit 105.

[0080] Optionally, the action 1005 of transmitting confirmation data 6 may include wirelessly sending confirmation data 6 to remote entity 9, wherein confirmation data 6 includes, for example, a difference output.

[0081] Additionally or alternatively, optionally, the action 1005 of transmitting the confirmation data 6 may include sending the confirmation data 6 to an onboard ADAS / ADS control system 24 adapted to control ADAS or ADS 21, the confirmation data 6 including an indication to at least partially disable ADAS or ADS 21.

[0082] Action 1006

[0083] In optional action 1006, when the difference output meets another difference exceeding condition, the performance comparison system 1 may, for example with the support of the confirmation transmission unit 105, transmit additional confirmation data 60, wherein the difference exceeding condition deviates from another difference exceeding condition.

[0084] Optionally, the action 1006 of transmitting additional confirmation data 60 may include wirelessly sending additional confirmation data 60 to remote entity 9, wherein the additional confirmation data 60 includes, for example, a difference output.

[0085] Additionally or alternatively, optionally, the action 1006 of transmitting additional confirmation data 60 may include sending additional confirmation data 60 to the ADAS / ADS control system 24, wherein the additional confirmation data 60 includes an indication to at least partially disable ADAS or ADS.

[0086] Those skilled in the art will recognize that this disclosure is by no means limited to the preferred embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims. Furthermore, it should be noted that the drawings are not necessarily drawn to scale, and the dimensions of certain features may be enlarged for clarity. Instead, the focus is on illustrating the principles of the embodiments described herein. Additionally, in the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality.

Claims

1. A method for evaluating the perception performance of a perception system (22) of an advanced driver assistance system (ADAS) or an automated driving system (ADS) (21) of a vehicle (2), performed by a perception comparison system (1) of the vehicle (2), the method comprising: Establish (1001) communication with the secondary vehicle (3), which is determined and / or estimated to be within the potential range of the environmental detection sensor (23) carried by the vehicle (2); Perception data (4) is derived (1002) from the perception system (22), which is adapted to estimate the environment of the vehicle (2); (1003) Secondary perception data (5) is received from the secondary perception system (32) of the secondary ADAS or ADS (31) of the secondary vehicle (3), the secondary perception system (32) being adapted to estimate the environment of the secondary vehicle (3). When the secondary vehicle (3) can be located in the sensing data (4), a difference output (1004) is determined based on a comparison of at least a portion of the sensing data (4) with at least a portion of the secondary sensing data (5), the difference output representing a sensing error, failure, and / or failure risk for the sensing system (22) and / or the potential secondary sensing system (32); and Transmit (1005) confirmation data (6) in the following situations: When at least a portion of the difference output satisfies the difference exceeding condition; or When the vehicle (2) can be sensed in the secondary sensing data (5) but the secondary vehicle (3) cannot be located in the sensing data (4), The confirmation data (6) represents, reflects, and / or includes at least the difference output and / or an indication that the vehicle (2) can be perceived in the secondary perception data (5) but the secondary vehicle (3) cannot be located in the perception data (4). The determination (1004) includes determining whether the positioning estimates (41, 42) of the vehicle (2) and the secondary vehicle (3) in the perception data (4) are consistent with the positioning estimates (51, 52) of the vehicle (2) and the secondary vehicle (3) in the secondary perception data (5), respectively. When the positioning estimates (41, 42) of the vehicle (2) and the secondary vehicle (3) in the perception data (4) are consistent with the positioning estimates (51, 52) of the vehicle (2) and the secondary vehicle (3) in the secondary perception data (5), respectively, the determination (1004) further includes determining at least a second portion of the difference output based on the following: A comparison of perception data (4) of free space estimation (44) including at least a first region (45) around the vehicle (2) with secondary perception data (5) of secondary free space estimation (54) including at least a first primary region (55) around the secondary vehicle (3), wherein the first region (45) at least partially overlaps (4555) with the first primary region (55); and / or A comparison of perception data (4) of drivable area estimation (46) in at least a second region (47) around the vehicle (2) with secondary perception data (5) of secondary drivable area estimation (56) in at least a second secondary region (57) around the secondary vehicle (3), wherein the second region (47) at least partially overlaps (4757) with the second secondary region (57). The transmission (1005) confirmation data (6) includes: The confirmation data (6) is sent to an on-board ADAS / ADS control system (24) adapted to control the ADAS or ADS (21), the confirmation data (6) including an indication to at least partially disable the ADAS or ADS (21).

2. The method according to claim 1, wherein, The determination (1004) includes determining at least a first portion of the difference output based on the following: Comparison of perception data (4) including object state estimation (41) of the vehicle (2) with secondary perception data (5) including secondary object state estimation (51) of the vehicle (2); and / or A comparison of perception data (4) including object state estimation (42) of the secondary vehicle (3) and secondary perception data (5) including secondary object state estimation (52) of the secondary vehicle (3).

3. The method according to claim 2, wherein, The determination of at least the second part of the difference output is also based on A comparison of perception data (4) including object state estimates (43) of at least the first environmental object and secondary perception data (5) including secondary object state estimates (53) of the at least the first environmental object.

4. The method according to any one of claims 1-3, further comprising: When the difference output satisfies another difference exceeding condition, transmit (1006) additional confirmation data (60), the difference exceeding condition deviating from the other difference exceeding condition.

5. The method according to claim 4, wherein, The additional confirmation data (60) of the transmission (1006) includes: The additional confirmation data (60) is wirelessly transmitted to the remote entity (9), the additional confirmation data (60) including the difference output, or The additional confirmation data (60) is sent to the ADAS / ADS control system (24), the additional confirmation data (60) including an indication to at least partially disable the ADAS or ADS (21).

6. A perception comparison system (1) for a vehicle (2) for evaluating the perception performance of a perception system (22) of an advanced driver assistance system (ADAS) or an automated driving system (ADS) (21) of the vehicle (2), the perception comparison system (1) comprising: A communication establishment unit (101) is used to establish (1001) communication with a secondary vehicle (3), which is determined and / or estimated to be within the potential range of an environmental detection sensor (23) mounted on the vehicle (2); A perception data export unit (102) is used to export (1002) perception data (4) from the perception system (22), which is adapted to estimate the environment of the vehicle (2); Secondary perception data receiving unit (1003) is used to receive (1003) secondary perception data (5) from the secondary perception system (32) of the secondary ADAS or ADS (31) of the secondary vehicle (3), the secondary perception system (32) being adapted to estimate the environment of the secondary vehicle (3). A difference output determination unit (1004) is used to determine (1004) a difference output based on a comparison of at least a portion of the perception data (4) with at least a portion of the secondary perception data (5) when the secondary vehicle (3) can be located in the perception data (4), the difference output representing a perception error, failure and / or failure risk for the perception system (22) and / or the potential secondary perception system (32); and The acknowledgment transmission unit (105) is adapted to transmit (1005) acknowledgment data (6) in the following situations: When at least a portion of the difference output satisfies the difference exceeding condition; or When the vehicle (2) can be sensed in the secondary sensing data (5) but the secondary vehicle (3) cannot be located in the sensing data (4), The confirmation data (6) represents, reflects, and / or includes at least the difference output and / or an indication that the vehicle (2) can be perceived in the secondary perception data (5) but the secondary vehicle (3) cannot be located in the perception data (4). Wherein, the difference output determination unit (1004) is adapted to determine whether the positioning estimates (41, 42) of the vehicle (2) and the secondary vehicle (3) in the perception data (4) are consistent with the estimates (51, 52) of the vehicle (2) and the secondary vehicle (3) in the secondary perception data (5), and when the positioning estimates (41, 42) of the vehicle (2) and the secondary vehicle (3) in the perception data (4) are consistent with the estimates (51, 52) of the vehicle (2) and the secondary vehicle (3) in the secondary perception data (5), the difference output determination unit (104) is adapted to further determine at least a second part of the difference output based on the following: A comparison of perception data (4) of free space estimation (44) including at least a first region (45) around the vehicle (2) with secondary perception data (5) of secondary free space estimation (54) including at least a first primary region (55) around the secondary vehicle (3), wherein the first region (45) at least partially overlaps (4555) with the first primary region (55); and / or A comparison of perception data (4) of drivable area estimation (46) in at least a second region (47) around the vehicle (2) with secondary perception data (5) of secondary drivable area estimation (56) in at least a second secondary region (57) around the secondary vehicle (3), wherein the second region (47) at least partially overlaps (4757) with the second secondary region (57). The confirmation transmission unit (105) is further adapted to: The confirmation data (6) is sent to an on-board ADAS / ADS control system (24) adapted to control the ADAS or ADS (21), the confirmation data (6) including an indication to at least partially disable the ADAS or ADS (21).

7. The perception comparison system (1) according to claim 6, wherein, The difference output determination unit (104) is adapted to determine at least a first portion of the difference output based on the following: Comparison of perception data (4) including object state estimation (41) of the vehicle (2) with secondary perception data (5) including secondary object state estimation (51) of the vehicle (2); and / or A comparison of perception data (4) including object state estimation (42) of the secondary vehicle (3) and secondary perception data (5) including secondary object state estimation (52) of the secondary vehicle (3).

8. The perception comparison system (1) according to claim 7, wherein, The difference output determination unit (104) determines at least the second part of the difference output based on a comparison of perception data (4) including object state estimates (43) of at least the first environmental object and secondary perception data (5) including secondary object state estimates (53) of the at least the first environmental object.

9. The perceptual comparison system (1) according to any one of claims 6-8, wherein, The confirmation transmission unit (105) is also adapted to: When the difference output satisfies another difference exceeding condition, transmit (1006) additional confirmation data (60), the difference exceeding condition deviating from the other difference exceeding condition.

10. The sensing comparison system (1) according to claim 9, wherein, The confirmation transmission unit (105) is adapted to: The additional confirmation data (60) is wirelessly transmitted to the remote entity (9), the additional confirmation data (60) including the difference output, or The additional confirmation data (60) is sent to the ADAS / ADS control system (24), the additional confirmation data (60) including an indication to at least partially disable the ADAS or ADS (21).

11. A vehicle (2) including ADAS or ADS (21) and including a perception comparison system (1) according to any one of claims 6-10.

12. A computer program product comprising a computer program containing computer program code means arranged to cause a computer or processor to perform the steps of the method according to any one of claims 1-5, the computer program being stored on a computer-readable medium.

13. A non-volatile computer-readable storage medium having a computer program product according to claim 12 stored thereon.

Images