Systems, devices, and methods for identifying and updating a design applicability domain for an autonomous vehicle

Abstract

A system, method, and processor-readable medium for identifying the operational design domain (ODD) of an autonomous driving system (ADS). On one hand, a geographic dataset is generated using a proposed map. The performance of the ADS under a range of environmental conditions is evaluated based on the geographic dataset, thereby identifying a finite-risk portion of a proposed condition space defined by the proposed map and the range of environmental conditions. Based on the finite-risk portion of the proposed condition space, the design design domain is identified. The ODD can be updated when additional data is received. The driver can be alerted when a vehicle using the ADS may exit the ODD.

Description

[0001] Cross-referencing related applications

[0002] This application claims the benefit of U.S. Patent Application No. 16 / 749,602, filed January 22, 2020, entitled “System, Device and Method of Identifying and Updating the Operational Design Domain of an Automated Vehicle,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This invention relates to autonomous vehicles, and more particularly to a system, apparatus, and method for identifying and updating the Operational Design Domain (ODD) of an autonomous vehicle. Background Technology

[0004] An autonomous driving system (ADS) is used to drive a vehicle automatically or semi-autonomously. The domain in which a given ADS is to operate normally is called the operational design domain (ODD). The ODD is typically defined by geographic boundaries or a set of roads, and may also include additional conditions or constraints applicable to the operation of the ADS. The ODD can identify the type of road on which the ADS is to operate safely (highway, local road, etc.); the geographic type in which the ADS is to operate safely (urban, hilly, mountainous, desert, etc.); the safe operating speed range; and / or the environmental conditions under which the ADS is to operate (precipitation, road conditions, temperature, lighting conditions, etc.).

[0005] Typically, before deploying an Adaptive Controller (ADS) on public roads for testing, an ODD (Original Design Domain) is defined, including map boundaries and environmental conditions. A typical method for identifying the ODD is to geofence a geographic area based on business or strategic considerations. The geofenced area is then tested: vehicles equipped with the ADS are deployed to the geofenced area for road testing and validation, using the ADS while driving the vehicle, typically with a safety driver present to intervene in case of ADS failure. Once the ADS has been tested or validated, the geofenced area is considered part of the ODD.

[0006] These existing methods rely heavily on intervention by a safe driver when assessing ADS. This is problematic for at least four reasons.

[0007] First, this requires safety drivers to remain highly vigilant and prepared to intervene at any time, without being informed beforehand which conditions (e.g., which specific roads or locations, or which environmental conditions) increase or decrease the risk of ADS failure. Safety drivers may not realize ADS failure will occur before an accident happens, and may not be able to react quickly enough to avoid it. For example, it may take a safety driver some time to realize that ADS has failed to correctly identify an object before they can react to the failure, resulting in a longer overall reaction time for the vehicle.

[0008] Second, it is unfair to require safety drivers to remain fully alert while monitoring ADS, and they are likely to become exhausted and lose focus over time.

[0009] Third, when the functionality of ADS changes over time, such as through software or hardware updates, the safety driver's understanding of ADS functionality becomes unreliable.

[0010] Fourth, once an ADS is tested in a geofenced area, that area is typically referred to as an ODD, regardless of how much safety driver intervention is required during the test, because there is no systematic way to distinguish and define the safe and unsafe parts of a planned ODD.

[0011] Figure 1 The flowchart shown illustrates examples of these existing techniques for generating an ODD. This known method 10 first selects a geographic area 12 based on a predicted business strategy. In step 14, a high-resolution map of the area is created. Then, in step 16, an initial basic test of the ADS is run, typically in a simulation or on a closed route that does not represent the actual area. In step 18, a safety driver is informed about the capabilities of the ADS. In step 20, the ADS is deployed in the area for road testing, accompanied by a safety driver. The safety driver is responsible for intervening for safety reasons. When the road test is considered complete, in step 22, the entire area is referred to as the ODD, regardless of how much safety driver intervention was required during the test.

[0012] Many examples of these current technologies are well-known. General Motors provided a safety report ( https: / / www.gm.com / content / dam / company / docs / us / en / gmcom / gmsafetyreport.pdf The safety report (which is incorporated herein by reference) indicates that public roads were used as testing grounds for arbitrarily selected ODDs. The safety report explicitly states that high-risk areas of the ODD could only be identified through test drives on public roads. Similarly, Ford issued a safety report ( https: / / media.ford.com / content / dam / fordmedia / pdf / Ford_AV_LLC_FINAL_HR_2.pdfThis safety report (incorporated in this application by reference) indicates that a safety driver (referred to as the driver) needs to understand the system's functionality, and that the source of the ODD is merely expectation and prediction, not reality. Furthermore, a safety driver needs to continuously understand changes to the ADS functionality and remember the current functionality while driving the vehicle. Waymo has also released a safety report ( https: / / storage.googleapis.com / sdc-prod / v1 / safety-report / waymo- safety-report-2017.pdf (This safety report is incorporated herein by reference) and includes similar statements regarding the need to provide safety drivers with education and updates on ADS functionality. Summary of the Invention

[0013] This invention provides a system, method, and processor-readable medium for identifying the operational design domain (ODD) of an autonomous driving system (ADS). Compared to prior art defining the aforementioned ODD, this invention offers one or more advantages.

[0014] First, in this invention, the ODD can be defined based on the functionality of the ADS relative to an objective, systematic risk measure, thereby capturing the full functionality of the ADS and aligning the ODD with business objectives, rather than simply defining the ODD a priori based on a predictive business model.

[0015] Second, when the ADS operates outside of limited risk parameters and is therefore more likely to require intervention, the ODD can enable the ADS to notify the safety driver, rather than relying on whether the safety driver can remember the complex and potentially changing functions of the ADS, as described in the above techniques.

[0016] Third, unlike existing technologies that can deploy autonomous vehicles for road testing in areas where the risk exceeds any predetermined risk parameters, the present invention can avoid premature unsafe road testing, which is similar to blind testing in terms of the degree of danger.

[0017] Fourth, unlike existing technologies, the present invention can provide a formal way to update the ODD when collecting further data and comparing it with the performance of the ADS and the risk tolerance of the entity managing the project.

[0018] Based on these potential advantages, the present invention allows for the reliable identification of the ODD prior to the deployment of the ADS for road testing. By demonstrating that the ODD poses a limited risk before road testing begins, the present invention avoids the prior art method of initially overestimating confidence and then relying on intervention by the safe driver when that overestimation manifests as a potential accident. The present invention enables the ADS to operate within the defined ODD, thereby ensuring that the expected value of any losses incurred is limited to a predetermined threshold (e.g., a dollar value threshold).

[0019] According to a first aspect of the present invention, a method is provided for identifying the design suitability domain for the operation of an autonomous driving system (ADS). The method includes: receiving proposed condition space data, which includes data representing a proposed map; generating a geographic dataset using the proposed condition space data; evaluating the performance of the ADS using the geographic dataset; identifying a finite risk portion of the proposed condition space based on the ADS performance; and identifying the design suitability domain based on the finite risk portion of the proposed condition space.

[0020] According to a second aspect of the invention, a system is provided for identifying the operational design domain (ODD) of an autonomous driving system (ADS) for operation.

[0021] According to an embodiment of the second aspect of the present invention, the system includes a processor system and a memory coupled to the processor system. The memory tangibly stores executable instructions thereon, which, when executed by the processor system, cause the system to perform the following operations: receiving proposed conditional spatial data, including data representing a proposed map; generating a geographic dataset using the proposed conditional spatial data; evaluating the performance of the ADS using the geographic dataset; identifying a limited risk portion of the proposed conditional space based on the ADS performance; and identifying the design applicability domain based on the limited risk portion of the proposed conditional space.

[0022] According to some embodiments of the first or second aspect of the present invention, the suggested condition space further includes representing a series of suggested environmental conditions; evaluating the ADS includes using the geographic dataset to evaluate the performance of the autonomous driving system under the series of suggested environmental conditions; the limited risk portion of the suggested condition space includes a combination of a location in the suggested map and an environmental condition from the series of suggested environmental conditions where a limited risk exists. The use of environmental conditions improves the robustness of the system or method in assessing risk.

[0023] According to some embodiments of the first or second aspect of the present invention, generating the geographic dataset using the proposed conditional spatial data includes receiving a plurality of map features of the proposed map. Using map features improves the robustness of the system or method in assessing the risks presented by specific features that a vehicle may encounter.

[0024] According to some embodiments of the first or second aspect of the present invention, the map features of the proposed map include: multiple nodes corresponding to locations on the proposed map; multiple road segments, each road segment corresponding to a path between two of the nodes; multiple routes, each route including one or more of the road segments; and multiple object types, each object type having one or more encounter probabilities, each encounter probability being associated with one of the road segments. Using the encounter probabilities of specific object types improves the robustness of the system in assessing risk based on the prevalence of each object type in a given road segment within the proposed map area.

[0025] According to some embodiments of the first or second aspect of the present invention, the limited risk includes a maximum risk metric value below a risk threshold, the maximum risk metric value being calculated as the highest route risk metric value among multiple route risk metrics corresponding to multiple routes with limited risk within the limited risk portion of the proposed map. Using a maximum risk metric value below a risk threshold ensures that the maximum risk of any portion of a route within the limited risk portion of the proposed map will be below a risk threshold defined by the project risk tolerance.

[0026] According to some embodiments of the first or second aspect of the present invention, each route risk metric is calculated by summing the expected risk for each of a plurality of object types present on the corresponding route. The expected risk for each object type is calculated as a product of the following: a severity value, indicating the expected severity of the ADS's failure to respond appropriately to the object type on the route; an exposure value, indicating the prevalence of the object type on the route; and a failure probability value, indicating the likelihood that the ADS will fail to respond appropriately to the object type on the route. The failure probability value is based on an assessment of the ADS's performance using the geographic dataset under the proposed environmental conditions. The use of the severity value, exposure value, and failure probability value produces a robust risk metric corresponding to the overall failure probability of the ADS.

[0027] According to some embodiments of the first aspect of the invention, the system further includes a vehicle, the vehicle including the ADS and for being driven by a driver, wherein, when executed by the processor system, the instructions further cause the system to perform the following operations: determine that the vehicle may exit the design domain; and alert the driver in response to determining that the vehicle may exit the design domain. As described above, being able to alert the vehicle driver to exit the ODD allows the driver to focus on the unlimited risk portion of vehicle operation.

[0028] According to some embodiments of the second aspect of the present invention, the method further includes: determining that a vehicle using the ADS for autonomous driving may exit the design application domain; and alerting the driver of the vehicle in response to determining that the vehicle may exit the design application domain. As described above, being able to alert the vehicle driver to exit the ODD allows the driver to focus on the unlimited risk portion of vehicle operation.

[0029] According to some embodiments of the first aspect of the invention, when executed by the processor system, the instructions further cause the system to perform the following operations after identifying the design application domain: receiving additional data; re-evaluating the performance of the ADS using the geographic dataset and the additional data, thereby updating the limited risk portion of the proposed condition space; and updating the design application domain based on the updated limited risk portion of the proposed condition space. As described above, this enables the ODD to be updated upon receiving updated data to keep the ODD accurate and consistent with the latest information.

[0030] According to some embodiments of the second aspect of the invention, the method further includes, after identifying the design application domain: receiving additional data; re-evaluating the performance of the ADS using the geographic dataset and the additional data, thereby updating the limited risk portion of the proposed condition space; and updating the design application domain based on the updated limited risk portion of the proposed condition space. As described above, this enables the ODD to be updated upon receiving updated data to keep the ODD accurate and consistent with the latest information.

[0031] According to some embodiments of the first or second aspect of the present invention, the additional data is selected from the group consisting of: updated suggested map data, updated geographic dataset data, updated expected risk data, updated risk threshold data, and updated ADS performance data. All of the above data types may be related to updating the ODD.

[0032] According to another aspect of the invention, a non-transitory processor-readable medium is provided, on which executable instructions are tangibly stored, which, when executed by a processor, cause the processor to perform the method described in one embodiment of the second aspect of the invention. Attached Figure Description

[0033] Figure 1 A flowchart is shown for a known method for identifying the ODD of an ADS;

[0034] Figure 2 A schematic diagram of an autonomous vehicle in an environment including a communication system, provided by an exemplary embodiment of the present invention, is shown.

[0035] Figure 3A An exemplary embodiment of the present invention is shown. Figure 2 A block diagram of the vehicle shown;

[0036] Figure 3B A block diagram of an ODD recognition autonomous driving system provided by an exemplary embodiment of the present invention is shown;

[0037] Figure 4A A system diagram illustrating the outline operation of a system for identifying an ODD of an ADS, provided by an exemplary embodiment of the present invention, is shown.

[0038] Figure 4B A system diagram illustrating the outline operation of a system for identifying an ODD that includes an ADS containing environmental conditions, provided by an exemplary embodiment of the present invention, is shown.

[0039] Figure 5 A flowchart illustrating the outline operation of a method for identifying an ODD that includes environmental conditions, provided by an exemplary embodiment of the present invention, is shown.

[0040] Figure 6 A flowchart illustrating the outline operation of a first method for identifying the ODD of an ADS by comparing each of the location, perception, and planning risks with its own risk threshold, provided by an exemplary embodiment of the present invention, is shown.

[0041] Figure 7 A flowchart illustrating a summary operation of a second method for identifying the ODD of an ADS by comparing the total of combined location, perception, and planning risks with a risk threshold, provided by an exemplary embodiment of the present invention, is shown.

[0042] Figure 8 A flowchart illustrating a summary operation of a third method for identifying ODDs of ADS by comparing total risk with a risk threshold, provided by an exemplary embodiment of the present invention, is shown.

[0043] Figure 9 A flowchart illustrating a summary operation of a method for generating and / or enhancing conditional data for evaluating ADS performance, provided by an exemplary embodiment of the present invention, is shown. Detailed Implementation

[0044] This invention is carried out with reference to the accompanying drawings, in which embodiments are illustrated. However, many different embodiments may be used, and therefore the description should not be construed as limiting oneself to the embodiments set forth herein. Rather, these embodiments are provided to make the invention thorough and complete. Where possible, the same reference numerals are used to denote the same elements in the drawings and detailed descriptions, and in alternative embodiments, apostrophes are used to denote similar elements, operations, or steps. The separate blocks or separations of functional elements of the illustrated systems, modules, and devices do not necessarily require physical separation of these functions, as communication between these elements can occur without any such physical separation via message passing, function calls, shared memory spaces, etc. Therefore, although these functions are described separately herein for ease of explanation, these functions do not need to be implemented in physically or logically separated platforms. Different devices may have different designs such that while some devices implement some functions in fixed-function hardware, others may implement these functions in a programmable processor with code available from a machine-readable medium. Finally, elements expressed in the singular may have a plural meaning, and vice versa, unless the context explicitly or inherently indicates otherwise.

[0045] For convenience, this invention describes exemplary embodiments of methods and systems relating to motor vehicles, such as automobiles, trucks, buses, small boats or ships, submarines, aircraft, warehouse equipment, construction equipment, tractors, or other farm equipment. The teachings of this invention are not limited to any particular type of vehicle and can be applied to vehicles that do not carry passengers as well as vehicles that do carry passengers. The ideas of this invention can also be implemented in mobile robotic vehicles, including but not limited to autonomous vacuum cleaners, detectors, lawnmowers, unmanned aerial vehicles (UAVs), and other objects.

[0046] Figure 2 A schematic diagram of an environment 100 in which a vehicle 105 is driven is shown. The environment includes a communication system 100 that communicates with the vehicle 105. The vehicle 105 includes a vehicle control system 115. As described below, Figure 3A The vehicle control system 115, shown in more detail, is coupled to the drive control system 150 and the electromechanical system 190 of the vehicle 105. In various embodiments, the vehicle control system 115 may allow the vehicle 105 to operate in one or more modes of fully autonomous, semi-autonomous, or fully user-controlled operation.

[0047] The vehicle 105 may include sensors, shown herein as a plurality of environmental sensors 110 (hereinafter referred to as environmental sensors 110) for collecting data about the external environment 100 surrounding the vehicle 105, and a plurality of sensors 111 (hereinafter referred to as vehicle sensors 111) for collecting data about the operating status of the vehicle 105. For example, the environmental sensors 110 may include one or more camera units 112, one or more LiDAR (light detection and ranging) units 114, and one or more radar units, such as synthetic aperture radar (SAR) units 116. As described below, the camera (unit) 112, LiDAR unit 114, and SAR unit 116 are mounted on the vehicle 105 and located around the vehicle 105, and are each coupled to the vehicle control system 115. In one exemplary embodiment, the camera unit 112, LiDAR unit 114, and SAR unit 116 are mounted on the vehicle 105 and located at the front, rear, left, and right sides of the vehicle 105 to collect data about the external environment 100 located at the front, rear, left, and right sides of the vehicle 105. For each type of environmental sensor 110, the respective units are mounted or otherwise positioned to have different fields of view (FOV) or coverage areas to capture data about the environment surrounding the vehicle 105. In some examples, for each type of environmental sensor 110, the FOV or coverage areas of some or all adjacent environmental sensors 110 partially overlap. Accordingly, the vehicle control system 115 receives the data about the external environment of the vehicle 105 collected by the camera unit 112, LiDAR unit 114, and SAR unit 116.

[0048] Vehicle sensors 111 may include an inertial measurement unit (IMU) 118, an electronic compass 119, and other vehicle sensors 120, such as speedometers, tachometers, wheel traction sensors, transmission gear sensors, throttle and brake position sensors, and steering angle sensors. The IMU 118 uses a combination of accelerometers and gyroscopes to sense the specific force and angular velocity of the vehicle 105 and provides the vehicle's orientation based on the sensed specific force and angular velocity. Vehicle sensors 111 repeatedly (e.g., periodically) sense the environment when activated and provide data about the operating status of the vehicle 105 to the vehicle control system 115 in real-time or near real-time. For example, the vehicle control system 115 may use signals received from satellite receiver 132 to collect data about the position of the vehicle 105. The vehicle control system 115 may also receive data about the orientation of the vehicle 105 from the IMU 118. The vehicle control system 115 can use data about the operation of the vehicle 105 provided by one or more of the satellite receiver 132, the IMU 118, and other vehicle sensors 120 to determine factors such as the linear velocity, angular velocity, acceleration, engine speed, transmission gears, and tire grip of the vehicle 105.

[0049] The vehicle control system 115 may further include one or more wireless transceivers 130, which enable the vehicle control system 115 to exchange data and (optionally) voice communications with the wireless wide area network (WAN) 210 of the communication system 100. The vehicle control system 115 can use the wireless WAN 210 to access a server 240, such as a driver assistance server, via one or more communication networks 220 (e.g., the Internet). The server 240 may be implemented as one or more server modules in a data center and is typically located behind a firewall 230. The server 240 is connected to network resources 250, such as supplementary data sources that the vehicle control system 115 can use.

[0050] In addition to the wireless WAN 210, the environment 100 also includes a satellite network 260, which comprises multiple satellites. The vehicle control system 115 includes the satellite receiver 132. Figure 2The satellite receiver 132 can determine its position using signals received from the plurality of satellites in the satellite network 260. The satellite network 260 typically includes multiple satellites that are part of at least one Global Navigation Satellite System (GNSS) that provides autonomous geospatial positioning globally. For example, the satellite network 260 may be a group of GNSS satellites. Exemplary GNSS include the US NAVSTAR Global Positioning System (GPS) or the Russian GLONASS Global Navigation Satellite System. Other satellite navigation systems that have been deployed or are under development include the European Union's Galileo positioning system, China's BeiDou Navigation Satellite System (BDS), India's Regional Satellite Navigation System, and Japan's satellite navigation system.

[0051] Figure 3ASelected components of a vehicle 105 provided by an exemplary embodiment of the present invention are shown. As described above, the vehicle 105 includes a vehicle control system 115 connected to a drive control system 150 and an electromechanical system 190, as well as environmental sensors 110 and vehicle sensors 111. The vehicle 105 also includes various structural elements, such as frames, doors, panels, seats, windows, mirrors, etc., known in the art but omitted in this invention to avoid obscuring the viewpoint of the invention. The vehicle control system 115 includes a processor system 102 coupled to the components via a communication bus (not shown), the communication bus providing communication paths between multiple components and the processor system 102. The processor system 102 is coupled to: a drive control system 150; random access memory (RAM) 122; read-only memory (ROM) 124; permanent (non-volatile) memory 126, such as flash erasable programmable read-only memory (EPROM); one or more wireless transceivers 130 for exchanging radio frequency signals with the wireless network 210; a satellite receiver 132 for receiving satellite signals from the satellite network 260; a real-time clock 134; and a touchscreen 136. The processor system 102 may include one or more processing units, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), and other processing units.

[0052] The one or more wireless transceivers 130 may include one or more cellular (RF) transceivers for communicating with multiple different wireless access networks (e.g., cellular networks) using different wireless data communication protocols and standards. The vehicle control system 115 may communicate with multiple fixed transceiver base stations (one of which is located within its geographical coverage area of ​​the wireless WAN 210 (e.g., cellular network) of the WAN 210) as well. Figure 1 The one or more wireless transceivers 130 can communicate with any of the wireless WAN 210. The one or more wireless transceivers 130 can transmit and receive signals via the wireless WAN 210. The one or more wireless transceivers 130 may include multi-band cellular transceivers supporting multiple radio frequency bands.

[0053] The one or more wireless transceivers 130 may further include a wireless local area network (WLAN) transceiver for communicating with a WLAN (not shown) via a WLAN access point (AP). The WLAN may include components conforming to the IEEE 802.11x standard (sometimes referred to as...). Wi-Fi wireless networks using other communication protocols.

[0054] The one or more wireless transceivers 130 may also include short-range wireless transceivers, such as... A transceiver for communicating with mobile computing devices such as smartphones or tablets. The one or more wireless transceivers 130 may also include other short-range wireless transceivers, including but not limited to Near Field Communication (NFC), IEEE 802.15.3a (also known as Ultra Wideband (UWB)), Z-Wave, ZigBee, ANT / ANT+, or infrared (e.g., Infrared Data Association (IrDA) communications).

[0055] The real-time clock 134 may include a crystal oscillator that provides accurate real-time time data. The time data may be periodically adjusted based on time data received via satellite receiver 132 or based on time data received from network resources 250 that perform a network time protocol.

[0056] The touchscreen 136 includes a display, such as a color liquid crystal display (LCD), a light-emitting diode (LED) display, or an active-matrix organic light-emitting diode (AMOLED) display, having a touch-sensitive input surface or overlay connected to an electronic controller. Additionally, additional input devices (not shown) coupled to the processor system 102 may be provided, including buttons, switches, and display panels.

[0057] The vehicle control system 115 also includes one or more speakers 138, one or more microphones 140, and one or more data ports 142, such as serial data ports (e.g., Universal Serial Bus (USB) data ports). The vehicle control system 115 may also include other sensors 120, such as tire pressure sensors (TPS), door contact switches, light sensors, and proximity sensors.

[0058] The drive control system 150 is used to control the movement of the vehicle 105. The drive control system 150 includes a steering unit 152, a braking unit 154, and a throttle (or acceleration) unit 156, each of which can be implemented as a software module or control block within the drive control system 150. In fully automatic or semi-automatic driving mode, the steering unit 152, braking unit 154, and throttle unit 156 process and receive navigation commands from the autonomous driving system (ADS) 170 (for autonomous / semi-automatic driving mode), generating control signals to control one or more of the steering, braking, and throttle of the vehicle 105. The drive control system 150 may include additional components for controlling other aspects of the vehicle 105, such as turn signals and brake lights.

[0059] The electromechanical system 190 receives control signals from the drive control system 150 to operate the electromechanical components of the vehicle 105. The electromechanical system 190 enables the physical operation of the vehicle 105. The electromechanical system 190 includes an engine 192, a transmission 194, and wheels 196. For example, the engine 192 may be a gasoline engine, a battery-powered engine, or a hybrid engine. Other components may be included in the electromechanical system 190, such as turn signals, brake lights, a fan, and windows.

[0060] The graphical user interface (GUI) of the vehicle control system 115 is presented by the processor system 102 and displayed on the touchscreen 136. Users can interact with the GUI using the touchscreen 136 and optional other input devices (e.g., buttons, dials) to select the driving mode of the vehicle 105 (e.g., fully automated driving mode or semi-automated driving mode) and display relevant data and / or information, such as navigation information, driving information, parking information, media player information, and climate control information. The GUI may include a series of traversable content-specific menus.

[0061] In addition to the GUI, the vehicle control system 115 stores a plurality of software systems 161 on its memory 126, each software system 161 including instructions executable by the processor system 102. The software system 161 includes an operating system 160 and an autonomous driving system (ADS) 170 for fully automatic and / or semi-automatic driving. In some embodiments, the ADS 170 may include separate sub-modules for operating in each of one or more of five recognized automatic or semi-automatic vehicle driving modes: a driver assistance sub-module for operating in driver assistance mode (Level 1); a partial automation sub-module for operating in partial automation mode (Level 2); a conditional automation sub-module for operating in conditional automation mode (Level 3); a high automation sub-module for operating in high automation mode (Level 4); and / or a full automation sub-module for operating in full automation mode (Level 5).

[0062] The autonomous driving system 170 may include one or more software modules 168, including: a computer vision module 172; an ODD recognition module 174 for recognizing ODDs according to the exemplary embodiments described herein; a localization module 177; a perception module 178; a planning module; and other modules 176. The memory 126 also stores instructions for each of the software modules 168 that can be invoked by the autonomous driving system 170. For example, the other modules 176 may include a mapping module, a navigation module, a climate control module, a media player module, a telephone module, and a message processing module. When executed by the processor system 102, the instructions of the ODD recognition module 174 cause the operation of the methods described herein to be performed.

[0063] Although the ODD identification module 174 is shown as a separate module, in some embodiments, one or more of the software modules 168 (including the ODD identification module 174) may be combined with one or more other modules 176.

[0064] The memory 126 also stores various types of data 180. The data 180 may include: data 182 received from the environmental sensor 110; user data 184, including user preferences, settings, and optional personal media files (e.g., music, videos, and orientation); and download cache 186, including data downloaded via the wireless transceiver 130, including data downloaded from network resources 250, etc. The sensor data 182 may include camera data received from the camera 112, LiDAR data received from the LiDAR unit 114, RADAR data from the SAR unit 116, IMU data from the IMU 118, compass data from the electronic compass 119, and other sensor data from other vehicle sensors 120. The camera data represents an image of the environment 100 captured by the camera 112. The LiDAR data represents a point cloud of the environment generated by the LiDAR unit 114. The RADAR data also represents a point cloud of the environment generated by the SAR unit 116. The download cache 186 can be periodically deleted, for example, after a predetermined time. System software, software modules, specific device applications, or portions thereof can be temporarily loaded into volatile memory (e.g., RAM 122) for storing runtime data variables and other types of data and / or information. Data received by the vehicle control system 115 can also be stored in the RAM 122. While specific functions are described for various types of memory, this is only an example, and different types of memory can also be used with different functional allocations.

[0065] Identify the ODD of ADS

[0066] The identification and updating of the operational design domain (ODD) for autonomous driving systems (ADS) will now be described.

[0067] In some embodiments, the ODD identification module 174 may be stored and executed on a system separate from the vehicle control system 115 (e.g., a computer system remote from the vehicle 105). For example, in some embodiments, the ODD identification module 174 resides in memory within the server 240 and is executed by the processor system of the server 240 to identify and / or update ODDs for use by the ADS 170. The ODD definition data 414, 464 generated by the server 240 Figure 4A and Figure 4B ) can be Figure 2The communication network or other data transmission methods shown are transmitted to the vehicle control system 115. In other embodiments, the ODD identification module 174 can be implemented on one or more servers or processors on a distributed computing platform, or it can be a virtual machine set up on a cloud computing platform. Figure 3B An example of a system separate from the vehicle control system 115 (hereinafter referred to as ODD identification system 300) is shown.

[0068] Next reference Figure 3B This figure illustrates a block diagram of an ODD identification system 300 for identifying the operational design domain (ODD) of an autonomous driving system (ADS) according to an exemplary embodiment of the present invention. The ODD identification system 300 is located remotely from the vehicle control system 115 and communicates with the vehicle control system 115 via a communication network (e.g., a wireless communication network), which will be described in further detail below.

[0069] The ODD identification system 300 includes a communication system 330 and a processor system 302 coupled to a memory 326. The memory 326 stores the ODD identification module 174 and data 380. The data 380 includes geographic data 382, ​​environmental data 384, and / or supplementary data 386 relating to various methods of identifying and / or updating ODDs, as described further below. When executing instructions from the ODD identification module 174, the ODD identification system 300 may utilize the data 380 stored in the memory 326 and / or received from other sources (e.g., one or more communication systems 330). The ODD identification system 300 may be located on, or be part of, the vehicle 105, or may communicate with the vehicle control system 115 of the vehicle 105.

[0070] The ODD identification module 174 and the examples of the method described herein can use statistical data and risk tolerance defined in monetary (or some other) values ​​to identify and determine whether a given map and a set of environmental conditions can be considered part of the ODD, even before the ADS 170 is deployed in the vehicle 105. This will be described in further detail below. Figure 4A and Figure 4B As shown, data representing an environment map in which the vehicle 105 will be driven (hereinafter referred to as map data) and data representing a range of environmental conditions (hereinafter referred to as environmental range data) are used as the starting point for the ODD considered as part of the ADS 170. In some embodiments, for example Figure 4AIn the embodiment shown, the map data and the environmental extent data can be combined with each other and / or with other conditional data to form suggested conditional spatial data 402, which will be described in further detail below. Data generator 404 receives the suggested conditional spatial data 402 as input and generates a geographic dataset 406, which includes the map data with constantly changing environmental conditions. The geographic dataset 406 reflects the actual exposure (i.e., the probability of finding objects) of various objects on roads corresponding to the map data. Then, the ADS 170 is tested on the geographic dataset 406 using an ADS evaluator 408, which calculates the total risk for each segment of the suggested ODD under the series of environmental conditions. The total risk is then compared to a preset risk threshold, identifying segments with a risk less than the threshold under a subsequence of the series of environmental conditions, and updating these segments in the ODD, while simultaneously identifying the subsequence of environmental conditions corresponding to each segment. The identified ODD then generates scenarios such that the expected loss value is less than a predetermined threshold in the event of ADS 170 failure.

[0071] Figure 4A A schematic system diagram of a system 400 for identifying the operational design domain (ODD) of an ADS (e.g., ADS 170) according to an exemplary embodiment of the present invention is shown. It should be understood that the ODD identification module 174 includes the system 400. In some embodiments, the system 400 is a software system including computer-readable instructions stored in the memory 126 and executed by the processor system 102 of the vehicle control system 115 for identifying the ODD of the ADS 170. In other embodiments, as referenced above… Figure 3B The ODD identification system 300 shown is a computer system located spatially and / or temporally distant from the vehicle 105 to be driven using the ADS 170. The ODD identification system 300 includes an ODD identification module 174, which includes the system 400 for identifying the ODD. The ODD of the ADS 170 can be included asynchronously with the operation of the ADS 170, and the identified or updated ODD definition is installed in the ADS 170 before the vehicle 105 is deployed. This can be achieved through... Figure 2 The installation is performed using the communication network shown, or it can be performed using any other data transmission technology, such as manually uploading the ODD definition data via the data port 142 using a physical data storage medium.

[0072] In some embodiments, instructions for performing the method are tangibly stored in a non-transitory processor-readable medium, as described further below. When executed by a processor, these instructions cause the processor to perform the method described herein.

[0073] Figure 4A The system 400 shown begins with the data generator 404, which receives suggestion condition space data 402. The suggestion condition space data 402 is data representing a suggestion condition space. The suggestion condition space is a multidimensional space defined by a series or set of suggestion conditions in which the ADS 170 can operate. In some embodiments, the suggestion condition space includes geographic conditions such as location (e.g., a specific road, road segment, or map boundary), terrain type, or road maintenance conditions. The location may include a suggested map, which may include initial map boundaries. In some other embodiments, the suggestion condition space may also include map features as described below.

[0074] The suggested condition space may also include other suggested conditions under which the ADS 170 can operate, including environmental conditions, vehicle status conditions, and driver conditions. Examples of environmental conditions include lighting conditions, weather conditions, and time of day conditions. Examples of vehicle status conditions include the operational or performance status of various sensors and other vehicle systems. Examples of driver conditions include the driver's identity, mental state, or physical condition. Therefore, the suggested condition space defines the outer boundary of the ODD identified by the system 400. The output of the system 400 is data representing the ODD definition (hereinafter referred to as ODD 414 / 464). The ODD definition is a condition subspace that includes a portion of the suggested condition space. Therefore, for example, when the suggested condition space includes location and environmental conditions containing multiple road segments (e.g., block 1200 of street X + highway 10 between highway exit 5 and highway exit 6) and multiple weather conditions (sunny + cloudy + light rain + heavy rain), the generated ODD 414 / 464 identified by the system 400 may include a subset of (road segment x weather conditions): for example, the ODD 414 / 464 may include ((block 1200 of street X) x (sunny + cloudy + light rain)) + ((highway 10 between highway exit 5 and highway exit 6) x (sunny + cloudy)).

[0075] Data generator 404 receives the suggested conditional spatial data 402 and uses the suggested conditional spatial data 402 to generate a geographic dataset 406. In some embodiments, the suggested conditional spatial data 452 includes data representing map features (hereinafter referred to as map feature data). Map feature data can be generated when map features are derived from the suggested map, as will be described in further detail below. The map features can be derived from a pre-existing map. Alternatively or additionally, the map features can be generated from data representing the environment around a measuring vehicle (hereinafter referred to as measurement data), which is received from the environmental sensors of the measuring vehicle while the measuring vehicle is driving on a road segment within the boundary of the suggested map. In some embodiments, the map features can be generated from a virtual representation of the environment on a road segment within the boundary of the suggested map (hereinafter referred to as simulated data, which will be described in further detail below). In some embodiments, the geographic dataset 406 is generated solely based on the suggested map data 452, while in other embodiments, it is generated by combining the environmental extent data 453 with the suggested map data 452 (see Figure 4B The geographic dataset 406 is generated by combining the following methods: First, the geographic dataset 406 includes object data indicating the types of objects that may exist on the road, along with relevant universality values ​​for each object type. The object data can be derived from a combination of pre-existing map feature data, simulation data, and measurement data. Second, the geographic dataset 406 includes sensor data and / or simulated sensor data, the sensor data indicating data received by the sensors of a measuring vehicle while it is traveling on the road, and the simulated sensor data being received by a simulated vehicle virtually traveling on the road. The sensor data can be acquired by a measuring vehicle equipped with sensors of the environmental sensors 110 of the vehicle 105 that have higher accuracy than those of the vehicle 105; in such cases, the sensor data can be downsampled before being included in the geographic dataset 406 to represent the accuracy level of the environmental sensors 110 of the vehicle 105. The sensor data includes metadata indicating the real or virtual location of the measuring vehicle and the real or virtual conditions at each point in time represented by the sensor data (e.g., environmental conditions around the measuring vehicle, driver conditions, and / or vehicle state conditions).

[0076] After the data generator 404 generates the geographic dataset 406, the geographic dataset 406 can be supplemented with augmented data by applying simulated changes in environmental conditions to the sensor data, thereby further enhancing the geographic dataset 406, as will be described in further detail below. Therefore, the geographic dataset 406 output by the data generator 404 includes measurement data, augmented data, simulated data, or any combination of measurement data, augmented data, and simulated data.

[0077] After generating the geographic dataset 406, the ADS evaluator 408 uses the geographic dataset 406 to evaluate the performance of the ADS 170, as will be described in further detail below. The evaluation of the geographic dataset 406 by the ADS evaluator 408 results in the calculation of an ADS risk metric for each road segment in the proposed map under each combination of the conditions (e.g., environmental conditions, vehicle condition conditions, and / or driver condition), shown as the ADS risk 410 for each condition output by the ADS evaluator 408. The risk comparator 412 compares a risk threshold 411 with the ADS risk of each road segment and identifies the ODD 414 based on which road segments meet the comparison under which conditions. The risk threshold 411 can be defined according to risk tolerance, for example, by a monetary (or some other) value, as will be described in further detail below. The portion of the proposed condition space with an ADS risk 410 below the risk threshold 411 (e.g., road segments in the proposed map, road segments under a specified set of conditions) defines a finite risk portion of the proposed condition space. The identified ODD 414 is output by the risk comparator 412, wherein the ODD 414 is composed of or based on the finite risk portion of the proposed condition space.

[0078] refer to Figure 4B This figure illustrates a schematic system diagram of an example of a second system 450 for identifying ODDs. It should be understood that the ODD identification module 174 includes the system 450. In some embodiments, the system 450 is a software system including computer-readable instructions stored in the memory 126 and executed by the processor system 102 of the vehicle control system 115 for identifying ODDs operated by the ADS 170. In some embodiments, the system 300 includes the ODD identification module 174, which includes the system 450 for identifying the ODDs.

[0079] refer to Figure 4BThe data generator 454 receives suggested map data 452 and data representing the series of suggested environmental conditions 453 (hereinafter referred to as environmental extent data 453). The suggested map data 452 and the environmental extent data 453 are used to generate a geographic dataset 456, which includes the above-mentioned references. Figure 4A The geographic dataset 406 generated in the process is a partial combination of measurement data, augmented data, and / or simulation data. The ADS evaluator 458 receives the geographic dataset 456 and uses the above reference... Figure 4A and Figure 4B The geographic dataset 456 evaluates the performance of the ADS 170. The ADS evaluator 458 outputs ADS risk data 460, which indicates the risk value of each road segment in the proposed map under each of the series of proposed environmental conditions. The risk comparator 462 receives the ADS risk data 460 and risk threshold data 461 indicating a risk threshold, compares the risk threshold with the maximum risk metric value of each road segment under each environmental condition, and identifies an ODD 464 based on the comparison, which will be described in further detail below. The ODD 464 is a multidimensional conditional subspace with finite risk (road segment x environmental condition) (i.e., a subspace that does not contain combinations of road segments and environmental conditions with risk metrics higher than the risk threshold). Therefore, the finite risk portion of the proposed conditional space includes the finite risk portion of the proposed map that has finite risk under the finite risk subsequence of the series of proposed environmental conditions.

[0080] Therefore, as referenced Figure 4A and Figure 4B The described exemplary system for identifying ODDs involves receiving the proposed conditional spatial data 402 (which may be divided into proposed map data 452 and proposed environmental extent data 453) at a data generator 404 / 454, which uses the proposed conditional spatial data to generate a geographic dataset 406 / 456. An ADS evaluator 408 / 458 uses the geographic dataset to evaluate the performance of the ADS 170. A risk comparator 412 / 462 uses the evaluation results to identify a finite-risk portion of the proposed conditional space represented by the proposed conditional spatial data, which is the ODD 414 / 464.

[0081] In some embodiments, the proposed map data 452 includes data representing GPS coordinates of road nodes and data representing GPS coordinates of connections between selected road nodes (i.e., road segments), such that the ODD is identified as a subset of the proposed map. The proposed map may include or correspond to multiple map features, and generating the geographic dataset 406 / 456 using the proposed conditional spatial data includes receiving the proposed map data 452, which includes map feature data representing multiple map features of the proposed map. The map features may include road nodes, road segments, routes, objects, paths, and / or other features of the proposed map or the spatial region represented by the proposed map. The map features may include multiple road nodes and multiple road segments corresponding to locations on the proposed map, each road segment corresponding to a path between two road nodes. Therefore, the map features may also include multiple routes between road nodes, each route comprising a consecutive sequence of one or more road segments. For a given proposed map, there may be multiple routes between two road nodes. Any route between two road nodes on the proposed map (that only passes through road segments included in the proposed map) will be considered a route included in the proposed map.

[0082] The map features may also include multiple object types for identifying object types that the vehicle 105 may encounter while driving along a route in the proposed map. These object types may be represented as metadata attached to road nodes and road segments in the proposed map. The object types can be obtained by measuring the environment in the proposed map (i.e., by driving the vehicle 105 in an environment corresponding to the proposed map and sensing objects in the environment using the multiple environmental sensors 110 of the vehicle 105). Object types may include one or more of the following: stationary objects (e.g., buildings, poles, towers, trees, bridges, roadblocks, monuments, walls); moving objects (e.g., vehicles (e.g., cars, light trucks, trucks, semi-trailers, trams, trains), pedestrians, animals); traffic signs (e.g., stop, yield, parking, school zone); traffic lights (e.g., by color (e.g., red, amber, green, white), shape (e.g., circle, square, forward arrow, left arrow, right arrow, line). Traffic signs and road markings are categorized by person, "walking," hand, bicycle, counter (digits), and pattern (e.g., constantly lit, constantly off, flashing); road markings (e.g., HOV lanes, highway entrances / exits, two-lane roads, intersections, yielding); lane boundaries (e.g., by color (e.g., white, yellow, blue, orange), pattern (e.g., solid line, double solid line, dashed line, double dashed line, dashed / solid line, solid / dashed line, Botts-dots)); and / or lanes (e.g., driving, non-driving, bicycle, intersections). For a more complete list of traffic signs and road markings, please see [link to traffic sign and road marking information]. https: / / mutcd.fhwa.dot.gov / services / publications / fhwaop02084 / us_road_symbol_ signs.pdf This document is incorporated herein by reference.

[0083] Apart from the moving objects, all other objects listed above are generally static and therefore deterministic: they either exist on the proposed map or they do not. Moving objects are random and therefore defined based on the probability of encountering them. Thus, each object type has one or more encounter probabilities, each associated with one of the road segments in the proposed map. In the case of static objects, this probability is typically 0 or 1, while for moving objects, the probability is random and changes and updates as more measurement or test data is collected for that road segment. In the context of assessing ADS risk, the probability of encountering an object on a given road segment may be referred to herein as the “exposure” level of that object type.

[0084] The environmental conditions discussed herein may include several different characteristics of the environment in which the ADS 170 is intended to operate. Ambient illuminance [lx] may indicate the ambient light intensity in the environment in which the vehicle 105 is driven, and is parameterized to a discretization level (logarithmic scale) from 0.01 lx to 120,000 lx. Visibility [m] may indicate the length of atmosphere a beam travels before its luminous flux decreases to 5% of its original value, and is parameterized to a discretization level from 0 m to 40,000 m. Precipitation type may indicate rain, ice, snow, sleet, etc. Precipitation amount [mm / h] may indicate precipitation intensity, and is parameterized to a discretization level from 0.1 mm / h to 200 mm / h. Time of day may affect quantities not covered above, such as peak-hour traffic density versus traffic density at 3 a.m. Additionally, atmospheric pressure [Pa], temperature [K], and relative humidity [%) may also be included. It should be understood that these environmental conditions and parameterization techniques are used as examples only and may be changed, omitted or added in various embodiments.

[0085] The geographic datasets 406 / 456 may contain measurement data, augmented data, and / or simulation data. The geographic datasets may contain the proposed map data 452, as well as all necessary metadata required to calculate the performance of the ADS 170 (e.g., road nodes, road segments, and map features of objects exposed in each road segment of the proposed map), as described below.

[0086] As described above, identifying a portion of the proposed condition space with finite risk involves comparing the ADS risk with risk thresholds 411 / 461. In some embodiments, the ADS evaluator 408 / 458 uses a risk metric to evaluate the ADS risk, examples of which will be described in detail below. To identify the finite-risk portion of the proposed map, a risk metric value is calculated for each route in the proposed map. The risk metric value for a route is typically the sum of the risk metric values ​​for each segment of the route. Therefore, the finite-risk portion of the proposed map is a set of routes, each with a risk metric value below the risk threshold.

[0087] In an embodiment using the environmental extent data 453, the risk metric is calculated for each route included in the proposed map under each suggested environmental condition in the environmental extent data. Therefore, the finite risk portion of the suggested condition space is a set of (route x environmental condition) combinations, each with a risk metric value lower than the risk threshold.

[0088] This makes the finite risk portion of the proposed condition space defined by the maximum value of the risk metric below the risk threshold (i.e., the risk metric value of the highest-risk route or the highest-risk combination of (route x environmental conditions) within the finite risk portion of the proposed condition space). If the finite risk portion of the proposed condition space is used as the ODD, the ODD can be represented as a geographic dataset containing GPS coordinates (i.e., road nodes) and connections between these road nodes (i.e., road segments) and a set of environmental conditions, wherein the ADS 170 will be able to operate in fully automatic mode and theoretically have a risk value below a predetermined risk threshold. This means that any route that may exist between two road nodes in the ODD will also have a risk value below the risk threshold.

[0089] The risk metric can be a quantified loss value (expressed in monetary units or other units) of driving on a specific segment of the proposed map under specific environmental conditions. The risk metric is a brief description of the likelihood of an accident occurring based on the performance of the ADS 170 in the presence of objects or map features in the proposed map. In some embodiments, each route risk metric is calculated by summing the expected risk values ​​for each of a plurality of object types present on the corresponding route. In some embodiments, the expected risk value for an object type can be calculated as a product of three factors: severity, exposure, and probability of failure. Severity indicates the expected severity of the ADS 170's failure to respond appropriately to the object type on the route. As mentioned above, exposure indicates the prevalence or encounter probability of the object type on the route. Probability of failure indicates the likelihood that the ADS 170 will be unable to respond appropriately to the object type on the route. Therefore, the risk metric for a route can be mathematically expressed as:

[0090] Risk ($) = ∑ i Severity ($) i *Exposure(num) i *possibility(%) i

[0091] Wherein, the subscript i represents each object type in the suggested map or the suggested condition space.

[0092] Severity can represent the expected loss (quantified in monetary or other units) in the event that the ADS 170 fails to react to an object. For example, the severity of the ADS 170 failing to react to a traffic light could be high (hundreds to millions of dollars), while the severity of the ADS 170 failing to detect a "Welcome to XYZ City" sign could be much lower. In some embodiments, severity can vary depending on environmental conditions. Severity can be based on collected empirical data, such as historical insurance payouts, or it can be determined by business strategy.

[0093] The failure probability value of the ADS 170 for an object type can be based on a performance evaluation of the ADS 170 under the suggested environmental conditions using the geographic dataset. This failure probability is negatively correlated with the performance of the ADS 170. For example, if the perception module 178 of the ADS 170 performs poorly in detecting trucks, the ADS 170 is more likely to fail to detect trucks.

[0094] Exposure can be represented as a dimensionless normalized number proportional to the probability of encountering map features or the frequency of encountering a specific object in the proposed map. For example, if the proposed map consists of highways, pedestrian exposure is very low, while semi-trailer exposure is high. Exposure data representing this exposure can be generated during the process of measuring areas in the environment aligned with regions in the proposed map, and the exposure can be updated over time as the vehicle 105 drives in areas aligned with regions in the proposed map, collecting more data.

[0095] Therefore, if the failure severity of ADS 170 for an object is high and the exposure of ADS 170 is also high, the ADS risk for that road segment in the proposed map is very high. However, if the exposure of ADS 170 for the same object or feature is very low, the risk may be low. For example, pedestrian exposure on a lane-separated highway is very low. Therefore, if the proposed map consists only of lane-separated highways, the performance of ADS 170 in pedestrian detection may be low, and the risk may still be below the risk threshold.

[0096] In some embodiments, when the ADS 170 is operating in automatic or semi-automatic mode and the vehicle 105 may exit the ODD, the ODD identification module 174 also has the function of using the ADS 170 to alert the driver of the vehicle 105 (i.e., a safety driver). The ODD identification module 174 may determine that the vehicle 105 may exit the ODD because the vehicle 105 is physically traveling on a road segment or other location not included in the ODD, or because the environmental conditions or other conditions defining the suggested condition space are constantly changing or may change soon, causing the ADS 170 to exceed the limited risk portion of the suggested condition space defining the ODD of the vehicle 105. After detecting or determining that the vehicle 105 may exit the ODD, the ODD identification module 174 alerts the driver of the vehicle 105 using some form of user output (e.g., visual and / or voice alerts conveyed via the touchscreen 136 and / or speaker 138 of the vehicle 105). Similarly, the ODD identification module 174 can provide an alert or notification when the vehicle 105 may re-enter the ODD. When the vehicle 105 is driven under a limited risk subsequence of the suggested condition space (i.e., on a road segment and under a set of environmental conditions with limited risk), by notifying the driver (i.e., the safety driver), the driver (i.e., the safety driver) can remain highly vigilant only when necessary. Therefore, the high dependence on the driver (i.e., the safety driver) of the vehicle 105 shown in the prior art can be greatly reduced. The updated ODD is fed back to the vehicle 105, and the driver (i.e., the safety driver) can rely on the ADS 170 to alert the driver when the ODD exceeds its boundaries. Therefore, the driver (i.e., the safety driver) can focus on being alert to unusual events or special situations. Ideally, these events or situations should be the primary focus of the driver (i.e., the safety driver) in the vehicle 105, rather than relying on the driver (i.e., the safety driver) to speculate whether the ADS 170 will provide sensing feedback on the limited situation of the ODD.

[0097] In some embodiments, the ODD identification module 174 can continue to update the ODD after the initial identification of the ODD. The ODD identification module 174 may receive additional data, such as updated suggestion map data (e.g., map data identifying new suggested boundaries of the ODD), updated geographic datasets (e.g., updated geographic datasets including updated exposure data and / or updated sensor data), updated expected risk data (i.e., data representing the severity values ​​of updates for various object types), updated risk threshold data 411 / 461 (e.g., data representing new risk thresholds), or updated ADS performance data (i.e., data representing the probability of update failure related to object types on a road segment under certain environmental conditions). The ODD identification module 174 can then re-evaluate the performance of the ADS 170 using the data contained in the geographic dataset and the additional data, thereby updating the limited risk portion of the suggestion condition space. The ODD identification module 174 can then update the ODD based on the updated limited risk portion of the suggestion condition space.

[0098] Figure 5 The overall operation of an exemplary embodiment of a method 500 for identifying ODDs performed by the ODD identification module 174 is shown. The ODD identification module 174 may include a software system (e.g., Figure 4A and Figure 4B (as described in systems 400 and 450). The method can be performed by the processes of systems 400 and 450. The processes used to perform the steps of method 500, the coding of the ODD identification module 174 and / or the software systems 400 and 450 of the ODD identification module 174 are entirely within the scope of those skilled in the art. Method 500 may include more or fewer steps than those shown and described, and the steps may be performed in different orders.

[0099] refer to Figure 5In step 502, the ODD identification module 174 receives the suggested conditional spatial data 402, including the suggested map data 452. In step 504, the geographic dataset 406 / 456 is generated based on the suggested conditional spatial data 402. In some embodiments, the geographic dataset 406 / 456 is generated at least partially based on the suggested map data 452. The suggested map data 452 includes map feature data as described above. The map feature data 505 identifies road nodes, road segments, routes, paths, and object types, and is used by the ODD identification module 174 to generate the geographic dataset 406 / 456. In step 506, the performance of the ADS 170 is evaluated using the geographic dataset 406 / 456. A risk metric 507 (a product of severity x exposure x probability for each object type) is applied to the geographic datasets 406 / 456 to assess the ADS risk for each condition (e.g., each road segment or each road segment under each condition) within the proposed condition space, thereby generating a risk metric 508 for each route / condition within the proposed condition space. In step 510, the risk metric 508 is compared to a risk threshold 511 to identify a finite risk portion of the proposed condition space. For example, the finite risk portion of the proposed condition space may be a set of road segments and condition combinations where a risk defined by the risk threshold exists. In step 512, the ODD is identified based on the finite risk portion of the proposed condition space.

[0100] In step 514, method 500 detects that vehicle 105 may exit the limited risk condition subspace of the ODD. In step 516, in response to detecting the possibility of exiting the ODD, an alert is generated and output to notify the driver of vehicle 105 (i.e., the safety driver).

[0101] In step 518, additional data that may be related to identifying the ODD is received. The method 500 uses the existing data and the additional data to loop back to evaluation step 506 to re-evaluate the performance of the ADS 170 and propagates through the remaining steps until the ODD is updated in step 512.

[0102] By using a systematic approach to identify and update the ODD, safety can be quantified and specified in the manner necessary for testing and evaluating ADS 170. Even if the initial recommendation map or recommendation condition space is determined based on commercial or strategic considerations, the identified or updated ODD is supported by a wealth of statistical data. This form of ODD identification and updating reduces the risks of prematurely running automated tests on public roads, which are similar to blind testing in terms of hazard level, especially from the perspective of a safe driver. The ODD can be developed using data collected based on the recommendation map without the need for testing on public roads.

[0103] Now for reference Figure 6 , Figure 7 , Figure 8 and Figure 9 describe Figure 4B A more detailed description of the various operating methods of the system 450 described herein.

[0104] refer to Figure 6 The figure illustrates a first exemplary method of operation 600 of the system 450. Method 600 is performed by the system 450 as part of the ODD identification module 174. Method 600 evaluates the performance of the ADS (e.g., ADS 170) based on the following three modules: the localization module 177, the perception module 178, and the planning module 179. The perception module 178, a module of the ADS 170, is responsible for detecting and tracking objects in the environment in which the vehicle 105 is driven. The planning module 179, a module of the ADS 170, is responsible for determining the trajectory of the vehicle 105 in response to objects detected from the perception module 178. The localization module 177, a module of the ADS 170, is responsible for locating the vehicle 105 on a map (e.g., the suggested map).

[0105] The first method 600 begins with the data generator 604, which receives the proposed map data 602 and environmental extent data 611. The proposed map data 602 includes proposed map data representing proposed initial map boundaries (i.e., boundaries on the initial map), and the environmental extent data 611 includes data representing a series of proposed environmental conditions for the operation of the ADS (e.g., ADS 170). The data generator 604 then uses the proposed conditional spatial data 602 to generate a geographic dataset 612 in the following two steps. First, in step 606, any map feature data present in the proposed map data 602 is extracted, and a high-precision environmental sensor 110 is used to measure the environment corresponding to the area of ​​the proposed map to identify possible objects. (As described above, the measured objects 607 may include stationary objects, moving objects, traffic signs, traffic lights, road markings, and / or lane boundaries, etc.) The measurement step 606 generates an initial geographic dataset 608, which may include object data for each road segment in the proposed map. Object data may include object universality data, which indicates the probability (i.e., exposure) of encountering the object type on a given road segment.

[0106] Secondly, the object data for each road segment in the proposed map included in the initial geographic dataset 608 can be combined with the environmental extent data 611 to generate the geographic dataset 612 by collecting sensor data for each combination of road segments and environmental conditions through some combination of measurement, simulation, and enhancement in step 610. The environmental extent data 611 includes environmental condition data, which represents environmental conditions. As mentioned above, environmental conditions may include, for example, illuminance, visibility / fog, precipitation type, precipitation amount, etc.

[0107] In some embodiments, augmented data may be used to supplement or replace the environmental condition data. For example, data augmentation techniques can be used to simulate different environmental conditions, as they affect various types of input on the environmental sensor 110. Brightness and contrast in camera data can be adjusted to simulate different lighting conditions; noise filters can be applied to LiDAR data to simulate precipitation, and so on. For exemplary data augmentation techniques for simulating environmental conditions for the ADS 170 performance evaluation, please visit [website address]. https: / / www.freecodecamp.org / news / image- augmentation-make-it-rain-make-it-snow-how-to-modify-a-photo-with-machine- learning-163c0cb3843f / and https: / / github.com / UjjwalSaxena / Automold--Road- Augmentation-Library The link is incorporated herein by reference.

[0108] Similarly, a driving simulator can be used to simulate the driving process and evaluate the performance of the ADS 170. (GM (The Matrix, simulating San Francisco streets, see [link]). https: / / www.getcruise.com / technology and https: / / venturebeat.com / 2019 / 04 / 20 / gms-cruise-is-preparing-for-a-self-driving-future- in-the-cloud / CARLA http: / / carla.org / ), rFpro( http: / / www.rfpro.com / ) and nVidia ( https: / / www.nvidia.com / en-us / self-driving-cars / drive-constellation / Driving simulators have been created, the links above of which are incorporated herein by reference. These types of simulators can be used to evaluate the ADS by providing map feature data and / or by generating simulated geographic and / or environmental condition data, wherein the simulated sensor data refers to data collected by measuring the vehicle as it drives on a route within the proposed map boundaries under a range of environmental conditions.

[0109] The geographic dataset 612 can be generated such that the object data for each road segment in the proposed map of the geographic dataset 612 is the probability of encountering different object types in each road segment, and the probability reflects the true probability.

[0110] Then, the ADS evaluator 614 evaluates the ADS 170 using the geographic dataset 612. First, in step 618, in the case of a manually driven vehicle 105 deployed along a route in the proposed map (i.e., where the ADS 170 is not involved in driving the vehicle), data is collected using the ADS 170 and the environmental sensor 110. The data collected by the ADS 170 and the environmental sensor 110 (hereinafter referred to as ADS collection data 620) includes sensor data collected by the environmental sensor 110 and ADS performance data indicating the internal localization, perception, and planning processes of the ADS in response to the sensor data. In step 616, the ADS collection data 620 is used to evaluate the localization of the vehicle 105 by the positioning module 177 and determine the associated risks (i.e., risk metrics), wherein the failure probability of the ADS 170 is based solely on the probability of failure of the positioning module 177. In step 616, the location risk assessment generates a location risk 622 (hereinafter referred to as location risk 622) for each road segment in each environment, which is then compared with a location risk threshold 624 in step 626. If the location risk 622 is determined to be intolerable (i.e., the value of the location risk 622 is higher than the value of the location risk threshold 624), then in step 628, road segments and environments (or combinations thereof) with high risk are removed from the provisional ODD subspace. The provisional ODD subspace is an incomplete version of the ODD, which is still being optimized by the system 450. In step 628, the removal of road segments and environments (or combinations thereof) with high risk continues from the provisional ODD subspace until the location risk 622 is lower than the location risk threshold 624 (i.e., until the value of the location risk 622 is less than the value of the location risk threshold 634).

[0111] Once the location risk 622 is tolerable (i.e., the value of the location risk is less than the value of the location risk threshold 624), method 600 proceeds to step 630. The perception module 178 of the ADS 170 is evaluated to determine the relevant perceived risk (i.e., risk metric), wherein the failure probability of the ADS 170 is based solely on the probability of the perception module 178 failing. Similar to the location module 177, the ADS evaluator 614 generates a perceived risk (hereinafter referred to as perceived risk 632) for each road segment in each environment, and compares it to the perceived risk threshold 634 in step 636. If the perceived risk is intolerable (i.e., the value of the perceived risk 632 is greater than or equal to the risk threshold 634), then in step 638, road segments and environments (or combinations thereof) with high risk are removed from the provisional ODD subspace. In step 638, high-risk road segments and environments (or combinations thereof) continue to be removed from the provisional ODD subspace until the perceived risk 632 is lower than the perceived risk threshold 634 (i.e., until the value of the perceived risk 632 is less than the value of the perceived risk threshold 634).

[0112] Once the perceived risk 632 is tolerable (i.e., the value of the perceived risk 632 is less than the value of the risk threshold 634), the method 600 proceeds to step 640. The planning module 179 of the ADS 170 is evaluated to determine the relevant planning risk (i.e., risk metric), wherein the probability of failure of the ADS 170 is based solely on the probability of failure of the planning module 179. Similar to the positioning module 177 and the perception module 178, the ADS evaluator 614 generates a planning risk 642 (hereinafter referred to as planning risk 642) for each road segment in each environment, and compares it with the planning risk threshold 644 in step 646. If the planning risk 642 is intolerable (i.e., the value of the planning risk 642 is greater than or equal to the value of the planning risk threshold 644), then in step 648, road segments and environments (or combinations thereof) with high risk are removed from the provisional ODD subspace. In step 648, high-risk road segments and environments (or combinations thereof) continue to be removed from the provisional ODD subspace until the planning risk 642 is lower than the planning risk threshold 644 (i.e., until the value of the planning risk 642 is less than the value of the planning risk threshold 644).

[0113] In the final step 650, the ODD is identified as the remaining road segments and environmental or other conditions in the set of road segments and environments that have not yet been removed from the provisional ODD subspace in steps 628, 638, or 648. Therefore, for each of the positioning module 177, the sensing module 178, and the planning module, each combination (of road segments under environmental conditions) falls within a limited risk range.

[0114] The second method 700 also relies on the evaluation of the positioning module 177, the sensing module 178, and the planning module 179. However, method 700 compares the total risk from all three modules (positioning module 177, sensing module 178, and planning module 179) with a single risk threshold to identify the ODD. The initial steps and procedures are the same as those of the first method 600. However, instead of comparing the positioning risk 622, the sensing risk 632, and the planning risk 642 with individual positioning risk thresholds, sensing risk thresholds, and planning risk thresholds to remove road segments and environmental conditions from the provisional ODD subspace, in step 654, risk comparator 652 sums the positioning risk 622, the sensing risk 632, and the planning risk 642 to generate a total ADS risk 662 for each road segment in each environment (hereinafter referred to as total ADS risk 662). In step 656, the total ADS risk 662 is compared with an overall risk threshold 658. If the total ADS risk 662 is lower than the overall risk threshold 658 (i.e., the value of the total risk 662 is less than the value of the overall risk threshold 658), then in step 660, the ODD is identified to include the road segment and environmental condition. If the total ADS risk 662 is higher than the overall risk threshold 658 (i.e., the value of the total risk 662 is greater than or equal to the value of the overall risk threshold 658), then in step 664, road segments and environmental conditions with high risk are removed from the suggested condition space until the overall risk threshold 658 is reached.

[0115] Figure 8 The third method 800 shown is agnostic to the different modules constituting the ADS 170. Instead of assessing location risk, perceived risk, and planning risk separately as described above, any of the multiple risk assessment metrics (including an overall risk metric that measures the overall probability of failure of the ADS 170) can be used to assess ADS risk. The third method 800 repeats the steps and procedures of the second method 700, except in its ADS evaluator 814, where the third method 800 performs only a single step 802 in which the total ADS risk is determined using a risk metric.

[0116] The third method 800 may reduce the complexity of designing a separate ADS evaluator for each of the positioning module 177, sensing module 178, and planning module 179 of the ADS 170. Furthermore, this can alleviate the dependencies between the positioning module, sensing module, and planning module imposed by the ADS evaluators 408, 458, 514, and 614. The ADS whose ODD is identified by the ODD identification module 174 may have other modules than the traditional positioning module 177, sensing module 178, and planning module 179, or may have other modules besides the traditional positioning module 177, sensing module 178, and planning module 179, but will still function correctly in the third method 800 as long as the inputs and outputs remain unchanged. Moreover, when the failure probabilities of the positioning module 177, sensing module 178, and / or planning module 179 are not independent of each other and therefore can be combined in a non-linear manner, the third method 800 can more accurately capture the overall failure risk of the ADS 170.

[0117] Figure 9 The flowchart shown illustrates a method 900 for generating and / or enhancing a geographic dataset for evaluating an ADS (e.g., the ADS170). Method 900 is... Figure 4B A more detailed representation of the data generation process performed by the data generator 454 shown is provided. Method 900 illustrates the use of suggested map data 452 and suggested environmental extent data 453 (see [link to documentation]). Figure 4B Different techniques for generating geographic datasets using the same input are shown here as map data including initial proposed map boundaries 452 and environmental extent data conditions 453.

[0118] In step 906, data is collected under constantly changing environmental conditions by manually driving a measurement vehicle (e.g., vehicle 105) within the boundaries of the proposed map, wherein the measurement vehicle is equipped with environmental sensors 110 of higher or at least equal accuracy. This data collection may run for several days to efficiently collect data with the necessary redundancy and a variety of environmental conditions, thereby effectively evaluating the ADS (e.g., ADS 170). A potential advantage of using this method is that the collected data represents the actual environment around the vehicle.

[0119] In step 908, suggested conditional spatial data 452 and environmental extent data 453 can be generated through simulation as described above, wherein the suggested conditional spatial data 452 includes data representing geographical conditions. A potential advantage of using this method is that it saves time and allows for incremental changes to environmental conditions 904.

[0120] In step 910, for each possible object in a road segment, the road segments within the boundaries of the map are measured. After the map measurement is completed, the above reference can be generated. Figure 6 The initial geographic dataset 608 may include object data for each road segment in the proposed map. The object data may include object prevalence data, which indicates the probability (i.e., exposure) of encountering the object type on a given road segment.

[0121] Then, in step 914, additional data is collected by performing further measurements under constantly changing environmental conditions 904, but keeping the object universality data the same as or close to the object universality data in the initial geographic dataset 911.

[0122] There may already be a usable dataset within the map boundaries, which can be used as an initial geographic dataset. As described above, the usable dataset can be obtained in step 912, and then used in step 916 to enhance the existing usable dataset under the changing environmental conditions 904.

[0123] The data generation process can employ any combination of these different steps to further enhance the diversity and robustness of the measurement data, and generate a final geographic dataset 918 containing a sufficient combination of road segment and environmental condition data. To eliminate bias associated with a specific dataset and appropriately assess the ADS risk, the ADS evaluator 454 requires a large amount of data to evaluate the ADS 170. The geographic dataset can be generated using the various methods described above. The failure risk of the tested ADS 170 can then be rigorously assessed to identify the ODD.

[0124] As described above, in some embodiments, the systems and methods described above can be implemented offline without vehicles. The environmental condition data and map data do not necessarily need to be provided as real-time data. The entire system can operate on simulated data, real data, or synthetic data. For example, the sensitivity of each environmental variable can be calculated by adjusting the environmental condition data (e.g., ambient illuminance). This allows for fine-tuning of environmental condition parameters, and the system will also have more robust statistical data to determine thresholds.

[0125] Accurately acquiring the data needed to evaluate the ADS can be a significant challenge. The exemplary systems and methods described herein are capable of taking simulated or synthetic data (i.e., data generated using simulated environments or data generated by applying simulated environmental conditions to sensor data, as described above) as input and testing the ADS's functionality under different environmental conditions. This can potentially provide more robust statistics for the system. Furthermore, a variety of different scenarios can be generated by synthesizing the collected data.

[0126] The systems and methods described in this paper for identifying and updating ODDs can also be potentially extended to cover safety-critical applications of learned (e.g., machine learning, logistic regression) modules and / or applications that require maps and / or environmental conditions.

[0127] The steps and / or operations in the flowcharts and accompanying drawings described herein are for illustrative purposes only. These steps and / or operations can be varied in many ways without departing from the spirit of the invention. For example, the steps can be performed in a different order, or the steps can be added, deleted, or modified.

[0128] In considering the present invention, the coding of software for implementing the above-described methods is within the scope of those skilled in the art. Machine-readable code, executable by one or more processors of one or more corresponding devices to perform the above-described methods, can be stored in a machine-readable medium such as the memory of a data manager. In this invention, the terms "software" and "firmware" are interchangeable and include any computer program stored in memory for execution by a processor, including random access memory (RAM), read-only memory (ROM), EPROM, electrically EPROM (EEPROM), and non-volatile RAM (NVRAM). The above memory types are merely examples and are therefore not limited to the types of memory that can be used to store computer programs.

[0129] Overview

[0130] All values ​​and subranges within the disclosed scope are also disclosed. Furthermore, although the systems, devices, and processes disclosed and illustrated herein may include a particular plurality of elements, such systems, devices, and components may be modified to include more or fewer such elements. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, elements shown in the accompanying drawings may be replaced, added, or modified, and the exemplary methods described herein may be modified by replacing, reordering, or adding steps to the disclosed methods. Moreover, numerous specific details are set forth to provide a thorough understanding of the exemplary embodiments described herein. However, those skilled in the art will understand that the exemplary embodiments described herein can be practiced without these specific details. Furthermore, well-known methods, processes, and elements have not been described in detail to avoid obscuring the exemplary embodiments described herein. The subject matter described herein is intended to cover and include all suitable technical modifications.

[0131] Although the invention has been partially described in terms of method, those skilled in the art will understand that the invention also relates to various elements for performing at least some aspects and features of the methods, whether by hardware, software, or a combination thereof. Therefore, the technical solutions of the invention can be embodied in a non-volatile or non-transitory machine-readable medium (e.g., optical disc, flash memory, etc.) having executable instructions stored thereon, tangibly stored thereon, enabling a processing device to perform the method examples disclosed herein.

[0132] The term "processor" can include any programmable system, including systems using microprocessors / controllers or nanoprocessors / controllers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), reduced instruction set circuits (RISCs), logic circuits, and any other circuits or processors capable of performing the functions described herein. The term "database" can refer to a data subject, a relational database management system (RDBMS), or both. As used herein, a database can include any collection of data and any other structured collection of records or data stored in a computer system, including hierarchical databases, relational databases, flat file databases, object-relational databases, and object-oriented databases. The examples above are merely illustrative and are not intended to limit the definition and / or meaning of the terms "processor" or "database" in any way.

[0133] This invention may be embodied in other specific forms without departing from the spirit of the claims. The exemplary embodiments described are to be regarded in all respects as illustrative rather than restrictive. This invention is intended to cover and include all suitable variations in the art. Therefore, the scope of the invention is defined by the appended claims rather than by the foregoing description. The scope of the claims should not be limited to the embodiments set forth in the examples, but should be given the broadest interpretation consistent with the entire description.

Claims

1. A method for identifying the design applicability domain of an autonomous driving system (ADS) for operation, characterized in that, The method includes: Receive suggested condition spatial data, which includes data representing the suggested map and data representing a series of suggested environmental conditions; Generate a geographic dataset using the suggested conditional spatial data; The performance of the ADS is evaluated using the geographic dataset; the performance of the ADS includes ADS risk data, which indicates a risk metric for each road segment in the proposed map under each of a series of proposed environmental conditions. Based on the performance of the ADS, a finite risk portion of the proposed condition space is identified; the finite risk portion is obtained by comparing a risk threshold with a risk metric for each road segment under each environmental condition. Based on the limited risk portion of the proposed condition space, the design applicability domain is identified.

2. The method according to claim 1, characterized in that: Evaluating the ADS includes using the geographic dataset to assess the performance of the ADS under the recommended environmental conditions. The limited risk portion of the proposed condition space includes a location in the proposed map and a combination of environmental conditions from the series of proposed environmental conditions where limited risk exists.

3. The method according to claim 2, characterized in that, Generating the geographic dataset using the suggested conditional spatial data includes receiving multiple map features of the suggested map.

4. The method according to claim 3, characterized in that, The map features of the proposed map include: Multiple nodes, corresponding to locations on the proposed map; Multiple road segments, each corresponding to the path between two of the aforementioned nodes; Multiple routes, each route comprising one or more of the aforementioned road segments; There are multiple object types, each with one or more encounter probabilities, and each encounter probability is associated with one of the road segments.

5. The method according to claim 4, characterized in that, The limited risk includes the maximum risk metric value below the risk threshold, which is calculated as the highest route risk metric value among multiple route risk metrics corresponding to multiple routes with limited risk within the limited risk portion of the proposed map.

6. The method according to claim 5, characterized in that, The risk metric for each route is calculated by summing the expected risks for each of the various object types present on the corresponding route. The expected risk for each object type is calculated as the product of the following: The severity value indicates the expected severity of the ADS's failure to respond appropriately to the object type on the route; Exposure value, which indicates the prevalence of the object type on the route; The failure probability value indicates the likelihood that the ADS will be unable to respond appropriately to the object type on the route; The failure probability value is based on an evaluation of ADS performance under the suggested environmental conditions using the geographic dataset.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: It is determined that vehicles using the aforementioned ADS for autonomous driving may fall outside the scope of the design application. In response to determining that the vehicle may be out of the design application domain, the driver of the vehicle is alerted.

8. The method according to any one of claims 1 to 6, characterized in that, This also includes, after identifying the design's applicable domain: Receive additional data; The performance of the ADS is reassessed using the geographic dataset and the additional data, thereby updating the limited risk portion of the proposed condition space; The design applicability domain is updated based on the updated limited risk portion of the proposed condition space.

9. The method according to claim 8, characterized in that, The additional data is selected from the group consisting of: updated suggested map data, updated geographic dataset data, updated expected risk data, updated risk threshold data, and updated ADS performance data.

10. A system for identifying the design applicability domain of an autonomous driving system (ADS) for operation, characterized in that, The system includes: Processor system; A memory coupled to the processor system, wherein the memory tangibly stores executable instructions thereon, which, when executed by the processor system, cause the system to perform the following operations: Receive suggested condition spatial data, which includes data representing the suggested map and data representing a series of suggested environmental conditions; Generate a geographic dataset using the suggested conditional spatial data; The performance of the ADS is evaluated using the geographic dataset; the performance of the ADS includes ADS risk data, which indicates a risk metric for each road segment in the proposed map under each of a series of proposed environmental conditions. Based on the performance of the ADS, a finite risk portion of the proposed condition space is identified; the finite risk portion is obtained by comparing a risk threshold with a risk metric for each road segment under each environmental condition. Based on the limited risk portion of the proposed condition space, the design applicability domain is identified.

11. The system according to claim 10, characterized in that: Evaluating the ADS includes using the geographic dataset to assess the performance of the ADS under the recommended environmental conditions. The finite risk portion of the proposed condition space includes the finite risk portion of the proposed map that has finite risk under the finite risk subseries of the series of proposed environmental conditions.

12. The system according to claim 11, characterized in that, Generating the geographic dataset using the suggested conditional spatial data includes receiving multiple map features of the suggested map.

13. The system according to claim 12, characterized in that, The map features of the proposed map include: Multiple nodes, corresponding to locations on the proposed map; Multiple road segments, each corresponding to the path between two of the aforementioned nodes; Multiple routes, each route comprising one or more of the aforementioned road segments; There are multiple object types, each with one or more encounter probabilities, and each encounter probability is associated with one of the road segments.

14. The system according to claim 13, characterized in that, The limited risk includes the maximum risk metric value below the risk threshold, which is calculated as the highest route risk metric value among multiple route risk metrics corresponding to multiple routes with limited risk within the limited risk portion of the proposed map.

15. The system according to claim 14, characterized in that, The risk metric for each route is calculated by summing the expected risks for each of the various object types present on the corresponding route. The expected risk for each object type is calculated as the product of the following: The severity value indicates the expected severity of the ADS's failure to respond appropriately to the object type on the route; Exposure value, which indicates the prevalence of the object type on the route; The failure probability value indicates the likelihood that the ADS will be unable to respond appropriately to the object type on the route; The failure probability value is based on an evaluation of ADS performance under the suggested environmental conditions using the geographic dataset.

16. The system according to any one of claims 10 to 15, characterized in that, It also includes a vehicle, which includes the ADS and is driven by a driver, wherein the instructions, when executed by the processor system, also cause the system to perform the following operations: It has been determined that the vehicle may be excluded from the design's applicable domain; In response to determining that the vehicle may be out of the design application domain, the driver is alerted.

17. The system according to any one of claims 10 to 15, characterized in that, When executed by the processor system, the instructions also cause the system to perform the following operations after identifying the design's applicable domain: Receive additional data; The performance of the ADS is reassessed using the geographic dataset and the additional data, thereby updating the limited risk portion of the proposed condition space; The design applicability domain is updated based on the updated limited risk portion of the proposed condition space.

18. The system according to claim 17, characterized in that, The additional data is selected from the group consisting of: updated suggested map data, updated geographic dataset data, updated expected risk data, updated risk threshold data, and updated ADS performance data.

19. A computer-readable medium, characterized in that, The computer-readable medium includes instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.

20. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a processor of a processing system, cause the processing system to perform the method according to any one of claims 1 to 9.

Images