Method and device for supporting the generation of scenarios for testing autonomous driving and / or advanced driver assistance system functions

Abstract

A method and apparatus for supporting the generation of scenarios for testing autonomous driving and / or advanced driver assistance system (AD / ADAS) functionality of a real-world vehicle. A device (101; 500) provides (301) a virtual environment (200) that simulates an environment associated with the operation of a vehicle having the AD / ADAS functionality, and operates the following in the virtual environment (200): fully computer-controlled movable virtual objects (230a‑c), human-controlled movable virtual objects (220), and at least one virtual AD / ADAS vehicle (210) that operates according to the AD / ADAS functionality. The device allows (303) a user of the device (101; 500) to control the one or more human-controlled movable virtual objects (220) via a user interface (506) during operation, and thereby generates scenarios to which the virtual AD / ADAS vehicle (210) is subjected.

Description

Technical Field

[0001] Embodiments herein relate to a method and apparatus involving generating scenarios for testing autonomous driving and / or advanced driver assistance system (AD / ADAS) functionality of a vehicle. Background Art

[0002] Unsupervised autonomous driving (AD) functions as well as advanced driver assistance systems (ADAS) functions, which may be referred to as AD and / or ADAS (AD / ADAS) functions, need to be evaluated and demonstrated prior to deployment to handle all relevant traffic scenarios in their defined operational design domain (ODD). However, due to the low exposure to safety-critical scenarios, billions of kilometers of supervised road testing may be required to confidently verify safe behavior, especially for autonomous vehicles (AVs). Even under aggressive testing assumptions and / or conditions, existing fleets of vehicles implementing AD / ADAS functions may need to be driven in real time in the real world for decades or centuries to be exposed to corner case scenarios and learn them, and thereby be able to validate against them. This approach alone is clearly not feasible, which in turn will hinder market penetration, especially of AD technologies.

[0003] There are variations in the definition of extreme situations. As used herein, an extreme situation may correspond to what may be referred to as a long-tail event, and may be interpreted as a situation that exposes a test subject (e.g., a vehicle with AD / ADAS functionality) to an environment that is far outside of what the test subject should normally be exposed to and normally be able to handle (e.g., an environment that is unlikely to occur but still possible), and that the test subject should or must be able to handle (e.g., be able to detect) and take appropriate action if exposed to the environment. Thus, an extreme situation scenario related to a vehicle's AD / ADAS functionality is a scenario that exposes the vehicle's AD / ADAS functionality to an extreme situation.

[0004] It lies in the nature of extreme case scenarios that they are often difficult to predict or calculate theoretically, and they may need to be identified by experiencing them.

[0005] The many miles, time, and resources required for real-world testing of AD / ADAS functionality for vehicles have prompted leading companies in AD vehicle development to virtually simulate many autonomous miles before deploying their software implementing AD / ADAS functionality in real vehicles for use in real-world environments and on real roads. Driving billions of miles virtually is cost- and time-saving. Summary of the invention

[0006] In view of the above, it is an object to provide one or more improvements or alternatives related to the prior art regarding AD / ADAS functional testing in vehicles.

[0007] According to a first aspect of the embodiments herein, the purpose is achieved by a method performed by a device, the method being used to support the generation of scenarios for testing autonomous driving and / or advanced driver assistance systems (AD / ADAS) functions of one or more real-world vehicles. The device provides a virtual environment that simulates an environment associated with the operation of one or more vehicles having the AD / ADAS function. The following items are operated in the virtual environment: one or more fully computer-controlled movable virtual objects, one or more human-controlled movable virtual objects, and at least one virtual AD / ADAS vehicle operated according to the AD / ADAS function. The device further allows a user of the device to control the one or more human-controlled movable virtual objects via a user interface of the device during operation in the virtual environment, and thereby generates a scenario to which one or more of the at least one virtual AD / ADAS vehicle is subjected. The scenario is obtained from at least one of the one or more human-controlled movable virtual objects, and the one or more human-controlled movable virtual objects affect one or more of the following items: the one or more fully computer-controlled movable virtual objects, the at least one virtual AD / ADAS vehicle, and the virtual environment.

[0008] According to a second aspect of embodiments herein, the object is achieved by a computer program comprising instructions which, when executed by one or more processors, cause the apparatus to perform the method according to the first aspect.

[0009] According to a third aspect of embodiments herein, the object is achieved by a carrier comprising the computer program according to the second aspect.

[0010] According to a fourth aspect of the embodiments herein, the purpose is achieved by a device for supporting the generation of scenarios for testing autonomous driving and / or advanced driver assistance systems (AD / ADAS) functions of one or more real-world vehicles. The device is configured to provide a virtual environment that simulates an environment related to the operation of one or more vehicles with the AD / ADAS function. The following items are operated in the virtual environment: one or more fully computer-controlled movable virtual objects, one or more human-controlled movable virtual objects, and at least one virtual AD / ADAS vehicle operated according to the AD / ADAS function. The device is further configured to allow the user of the device to control the one or more human-controlled movable virtual objects via the user interface of the device during operation in the virtual environment, and thereby generate a scenario experienced by one or more of the at least one virtual AD / ADAS vehicle. The scenario is obtained from at least one of the one or more human-controlled movable virtual objects, and the one or more human-controlled movable virtual objects affect one or more of the following items: the one or more fully computer-controlled movable virtual objects, the at least one virtual AD / ADAS vehicle, and the virtual environment.

[0011] The device may further and / or may be further configured to: initiate identification that a certain scenario has occurred based on data generated in relation to one or more of the at least one virtual AD / ADAS vehicle. The identification may be based on user input via a user interface.

[0012] Furthermore, the device may and / or may be configured to: provide, via a user interface and in response to identification of the certain scenario, a notification to the user regarding participation in generating the certain scenario.

[0013] The data generated in relation to the one or more of the at least one virtual AD / ADAS vehicle may include data generated externally from the one or more of the at least one virtual AD / ADAS vehicle.

[0014] Furthermore, the device may and / or may be configured to: store data that enables at least a portion of the certain scenario to be recreated and thereby enables the same or another virtual AD / ADAS vehicle to be subjected to the at least a portion of the certain scenario on another occasion.

[0015] The fully computer-controlled movable virtual objects may include one or more of the at least one virtual AD / ADAS vehicle.

[0016] One or more of the at least one virtual AD / ADAS vehicle may be provided with a corresponding identifier during operation in the virtual environment, the identifier enabling a user to identify such virtual AD / ADAS vehicle as a certain type of virtual vehicle operating in the virtual environment via a user interface.

[0017] At least some of the one or more fully computer-controlled movable virtual objects may operate in the virtual environment with unpredictable behavior by implementing the computer control.

[0018] In some embodiments, the at least one virtual AD / ADAS vehicle is a plurality of virtual AD / ADAS vehicles that respectively operate according to different implementations of the AD / ADAS functions.

[0019] Moreover, in some embodiments, the at least one virtual AD / ADAS vehicle is a plurality of virtual AD / ADAS vehicles configured to operate according to different sub-functions of the AD / ADAS function, respectively.

[0020] In addition, the device can and / or can be configured to: allow a user to determine at least a portion of the virtual environment and / or at least a portion of the behavior and / or position of one or more of the computer-controlled movable virtual objects operating in the virtual environment via a user interface.

[0021] The embodiments herein expose a virtual AD / ADAS vehicle to a mix of human and computer-controlled movable virtual objects during operation in a virtual environment on a single device, and scenarios are generated in their context. A single user on a single device can contribute to scenario generation in a virtual environment, but there, thanks to the computer-controlled movable virtual objects, complex traffic situations are still enabled. The generated scenarios can be considered to correspond to test cases, which can be reused, for example, for testing of AD / ADAS functions. Therefore, storing data that allows scenarios to be recreated and therefore reused may be more valuable than data related to the behavior of the function (e.g., the testing of the AD / ADAS function itself).

[0022] Specifically, embodiments herein may increase the likelihood of generating and identifying extreme case scenarios. A single user on a single device is further the most common way for users to participate in various virtual situations (e.g., computer games) today, and embodiments herein enable leveraging this and thereby reaching many users, and allowing them to contribute to scenario generation and identification. This facilitates crowdsourcing execution of the method, which involves multiple devices with corresponding users, thereby further increasing the likelihood of generating interesting scenarios in a time-saving and cost-effective manner. This may be further facilitated by having the fully computer-controlled movable virtual object include the at least one virtual AD / ADAS vehicle.

[0023] In addition, using input from a user of the device in identifying the certain scenario may be particularly advantageous for identifying extreme case scenarios. The user is typically the user who knows best what he / she has done and can identify scenarios that "stand out" for some reason, which may typically be the case for extreme case scenarios. Extreme case scenarios are typically rare and have quite unique properties, which can make it difficult to automatically identify such scenarios. Identifying certain scenario(s) with or without user input further results in a reduced number of scenarios to store, at least store for a long period of time, and / or further automatically and / or manually evaluate or identify.

[0024] Thus, generally speaking, embodiments herein provide improvements with respect to testing of AD / ADAS functionality in vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Examples of embodiments herein are described in more detail with reference to the additional schematic diagrams briefly described below.

[0026] Figure 1 is a block diagram that schematically depicts a system for discussing embodiments herein.

[0027] FIG. 2A illustrates schematically and in a simplified manner an example of a virtual environment that will be used to illustrate embodiments herein.

[0028] FIG. 2B schematically and in a simplified manner illustrates the virtual environment of FIG. 2A populated with different types of virtual vehicles for operation therein.

[0029] Figure 3 is a flow chart schematically illustrating the actions of a method according to embodiments herein.

[0030] Figure 4 is another flow chart that schematically illustrates some actions associated with embodiments herein.

[0031] Figure 5 is used to illustrate an embodiment of a device according to an embodiment of the present invention and how it can be configured to perform Figure 3 Functional block diagrams describing methods and actions.

[0032] Figure 6 is a schematic diagram illustrating an embodiment related to a computer program and its carrier, wherein the computer program and its carrier are used to cause the device to execute Figure 3 Describes methods and actions. DETAILED DESCRIPTION

[0033] The embodiments herein are exemplary embodiments. It should be noted that these embodiments are not necessarily mutually exclusive. Components from one embodiment may exist in another embodiment by default, and it will be obvious to those skilled in the art how those components can be used in other exemplary embodiments.

[0034] As a development for the embodiments herein, the situation indicated in the background art will first be further clarified.

[0035] While virtually simulating many autonomous miles is a time and cost saver compared to real-world test driving, it cannot and should not replace real-world testing and testing involving humans, at least as long as there are actual humans involved in the real-world environments and scenarios that the AD / ADAS vehicle will be subjected to. The human factor should not be forgotten, and human creativity should not be overlooked.

[0036] Rule-based control algorithm design is limited by the initial settings of all rules, and when multiple rules need to be considered, it can be easy to miss or involve design errors. Formal methods are one of the ways to solve the verification challenges of testing complex rule-based control algorithms, but it requires good modeling of the algorithms. Machine learning-based control algorithms are limited by their training samples. AD / ADAS system testers usually design test scenarios or drive vehicles on public roads for collecting scenarios. However, some scenarios, including extreme case scenarios, cannot be well covered by design or real-world driving, but the scenarios are very valuable for AD / ADAS system and function verification.

[0037] In fully computer-controlled dynamic environments (e.g., with moving AI-controlled objects that are trained based on datasets generated from real-world driving), these typically do not include rare safety-critical behaviors, and there is typically no or insufficient data on edge cases. The interaction of computer-controlled objects with AVs can ultimately generate some combinations of interesting test scenarios, but it is not possible to generate all edge case scenarios that may occur in the real world and may be of interest. For example, it would be interesting to be able to increase the possibility of generating edge case scenarios without having to drive in the real world using real-world vehicles (at least not for decades or centuries).

[0038] Crowdsourcing is a process that involves outsourcing tasks to a decentralized group of people. The basic idea of ​​the embodiments herein is to facilitate crowdsourcing the task of verifying AD / ADAS functionality to individuals around the world who may be interested in, or motivated by, other means of facilitating imposing risky and potentially safety-critical scenarios to vehicles with AD / ADAS functionality, such as in a simulated environment or video game.

[0039] Figure 1 is a block diagram schematically depicting a system 100 for discussing embodiments herein.

[0040] System 100 relates to one or more clients or user devices, or simply devices, which are illustrated by devices 101-103 in the figure and can be operated according to the embodiments herein. Devices 101-103 are associated with users respectively and can belong to different types, such as smart phones, laptop or desktop personal computers, game consoles, etc. Devices 101-103 therefore have computer and data processing capabilities. In the example shown, in order to illustrate the principle, devices 101, 103 are illustrated as smart phones, and device 102 is illustrated as another type, such as a desktop computer. Therefore, one or more of devices 101-103 can be, but not limited to, devices with wireless communication capabilities, i.e. wireless communication devices. The methods and actions of the embodiments herein described below can be performed by any one of devices 101-103, but device 101 can be used as the main example below.

[0041] Devices 101-103 may further be connected to a computer network 104 (e.g., the Internet), and may be communicatively connected, for example, via the network to a server 105 for remote processing and / or storage of data regarding embodiments herein, as will be discussed further below. Server 105 may serve multiple devices, for example, all devices 101-103. A server may correspond to a computer or host with server capabilities, or a cluster of such computers, and may be part of or correspond to a so-called computer cloud, or simply a cloud, that provides server services.

[0042] As will be explained in further detail below, each of the devices 101-103 (e.g., device 101) may provide a virtual environment, for example by executing software on the device 101, or provide access to the virtual environment, for example provided by the server 105. The virtual environment simulates an environment related to the operation of (multiple) vehicles with AD / ADAS functions. The virtual environment may be provided by or through software executed on the device 101, for example in the form of a computer program (e.g., a so-called application or "app") installable and executable on the device 101, for example corresponding to a simulator and / or a computer game that may motivate the user of the software and the device 101 to virtually challenge a vehicle equipped with AD / ADAS software with difficult to manage and / or risky scenarios.

[0043] It should be understood that the types of user devices mentioned above each include a user interface that allows different kinds of user input and output to the device. The user interface can be of different types, including conventional types for interfacing with a user, such as a touch-sensitive display (i.e., screen), a keyboard, action buttons and / or sticks such as on a game controller, a speaker, for example a microphone with voice recognition, etc.

[0044] Figure 2a An example of such a virtual environment mentioned above, here virtual environment 200, is schematically illustrated. As illustrated, virtual environment 200 includes a virtual infrastructure including roads for driving virtual vehicles (e.g., virtual cars), and associated structures adjacent to and near the roads, such as in the form of buildings, crosswalks, sidewalks, traffic lights, etc.

[0045] Figure 2b Pictured Figure 2a2, which is now also populated with: computer-controlled movable virtual objects 230a-c, which may be exemplified as artificial intelligence (AI) controlled cars, virtual AD / ADAS vehicles 210, which may be exemplified as AV cars (which are configured to operate according to the AD / ADAS functions), and human-controlled movable virtual objects 220, which may be exemplified as human-controlled (i.e., human driver (HD)) cars. The objects and vehicles 210, 220, 230a-c are operating and / or are configured to operate in the virtual environment 200.

[0046] The virtual AD / ADAS vehicle 210 may execute and / or operate according to a particular version of AD / ADAS software that provides AD / ADAS functionality.

[0047] The computer-controlled movable virtual objects 230a-c can be fully or partially AI-controlled, that is, the behavior of these objects can be simulated by using AI algorithms. The AI ​​algorithm can change from a complex supervised machine learning algorithm trained by real-world data to a reinforcement learning technique that explores a simulated environment and / or can learn to challenge, for example, a virtual AD / ADAS vehicle 210. The computer-controlled movable virtual objects 230a-c can include, for example, pedestrians, cyclists, vehicles, etc. The computer-controlled movable virtual objects 230a-c can be fully or completely computer-controlled when operating, that is, they can act and move without human control during operation in the virtual environment 200. Alternatively, they can be referred to as fully computer-operated virtual objects or autonomous virtual vehicles. Therefore, the fully computer-controlled movable virtual objects 230a-c can be operated independently of human users. In a computer game context, they can correspond to so-called non-player characters (NPCs).

[0048] The user of the device 101 can control the human-controlled movable virtual object 220 in the virtual environment 200, for example, via the user interface of the device 101, and thereby cause it to interact with and / or affect the virtual environment 200 itself, such as its infrastructure, computer-controlled virtual objects 230a-c, and / or the virtual AD / ADAS vehicle 210. The interaction, or how the interaction affects the virtual environment 200 itself, other objects and / or vehicles operating therein, can directly or indirectly cause the AD / ADAS vehicle to be subjected to (i.e., exposed to and / or experience) various situations and scenarios when operating in the virtual environment 200. It is noted that it is not necessary for a human (e.g., a user of the device 101) to control every aspect of a movable virtual object in order for it to be human-controlled, but, for example, the user control can control the object's movement behavior in some aspect, and this causes the said influence and / or interaction during operation, and is therefore a result from the user control. In general, human control may be in real time or near real time, or with some delay during operation, and / or the control may be set or configured by a human (eg, a user of device 101 ) prior to operation.

[0049] The human-controlled movable virtual object(s) develop opportunities for the user to participate and effectively act as a tester with respect to AD / ADAS functionality, although this need not necessarily be the focus for the user, who may instead participate in a simulator and / or computer game. A user of a device (e.g., device 101) may, for example, control the human-controlled movable virtual object 220 as part of such a simulator and / or computer game, and, for example, define or determine at least a portion of the virtual environment 200 as part of configuring the simulator and / or computer game.

[0050] A user of a device (e.g., device 101) can further load and / or influence the behavior of a computer-controlled, movable virtual object (e.g., an AI road user) in a simulator and / or computer game, for example in order to set up a scenario where the goal is to expose the virtual AD / ADAS vehicle 210 to situations or scenarios of interest, such as corresponding to extreme case scenarios.

[0051] As should be appreciated, existing software in the form of simulators and / or computer games may be modified to provide a virtual environment as virtual environment 200 and implement embodiments herein, as described below.

[0052] Notice, Figure 12A-B are merely schematic and are used for illustration purposes, and not everything shown in each figure may be required for all embodiments herein, as should be clear to the technician. In addition, in practice, there can certainly be more devices, multiple servers, other more complex virtual environments, etc., but for the sake of simplicity, they are not shown herein. In fact, there can be, for example, a much larger number of devices and corresponding users, when the corresponding device executes the software, each of which can provide the same, similar or different virtual environments, etc. The software can be a copy of the same version of the software, such as the same simulator and / or computer game software, or some devices can execute different software, such as another simulator and / or video game, but it still implements the embodiments herein.

[0053] Figure 3 Depicted is a flow chart in combination with a signaling diagram which will be used to discuss embodiments herein.

[0054] The following actions may form a method for supporting generation of scenarios for testing autonomous driving and / or advanced driver assistance system (AD / ADAS) functionality of one or more real-world vehicles. The actions are performed by a device (eg, device 101 ).

[0055] The following actions may be performed in any suitable order, and / or with full or partial overlap in time when this is possible and appropriate.

[0056] Action 301

[0057] The device 101 provides a virtual environment (e.g., virtual environment 200), which simulates an environment related to the operation of one or more vehicles (e.g., the real-world vehicles) having the AD / ADAS function, for example, through a computer program executed on the device 101. In the virtual environment 200, it is operating the following: one or more fully computer-controlled movable virtual objects (e.g., computer-controlled movable virtual objects 230a-c), one or more human-controlled movable virtual objects (e.g., human-controlled movable virtual objects 220), and at least one virtual AD / ADAS vehicle (e.g., virtual AD / ADAS vehicle 210) operating according to the AD / ADAS function.

[0058] The AD / ADAS functionality may fully or partially correspond to, be associated with, or be programmed to be a real-world vehicle's AD / ADAS functionality. As should be understood, "real-world vehicle" as used herein refers to a physical vehicle that is intended to operate in the real physical world in which humans are born, live, and interact directly.

[0059] Computer-controlled and / or human-controlled movable virtual objects may, for example, correspond to or include vehicle(s) as discussed and illustrated with respect to FIG. 2A , and may additionally or alternatively include or be other object(s), such as virtual human(s), pedestrians walking and / or running, for example, with strollers(s), flying objects, such as drones, or thrown or fallen objects (such as rocks or trees) that can be moved by a user and placed, for example, on a road in a virtual environment, etc. As will be appreciated, movable virtual objects may correspond to different kinds of road users.

[0060] All or some of the one or more fully computer-controlled movable virtual objects can advantageously be operated by implementing the unpredictable behavior of the computer control. Therefore, repeated, predictable behavior can be avoided so that when the software (e.g., game) implementing the embodiments herein is being executed on a device (e.g., device 101) and operated by its user, these objects will not be repeatedly executed in the same way on different occasions. In order to achieve this, the computer-controlled movable virtual objects or at least their behavior can be implemented based on an AI algorithm, and / or implemented with a certain degree of randomized behavior. The former can have some additional advantages, wherein the algorithm can learn from previous occasions that, for example, cause the generation of certain (e.g., particularly interesting) scenes (e.g., extreme case scenes as described below). Machine learning algorithms can be used to influence and promote behavior, which increases the possibility of generating other similar scenes of interest. In any case, by avoiding repeated behavior, there will be greater changes among the generated scenes, as well as the increased possibility of generating scenes of interest (e.g., extreme case scenes).

[0061] The human-controlled movable virtual object may be or include a virtual AD / ADAS vehicle, for example if not fully autonomous but at least partially human-controlled, but may also be or include another vehicle or object.

[0062] In some embodiments, one or more fully computer-controlled movable virtual objects include one or more of the at least one virtual AD / ADAS vehicle, such as virtual AD / ADAS vehicle 210, i.e., the AD / ADAS vehicle is not user-centric at that time. In the case where the AD / ADAS functionality does not include autonomous driving functionality, such functionality may be provided separately, for example, the AD / ADAS functionality may be integrated into one of the fully computer-controlled movable virtual objects, in the form of a virtual vehicle that has been configured to allow this, for example. For further discussion and associated advantages, see below under action 303.

[0063] Action 302

[0064] Device 101 may allow a user to determine at least a portion of virtual environment 200 and / or at least a portion of the behavior and / or position of one or more of computer-controlled movable virtual objects 230a - c operating in virtual environment 200 via a user interface of device 101 .

[0065] Thus, a user can, for example, create traffic situations in a virtual environment that may be unique and will affect the operation of an AD / ADAS vehicle in the virtual environment, for example, so as to induce challenging and / or risky scenarios. This not only encourages the respective user to be creative when operating human-controlled virtual objects, thereby facilitating the generation of unique scenarios, but also may increase the probability of generating interesting scenarios (e.g., extreme case scenarios).

[0066] As used in the context of the present disclosure, a device or other entity that allows or is configured to allow a user of a device to do something via a user interface of the device may involve the device being configured, for example, by hardware and / or software, to be or be brought into a state in which the user can interact with the device and a computer program executed on the device and, for example, implementing the embodiments herein, via the user interface. The interaction is associated with the input and / or output of data via the user interface to thereby accomplish what the device allows or is configured to allow the user to do.

[0067] Thus, the present actions may, for example, involve the device 101 and / or software executing on the device 101 providing the required interface(s) and functionality(s) in hardware and / or software, such that a user may thereby, for example, visually via a display and a graphical user interface (GUI):

[0068] Determining or influencing a virtual environment 200, such as a city and / or different traffic conditions, situations, and / or scenarios, which may result in an AD / ADAS vehicle being exposed to events and which may generate scenarios of interest, and / or

[0069] Position or place one or more road users, such as one or more of the computer-controlled movable virtual objects (e.g., computer-controlled movable virtual objects 230a-c), the human-controlled movable virtual objects (e.g., human-controlled movable virtual objects 220), and the virtual AD / ADAS vehicles (e.g., virtual AD / ADAS vehicles 210) in the virtual environment, and / or determine or influence the behavior they will exhibit when operating therein.

[0070] Thus, allowing the user to determine at least a portion of the virtual environment may include allowing the user to determine a virtual infrastructure of the virtual environment 200. Allowing the user to determine the virtual infrastructure may, for example, include allowing the user to select from a map on the real-world environment, and thereby incorporating the virtual infrastructure into the virtual environment corresponding to the real-world infrastructure according to the selection. The user may, for example, select an area of ​​the map, and the virtual infrastructure, or even the virtual environment as a whole, will be based on and / or correspond to the real-world environment, such as infrastructure, of the selected area.

[0071] This makes it easier for users to create traffic situations, for example based on real life experience or knowledge of places where tricky or special traffic situations tend to occur. It further encourages users to be creative and helps attract more users. Thus, generating scenarios is further facilitated, and more interesting scenarios are generated.

[0072] Combine the following Figure 4 Discuss further examples related to this.

[0073] Action 303

[0074] The device 101 allows a user of the device 101 to control a human-controlled movable virtual object 220 during operation in the virtual environment 200 via a user interface of the device, and thereby causes generation of a scenario experienced by one or more of the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210). The scenario is derived from the human-controlled movable virtual object 220, which affects one or more of the following: one or more of the computer-controlled movable virtual objects 230a-c, the at least one virtual AD / ADAS vehicle 210, and the virtual environment 200.

[0075] Thus, the above scenario is caused by the user controlling (multiple) human-controlled movable virtual objects during operation, and, for example, directly interacting between these and the virtual AD / ADAS vehicle 210, or indirectly interacting via computer-controlled movable virtual objects 230a-c and / or the virtual environment 200. Direct or indirect interactions with the virtual AD / ADAS vehicle 210 may include interactions with AD / ADAS functions of the virtual AD / ADAS vehicle.

[0076] As should be understood, the scenario to which the virtual AD / ADAS vehicle 210 is subjected generally corresponds herein to a series of events during a continuous time period. In addition, as can be appreciated from the above, each scenario generated by the embodiments herein should be initiated by and can be considered to start with an event in which a user controls at least one of the human-controlled movable virtual objects through a user input via a user interface of the device, and thereby causes it to affect one or more of the fully computer-controlled movable virtual objects, the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210), and / or the virtual environment 200, thereby causing the virtual AD / ADAS vehicle 210 to become directly affected, or indirectly affected via a series of additional events. In order to determine whether the virtual AD / ADAS vehicle 210 is affected by (multiple) human-controlled movable objects, it can be compared with a situation (e.g., a hypothetical or simulated situation) in which no user control is involved and therefore no user-induced events, and for example, to see whether the virtual AD / ADAS vehicle and / or AD / ADAS function reacts and / or behaves differently. It should be appreciated that a plurality of such scenarios will typically be generated during a period of operation in the virtual environment 200, although not all will be of interest and fewer will correspond to corner case scenarios. However, the greater the volume of scenarios generated and the greater the variation among the generated scenarios, the greater the likelihood of generating interesting and corner case scenarios.

[0077] In the case where multiple virtual AD / ADAS vehicles are operating in the virtual environment 200, scenarios may be generated in relation to each virtual AD / ADAS vehicle individually, and / or the scenarios may be evaluated individually. That is, the scenarios may be for each virtual AD / ADAS vehicle, and the generated scenarios need not involve multiple virtual AD / ADAS vehicles. Therefore, the induced scenario generation may involve any one of the multiple virtual AD / ADAS vehicles, or may generally involve one or more of the multiple virtual AD / ADAS vehicles.

[0078] In addition, as should be understood from the above, the embodiments herein expose virtual AD / ADAS vehicles to mixed human and computer-controlled movable virtual objects in a virtual environment on a single device, and the scene is generated in its context, which enables a single user on a single device to contribute to the scene generation in the virtual environment, in which, thanks to the computer-controlled movable virtual objects, it is still possible to achieve quite complex traffic situations. The embodiments herein support and facilitate the generation of scenes that AD / ADAS vehicles will be subjected to and can be used for repeated testing of AD / ADAS functions. The generated scenes can be considered to correspond to test cases that can be reused, for example. Compared with the situation with only computer-controlled environments and behaviors, the embodiments herein can support the generation of test cases including scenes that can be considered to be closer to real-world scenes, and can also increase the possibility of generating extreme situation scenes. This is because humans and human behaviors exist in the real world, and their influence should also exist virtually, especially if the purpose is to generate scenes related to AD / ADAS functions for real-world vehicles. Human influence and unpredictability in control and behavior should enable greater changes in scenes that can be generated. Thus, human creativity can also be better exploited, which is not as easy to achieve in a fully computer-controlled situation. The greater variety and creativity involved is thought to increase the likelihood of extreme cases, especially in situations with many contributing users, such as in crowdsourcing situations.

[0079] It is noted that in some embodiments, the provided virtual environment (e.g., 200) may be at least partially visualized to the user(s) through a corresponding physical environment, and / or one or more of the virtual objects (e.g., vehicles) (e.g., 220a) may be at least partially visualized to the user(s) through a corresponding physical object (e.g., vehicle). For example, a user may control and / or view a physical (e.g., radio-controlled) vehicle in at least a portion of the physical environment corresponding to at least a portion of the virtual environment 200, rather than just virtually, through radio controls and / or augmented reality (AR) on a device (e.g., device 101). In such embodiments, the physical appearance of the virtual environment 200 and / or the objects therein may be considered part of the user interface, i.e., how the user(s) view and / or interact with the virtual environment and / or the objects therein (e.g., vehicles).

[0080] In some embodiments, the fully computer-controlled movable virtual objects (e.g., 230a-c) include the at least one virtual AD / ADAS vehicle (e.g., 210). In other words, the one or more human-controlled movable virtual objects (e.g., 220) may exclude the virtual AD / ADAS vehicle 210, which is therefore not controlled by the user, i.e., not user-centric. Such embodiments are indicated above under action 301. The advantage is that this facilitates the implementation of the embodiments herein in or with existing software or software platforms (e.g., computer games) (possibly in combination with associated hardware); and may include computer-controlled vehicles that are not necessarily user-centric, but instead may correspond to, for example, NPCs that the user can directly or indirectly affect and / or be affected by. For example, if such existing software is adapted to include one or more computer-controlled vehicles equipped with (e.g., via push download to software and / or hardware) AD / ADAS functionality, or such that the one or more computer-controlled vehicles are equipped with the AD / ADAS functionality and thereby become (multiple) virtual AD / ADAS vehicles, this should facilitate both the implementation of the embodiments herein and the crowdsourcing effect, thereby facilitating the generation of more and more varied scenarios, including extreme case scenarios. For example, because the at least one AD / ADAS vehicle is not controlled by a user, it can instead be controlled by multiple AD / ADAS vehicles in operation, simultaneously on a single device 101 operated by a single user.

[0081] In the embodiments herein implemented in or using existing software or software platforms (e.g., computer games) (possibly in conjunction with associated hardware), the involvement of (multiple) AD / ADAS vehicles and AD / ADAS functionality need not even be known to the user, but rather are agreed upon and implemented, for example, in collaboration with the provider of the computer game, and the generation of scenarios can ensue as a side effect as the user operates a character corresponding to a human-controlled movable virtual object in the virtual environment of the computer game.

[0082] One or more of the at least one virtual AD / ADAS vehicle (e.g., virtual AD / ADAS vehicle 210) may be provided with a corresponding identifier during operation in the virtual environment 200, the identifier enabling a user to identify such virtual AD / ADAS vehicle as a certain type of virtual vehicle operating in the virtual environment 200 via a user interface.

[0083] In this way, the virtual AD / ADAS vehicle 210 can be identified by the user, for example, as a virtual vehicle specifically for generating a scene related thereto. Therefore, the identifier should enable the user to identify the virtual AD / ADAS vehicle itself and thereby distinguish it from other similar vehicles that might otherwise be confused with, for example, other vehicles corresponding to human-controlled movable virtual objects (e.g., 220) and computer-controlled movable virtual objects (e.g., 230a-c).

[0084] In some embodiments, the user may be hinted which vehicle is the AD / ADAS vehicle without having to provide it with an explicit identifier as above, for example, when the AD / ADAS vehicle is the only vehicle or the vehicle in which the user has a viewpoint, the user may be hinted that this is the AD / ADAS vehicle, and thus, at least not from the user's perspective, there is no need to explicitly identify this vehicle as an AD / ADAS vehicle. In some embodiments, it may simply be to make the AD / ADAS vehicle like a real-world AV without a visible driver, and may be identified as such. However, in the case of multiple other vehicles in the virtual environment 200, user-induced scenario generation may be facilitated if, for example, identification of which vehicle is the AD / ADAS vehicle 210 is made easier by the identifier.

[0085] Action 304

[0086] The device 101 may initiate identification that a scenario has occurred, or identify that a scenario has occurred, based on data generated in relation to one or more of the at least one AD / ADAS vehicle (eg, the virtual AD / ADAS vehicle 210 ).

[0087] The identified certain scenario should be a scenario of interest. A certain scenario may be a certain type or kind of scenario, and may be determined based on predefined criteria and / or user input. For example, it may be a new scenario compared to previously generated and / or stored scenarios. A certain scenario may be a scenario or a candidate scenario for a test case(s), and / or may be an edge case scenario, or a candidate for an edge case scenario. The scenarios of interest are discussed further below.

[0088] Data generated in connection with the AD / ADAS vehicle 210 may be data generated by or in a portion of software that is related to the AD / ADAS vehicle and / or that provides AD / ADAS functionality, and / or is related to another vehicle, object, or infrastructure that interacts with the AD / ADAS vehicle or is within a predefined or predetermined vicinity of the AD / ADAS vehicle.

[0089] Thus, in some embodiments, the data generated in relation to the one or more of the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210) includes data generated externally from the one or more of the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210). That is, data generated externally from the virtual AD / ADAS vehicle(s) experiencing the certain scenario.

[0090] Moreover, as indicated below, externally generated data (e.g., sensory data) may also be used for some functions of the AD / ADAS vehicle 210, e.g., where it is not desirable or possible to provide it with full or complete AD / ADAS functionality as it would in a real vehicle. That is, instead of simulating real-world sensors that provide data to the AD / ADAS functionality (i.e., to the AD / ADAS functionality to be operated on) when in a real-world vehicle, such data may instead be provided by, e.g., extracted from, a device executing software implementing embodiments herein, and such data may be based, for example, on how the AD / ADAS vehicle (e.g., its boundaries, position, and / or speed) relates to the virtual environment and other (e.g., moving) objects therein.

[0091] The identification itself may be performed by the device 101, and / or may involve its user, and / or may involve remote processing, such as the transmission of data to / from a server (e.g., server 105) that assists or performs the identification. In the latter case, the device may initiate the identification by, for example, sending data generated in connection with the AD / ADAS vehicle to a server (e.g., server 105). The initiation of the identification may be performed with or without user support.

[0092] For example, the identification may be based on: the software providing AD / ADAS functionality when executed in device 101 signals that it cannot handle a situation as it should or that a problematic event (e.g., a collision) has occurred, and / or some portion of the software monitors what the AD / ADAS vehicle involved has caused and / or is causing when interacting with other vehicles, objects, and infrastructure in the virtual environment, and generates data identifying certain events (e.g., collisions). The identification may be further based on predefined criteria, which may, for example, include the occurrence of such a certain event and / or the occurrence of a certain signal from the software providing AD / ADAS functionality. In some embodiments, the predefined criteria may include comparing the generated data with a previously stored scenario and seeing if it is sufficiently different in one or more aspects and is therefore or may be of interest.

[0093] As already indicated above, the identification can be based on user input via the user interface. The user input for scene identification can be part of the information used to identify the scene of interest. There may be, for example, a replay presented to the user via the user interface, such as a replay during the following period: during this period, the user has controlled the human-controlled movable virtual object 220 in the virtual environment, and thereby may have directly or indirectly affected the virtual AD / ADAS vehicle 210. Then, for example, the user may be allowed to mark the (multiple) scenes that the user considers interesting, for example, by selecting a timestamp, via the user interface. Another option is that the device suggests scene candidates to the user via the user interface, for example, through the network 104 with or without remote processing support, via the user interface, with the aid of the server 105, and then the user can consult the candidate via the user interface, and thereafter select and / or rate (if any) the (multiple) scenes that the user considers interesting.

[0094] Using input from a user of the device (e.g., a user of device 101) (or even better, input from a user involved in causing the scenario) for identifying scenarios has several advantages, particularly for identifying edge case scenarios. The user is the one who knows best what he / she has done and identifies scenarios that "stand out" for some reason, which can typically be cases for edge case scenarios. As explained, these scenarios are typically rare and have a fairly unique nature, which can make it difficult to automatically identify many such scenarios.

[0095] Identifying certain scenes among a large number of scenes on each device, with or without user input, should further result in a reduced number of scenes to be stored, at least for a long period of time, and / or further automatically and / or manually evaluated or identified, for example after uploading to a server (e.g., server 105), for central processing that may involve comparison with data and / or scenes generated by other users and / or on other devices.

[0096] Action 305

[0097] Device 121 can provide a notification to the user about participating in generating the certain scene via the user interface and in response to the identification of the certain scene. That is, the user can be notified via the user interface that he / she participated in the generation of a certain scene that has been identified, such as a scene of special interest, such as an extreme case scene.

[0098] As should be appreciated, the identification itself may have been performed by the device 101 , with or without user involvement, and / or by the server 105 .

[0099] Feedback to the user like this further supports similar behavior by the user, which, for example, leads to the identified extreme case scenario, and therefore should increase the possibility of generating more and more interesting scenarios. It also enables gamification about scenario generation, in which the user associated with the feedback can be rewarded. For example, there may be a rating and / or point rewarded to the user, and it is related to the number of certain identified scenarios, such as the scenarios that the user has participated in generating and is identified as interesting. Rewards can also be based on the classification of scenarios, in which some categories are better rewarded. For example, if a scenario belongs to a category or type of scenario that is considered an extreme case and / or is more likely to occur in the real world, it may be rewarded to a greater extent. The scenario in which the user has been involved in creating a virtual environment related to the scenario may belong to a certain category as follows, which may also be rewarded to a greater extent. This enables the number of scenarios that occur for certain categories or types of scenarios to be affected, and thereby affects the possibility of generating interesting scenarios and / or generating certain desired categories or types of scenarios.

[0100] Action 306

[0101] The device 121 may store data that enables at least a portion of the certain scenario to be recreated, thereby enabling the same or another virtual AD / ADAS vehicle to experience the at least a portion of the certain scenario on another occasion.

[0102] At least a part of the certain scenario means that the certain scenario can be partially or completely recreated, for example, a part thereof that is considered sufficient for a test case and / or a certain test purpose.

[0103] For example, the recreation of a scenario corresponding to a test case enables repeated testing or further testing of the AD / ADAS functionality by reusing the scenario. The recreated scenario can be used to subject the same or another AD / ADAS vehicle having the same or another (e.g., updated) AD / ADAS functionality to a scenario corresponding to, for example, an extreme case scenario. When provided by another software and / or when related to another vehicle, the scenario of interest for testing a specific AD / ADAS functionality is typically also of interest for testing this AD / ADAS functionality as well as similar or related AD / ADAS functionality. Therefore, storing data that allows the scenario to be recreated and thereby reused may be more valuable than data related to the behavior of the functionality (e.g., testing of the AD / ADAS functionality itself).

[0104] It should be appreciated that in addition to supporting the generation of scenarios for concurrently testing AD / ADAS functionality, embodiments herein may also test and generate data related to the functionality itself implemented by the virtual AD / ADAS vehicle.

[0105] It is further within the capabilities of a skilled person, such as a programmer providing software implementing the embodiments herein and for execution on a device, to identify and, if necessary, generate the data to be stored and allow for said recreation.

[0106] It is noted that the current action, or in general the storage of scene data (such as data enabling at least part of the scene to be recreated), can be performed in response to an identification that a scene of interest has occurred and is related to the identified scene of interest, i.e., can be in response to action 304. It may be sufficient and beneficial to store only data for recreating the scene of interest. However, in some embodiments, the scene data may first be stored at least temporarily, and then, for example, in the case of a comparison involving other stored scene data, it is determined whether to delete or retain the stored scene data, for example, in the case where it is identified as being of interest. In these embodiments, the storage may be remote from the device, for example, on a remote server (such as, server 105). Moreover, in these cases, the identification that the scene is a scene of interest may be performed completely or partially remotely (e.g., on server 105).

[0107] As should be appreciated, the embodiments herein provide an opportunity to test and use AD / ADAS functionality of a vehicle using, for example, a simulation platform and at different levels. Perhaps the simplest and possibly most direct level is to equip a virtual AD / ADAS vehicle (e.g., virtual AD / ADAS vehicle 210) with decision and control algorithms to test various functions, such as ADAS functions, such as an automated emergency braking (AEB) system or adaptive cruise control (ACC), etc. At this level, the virtual AD / ADAS vehicle does not need to be equipped with a sensing (such as a perception) system, since information describing this (e.g., information about the surrounding environment, such as other road users and infrastructure, distances, relative speeds, etc.) can be extracted from and / or within the virtual environment, or more precisely, from software executed on a device, such as software that implements an embodiment and provides a virtual environment, etc.

[0108] It should be appreciated that AD / ADAS software companies thereby have the opportunity to test, for example, different dimensions of software in a virtual environment and equip a virtual AD / ADAS vehicle with certain AD / ADAS functionality. For example, where it is desired to test or validate a perception algorithm, such an algorithm may be inserted (i.e., loaded into) the virtual AD / ADAS vehicle 210.

[0109] Embodiments herein also enable deployment of several versions of the same AD / ADAS functionality, for example, provided by different AD / ADAS software, in the same virtual environment (e.g., in the virtual environment 200 on the device 101), and / or ensure that different versions are used on different devices, allowing them to operate in the corresponding virtual environment, thereby testing the functionality, and, for example, comparing the results. If, for example, the virtual environment is part of a simulator and / or computer game software as discussed above, different versions of the AD / ADAS functionality can be pushed to the device executing the simulator and / or game, for example, pushed to different (multiple) fully computer-controlled movable virtual objects in the form of (multiple) vehicles, and / or pushed to (multiple) human-controlled virtual objects in the form of (multiple) vehicles, each of which corresponds to an AD / ADAS vehicle. Loading and / or configuring the AD / ADAS functionality (e.g., in the form of code that implements the functionality) can be done with the user knowing it and / or being explicitly notified of it, or without the user knowing it and / or being explicitly notified of it. It may be implemented, for example, by downloading from the server 105 to the device 101 during and / or as part of action 301. For example, as a push download, as mentioned above.

[0110] Therefore, in some embodiments, the at least one virtual AD / ADAS vehicle (e.g., 210) is a plurality of virtual AD / ADAS vehicles configured to operate according to different implementations of the AD / ADAS functionality, respectively. Different implementations may correspond to different versions of software that provide the AD / ADAS functionality. Therefore, in these embodiments, for each of the plurality of virtual AD / ADAS vehicles, the AD / ADAS functionality may be the same or substantially the same.

[0111] As indicated above, the advantage is that, for example, different software versions can be used and / or tested simultaneously in the virtual environment 200, and data and / or results compared. This also increases the impact from each user or device, and the crowdsourcing impact is further increased because each user or device contributes to more exposure of the AD / ADAS vehicle to scenarios for testing AD / ADAS functions.

[0112] In addition, in some embodiments, the at least one virtual AD / ADAS vehicle (e.g., 210) is a plurality of virtual AD / ADAS vehicles configured to operate according to different sub-functions of the AD / ADAS function, respectively. Therefore, in these embodiments, the AD / ADAS function includes different sub-functions. In general, the AD / ADAS function as used herein may include several sub-functions. For example, in these embodiments, one or more of the plurality of virtual AD / ADAS vehicles may implement, for example, an ADAS function (such as AEB), and one or more of the other plurality of virtual AD / ADAS vehicles may implement, for example, an AD function. This increases the flexibility and versatility of the embodiments herein, and increases the possibility of generating interesting scenarios. Functions can be tested in a virtual environment with respect to each other and how they may affect each other. This can also be utilized to increase the changes in generating interesting scenarios.

[0113] As already indicated, embodiments herein and as discussed above support or even facilitate the generation of scenarios, including so-called extreme case scenarios, to which it may be of interest to subject the AD / ADAS vehicle 210 and also other vehicles with AD / ADAS functionality. For example, by being in a virtual environment, (human) users will be less willing to cause potentially harmful or dangerous scenarios than in the real world, and this therefore increases the probability of generating such scenarios, i.e., scenarios that may be important for testing and correspond to extreme case scenarios, which are difficult and are generally desirable to avoid in the real world for obvious reasons. As explained above, extreme case scenarios are scenarios that generally occur so rarely in the real world that, for example, existing test fleets would need to drive for decades or hundreds of years in order to be exposed to such scenarios. In all computer-controlled environments (which have objects trained based on data sets that generally do not include rare safety-critical behaviors), the interaction of computer-controlled objects with AVs may ultimately generate some combined interesting test scenarios, but it may not be able to generate extreme case scenarios that may occur in the real world. This is due to the fact that computer-controlled objects can only behave based on training data sets that do not include information about or behind edge case scenarios. Therefore, augmenting all computer-controlled environments with human-controlled objects as in the embodiments herein provides the opportunity to expose AD / ADAS vehicles and computer-controlled objects in all computer-controlled environments that have behaviors that are outside the domain of what can be achieved with conventional training data sets.

[0114] Additionally, it should be appreciated that the mixture of computer-controlled vehicles and human-controlled movable objects as in the embodiments herein can enable the generation of scenarios that are closer to real-world scenarios with human interaction, at least in some respects, than scenarios that may be generated in conventional virtual test environments that are only computer-controlled. All computer-controlled environments, even if based on data from real human behavior in real-world situations and artificial intelligence that simulates human behavior, usually must be based on data collected from a very large number of individuals over a very long period of time in order to be able to generate behaviors similar to human behavior. Such data is usually not available, or will require a very long period of time, and / or is very difficult to generate in the real world. And even if this will be attempted, the mixture of AD / ADAS vehicles and human-controlled vehicles is expected to change over time in the real world, thereby increasing the preference for AD / ADAS vehicles, and this ratio (i.e., the commonness of AD / ADAS vehicles) will affect human behavior in traffic. Scenarios that consider human behavior in situations with a higher number of AD / ADAS vehicles than are currently available in the real world cannot rely on real-world data, but thanks to the embodiments herein, such scenarios can be generated.

[0115] In addition, as indicated above, the embodiments herein allow for simpler implementations in other, for example, already existing virtual environment platforms or programs, and / or co-implementations with other, for example, already existing virtual environment platforms or programs, such as those associated with simulators (e.g., vehicle simulators) and / or computer games. In other words, the embodiments herein facilitate implementation in different scenarios, in different virtual environments, and / or on different types of devices. This in turn makes it possible to attract more and different users to implement the method. The more people involved, the more different and varied the situations and scenarios, and the greater the possibility of generating valuable scenarios (such as, extreme case scenarios) thereby.

[0116] Thus, embodiments herein facilitate crowdsourced generation of scenarios for testing AD / ADAS functionality.

[0117] Furthermore, the embodiments herein should not only enable companies that develop software that provides AD / ADAS functionality to accelerate the evaluation of their software, but also significantly reduce costs by reducing the number of miles of human-supervised road testing required to validate AVs.

[0118] In summary, embodiments herein provide improvements with respect to testing of AD / ADAS functionality in vehicles.

[0119] Figure 4is a flow chart to further discuss and illustrate the embodiments herein from a more user-centric perspective, where the user here is illustrated as a user of device 101 .

[0120] The acts may be performed in any suitable order, and / or fully or partially overlapping in time when this is possible and appropriate.

[0121] Action 401

[0122] The user may determine at least a portion of virtual environment 200 via a user interface of device 101. For example, the user may design and / or load at least a portion of the virtual environment, such as from within software executing on device 101, and / or from a remote location (e.g., from server 105).

[0123] This action may correspond in whole or in part to actions 301 and 302 above, and may result in what is schematically illustrated and illustrated in FIG. 2A .

[0124] For example, this may involve a user loading a portion of a map belonging to an area with challenging traffic situations, the user being aware of, for example, a high rate of traffic incidents or accidents in the area, which may lead the user to assume that a challenging scenario may be designed based on this.

[0125] Action 402

[0126] A user may select one or more human-controlled, movable virtual objects (e.g., human-controlled, movable virtual objects 220 ) via a user interface of device 101 for control by the user while they are operating in virtual environment 200 (i.e., during operation), for example as part of a simulator and / or computer game executing on device 101 .

[0127] This action may partially correspond to actions 301 and 303 above.

[0128] Action 403

[0129] A user may define, via a user interface of device 101 , the behavior of (multiple) computer-controlled, movable virtual objects (e.g., computer-controlled, movable virtual objects 230 a - c ) relating to how they will or should operate in a virtual environment 200 , e.g., as part of a simulator and / or computer game executing on device 101 .

[0130] It is noted that when a user defines the behavior of a computer-controlled movable virtual object and as a result the user thereby actually controls the object in the virtual environment, such an object may be considered both human-controlled and computer-controlled, although it may be fully computer-controlled during operation.

[0131] This action may partially correspond to actions 301 and 303 above.

[0132] Action 404

[0133] A user may load and / or position virtual AD / ADAS vehicle(s) (e.g., virtual AD / ADAS vehicle 210) via a user interface of device 101 to operate in virtual environment 200, e.g., as part of a simulator and / or computer game executing on device 101. As previously indicated, such AD / ADAS vehicles may be both computer-controlled virtual vehicles or human-controlled virtual vehicles.

[0134] This action may partially correspond to action 301 above.

[0135] Actions 402 - 404 may result in what is schematically illustrated and illustrated in FIG. 2B .

[0136] Actions 401-404 provide the user with the opportunity to simulate or even recreate challenging or risky scenarios that vehicles have encountered in the real world, or be inspired by the scenarios, and possibly even make those scenarios even more challenging. This may include the virtual environment itself in action 401, and / or road users such as pedestrians, vehicles, and AV(s) with and without AI support, such as the control in actions 402-404 which may be based, for example, in whole or in part, on an AI algorithm. For example, a user may define and / or control the behavior of road user(s), and have other road users controlled in whole or in part by an AI algorithm.

[0137] Action 405

[0138] Based on actions 401-404, movable virtual object(s) and vehicle(s) may be operated in the virtual environment, e.g., computer-controlled movable virtual objects 230a-c, human-controlled movable virtual object(s) 220, and virtual AD / ADAS vehicle 210 may be operated in the virtual environment 200. As mentioned above, the operation may be part of a simulator and / or computer game executed on the device 101.

[0139] This action may partially correspond to action 301 above.

[0140] Action 406

[0141] During operation in a virtual environment (e.g., during action 405), the device 101 may store data, which may be referred to as data logging. The data may relate to the behavior of (multiple) virtual AD / ADAS vehicles (e.g., virtual AD / ADAS vehicles 210), and / or data that enables partial or complete re-creation of (multiple) generated scenarios. (Multiple) scenarios are scenarios that (multiple) AD / ADAS vehicles experience during operation that may be part of a simulator and / or computer game executed on the device 101. The storage may be performed locally on the device 101 and / or remotely, for example, on the server 105.

[0142] This action may partially correspond to action 305 above.

[0143] Action 407

[0144] During or after operation in the virtual environment (e.g., during or after action 405), one or more scenes may be identified. The identification may be performed from the stored data resulting from action 406. Although real-time or near real-time identification may be possible (e.g., based on data temporarily stored in a fast access memory), it may be preferred to store the data in action 406 as long as possible during a longer operation period or based on storage capacity, and thereafter (e.g., after action 405), identify the scene from the stored data. The scene or its portion(s) that is identified and considered to be of interest may then be saved, e.g., stored more permanently, which may involve moving data associated with such scenes from, e.g., temporary storage locally to more permanent or long-term storage for further use of the data, e.g., remotely (e.g., on server 105). Therefore, at least in some embodiments, the storage in action 406 may be responsive to the current action.

[0145] This action may correspond in whole or in part to action 304 above.

[0146] The above-mentioned user may, for example, be or correspond to a crowdsourcing user who may act as a tester and, for example, by means of actions 401-404, may have the possibility to establish or select, or at least plan, a traffic situation or even a traffic scenario that will or may lead to a scenario during operation in the virtual environment: the scenario may be identified and may involve data stored in actions 406-407. The user may do this in order to cause risky or potentially risky situations or scenarios for the AD / ADAS vehicle(s) (e.g., the virtual AD / ADAS vehicle 210), thereby challenging or testing its behavior in such scenarios.

[0147] Even without the execution of one or more of the actions 401-404, for example in the case of a predefined or predetermined virtual environment 200 and / or (multiple) computer-controlled movable virtual objects and / or (multiple) human-controlled movable virtual objects 220 and / or a virtual AD / ADAS vehicle, the actions 405 and 406 and embodiments herein can still be useful. If the execution involves many users of the device and a crowdsourcing approach, valuable data corresponding to many driving miles can be provided and / or stored, and the data increases the likelihood of generating, identifying and storing interesting scenarios (e.g., extreme case scenarios).

[0148] Note that while the embodiments herein may be particularly suitable for generating corner case scenarios that can be recreated and reused in test cases, for example, they may also be used to test specific AD / ADAS functionality itself in addition to conventional real-world test drives and testing, for example in fully computer-controlled and simulated scenarios.

[0149] Figure 5 1 is a schematic block diagram for illustrating an embodiment of a device 500 that may correspond to the device 101 or any one of the devices 101-103. The schematic block diagram is also used to illustrate how the device 500 (eg, the device 101) may be configured to perform the above-mentioned Figure 3 Embodiments of the methods and actions discussed.

[0150] Thus, the apparatus 500 is used to support generation of scenarios for testing autonomous driving and / or advanced driver assistance systems (AD / ADAS) functionality of one or more real-world vehicles.

[0151] The apparatus 500 may include a processing module 501, such as means, one or more hardware modules including, for example, one or more processors, and / or one or more software modules for performing the methods and / or actions.

[0152] The device 500 may further include a memory 502, which may include (such as, contain or store) a computer program 503. The computer program 503 includes "instructions" or "codes" that are directly or indirectly executable by the device 500 to perform the methods and / or actions. The memory 502 may include one or more memory units and may be further arranged to store data, such as configurations and / or applications involved in the embodiments herein or used to perform the functions and actions of the embodiments herein.

[0153] Furthermore, the device 500 may include a processor(s) 504, i.e. one or more processors, as (multiple) exemplified hardware modules, and may include or correspond to one or more processing circuits. In some embodiments, the processing module(s) 501 may include the processor(s) 504, for example, embodied in the form of the processor(s) 504, or implemented by the processor(s) 504. In these embodiments, the memory 502 may include a computer program 503 executable by the processor(s) 504, whereby the device 500 is operative or configured to perform the method and / or its actions.

[0154] Typically, the device 500 (e.g., processing module(s) 501) includes input / output (I / O) module(s) 505 that are configured to be involved in (e.g., by performing) any communication to and / or from other units and / or devices, such as sending and / or receiving information to and / or from other devices (e.g., servers such as server 105). When applicable, the I / O module(s) 505 may be instantiated by obtaining (e.g., receiving) module(s) and / or providing (e.g., sending) module(s).

[0155] The device 500 further includes a user interface 506, which may be integrated with and / or communicatively connected to the device 500. The user interface 506 may allow different kinds of user input and output to / from the device 500. The user interface 506 may be of different types, including any conventional kind for interfacing with a user, including, for example, a display (i.e., screen) that may be touch-sensitive, a keyboard, action buttons and / or sticks such as on a game controller, speakers, a megaphone that allows communication using, for example, voice recognition, etc. As implemented, when including a display or screen, the user interface 506 is typically associated with a GUI implemented by software (i.e., by computer program code).

[0156] In addition, in some embodiments, the device 500 (e.g., (multiple) processing modules 501) includes (multiple) providing modules, (multiple) allowing modules, (multiple) storage modules, and (multiple) initiating modules. One or more of the modules are used as (multiple) illustrative hardware and / or software modules for performing the actions of the embodiments herein. These modules can be implemented in whole or in part by (multiple) processors 504.

[0157] The device 500, and / or (multiple) processing modules 501, and / or (multiple) processors 504, and / or (multiple) I / O modules 505, and / or (multiple) providing modules can therefore be operative or configured to provide the virtual environment (e.g., virtual environment 200), which simulates the environment related to the operation of the one or more vehicles having the AD / ADAS function, and in which it operates the following: the one or more computer-controlled movable virtual objects (e.g., 230a-c), the one or more human-controlled movable virtual objects (e.g., 220), and the at least one virtual AD / ADAS vehicle (e.g., 210) that operates according to the AD / ADAS function.

[0158] Furthermore, the device 500, and / or the processing module(s) 501, and / or the processor(s) 504, and / or the I / O module(s) 505, and / or the enabling module(s) may be operable or configured to enable the user of the device 500 to control the human-controlled movable virtual object 220 via the user interface 506 of the device 500 during operation in the virtual environment 200, and thereby cause the generation of the scene experienced by the one or more of the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210). The scene is derived from the human-controlled movable virtual object 220, which affects one or more of the following: the one or more fully computer-controlled movable virtual objects 230a-c, the at least one virtual AD / ADAS vehicle 210, the virtual environment 200. The device 500, and / or the processing module(s) 501, and / or the processor(s) 504, and / or the I / O module(s) 505, and / or the enabling module(s) may further be operative or configured to allow a user to determine, via a user interface 506 of the device 500, said at least a portion of the virtual environment 200, and / or said at least a portion of the behavior and / or said position of one or more of the computer-controlled movable virtual objects 230a-c operating in the virtual environment 200. This may, for example, include allowing the user to determine said virtual infrastructure of the virtual environment 200.

[0159] In addition, the device 500, and / or (multiple) processing modules 501, and / or (multiple) processors 504, and / or (multiple) I / O modules 505, and / or (multiple) initiating modules can be operative or configured to initiate the identification that the certain scenario has occurred based on data generated in connection with the one or more of the at least one AD / ADAS vehicles (e.g., the virtual AD / ADAS vehicle 210).

[0160] The device 500, and / or (multiple) processing modules 501, and / or (multiple) processors 504, and / or (multiple) I / O modules 505 and / or (multiple) providing modules may further be operative or configured to provide, via the user interface 506 and in response to the identification of the certain scene, the notification directed to the user regarding participation in generating the certain scene.

[0161] In addition, the device 500, and / or (multiple) processing modules 501, and / or (multiple) processors 504, and / or (multiple) I / O modules 505, and / or (multiple) storage modules may be operative or configured to store said data enabling the re-creation of said at least a portion of said certain scene.

[0162] Figure 6is a schematic diagram illustrating some embodiments related to a computer program and its carrier, so that the device 500 discussed above performs the methods and actions. The computer program may be a computer program 503 and includes instructions that, when executed by (multiple) processors 504 and / or (multiple) processing modules 501, cause the device 500 to perform as described above. In some embodiments, a carrier, or more specifically, a data carrier, such as a computer program product, including a computer program is provided. The carrier may be one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium, such as a computer-readable storage medium 601 as schematically illustrated in the figure. Thus, the computer program 503 may be stored on the computer-readable storage medium 601. By carrier, transitory propagating signals may be excluded, and the data carrier may be correspondingly named as a non-transitory data carrier. Non-limiting examples of data carriers as computer-readable storage media are memory cards or memory sticks, disk storage media such as CDs or DVDs, or mass storage devices typically based on (multiple) hard drives or (multiple) solid-state drives (SSDs). The computer-readable storage medium 601 can be used to store data accessible through a computer network 602 (e.g., the Internet or a local area network (LAN)). In addition, the computer program 503 can be provided as (multiple) pure computer programs, or included in one or more files. One or more files can be stored on the computer-readable storage medium 601 and can be obtained, for example, by downloading, for example, through the computer network 602 indicated in the figure, for example, via a server. The server can be, for example, a web or file transfer protocol (FTP) server. One or more files can be, for example, executable files for downloading directly or indirectly to the first node and executing on the first node, so that it is executed as described above, for example, by the execution of (multiple) processors 504. One or more files can also or alternatively be used for intermediate downloading and compiling involving (multiple) the same or another processor, so that they are executable before further downloading and execution, so that the device 500 is executed as described above.

[0163] It is to be noted that any processing module(s) and circuit(s) mentioned above may be implemented as software and / or hardware modules, for example in existing hardware.

[0164] Those skilled in the art will also appreciate that the modules and circuits discussed herein may refer to a combination of hardware modules, software modules, analog and digital circuits, and / or one or more processors configured with software and / or firmware stored, for example, in a memory, which, when executed by one or more processors, may cause (multiple) nodes and (multiple) devices to be configured to and / or perform the above-described methods and actions.

[0165] The identification by any identifier herein may be implicit or explicit. The identification may be unique in a certain context, for example, in a device or system, or at least in a relevant part or area thereof.

[0166] It should also be noted that, although the terminology used herein may be particularly associated with and / or exemplified by certain communication systems or networks, this in itself should not be construed as limiting the scope of the embodiments herein to only such certain systems or networks, etc.

[0167] As used herein, the term "memory" may refer to a data storage device for storing digital information, typically a hard disk, magnetic storage, medium, portable computer floppy disk or disk, flash memory, random access memory (RAM), etc. In addition, the memory may be an internal register memory of a processor.

[0168] It should also be noted that any enumerated terms (such as first device, second device, etc.) should be considered non-limiting in themselves, and the terms themselves do not imply a certain hierarchical relationship. In the absence of any explicit information to the contrary, naming by enumeration should be considered as just a way to implement different names.

[0169] As used herein, the expression “configured to” may mean that the device and / or the processing circuit or module is configured or adapted by means of software and / or hardware configuration to perform one or more of the actions described herein.

[0170] As used herein, the term "number" or "value" may refer to any kind of number, such as a binary number, a real number, an imaginary number, or a rational number, etc. In addition, a "number" or "value" may be one or more characters, such as a letter or a string of letters. Moreover, a "number" or "value" may be represented by a bit string.

[0171] As used herein, the expressions "may" and "in some embodiments" have been generally used to indicate that the features described can be combined with any other embodiments disclosed herein.

[0172] In the drawings, features that may be present only in some embodiments are often drawn using dotted or dashed lines.

[0173] As used herein, the expressions "transmit" and "send" are generally interchangeable. These expressions may include transmissions by broadcast, unicast, multicast, etc. In this context, a transmission by broadcast may be received and decoded by any authorized device within range. In the case of unicast, one specifically addressed device may receive and encode the transmission. In the case of groupcast (e.g., multicast), a group of specifically addressed devices may receive and decode the transmission.

[0174] When the word "comprising" or "including" is used, it shall be interpreted in a non-limiting sense, meaning "consisting at least of."

[0175] The embodiments herein are not limited to the embodiments described above. Various substitutions, modifications and equivalents may be used. Therefore, the above embodiments should not be considered to limit the scope of the present disclosure, which is defined by the appended claims.

Claims

1. A method performed by a device (101; 500) for supporting generation of scenarios for testing software-implemented autonomous driving and / or advanced driver assistance system (AD / ADAS) functions of one or more real-world vehicles, wherein the method comprises: - providing (301) a virtual environment (200) simulating an environment associated with the operation of one or more vehicles having the software-implemented AD / ADAS functionality, and in which the following items are operated: one or more fully computer-controlled movable virtual objects (230a-c), one or more human-controlled movable virtual objects (220), and at least one virtual AD / ADAS vehicle (210) operating according to the software-implemented AD / ADAS functionality, and - allowing (303) a user of the device (101; 500) to control the one or more human-controlled movable virtual objects (220) via a user interface (506) of the device (101; 500) during operation in the virtual environment (200), and thereby causing generation of a scene experienced by one or more of the at least one virtual AD / ADAS vehicle (210), and the scene being derived from at least one of the one or more human-controlled movable virtual objects (220), the one or more human-controlled movable virtual objects (220) affecting one or more of the following: the one or more fully computer-controlled movable virtual objects (230a-c), the at least one virtual AD / ADAS vehicle (210), the virtual environment (200), and - Initiating an identification (304) that a scenario of interest has occurred based on data generated in relation to one or more of the at least one virtual AD / ADAS vehicles (210), wherein the identification can be based on software providing AD / ADAS functionality when executed in the device (101) signaling that it is unable to handle the situation as it should or that a problematic event has occurred.

2. The method of claim 1, wherein the identifying is based on user input via a user interface (506).

3. The method according to any one of claims 1-2, wherein the method further comprises: - providing (305) via a user interface (506) and in response to the identification of said certain scene of interest, a notification directed to the user regarding participation in generating said certain scene of interest.

4. The method according to any one of claims 1-2, wherein the data generated in relation to the one or more of the at least one virtual AD / ADAS vehicle (210) comprises: Data generated from the one or more externally in the at least one virtual AD / ADAS vehicle (210).

5. The method according to any one of claims 1-2, wherein the method further comprises: - storing (306) data that enables at least a portion of the certain scenario to be recreated and thereby enables the same or another virtual AD / ADAS vehicle (210) to be subjected to the at least a portion of the certain scenario on another occasion.

6. The method according to any one of claims 1-2, wherein the fully computer-controlled movable virtual objects (230a-c) include one or more of the at least one virtual AD / ADAS vehicle (210).

7. A method according to any one of claims 1-2, wherein one or more of the at least one virtual AD / ADAS vehicle (210) is provided with a corresponding identifier during operation in the virtual environment (200), the identifier enabling a user to identify such virtual AD / ADAS vehicle as a certain type of virtual vehicle operating in the virtual environment (200) via a user interface (506).

8. The method according to any one of claims 1-2, wherein at least some of the one or more fully computer-controlled movable virtual objects (230a-c) operate in the virtual environment (200) with unpredictable behavior by implementing the computer control.

9. The method according to any one of claims 1-2, wherein the at least one virtual AD / ADAS vehicle (210) is a plurality of virtual AD / ADAS vehicles (210) that operate according to different implementations of the AD / ADAS functions, respectively.

10. The method according to any one of claims 1-2, wherein the at least one virtual AD / ADAS vehicle (210) is configured to be a plurality of virtual AD / ADAS vehicles (210) that operate respectively according to different sub-functions of the AD / ADAS function.

11. The method according to any one of claims 1-2, wherein the method further comprises: - allowing (302) a user to determine, via a user interface (506), at least a portion of a virtual environment (200), and / or at least a portion of the behavior and / or position of one or more of computer-controlled movable virtual objects (230a-c) operating in the virtual environment (200).

12. A computer program product comprising computer instructions, characterized in that: When executed by a processor (504), the computer instructions implement the steps of the method according to any one of claims 1-11.

13. A computer-readable storage medium storing computer instructions, characterized in that: When executed by a processor, the computer instructions implement the steps of the method according to any one of claims 1-11.

14. A device (101; 500) for supporting generation of scenarios for testing software-implemented autonomous driving and / or advanced driver assistance systems (AD / ADAS) functionality of one or more real-world vehicles, wherein the device (101; 500) is configured to: providing (301) a virtual environment (200) simulating an environment associated with the operation of one or more vehicles having the software-implemented AD / ADAS functionality, and in which it operates: one or more fully computer-controlled movable virtual objects (230a-c), one or more human-controlled movable virtual objects (220), and at least one virtual AD / ADAS vehicle (210) operating according to the software-implemented AD / ADAS functionality, and allowing (303) a user of the device (101; 500) to control the one or more human-controlled movable virtual objects (220) via a user interface (506) of the device (101; 500) during operation in the virtual environment (200), and thereby causing a scene to be generated that is experienced by one or more of the at least one virtual AD / ADAS vehicle (210), and the scene is derived from at least one of the one or more human-controlled movable virtual objects (220), the one or more human-controlled movable virtual objects (220) affecting one or more of the following: the one or more fully computer-controlled movable virtual objects (230a-c), the at least one virtual AD / ADAS vehicle (210), the virtual environment (200), and - Initiating an identification (304) that a scenario of interest has occurred based on data generated in relation to one or more of the at least one virtual AD / ADAS vehicles (210), wherein the identification can be based on software providing AD / ADAS functionality when executed in the device (101) signaling that it is unable to handle the situation as it should or that a problematic event has occurred.

Images