Method and server for supporting the generation of scenarios for testing autonomous driving and / or advanced driver assistance system functions

Abstract

A method and server for supporting the generation of scenarios for testing autonomous driving and / or advanced driver assistance system (AD / ADAS) functions of real-world vehicles. A server (101; 500) provides (301) a virtual environment (200) that simulates an environment associated with the operation of a vehicle having the AD / ADAS functions, and operates the following in the virtual environment (200): fully computer-controlled movable virtual objects (230a-c), human-controlled movable virtual objects (220a-c), and at least one virtual AD / ADAS vehicle (210) operating according to the AD / ADAS functions. The server (101; 500) allows a device (101-103) remotely connected to the server (101; 500) and a user of the device (101-103) to control the human-controlled movable virtual objects (220a-c) in a virtual environment (200) via a user interface of the device (101-103), respectively, and thereby cause generation of a scenario experienced by the at least one virtual AD / ADAS vehicle (210).

Description

Technical Field

[0001] Embodiments herein relate to a method and server that involve generating scenarios for testing autonomous driving and / or advanced driver assistance system (AD / ADAS) functionality of a vehicle. Background Art

[0002] Unsupervised autonomous driving (AD) functions as well as advanced driver assistance systems (ADAS) functions, which may be referred to as AD and / or ADAS (AD / ADAS) functions, need to be evaluated and demonstrated to handle all relevant traffic scenarios within their defined operational design domain (ODD) before deployment. However, due to the low exposure to safety-critical scenarios, billions of kilometers of supervised road testing may be required to confidently verify safe behavior, especially for autonomous vehicles (AVs). Even under aggressive testing assumptions and / or conditions, existing vehicle fleets implementing AD / ADAS functions may need to be driven in real time in the real world for decades or centuries to be exposed to corner case scenarios and learn from them, and thereby be able to validate against them. This approach alone is clearly not feasible, which in turn will hinder market penetration, especially of AD technologies.

[0003] There are variations in the definition of an extreme case. As used herein, an extreme case may correspond to what may be referred to as a long-tail event, and may be interpreted as a situation that exposes a test subject (e.g., a vehicle having AD / ADAS functionality) to an environment that is far outside of what the test subject should normally be exposed to and normally be able to handle (e.g., an environment that is unlikely to occur but still possible), and that the test subject should or must be able to handle (e.g., be able to detect) and take appropriate action if exposed to that environment. Thus, an extreme case scenario related to a vehicle's AD / ADAS functionality is a scenario that exposes the vehicle's AD / ADAS functionality to an extreme case.

[0004] It lies in the nature of extreme case scenarios that they are often difficult to predict or calculate theoretically, and they may need to be identified by experiencing them.

[0005] The many miles, time, and resources required for real-world testing of AD / ADAS functionality in vehicles have prompted leading companies in AD vehicle development to virtually simulate many autonomous miles before deploying their software implementing AD / ADAS functionality in real vehicles for use in real-world environments and on real roads. Driving billions of miles virtually saves both cost and time. Summary of the Invention

[0006] In view of the above, it is an object to provide one or more improvements or alternatives to the prior art regarding AD / ADAS functionality testing in vehicles.

[0007] According to a first aspect of the embodiments herein, the purpose is achieved by a method executed by a server, the method being used to support the generation of scenarios for testing autonomous driving and / or advanced driver assistance systems (AD / ADAS) functions of one or more real-world vehicles. The server provides a virtual environment that simulates an environment related to the operation of one or more vehicles with the AD / ADAS functions. In the virtual environment, it operates the following: one or more fully computer-controlled movable virtual objects, one or more human-controlled movable virtual objects, and at least one virtual AD / ADAS vehicle that operates according to the AD / ADAS functions. The server allows a device to be remotely connected to the server and a user of the device to control the human-controlled movable virtual objects separately in the virtual environment via a user interface of the device, and thereby generates a scenario experienced by one or more of the at least one virtual AD / ADAS vehicle. The scenario is obtained from at least one of the one or more human-controlled movable virtual objects, and the one or more human-controlled movable virtual objects affect one or more of the following: the one or more fully computer-controlled movable virtual objects, the at least one virtual AD / ADAS vehicle, and the virtual environment.

[0008] According to a second aspect of embodiments herein, the object is achieved by a computer program comprising instructions which, when executed by one or more processors, cause the server to perform the method according to the first aspect.

[0009] According to a third aspect of embodiments herein, the object is achieved by a carrier comprising the computer program according to the second aspect.

[0010] According to a fourth aspect of the embodiments herein, the purpose is achieved by a server for supporting the generation of scenarios for testing autonomous driving and / or advanced driver assistance systems (AD / ADAS) functions of one or more real-world vehicles. The server is configured to provide a virtual environment that simulates an environment related to the operation of one or more vehicles with the AD / ADAS functions. In the provided virtual environment, it operates the following: one or more fully computer-controlled movable virtual objects, one or more human-controlled movable virtual objects, and at least one virtual AD / ADAS vehicle that operates according to the AD / ADAS functions. The server is further configured to allow a device to be remotely connected to the server and the user of the device to control the human-controlled movable virtual objects separately in the virtual environment via the user interface of the device, and thereby generate a scenario experienced by one or more of the at least one virtual AD / ADAS vehicle. The scenario is obtained from at least one of the one or more human-controlled movable virtual objects, and the one or more human-controlled movable virtual objects affect one or more of the following: the one or more fully computer-controlled movable virtual objects, the at least one virtual AD / ADAS vehicle, and the virtual environment.

[0011] The server may further and / or may be further configured to: initiate identification that a certain scenario has occurred. The identification may be based on data generated in relation to one or more of the at least one virtual AD / ADAS vehicles. The data generated in relation to one or more of the at least one virtual AD / ADAS vehicles may include data generated externally from one or more of the at least one virtual AD / ADAS vehicles.

[0012] In some embodiments, the identifying is based on identifying a context of information received from one or more of the devices.

[0013] Furthermore, the server may and / or may be configured to send information identifying participation in generating the certain scene to a device, the device being used to control a movable virtual object controlled by a human and thereby causing the certain scene to be generated.

[0014] Furthermore, the server may and / or may be configured to: store data enabling at least a portion of the certain scenario to be recreated and thereby enabling the same or another virtual AD / ADAS vehicle to be subjected to the at least a portion of the certain scenario on another occasion.

[0015] The fully computer-controlled movable virtual object may include the at least one virtual AD / ADAS vehicle.

[0016] In some embodiments, one or more of the at least one virtual AD / ADAS vehicle is provided with a corresponding identifier during operation in the virtual environment (200), the identifier enabling a user of the device to identify such virtual AD / ADAS vehicle as a certain type of virtual vehicle operating in the virtual environment via a user interface of the device.

[0017] Furthermore, in some embodiments, one or more of the fully computer-controlled movable virtual objects are fully computer-controlled virtual vehicles that are each provided with an identifier during operation in the virtual environment, the identifier enabling a user of the device to identify, via a user interface of the device, that they are fully computer-controlled. In some embodiments, one or more of the human-controlled movable virtual objects are human-controlled movable virtual vehicles that are each provided with an identifier during operation in the virtual environment, the identifier enabling a user to identify, via a user interface of the device, that they are human-controlled.

[0018] Furthermore, in some embodiments, one or more of the human-controlled movable virtual objects are human-controlled virtual vehicles, and one or more of the fully computer-controlled virtual movable objects are fully computer-controlled virtual vehicles. In these embodiments, the server can and / or can be configured to control the ratio of human-controlled virtual vehicles to fully computer-controlled virtual vehicles operating in the virtual environment. The ratio can be controlled to maintain a certain level.

[0019] The embodiments herein expose virtual AD / ADAS vehicles to a mix of human and computer-controlled movable virtual objects during operation in a virtual environment, and scenarios are generated within their context, enabling scenario generation in a mix of human-controlled vehicles and AVs that is not yet available in the real world. Furthermore, multiple users can contribute to and assist in scenario generation and identification. The generated scenarios can be considered to correspond to test cases that can be reused, for example, for testing AD / ADAS functionality. Specifically, the embodiments herein can increase the likelihood of generating and identifying extreme case scenarios. A single user on a single device is furthermore the most common way for users to participate in various virtual scenarios (e.g., computer games) today, and computer networks are increasingly popular for providing virtual environments with (multiple) central processors and allowing users to connect remotely. The embodiments herein enable this to be exploited and thereby facilitate reaching many users who can contribute to scenario generation and identification. Thus, crowdsourcing execution of the method is facilitated, involving multiple devices with corresponding users, thereby further increasing the likelihood of generating interesting scenarios in a time-efficient and cost-effective manner.

[0020] Thus, generally speaking, embodiments herein provide improvements with respect to testing of AD / ADAS functionality in vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Examples of embodiments herein are described in more detail with reference to the additional schematic diagrams briefly described below.

[0022] Figure 1 is a block diagram that schematically depicts a system for discussing embodiments herein.

[0023] Figure 2a An example of a virtual environment that will be used to illustrate embodiments herein is illustrated schematically and in a simplified manner.

[0024] Figure 2b Schematically and in a simplified manner Figure 2a A virtual environment populated with different types of virtual vehicles for operating therein.

[0025] Figure 3 is a flow chart schematically illustrating acts of a method according to embodiments herein.

[0026] Figure 4 is another flow chart that schematically illustrates some actions associated with embodiments herein.

[0027] Figure 5is used to illustrate an embodiment of a server according to embodiments herein and how it may be configured to perform Figure 3 Functional block diagrams describing methods and actions.

[0028] Figure 6 The present invention is a schematic diagram illustrating an embodiment related to a computer program and its carrier, wherein the computer program and its carrier are used to cause the device to execute Figure 3 Describes methods and actions. DETAILED DESCRIPTION

[0029] The embodiments herein are exemplary embodiments. It should be noted that these embodiments are not necessarily mutually exclusive. Components from one embodiment may exist in another embodiment by default, and it will be apparent to those skilled in the art how those components may be used in other exemplary embodiments.

[0030] As a development for the embodiments herein, the situation indicated in the background will first be further clarified.

[0031] While virtually simulating many autonomous miles is time- and cost-effective compared to real-world test driving, it cannot and should not replace real-world testing and testing involving humans, at least as long as actual humans are involved in the real-world environments and scenarios that AD / ADAS vehicles will be subjected to. The human factor should not be forgotten, and human creativity should not be overlooked.

[0032] Rule-based control algorithm design is limited by the initial settings of all rules, and when multiple rules need to be considered, it can be easy to omit or involve design errors. Formal methods are one of the ways to solve the verification challenges of testing complex rule-based control algorithms, but it requires good modeling of the algorithms. Machine learning-based control algorithms are limited by their training samples. AD / ADAS system testers usually design test scenarios or drive vehicles on public roads to collect scenarios. However, some scenarios, including extreme case scenarios, cannot be well covered by design or real-world driving, but they are very valuable for AD / ADAS system and function verification.

[0033] In fully computer-controlled dynamic environments (e.g., with moving AI-controlled objects trained based on datasets generated from real-world driving), these typically do not include rare safety-critical behaviors, and there is typically no or insufficient data on corner cases. The interaction of computer-controlled objects with AVs can ultimately generate some combinatorially interesting test scenarios, but it cannot generate all corner case scenarios that might occur in the real world and might be of interest. For example, it would be interesting to be able to increase the possibility of generating corner case scenarios without having to drive in the real world using real-world vehicles (at least not for decades or centuries).

[0034] Crowdsourcing is a process that involves outsourcing tasks to a decentralized group of people. The basic idea of ​​the embodiments herein is to facilitate crowdsourcing the task of validating AD / ADAS functionality to individuals around the world who may be interested in or motivated by other means of facilitating the imposition of risky and potentially safety-critical scenarios on vehicles with AD / ADAS functionality, such as in simulation environments or video games.

[0035] Figure 1 is a block diagram schematically depicting a system 100 for discussing embodiments herein.

[0036] System 100 involves one or more client or user devices, or simply devices, illustrated by devices 101-103 in the figure, and can operate according to the embodiments herein. Devices 101-103 are each associated with a user and can be of different types, such as smartphones, laptop or desktop personal computers, game consoles, etc. Devices 101-103 therefore have computer and data processing capabilities. In the example shown, to illustrate the principle, devices 101 and 103 are illustrated as smartphones, and device 102 is illustrated as another type, such as a desktop computer. Therefore, one or more of devices 101-103 can be, but are not limited to, devices with wireless communication capabilities, i.e., wireless communication devices. When discussing the embodiments, device 101 will be used as the primary example below. Devices 101-103 can further be connected to a computer network 104 (e.g., the Internet) and can, for example, be communicatively connected to a server 105 or server device via the network. The server 105 can be used for remote processing and / or storage of data related to devices 101-103, as will be discussed further below. The server 105 may serve a plurality of devices, for example all devices 101 - 103. The server may correspond to a computer or host with server capabilities, or a cluster of such computers, and may be part of or correspond to a so-called computer cloud, or simply a cloud, that provides server services.

[0037] As will be explained in further detail below, each of the devices 101-103 (e.g., device 101) can provide a virtual environment, for example by executing software on the device 101, or provide access to the virtual environment, for example, provided by the server 105. The virtual environment simulates an environment related to the operation of (multiple) vehicles with AD / ADAS functions. The virtual environment can be provided by or through the software executed on the device 101, for example in the form of a computer program (e.g., a so-called application or "app") that can be installed and executed on the device 101, for example corresponding to a simulator and / or computer game that can motivate the user of the software and the device 101 to virtually challenge a vehicle equipped with AD / ADAS software with difficult to manage and / or risky scenarios.

[0038] It should be understood that the types of user devices mentioned above each include a user interface that allows different types of user input and output to the device. The user interface can be of different types, including conventional types for interfacing with a user, such as a touch-sensitive display (i.e., a screen), a keyboard, action buttons and / or sticks such as on a game controller, a speaker, for example, a microphone with voice recognition, etc.

[0039] Figure 2a An example of such a virtual environment mentioned above, here virtual environment 200, is schematically illustrated. As illustrated, virtual environment 200 includes a virtual infrastructure comprising a road for driving a virtual vehicle (e.g., a virtual car), and associated structures adjacent to and near the road, such as in the form of buildings, crosswalks, sidewalks, traffic lights, etc.

[0040] Figure 2b Pictured Figure 2a 2 , the virtual environment 200 is now populated with computer-controlled, movable virtual objects 230 a-c, which may be exemplified as artificial intelligence (AI)-controlled cars, virtual AD / ADAS vehicles 210, which may be exemplified as AV cars (configured to operate according to the AD / ADAS functionality), and human-controlled, movable virtual objects 220 a-c, which may be exemplified as human-controlled (i.e., human driver (HD)) cars. The objects and vehicles 210, 220 a-c, 230 a-c are operating and / or are configured to operate in the virtual environment 200.

[0041] The virtual AD / ADAS vehicle 210 may execute and / or operate according to a particular version of AD / ADAS software that provides AD / ADAS functionality.

[0042] The computer-controlled movable virtual objects 230a-c can be fully or partially AI-controlled, meaning their behavior can be simulated using AI algorithms. AI algorithms can range from complex supervised machine learning algorithms trained on real-world data to reinforcement learning techniques that explore simulated environments and / or can learn to challenge, for example, a virtual AD / ADAS vehicle 210. The computer-controlled movable virtual objects 230a-c can include, for example, pedestrians, cyclists, vehicles, and the like. The computer-controlled movable virtual objects 230a-c can be fully or completely computer-controlled during operation, meaning they can perform actions and movements without human control during operation in the virtual environment 200. Alternatively, they can be referred to as fully computer-operated virtual objects or autonomous virtual vehicles. Therefore, the fully computer-controlled movable virtual objects 230a-c can be operated independently of a human user. In a computer game context, they can correspond to so-called non-player characters (NPCs).

[0043] A user of the devices 101-103 can control, for example, the human-controlled movable virtual objects 220a-c in the virtual environment 200 via a user interface of the device, and thereby cause one or more of these to interact with and / or affect the virtual environment 200 itself, such as its infrastructure, computer-controlled virtual objects 230a-c, and / or the virtual AD / ADAS vehicle 210 and / or other human-controlled movable virtual object(s). The interactions or how the human-controlled movable virtual object(s) affect the virtual environment 200 itself, other objects and / or vehicles operating therein can directly or indirectly cause the AD / ADAS vehicle to be subjected to (i.e., be exposed to and / or experience) various situations and scenarios when operating in the virtual environment 200. It is noted that a human (e.g., a user of device 101) is not required to control every aspect of a movable virtual object (e.g., human-controlled movable virtual object 220a) in order for it to be human-controlled, but rather, for example, the user control may control the object's movement behavior in some aspect, and this may cause the effects and / or interactions described during operation and thus be a result of the user control. Generally speaking, during operation, human control may be real-time or near real-time, or have some delay, and / or the control may be set or configured by a human (e.g., a user of device 101) prior to operation.

[0044] (Multiple) human-controlled movable virtual objects develop opportunities for users to participate and effectively act as testers with respect to AD / ADAS functionality, although this does not necessarily need to be the focus of the user, who may instead participate in a simulator and / or computer game. A user of a device (e.g., device 101) can, for example, control the human-controlled movable virtual object 220a as part of such a simulator and / or computer game, and, for example, determine (such as select) at least a portion of the virtual environment 200 in which the human-controlled movable virtual object 220a will operate, for example as part of a configuration of the simulator and / or computer game that affects how it will operate in the virtual environment 200.

[0045] A user of a device (e.g., device 101) can further load and / or influence the behavior of computer-controlled movable virtual objects (e.g., 230a-c, such as AI road users) in a simulator and / or computer game, for example, in order to set up scenarios where the goal is to expose the virtual AD / ADAS vehicle 210 to situations or scenarios of interest, for example corresponding to extreme case scenarios.

[0046] As will be appreciated, existing software in the form of simulators and / or computer games provided through or by server(s) may be modified to provide a virtual environment as virtual environment 200 and implement the embodiments herein as described below.

[0047] Notice, Figure 1 and Figure 2a 、 Figure 2b The figures are merely schematic and for illustrative purposes, and not everything shown in the figures may be required for all embodiments herein, as will be apparent to those skilled in the art. Furthermore, in practice, there may of course be more devices, multiple servers, other more complex virtual environments, etc., but for simplicity, these are not shown herein. In practice, there may be, for example, a much greater number of devices and corresponding users.

[0048] Figure 3 Depicted is a flow chart in combination with a signaling diagram that will be used to discuss embodiments herein.

[0049] The following actions may form a method for supporting generation of scenarios for testing autonomous driving and / or advanced driver assistance systems (AD / ADAS) functionality of one or more real-world vehicles. The actions are performed by a server (eg, server 105).

[0050] The following actions may be performed in any suitable order, and / or with full or partial overlap in time when this is possible and appropriate.

[0051] Action 301

[0052] The server 105 provides a virtual environment (e.g., virtual environment 200) that simulates an environment associated with the operation of one or more vehicles (e.g., the real-world vehicles) having the AD / ADAS functionality. In the virtual environment 200, the following are being operated: one or more fully computer-controlled movable virtual objects (e.g., computer-controlled movable virtual objects 230a-c), one or more human-controlled movable virtual objects (e.g., human-controlled movable virtual objects 220a-c), and at least one virtual AD / ADAS vehicle (e.g., virtual AD / ADAS vehicle 210) operating according to the AD / ADAS functionality.

[0053] AD / ADAS functionality may fully or partially correspond to AD / ADAS functionality of a real-world vehicle, or be associated with the real-world vehicle, or be programmed to be a real-world vehicle. As will be understood, "real-world vehicle" as used herein refers to a physical vehicle designed to operate in the real physical world in which humans are born, live, and directly interact.

[0054] The computer-controlled and / or human-controlled movable virtual object may, for example, correspond to or include a Figure 2a The discussed and illustrated vehicle(s) may additionally or alternatively include or be other object(s), such as virtual human(s), for example, walking and / or running pedestrians with, for example, strollers(s), flying objects, such as drones, or thrown or fallen objects (such as rocks or trees) that can be moved by a user and placed, for example, on a road in a virtual environment. As will be appreciated, movable virtual objects may correspond to different kinds of road users.

[0055] All or some of the one or more fully computer-controlled movable virtual objects can advantageously be operated by implementing the computer-controlled unpredictable behavior. Therefore, repeated, predictable behavior can be avoided so that when the software (e.g., game) implementing the embodiments herein is executed on a device operated by a user, these objects will not be repeatedly executed in the same way on different occasions. In order to achieve this, the computer-controlled movable virtual objects or at least their behavior can be implemented based on an AI algorithm, and / or implemented with a certain degree of randomized behavior. The former can have some other advantages, in that the algorithm can learn from previous occasions that, for example, generate certain (e.g., particularly interesting) scenes (e.g., extreme case scenes as described below). Machine learning algorithms can be used to influence and promote behavior, which increases the possibility of generating other similar scenes of interest. In any case, by avoiding repetitive behavior, there will be greater variation among the generated scenes, as well as the increased possibility of generating scenes of interest (e.g., extreme case scenes).

[0056] The human-controlled movable virtual object may be or include a virtual AD / ADAS vehicle, eg, at least partially human-controlled if not fully autonomous, but may also be or include another vehicle or object.

[0057] In some embodiments, one or more fully computer-controlled movable virtual objects include one or more of the at least one virtual AD / ADAS vehicle, such as virtual AD / ADAS vehicle 210, i.e., the AD / ADAS vehicle is not user-centric at that time. Where AD / ADAS functionality does not include autonomous driving functionality, such functionality may be provided separately, for example, by integrating AD / ADAS functionality into one of the fully computer-controlled movable virtual objects, such as in the form of a virtual vehicle that has been configured to allow for this. For further discussion and associated advantages, see below under action 303.

[0058] Action 302

[0059] The server 105 allows devices (e.g., devices 101-103) to remotely connect to the server 105 and users of the devices 101-103 to control the human-controlled movable virtual objects 220a-c, respectively, in the virtual environment 200 via the user interfaces of the devices 101-103. This enables the generation of a scenario that one or more of the at least one virtual AD / ADAS vehicle 210 experiences, and that is derived from at least one of the one or more human-controlled movable virtual objects 220a-c, that affects one or more of: the one or more fully computer-controlled movable virtual objects 230a-c, the at least one virtual AD / ADAS vehicle 210, and the virtual environment 200.

[0060] Thus, the above scenario is caused by at least one user controlling a human-controlled movable virtual object (e.g., any of 220a-c) during operation. This causes direct interaction between the virtual AD / ADAS vehicle 210 or the virtual objects 220a-c, or indirectly via computer-controlled movable virtual objects 230a-c, another human-controlled movable virtual object (e.g., controlled by another user), and / or the virtual environment 200. Direct or indirect interaction with the virtual AD / ADAS vehicle 210 may include interaction with AD / ADAS functionality of the virtual AD / ADAS vehicle.

[0061] As should be understood, the scenario experienced by the virtual AD / ADAS vehicle 210 generally corresponds to a series of events during a continuous time period in this document. In addition, as can be appreciated from the above, each scenario generated by the embodiments herein should be initiated by and can be considered to begin with an event in which a user controls at least one of the human-controlled movable virtual objects through user input via a user interface of the device, thereby causing it to affect the following: one or more of the fully computer-controlled movable virtual objects, the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210), and / or the virtual environment 200, thereby causing the virtual AD / ADAS vehicle 210 to become directly affected or indirectly affected via a series of other events. In order to determine whether the virtual AD / ADAS vehicle 210 is affected by (multiple) human-controlled movable objects, it can be compared with a situation (e.g., a hypothetical or simulated situation) in which no user control is involved and therefore no user-induced events are present, and for example, to see whether the virtual AD / ADAS vehicle and / or AD / ADAS function react and / or behave differently. It should be appreciated that during a period of operation in the virtual environment 200, a plurality of such scenarios will typically be generated, although not all will be of interest, and a smaller number will correspond to extreme case scenarios. However, the greater the number of scenarios generated and the greater the variation among the generated scenarios, the greater the likelihood of generating interesting and extreme case scenarios. Multiple users on multiple devices (where the user of each device controls one or more human-controlled, movable virtual objects in the virtual environment 200, along with fully computer-controlled, movable virtual objects also operating therein) will facilitate and support the generation of such scenarios, respectively.

[0062] In the case where multiple virtual AD / ADAS vehicles are operating in the virtual environment 200, scenarios can be generated and / or evaluated separately for each virtual AD / ADAS vehicle. That is, a scenario can be generated for each virtual AD / ADAS vehicle, and the generated scenario need not involve multiple virtual AD / ADAS vehicles. Thus, the induced scenario generation may involve any one of the multiple virtual AD / ADAS vehicles, or may generally involve one or more of the multiple virtual AD / ADAS vehicles.

[0063] Furthermore, as should be understood from the above, embodiments herein expose a virtual AD / ADAS vehicle to a mix of human and computer-controlled movable virtual objects in a virtual environment, and scenarios are generated within their context.

[0064] The embodiments herein support and facilitate the generation of scenarios that AD / ADAS vehicles will experience and that can be used for repeated testing of AD / ADAS functionality. The generated scenarios can be considered to correspond to, for example, reusable test cases. Compared to scenarios with only computer-controlled environments and behaviors, the embodiments herein can support the generation of test cases that include scenarios that can be considered to be closer to real-world scenarios, and can also increase the possibility of generating extreme case scenarios. This is because humans and human behavior exist in the real world, and their influence should also exist virtually, especially if the goal is to generate scenarios related to AD / ADAS functionality for real-world vehicles. The human influence and unpredictability in control and behavior should enable greater variation in the scenarios that can be generated. As a result, human creativity can also be better utilized, which is not easy to achieve in scenarios with full computer control. The greater variation and creativity involved is believed to increase the possibility of extreme cases, especially in scenarios with many contributing users, such as in crowdsourcing scenarios.

[0065] It should be noted that in some embodiments, the provided virtual environment (e.g., 200) can be at least partially visualized to the user(s) through a corresponding physical environment, and / or one or more of the virtual objects (e.g., vehicles) (e.g., 220a) can be at least partially visualized to the user(s) through a corresponding physical object (e.g., vehicle). For example, a user can control and / or view a physical (e.g., radio-controlled) vehicle in at least a portion of the physical environment corresponding to at least a portion of the virtual environment 200, rather than just virtually, through radio controls and / or augmented reality (AR) on a device (e.g., device 101). In such embodiments, the physical representation of the virtual environment 200 and / or objects therein can be considered part of the user interface, i.e., how the user(s) view and / or interact with the virtual environment and / or objects therein (e.g., vehicles).

[0066] In some embodiments, the fully computer-controlled movable virtual objects (e.g., 230a-c) include the at least one virtual AD / ADAS vehicle (e.g., 210). In other words, the one or more human-controlled movable virtual objects (e.g., 220a-c) can exclude the virtual AD / ADAS vehicle 210, which is therefore not controlled by the user, i.e., not user-centric. Such embodiments are indicated above under action 301. The advantage is that this facilitates implementation of the embodiments herein in or using existing software or software platforms (e.g., computer games) (possibly in combination with associated hardware); and can include computer-controlled vehicles that are not necessarily user-centric, but instead can correspond to NPCs that the user can directly or indirectly influence and / or be influenced by. For example, if such existing software is adapted to include one or more computer-controlled vehicles equipped with (e.g., via a push download to the software and / or hardware) AD / ADAS functionality, or such that the one or more computer-controlled vehicles are equipped with the AD / ADAS functionality and thereby become (multiple) virtual AD / ADAS vehicles, this should facilitate both the implementation of the embodiments herein and a crowdsourcing effect, thereby facilitating the generation of increasingly varied scenarios, including extreme case scenarios. For example, if the at least one AD / ADAS vehicle or some(s) of the AD / ADAS vehicles are not controlled by (multiple) users, there can be more operating AD / ADAS vehicles than users, simultaneously in the virtual environment 200, and the number of AD / ADAS vehicles does not need to scale with the number of users.

[0067] In the embodiments herein implemented in or using existing software or software platforms (e.g., a computer game) (possibly in conjunction with associated hardware), the involvement of (multiple) AD / ADAS vehicles and AD / ADAS functionality need not even be known to the user, but rather are agreed upon and implemented, for example, in collaboration with the provider of the computer game, and the generation of scenarios can then follow as a side effect as the user operates a character corresponding to a human-controlled movable virtual object in the virtual environment of the computer game.

[0068] One or more of the at least one virtual AD / ADAS vehicle (e.g., virtual AD / ADAS vehicle 210) can be provided with a corresponding identifier during operation in the virtual environment 200, which enables a user of the devices 101-103 to identify such virtual AD / ADAS vehicle as a certain type of virtual vehicle operating in the virtual environment 200 via the user interface of the device.

[0069] In this way, the virtual AD / ADAS vehicle 210 can be identified by the user, for example, as a virtual vehicle specifically for generating a scenario related thereto. Therefore, the identifier should enable the user to identify the virtual AD / ADAS vehicle itself and thereby distinguish it from other similar vehicles that might otherwise be confused with, for example, other vehicles corresponding to human-controlled movable virtual objects (e.g., 220 a-c) and computer-controlled movable virtual objects (e.g., 230 a-c).

[0070] In some embodiments, the user may be hinted as to which vehicle is the AD / ADAS vehicle without having to provide it with an explicit identifier as described above. For example, when the AD / ADAS vehicle is a vehicle in which the user has a viewpoint, the user may be hinted as being the AD / ADAS vehicle, and thus, at least not from the user's perspective, there is no need to explicitly identify the vehicle as an AD / ADAS vehicle. In some embodiments, it may simply be to make the (multiple) AD / ADAS vehicles resemble real-world AVs without visible drivers and may be identified as such. However, in the case of multiple other vehicles in the virtual environment 200, user-induced scenario generation may be facilitated if, for example, identification of which vehicle is the AD / ADAS vehicle 210 is made easier by the identifier.

[0071] Action 303

[0072] As indicated, one or more of the human-controlled movable virtual objects 220a-c can be human-controlled virtual vehicles, and one or more of the fully computer-controlled virtual movable objects 230a-c can be fully computer-controlled virtual vehicles. In such embodiments, the server 105 can control the ratio of human-controlled virtual vehicles to fully computer-controlled virtual vehicles operating in the virtual environment 200.

[0073] In other words, control belongs to the ratio itself, i.e., it should be controlled purposefully and not as a side effect. Control can advantageously be achieved by adding or removing fully computer-controlled virtual vehicles from the virtual environment based on the number of human-controlled virtual vehicles. It can also be achieved by alternatively or additionally controlling the number of human-controlled virtual vehicles (e.g., the number of such vehicles allowed per user) and / or controlling the number of remotely connected devices.

[0074] Even if based on data from real human behavior in real-world situations and, for example, artificial intelligence that simulates human behavior, all computer-controlled environments must typically be based on data collected from a very large number of individuals over a very long period of time in order to be able to generate behavior that resembles human behavior. Such data is typically not available or would require a very long period of time and / or is very difficult to generate in the real world.

[0075] The amount of AD / ADAS vehicles, and particularly AVs, in traffic will affect human behavior in traffic. It should be understood that this is particularly difficult to test in the real world, as well as in a fully computer-controlled virtual environment. The mix of AD / ADAS vehicles and AVs, as well as human-controlled vehicles, is expected to change over time in the real world, increasingly favoring AD / ADAS vehicles and AVs. Therefore, it is of great interest to be able to test and generate scenarios at different ratios between human-controlled virtual vehicles and fully computer-controlled virtual vehicles before they appear in the real world. For example, a certain ratio can be maintained over a period of time and scenarios generated. In other words, the ratio can be controlled to maintain a certain level. The level can be predefined or predetermined and, for example, remain the same over a certain period of time during which the scenario is generated. For example, the desired ratio can be set to a certain number or range and then controlled to remain within this number or range.

[0076] For the reasons indicated above, and to better simulate real-world situations in which human drivers are typically able to identify AVs, it may be advantageous if a user can distinguish between fully computer-controlled vehicles and human-controlled vehicles in the virtual environment 200. Thus, in some embodiments, to facilitate this, one or more of the fully computer-controlled movable virtual objects (e.g., 230a-c) are fully computer-controlled virtual vehicles that are each provided with an identifier during operation in the virtual environment 200 that enables a user of a device 101-103 to identify, via a user interface of the device 101-103, that these are fully computer-controlled. And / or, in some embodiments, one or more of the human-controlled movable virtual objects (e.g., 220a, c) are vehicles that are each provided with an identifier during operation in the virtual environment 200 that enables a user to identify, via a user interface of the device (101-103), that these are human-controlled.

[0077] Action 304

[0078] The server 105 may initiate identification or identify that a certain scenario has occurred. The identification may be based on data generated in relation to one or more of the at least one virtual AD / ADAS vehicle (eg, the virtual AD / ADAS vehicle 210).

[0079] The identified scenario should be a scenario of interest. A scenario can be a scenario of a certain type or kind and can be determined based on predefined criteria and / or user input. For example, it can be a new scenario compared to previously generated and / or stored scenarios. A scenario can be a scenario or candidate scenario for a test case(s) and / or can be a corner case scenario or a candidate for a corner case scenario. Scenarios of interest are discussed further below.

[0080] Data generated in relation to the AD / ADAS vehicle 210 may be data generated by or in a portion of software that is related to the AD / ADAS vehicle and / or that provides AD / ADAS functionality, and / or is related to another vehicle, object, or infrastructure that interacts with the AD / ADAS vehicle or is within a predefined or predetermined vicinity of the AD / ADAS vehicle.

[0081] Thus, in some embodiments, the data generated in relation to the one or more of the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210) includes data generated externally from the one or more of the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210). That is, data generated externally from the virtual AD / ADAS vehicle(s) experiencing the certain scenario.

[0082] Moreover, as indicated below, externally generated data (e.g., sensory data) can also be used for some functions of the AD / ADAS vehicle 210, for example, where it is not desirable or possible to provide it with full or complete AD / ADAS functionality as it would in a real vehicle. That is, instead of simulating real-world sensors that provide data to the AD / ADAS functionality (i.e., to the AD / ADAS functionality to be operated on) when in a real-world vehicle, such data can instead be provided by, e.g., extracted from, a device executing software implementing embodiments herein, and such data can be based, for example, on how the AD / ADAS vehicle (e.g., its boundaries, position, and / or speed) relates to the virtual environment and other (e.g., moving) objects therein.

[0083] The identification itself can be performed by server 105 and / or performed in whole or in part by a device involved in scene generation (e.g., device 101), with or without user participation. When a device is involved, data can be transferred to / from the device (e.g., 101). In some embodiments, identification can involve remote processing, such as transferring that data to / from another server that assists or performs the identification.

[0084] In some embodiments, a device (e.g., device 101) may initiate identification, e.g., through user engagement via a user interface of device 101 (e.g., exemplified below), and then report and transmit information about this to server 105, which may then initiate and / or perform the actual identification based on data (e.g., temporarily stored) on server 105. Server 105 may evaluate and, e.g., rank such candidate scenarios in comparison to other scenarios that have been previously generated, evaluated, and / or stored.

[0085] For example, the identification can be based on: the software providing AD / ADAS functionality when executed on the server 105 or on the device 101 signals that it is unable to handle a situation as it should or that a problematic event has occurred (e.g., a collision), and / or some portion of the software monitors what the AD / ADAS vehicle involved has caused and / or is causing when interacting with other vehicles, objects, and infrastructure in the virtual environment, and generates data identifying certain events (e.g., a collision). The identification can be further based on predefined criteria, which can, for example, include the occurrence of such a certain event and / or the occurrence of a certain signal from the software providing AD / ADAS functionality. In some embodiments, the predefined criteria can include comparing the generated data with a previously stored scenario and seeing if it is sufficiently different in one or more aspects and is therefore or could be of interest.

[0086] As already indicated above, the identification can be based on user input via the user interface. The user input for scene identification can be part of the information used to identify the scene of interest. There may be, for example, a replay presented to the user via the user interface of the device 101, such as a replay within the following period: during this period, the user has controlled the human-controlled movable virtual object 220a in the virtual environment 200 and thereby may have directly or indirectly affected the virtual AD / ADAS vehicle 210. Then, for example, the user may be allowed to mark the (multiple) scenes that the user considers interesting via the user interface, for example by selecting a timestamp. Another option is that the server 105 suggests scene candidates to the user via the user interface, and the user can then review the candidates via the user interface and thereafter select and / or rate (if any) the (multiple) scenes that the user considers interesting. The results may then be transmitted to the server 105 for further evaluation and / or storage.

[0087] In summary, the identification of the certain scenario may be based on identifying the scenario of information received from one or more of the devices 101 - 103 .

[0088] To identify a scenario, particularly if it is based on input from the user(s) involved in causing the scenario, using input from the user of the device (e.g., the user of device 101) has several advantages, particularly for identifying edge case scenarios. The user is the one who best understands what he / she has done and identifies scenarios that "stand out" for some reason, which can typically be cases for edge case scenarios. As explained, these scenarios are often rare and have quite unique properties, which can make it difficult to automatically identify many such scenarios.

[0089] Identifying certain scenarios among a large number of scenarios, with or without user input, should further result in a reduced number of scenarios to store, at least for a long period of time, and / or further automatically and / or manually evaluate or identify, for example, on server 105.

[0090] Action 305

[0091] The server 105 may send information identifying the device participating in generating the certain scene to the device (eg, device 101). The device should be a device that is used to control a movable virtual object (eg, 220a) controlled by a human, thereby generating the certain scene.

[0092] The device 101 may receive this information via its user interface and in response to the received information, and then, for example, provide a notification directed to the user regarding the user's participation in generating the certain scenario. That is, the user may be notified via the user interface that he / she has participated in generating a certain scenario that has been identified, such as a scenario of particular interest, such as an extreme case scenario.

[0093] As will be appreciated, the identification itself may have been performed by the device 101 , with or without user participation, and / or by the server 105 .

[0094] Feedback to the user like this further supports similar behavior by the user, which, for example, leads to the identified extreme case scenario. It also enables gamification with respect to scenario generation, where the user associated with the feedback can be rewarded. For example, there may be ratings and / or points awarded to the user, and this is related to the number of certain identified scenarios, such as scenarios that the user has participated in generating that are identified as interesting. Rewards can also be based on the classification of scenarios, with some categories being rewarded better. For example, if a scenario belongs to a category or type of scenario that is considered an extreme case and / or is more likely to occur in the real world, it may be rewarded to a greater extent. Scenarios in which the user has participated in creating a virtual environment related to the scenario may belong to a category that may also be rewarded to a greater extent. This makes it possible to influence the number of scenarios that occur for certain categories or types of scenarios, and thereby influence the likelihood of generating interesting scenarios and / or generating certain desired categories or types of scenarios.

[0095] Action 306

[0096] The server 105 may store data that enables the re-creation of at least a portion of the certain scenario, thereby enabling the same or another virtual AD / ADAS vehicle to experience the at least a portion of the certain scenario on another occasion.

[0097] The at least part of a certain scenario means that the certain scenario can be partially or completely recreated, for example, a part thereof that is considered sufficient for a test case and / or a certain test purpose.

[0098] For example, the recreation of a scenario corresponding to a test case enables repeated testing or further testing of the AD / ADAS function by reusing the scenario. The recreated scenario can be used to subject the same or another AD / ADAS vehicle having the same or another (e.g., updated) AD / ADAS function to a scenario corresponding to, for example, an extreme case scenario. When provided by another software and / or when related to another vehicle, a scenario of interest for testing a specific AD / ADAS function is typically also of interest for testing this AD / ADAS function and similar or related AD / ADAS functions. Therefore, storing data that allows the scenario to be recreated and thereby reused may be more valuable than data related to the behavior of the function (e.g., testing of the AD / ADAS function itself).

[0099] It should be appreciated that, in addition to supporting the generation of scenarios for concurrently testing AD / ADAS functionality, embodiments herein may also test and generate data related to the functionality itself implemented by the virtual AD / ADAS vehicle.

[0100] It is further within the capabilities of a skilled person, such as a programmer providing software implementing the embodiments herein and for execution on a device, to identify and, if necessary, generate data to be stored and to allow for said recreation.

[0101] It is noted that the present action, or generally the storage of scene data (such as data enabling the reconstruction of at least part of the scene), can be performed in response to an identification that a scene of interest has occurred and is related to the identified scene of interest, i.e., can be in response to action 304. It may be sufficient and beneficial to store only the data for recreating the scene of interest. However, in some embodiments, the scene data may first be stored at least temporarily, and then, for example, in the case of a comparison with other stored scene data, it may be determined whether to delete or retain the stored scene data, for example in the case where it is thereby identified as being of interest. In these embodiments, the storage may be remote from the device, for example on a remote server (such as server 105). Moreover, in these cases, the identification that the scene is a scene of interest may be performed completely or partially remotely (e.g., on server 105).

[0102] As will be appreciated, the embodiments herein provide an opportunity to test and utilize AD / ADAS functionality of a vehicle using, for example, a simulation platform at different levels. Perhaps the simplest and potentially most straightforward level is to equip a virtual AD / ADAS vehicle (e.g., virtual AD / ADAS vehicle 210) with decision and control algorithms to test various functionality, such as ADAS functionality such as an automated emergency braking (AEB) system or adaptive cruise control (ACC), etc. At this level, the virtual AD / ADAS vehicle does not need to be equipped with a sensing (such as a perception) system, since information describing this (e.g., information about the surrounding environment, such as other road users and infrastructure, distances, relative speeds, etc.) can be extracted from and / or within the virtual environment, or more precisely, from within software executed on a device, such as software that implements the embodiments and provides the virtual environment, etc.

[0103] It should be appreciated that AD / ADAS software companies thus have the opportunity to test, for example, different dimensions of software in a virtual environment and equip a virtual AD / ADAS vehicle with certain AD / ADAS functionality. For example, if it is desired to test or validate a perception algorithm, such an algorithm can be inserted (i.e., loaded) into the virtual AD / ADAS vehicle 210.

[0104] The embodiments herein also enable deployment of several versions of the same AD / ADAS functionality, for example, provided by different AD / ADAS software, in the same virtual environment, such as in the virtual environment 200 on the server 105. If, for example, the virtual environment is part of a simulator and / or computer game software as discussed above, then different versions of the AD / ADAS functionality can be pushed to the server executing the simulator and / or computer game corresponding to the server 105 and used in different AD / ADAS vehicles operating in the simulator and / or game in the virtual environment corresponding to the virtual environment 200.

[0105] As described above or generally, loading and / or configuring the AD / ADAS functionality of the AD / ADAS vehicle (e.g., in the form of code implementing it) can be done with or without the user's knowledge and / or being explicitly informed of it or without being explicitly informed of it. It may, for example, be done only on the server 105 and / or by download, for example, pushed by the server 105 to the (multiple) device, for example during and / or as part of action 301. Whether the download should be performed to the (multiple) device, or whether it is sufficient to involve only the server 105, can be implementation-specific and, for example, depends on how the simulator and / or computer game software with which the embodiments herein may be integrated is implemented.

[0106] Therefore, in some embodiments, the at least one virtual AD / ADAS vehicle (e.g., 210) is a plurality of virtual AD / ADAS vehicles. These may be configured to operate according to different implementations of the AD / ADAS functionality. Different implementations may correspond to different versions of software that provide the AD / ADAS functionality. Therefore, in these embodiments, the AD / ADAS functionality may be the same or substantially the same for each of the plurality of virtual AD / ADAS vehicles.

[0107] As indicated above, the advantage is that, for example, different software versions can be used and / or tested simultaneously in the virtual environment 200 and the data and / or results compared. This also increases the impact from each user or device, and the crowdsourcing impact is further increased because each user or device contributes to a greater exposure of the AD / ADAS vehicle to scenarios for testing AD / ADAS functionality.

[0108] In addition, in some embodiments, the at least one virtual AD / ADAS vehicle (e.g., 210) is a plurality of virtual AD / ADAS vehicles configured to operate according to different sub-functions of the AD / ADAS function, respectively. Therefore, in these embodiments, the AD / ADAS function includes different sub-functions. Generally speaking, the AD / ADAS function as used herein may include several sub-functions. For example, in these embodiments, one or more of the plurality of virtual AD / ADAS vehicles may implement, for example, an ADAS function (such as AEB), and one or more of the other plurality of virtual AD / ADAS vehicles may implement, for example, an AD function. This increases the flexibility and versatility of the embodiments herein and increases the possibility of generating scenarios of interest. Functions can be tested in a virtual environment with respect to each other and how they may affect each other. This can also be utilized to increase the changes in generating scenarios of interest.

[0109] As already indicated, the embodiments herein, as well as those discussed above, support or even facilitate the generation of scenarios, including so-called corner case scenarios, that may be of interest to subject AD / ADAS vehicle 210, and other vehicles with AD / ADAS functionality, to. For example, by being in a virtual environment, (human) users will be less inclined to engage in potentially harmful or dangerous scenarios than in the real world, and this therefore increases the probability of generating such scenarios, which may be important for testing and correspond to corner case scenarios that are difficult and, for obvious reasons, are generally avoided in the real world. As explained above, corner case scenarios are scenarios that typically occur so rarely in the real world that, for example, an existing test fleet would need to be driven for decades or centuries in order to be exposed to such scenarios. In any computer-controlled environment (with objects trained based on a dataset that typically does not include rare safety-critical behaviors), the interaction of computer-controlled objects with AVs can ultimately generate some combination of interesting test scenarios, but it may not be able to generate corner case scenarios that are likely to occur in the real world. This is due to the fact that computer-controlled objects can only behave based on training datasets that do not include information about or behind edge case scenarios. Therefore, augmenting all computer-controlled environments with human-controlled objects, as in the embodiments herein, provides the opportunity to expose AD / ADAS vehicles and computer-controlled objects in all computer-controlled environments with behaviors that are outside the domain of what can be achieved with conventional training datasets.

[0110] Additionally, it should be appreciated that the mix of computer-controlled vehicles and human-controlled movable objects, as in the embodiments herein, can enable the generation of scenarios that, at least in some respects, more closely resemble real-world scenarios with human interaction than scenarios that could be generated in conventional, computer-controlled virtual test environments. All computer-controlled environments, even those based on data from actual human behavior in real-world situations and, for example, artificial intelligence that simulates human behavior, typically must be based on data collected from a very large number of individuals over a very long period of time in order to generate behavior that resembles human behavior. Such data is often unavailable, would require very long periods of time, and / or is very difficult to generate in the real world. And even if this were attempted, the mix of AD / ADAS vehicles and human-controlled vehicles in the real world is expected to change over time, thereby increasing the preference for AD / ADAS vehicles, and this ratio (i.e., the prevalence of AD / ADAS vehicles) will affect human behavior in traffic, as already mentioned above. Scenarios that consider human behavior in situations with a higher number of AD / ADAS vehicles than are currently available in the real world cannot rely on real-world data, but thanks to the embodiments herein they can be generated.

[0111] In addition, as indicated above, the embodiments herein allow for other such as already existing virtual environment platforms or simpler implementations in the program and / or with other such as already existing virtual environment platforms or the common implementation of the program, which are such as relevant to simulators (e.g., vehicle simulators) and / or computer games, particularly network-based gaming platforms. In other words, the embodiments herein are convenient for realizing in different situations, in different virtual environments and / or on different types of devices. This then makes it possible to attract more and different users to implement the method. The more personnel involved, the more different and varied the situations and scenes are, and the greater the possibility of generating valuable scenes (such as, extreme case scenes) thereby.

[0112] Thus, embodiments herein facilitate crowdsourced generation of scenarios for testing AD / ADAS functionality.

[0113] Furthermore, the embodiments herein should not only enable companies that develop software that provides AD / ADAS functionality to accelerate the evaluation of their software, but also significantly reduce costs by reducing the number of miles of human-supervised road testing required to validate AVs.

[0114] In summary, the embodiments herein provide improvements with respect to testing of AD / ADAS functionality in vehicles.

[0115] Figure 4 is a flowchart to further discuss and illustrate the embodiments herein from a more user-centric perspective, where the user here is illustrated as a user of device 101 .

[0116] The acts may be performed in any suitable order, and / or with full or partial overlap in time when this is possible and appropriate.

[0117] Action 401

[0118] The user may determine at least a portion of the virtual environment 200 via a user interface of the device 101 , such as selecting among alternatives available on the server 105 .

[0119] This action may correspond in whole or in part to action 301 above, and may result in Figure 2a Things schematically illustrated and exemplified in.

[0120] Action 402

[0121] A user may select one or more human-controlled movable virtual objects (e.g., human-controlled movable virtual object 220a) via a user interface of device 101 for control by the user while they operate in the virtual environment 200 (i.e., during operation), for example as part of a simulator and / or computer game executed on server 105 and a client portion thereof executed on device 101.

[0122] This action may partially correspond to actions 301 and 302 above.

[0123] Action 403

[0124] A user can define, via a user interface of device 101, the behavior of certain (multiple) computer-controlled movable virtual objects (e.g., computer-controlled movable virtual objects 230a-c) relating to how they will or should operate in the virtual environment 200, for example as part of a simulator and / or computer game executed on server 105 and its client portion executed on the device.

[0125] It is noted that when a user defines the behavior of a computer-controlled movable virtual object and as a result the user thereby actually controls the object in the virtual environment, such object may be considered both human-controlled and computer-controlled, although it may be fully computer-controlled during operation.

[0126] This action may partially correspond to actions 301 and 303 above.

[0127] Action 404

[0128] The user may load and / or position virtual AD / ADAS vehicle(s) (e.g., virtual AD / ADAS vehicle 210) via the user interface of the device 101 to operate in the virtual environment 200, e.g., as part of a simulator and / or computer game executing on the server 105 and a client portion thereof executing on the device 101. As previously indicated, such AD / ADAS vehicles may be both computer-controlled virtual vehicles and human-controlled virtual vehicles.

[0129] This action may partially correspond to action 301 above.

[0130] Actions 402-404 can result in Figure 2b Things schematically illustrated and exemplified in.

[0131] Actions 401-404 provide users with the opportunity to simulate or even recreate challenging or risky scenarios that vehicles have encountered in the real world, or to be inspired by such scenarios, and potentially even make those scenarios even more challenging. This can include the virtual environment itself in action 401, and / or road users such as pedestrians, vehicles, and AV(s) with and without AI support, such as in actions 402-404, which can be fully or partially based on the control in actions 402-404 of AI algorithms. For example, a user can define and / or control the behavior of certain road users and have other road users fully or partially controlled by AI algorithms and / or other users.

[0132] Action 405

[0133] Based on actions 401-404, movable virtual object(s) and vehicle(s) may be operated in the virtual environment, for example, computer-controlled movable virtual objects 230a-c, human-controlled movable virtual objects 220a-c, and virtual AD / ADAS vehicle 210 may be operated in the virtual environment 200. As mentioned above, this operation may be part of a simulator and / or computer game executed on the server 105.

[0134] This action may partially correspond to action 301 above.

[0135] Action 406

[0136] During operation in the virtual environment (e.g., during action 405), the server 105 and / or (multiple) devices (e.g., 101) can store data, which can be referred to as data logging. This data can relate to the behavior of (multiple) virtual AD / ADAS vehicles (e.g., virtual AD / ADAS vehicles 210), and / or data that enables partial or complete re-creation of (multiple) generated scenarios. (Multiple) scenarios are scenarios that (multiple) AD / ADAS vehicles experience during operation as part of a simulator and / or computer game that can be executed on the server 105 and its client portion executed on the device 101. The storage can therefore be performed locally, for example, on the device 101, and / or remotely, for example, on the server 105.

[0137] This action may partially correspond to action 305 above.

[0138] Action 407

[0139] During or after operation in the virtual environment (e.g., during or after action 405), one or more scenes may be identified. The identification may be performed from the stored data obtained by action 406. While real-time or near real-time identification may be possible (e.g., based on data temporarily stored in fast access memory), it may be preferred to store the data in action 406 as long as possible or as long as appropriate during a longer operation period or based on storage capabilities, and thereafter (e.g., after action 405), identify scenes from the stored data. Scenes identified and deemed of interest, or portions thereof, may then be saved, e.g., stored more permanently, which may involve moving data associated with such scenes from, e.g., local temporary storage to more permanent or long-term storage for further use of the data, e.g., on server 105 or at some other location. Thus, at least in some embodiments, the storage in action 406 may be responsive to the current action.

[0140] This action may correspond in whole or in part to action 304 above.

[0141] The above-mentioned user may, for example, be or correspond to a crowdsourcing user who acts as a tester and, for example, by means of actions 401-404, may have the possibility of creating or selecting, planning or at least participating in a traffic situation or traffic scenario that will or may lead to a scenario during operation in the virtual environment in action 405. The scenario may then be identified and related to the data stored as in actions 406-407.

[0142] A user may utilize embodiments herein to induce risky or potentially risky situations or scenarios for AD / ADAS vehicle(s) (eg, virtual AD / ADAS vehicle 210 ) to thereby challenge or test its behavior in such scenarios.

[0143] Even without the execution of one or more of actions 401-404, actions 405 and 406 and the embodiments herein can still be useful, for example, in the case of a predefined or predetermined virtual environment 200 and / or (multiple) computer-controlled movable virtual objects and / or human-controlled movable virtual objects 220a-c and / or virtual AD / ADAS vehicles. If the execution involves many devices and many users in a crowdsourcing manner, valuable data corresponding to many driving miles can be provided and / or stored, and this data increases the likelihood of generating, identifying, and storing interesting scenarios (e.g., edge case scenarios).

[0144] It is noted that while the embodiments herein may be particularly suitable for generating corner case scenarios that can be recreated and reused in test cases, for example, they may also be used to test specific AD / ADAS functionality itself in addition to conventional real-world test drives and testing, for example in fully computer-controlled and simulated scenarios.

[0145] Figure 5 is a schematic block diagram illustrating an embodiment of a server 500 that may correspond to server 105. The schematic block diagram also illustrates how server 500 (eg, server 105) may be configured to perform the above-described Figure 3 Examples of methods and actions discussed.

[0146] Thus, the server 500 is used to support the generation of scenarios for testing autonomous driving and / or advanced driver assistance systems (AD / ADAS) functionality of one or more real-world vehicles.

[0147] The server 500 may include a processing module 501, such as a device, one or more hardware modules including, for example, one or more processors, and / or one or more software modules for performing the methods and / or actions.

[0148] The server 500 may further include a memory 502, which may include (e.g., contain or store) a computer program 503. The computer program 503 includes "instructions" or "codes" that are directly or indirectly executable by the server 500 to perform the methods and / or actions. The memory 502 may include one or more memory units and may be further arranged to store data, such as configurations and / or applications involved in the embodiments herein or used to perform the functions and actions of the embodiments herein.

[0149] In addition, the server 500 may include (multiple) processors 504, i.e., one or more processors, as (multiple) exemplary hardware modules, and may include or correspond to one or more processing circuits. In some embodiments, the (multiple) processing modules 501 may include (multiple) processors 504, for example, be embodied in the form of (multiple) processors 504, or be implemented by (multiple) processors 504. In these embodiments, the memory 502 may include a computer program 503 executable by the (multiple) processors 504, whereby the server 500 is operative or configured to perform the method and / or its actions.

[0150] Typically, the server 500 (e.g., processing module(s) 501) includes input / output (I / O) module(s) 505 that are configured to be involved in (e.g., by executing) any communication to and / or from other units and / or devices, such as sending and / or receiving information to and / or from devices 101-103. Where applicable, the I / O module(s) 505 can be instantiated by obtaining (e.g., receiving) modules and / or providing (e.g., sending) modules.

[0151] The server 500 may further include a user interface that may be integrated with and / or communicatively connected to the server 500. The user interface may allow for different kinds of management actions (e.g., different kinds of control and configuration) via inputs and outputs to / from the server 500. The user interface may itself be of a conventional type for interfacing with a server.

[0152] In addition, in some embodiments, the server 500 (e.g., processing module(s) 501) includes one or more providing modules, allowing modules, storing modules, initiating modules, controlling modules, and sending modules as hardware and / or software modules for performing the actions of the embodiments described herein. These modules may be implemented in whole or in part by the processor(s) 504.

[0153] The server 500, and / or (multiple) processing modules 501, and / or (multiple) processors 504, and / or (multiple) I / O modules 505, and / or (multiple) providing modules can therefore be operative or configured to provide the virtual environment (e.g., virtual environment 200), which simulates the environment associated with the operation of the one or more vehicles having the AD / ADAS function, and in which it operates the following: the one or more computer-controlled movable virtual objects (e.g., 230a-c), the one or more human-controlled movable virtual objects (e.g., 220a-c), and the at least one virtual AD / ADAS vehicle (e.g., 210) operating according to the AD / ADAS function.

[0154] Furthermore, the server 500, and / or the processing module(s) 501, and / or the processor(s) 504, and / or the I / O module(s) 505, and / or the enabling module(s) may be operable or configured to enable the devices (e.g., devices 101-103) remotely connected to the server 500 and the users of the devices to control the human-controlled movable virtual objects 220a-c, respectively, via the user interfaces of the devices, during the operation in the virtual environment 200, thereby causing the generation of a scenario experienced by one or more of the at least one virtual AD / ADAS vehicle (e.g., the virtual AD / ADAS vehicle 210). The scenario is derived from at least one of the human-controlled movable virtual objects 220a-c that affects one or more of the following: the one or more fully computer-controlled movable virtual objects 230a-c, the at least one virtual AD / ADAS vehicle 210, and the virtual environment 200.

[0155] In addition, the server 500, and / or (multiple) processing modules 501, and / or (multiple) processors 504, and / or (multiple) I / O modules 505, and / or (multiple) initiating modules can be operative or configured to initiate the identification that the certain scenario has occurred.

[0156] The server 500, and / or (multiple) processing modules 501, and / or (multiple) processors 504, and / or (multiple) I / O modules 505 and / or (multiple) sending modules can be further operative or configured to send the information identifying the participation in generating the certain scene to the device (e.g., 101), and the device (e.g., 101) is used to control the human-controlled movable virtual object (e.g., 220a) and thereby generate the certain scene.

[0157] In some embodiments, the server 500, and / or processing module(s) 501, and / or processor(s) 504, and / or I / O module(s) 505, and / or control module(s) are further operative or configured to control the said ratio of human-controlled virtual vehicles to fully computer-controlled virtual vehicles operating in the virtual environment 200.

[0158] In addition, the server 500, and / or (multiple) processing modules 501, and / or (multiple) processors 504, and / or (multiple) I / O modules 505, and / or (multiple) storage modules may be operative or configured to store said data enabling the re-creation of said at least a portion of said certain scene.

[0159] Figure 6is a schematic diagram illustrating some embodiments related to a computer program and its carrier, so that the server 500 discussed above performs the methods and actions. The computer program may be computer program 503 and includes instructions that, when executed by processor(s) 504 and / or processing module(s) 501, cause the server 500 to perform as described above. In some embodiments, a carrier, or more specifically, a data carrier, such as a computer program product, is provided that includes the computer program. The carrier may be one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium, such as computer-readable storage medium 601, as schematically illustrated in the figure. Thus, the computer program 503 may be stored on the computer-readable storage medium 601. By carrier, transitory propagating signals may be excluded, and the data carrier may be correspondingly designated as a non-transitory data carrier. Non-limiting examples of data carriers that are computer-readable storage media are memory cards or sticks, disk storage media such as CDs or DVDs, or mass storage devices typically based on hard drive(s) or solid-state drive(s) (SSDs). Computer readable storage medium 601 can be used for storing data accessible by computer network 602 (for example, the Internet or local area network (LAN)). In addition, computer program 503 can be provided as (multiple) pure computer programs, or be included in one or more files. One or more files can be stored on computer readable storage medium 601, and for example, can be obtained by downloading, for example, by computer network 602 as indicated in the figure, for example, via a server. The server can be, for example, a web or file transfer protocol (FTP) server. One or more files can be, for example, executable files for downloading directly or indirectly to the first node and executing on the first node, so that it can be, for example, executed as described above by the execution of (multiple) processor 504. One or more files can also or alternatively be used for intermediate downloading and compiling involving (multiple) identical or other processors, so that they are executable before further downloading and execution, so that the server 500 can be executed as described above.

[0160] It is to be noted that any processing module(s) and circuit(s) mentioned above may be implemented as software and / or hardware modules, for example in existing hardware.

[0161] Those skilled in the art will also appreciate that the modules and circuits discussed herein may refer to a combination of hardware modules, software modules, analog and digital circuits, and / or one or more processors configured with, for example, software and / or firmware stored in a memory, which, when executed by one or more processors, may cause (multiple) nodes and (multiple) devices to be configured to and / or perform the above-described methods and actions.

[0162] The identification by any identifier herein may be implicit or explicit. The identification may be unique in a certain context, such as in a device, server or system, or at least in a relevant part or area thereof.

[0163] It is also noted that, while the terminology used herein may be particularly associated with and / or exemplified by certain communication systems or networks, this in itself should not be considered as limiting the scope of the embodiments herein to only such certain systems or networks, etc.

[0164] As used herein, the term "memory" may refer to a data storage device for storing digital information, typically a hard disk, magnetic storage, media, a portable computer floppy disk or disk, flash memory, random access memory (RAM), etc. In addition, the memory may be an internal register memory of a processor.

[0165] It should also be noted that any enumerated terms (such as first device, second device, etc.) should be considered non-restrictive, and the terms themselves do not imply a certain hierarchical relationship. In the absence of any clear information to the contrary, naming by enumeration should be considered as just a way to achieve different names.

[0166] As used herein, the expression “configured to” may mean that the device and / or processing circuit or module is configured or adapted by means of software and / or hardware configuration to perform one or more of the actions described herein.

[0167] As used herein, the term "number" or "value" may refer to any kind of number, such as a binary number, a real number, an imaginary number, or a rational number. In addition, a "number" or "value" may be one or more characters, such as a letter or a string of letters. Furthermore, a "number" or "value" may be represented by a string of bits.

[0168] As used herein, the expressions "may" and "in some embodiments" have been generally used to indicate that the features described can be combined with any other embodiments disclosed herein.

[0169] In the drawings, features that may be present only in some embodiments are often drawn using dotted or dashed lines.

[0170] As used herein, the terms "transmit" and "send" are generally interchangeable. These terms may include transmissions via broadcast, unicast, multicast, etc. In this context, a transmission via broadcast can be received and decoded by any authorized device within range. In the case of unicast, a specifically addressed device can receive and decode the transmission. In the case of multicast (e.g., multicast), a group of specifically addressed devices can receive and decode the transmission.

[0171] When the word "comprise" or "comprising" is used, it shall be interpreted as non-limiting, ie meaning "consisting at least of."

[0172] The embodiments herein are not limited to the embodiments described above. Various alternatives, modifications, and equivalents may be used. Therefore, the above embodiments should not be considered to limit the scope of the present disclosure, which is defined by the appended claims.

Claims

1. A method performed by a server (105; 500) for supporting generation of scenarios for testing software-implemented autonomous driving and / or advanced driver assistance systems (AD / ADAS) functionality of one or more real-world vehicles, wherein the method comprises: - providing (301) a virtual environment (200) simulating an environment associated with the operation of one or more vehicles having the software-implemented AD / ADAS functionality, and in which the following are operated: one or more fully computer-controlled movable virtual objects (230a-c), one or more human-controlled movable virtual objects (220a-c), and at least one virtual AD / ADAS vehicle (210) operating in accordance with the software-implemented AD / ADAS functionality, and - allowing (302) devices (101-103) remotely connected to the server (105; 500) and users of the devices (101-103) to control the human-controlled movable virtual objects (220a-c) respectively in the virtual environment (200) via user interfaces of the devices (101-103), and thereby causing a scene to be generated which is experienced by one or more of the at least one virtual AD / ADAS vehicle (210), and the scene being derived from at least one of the one or more human-controlled movable virtual objects (220a-c), the one or more human-controlled movable virtual objects (220a-c) affecting one or more of the following: the one or more fully computer-controlled movable virtual objects (230a-c), the at least one virtual AD / ADAS vehicle (210), the virtual environment (200), and - Initiating an identification that a certain scenario of interest has occurred (304), where the identification may be based on software providing AD / ADAS functionality when executed in the device (101) signaling that it is unable to handle the situation as it should or that a problematic event has occurred.

2. The method of claim 1, wherein the identification is based on data generated in relation to one or more of the at least one virtual AD / ADAS vehicle (210).

3. The method of claim 2, wherein the data generated in relation to the one or more of the at least one virtual AD / ADAS vehicle (210) comprises: Data generated externally from one or more of the at least one virtual AD / ADAS vehicle (210).

4. The method according to any one of claims 1-3, wherein said identifying is based on identifying a context of information received from one or more of said devices (101-103).

5. The method according to any one of claims 1 to 3, wherein the method further comprises: Information identifying a participant in generating the certain scene of interest is sent (305) to a device (101) that is used to control a human-controlled movable virtual object (220a) and thereby causes the generation of the certain scene of interest.

6. The method according to any one of claims 1 to 3, wherein the method further comprises: - Storing (306) data that enables re-creation of at least a portion of the certain scenario of interest and thereby enables the same or another virtual AD / ADAS vehicle (210) to be subjected to the at least a portion of the certain scenario of interest on another occasion.

7. The method according to any one of claims 1-3, wherein the fully computer-controlled movable virtual objects (230a-c) include the at least one virtual AD / ADAS vehicle (210).

8. A method according to any one of claims 1-3, wherein one or more of the at least one virtual AD / ADAS vehicle (210) is provided with a corresponding identifier during operation in the virtual environment (200), the identifier enabling a user of the device (101-103) to identify such virtual AD / ADAS vehicle as a certain type of virtual vehicle operating in the virtual environment (200) via a user interface of the device (101-103).

9. A method according to any one of claims 1 to 3, wherein one or more of the fully computer-controlled movable virtual objects (230a-c) are fully computer-controlled virtual vehicles, which are respectively provided with identifiers during operation in the virtual environment (200), said identifiers enabling a user of the device (101-103) to identify these as being fully computer-controlled via a user interface of the device (101-103), and / or wherein one or more of the human-controlled movable virtual objects (220a-c) are human-controlled movable virtual vehicles, which are respectively provided with identifiers during operation in the virtual environment (200), said identifiers enabling a user to identify these as being human-controlled via a user interface of the device (101-103).

10. The method of any one of claims 1 to 3, wherein one or more of the human-controlled movable virtual objects (220a-c) are human-controlled virtual vehicles (220a-c) and one or more of the fully computer-controlled virtual movable objects (230a-c) are fully computer-controlled virtual vehicles (230a-c), and wherein the method further comprises: Control (303) the ratio of human-controlled virtual vehicles (220a-c) to fully computer-controlled virtual vehicles (230a-c) operating in the virtual environment (200). The method according to claim 10 , wherein the ratio is controlled to be maintained at a certain level.

12. A computer program product (503) comprising instructions which, when executed by one or more processors (504), cause a server (500) to perform a method according to any one of claims 1-11.

13. A computer-readable storage medium storing computer instructions, characterized in that: When executed by a processor, the computer instructions implement the method according to any one of claims 1-11.

14. A server (105; 500) for supporting generation of scenarios for testing software-implemented autonomous driving and / or advanced driver assistance systems (AD / ADAS) functionality of one or more real-world vehicles, wherein the server (105; 500) is configured to: providing (301) a virtual environment (200) simulating an environment associated with the operation of one or more vehicles having the software-implemented AD / ADAS functionality, and in which the following are operated: one or more fully computer-controlled movable virtual objects (230a-c), one or more human-controlled movable virtual objects (220a-c), and at least one virtual AD / ADAS vehicle (210) operating in accordance with the software-implemented AD / ADAS functionality, and allowing (302) devices (101-103) to be remotely connected to the server (105; 500) and users of the devices (101-103) to control the human-controlled movable virtual objects (220a-c) respectively in the virtual environment (200) via user interfaces of the devices (101-103), and thereby causing a scene to be generated that is experienced by one or more of the at least one virtual AD / ADAS vehicle (210), and the scene being derived from at least one of the one or more human-controlled movable virtual objects (220), the one or more human-controlled movable virtual objects (220) affecting one or more of the following: the one or more fully computer-controlled movable virtual objects (230a-c), the at least one virtual AD / ADAS vehicle (210), the virtual environment (200), and - Initiating an identification that a certain scenario of interest has occurred (304), where the identification may be based on software providing AD / ADAS functionality when executed in the device (101) signaling that it is unable to handle the situation as it should or that a problematic event has occurred.

Images