Method and apparatus for generating driving scenarios

Abstract

To automatically generate a vast number of driving scenarios for verifying the autonomous driving performance of autonomous vehicles, robots, and other devices. [Solution] A method for generating a driving scenario according to one embodiment of the present disclosure includes the steps of: using artificial intelligence to extract a situation in text form from collected data; configuring each of the situations extracted in text form as a node, assigning weights to the edges between the nodes, and generating a situation graph; and generating a driving scenario using the situation graph.

Description

Technical Field

[0001] The present disclosure relates to a driving scenario generation technology, and more specifically, to a method and apparatus for generating a driving scenario for generating a driving scenario based on generative artificial intelligence (generative AI).

Background Art

[0002] Autonomous driving is a technology that autonomously performs recognition, judgment, and control to safely operate a vehicle and people to a destination without accidents. Autonomous driving vehicles utilize various sensors such as cameras, radars, ultrasonic sensors, and lidars to sense the surrounding environment, recognize surrounding objects using artificial intelligence (AI) technology, and control the behavior of the vehicle itself, enabling safe driving without accidents. Automobile manufacturers are conducting various verifications to meet intended functions / performances to ensure safe autonomous driving performance and providing products to consumers.

[0003] Generative artificial intelligence is the next step technology of artificial intelligence, and is a field of artificial intelligence technology that learns various domains such as images, texts, videos, etc. and generates new content. The learned model has the merit of providing high-level results corresponding to various requirements and even providing outputs beyond the level of human creativity. In the field of large language models (LLMs), OpenAI's ChatGPT (registered trademark) is a representative example of generative AI. When receiving an input of a user's request item, the ChatGPT generative AI analyzes the request item by itself and gives the most suitable answer thereto, and is widely used by many people.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technical problem of the present disclosure is to provide a method and apparatus for generating a driving scenario for generating a driving scenario based on generative artificial intelligence.

[0005] The technical problem addressed in this disclosure is to provide a method and apparatus for generating driving scenarios that can automatically generate a vast number of driving scenarios for verifying the autonomous driving performance of devices such as autonomous vehicles and robots.

[0006] The technical problem addressed in this disclosure is to provide a method and apparatus for generating driving scenarios that can automatically generate driving scenarios for verifying autonomous driving performance, and corresponding video footage.

[0007] The technical challenges that this disclosure seeks to address are not limited to those mentioned above, and any other technical challenges not mentioned can be clearly understood by a person with ordinary skill in the art to which this disclosure pertains from the following description. [Means for solving the problem]

[0008] A method for generating a driving scenario according to one embodiment of the present disclosure includes the steps of: using artificial intelligence to extract a situation in text form from collected data; configuring each of the situations extracted in text form as a node, assigning weights to the edges between the nodes, and generating a situation graph; and generating a driving scenario using the situation graph.

[0009] According to one embodiment, the extraction step can be performed using generative artificial intelligence to extract the situation in text form from the collected data related to the accident situation.

[0010] According to one embodiment, the extraction step can extract the situation from the collected data as a phrase or sentence.

[0011] According to one embodiment, the step of generating the situation graph involves configuring the situation extracted in text form as situation nodes, configuring information about each of the collected data as accident nodes, connecting edges between the situation nodes and the accident nodes, assigning weights to the edges, and then generating the situation graph.

[0012] According to one embodiment, the step of generating the situation graph involves connecting a first edge between situation nodes that have a relationship with the accident node, connecting a second edge between mutually exclusive situation nodes, and then assigning the weights to the first and second edges respectively to generate the situation graph.

[0013] According to one embodiment, the first edge may be given different weights depending on the degree of relationship between the accident node and the situation node, and the second edge may be given different weights depending on the degree of exclusivity between the situation nodes.

[0014] According to one embodiment, the step of generating the driving scenario involves determining at least one situation node from among the situation nodes connected to the accident node by the first edge, based on the weight assigned to the first edge, and generating the driving scenario using the at least one situation node.

[0015] Furthermore, a method for generating a driving scenario according to one embodiment of the present disclosure may further include the step of generating video corresponding to the driving scenario using a video generation model.

[0016] A method for generating a driving scenario according to another embodiment of the present disclosure includes the steps of: using artificial intelligence to extract a situation in text form from collected data; configuring each of the situations extracted in text form as a situation node and configuring information about each of the collected data as a central node; connecting edges between the situation nodes and the central node and assigning weights to the edges; and generating a driving scenario based on the weights assigned to the edges.

[0017] According to one embodiment, the weighting step involves connecting a first edge between situation nodes that have a relationship with the central node, connecting a second edge between mutually exclusive situation nodes, and then assigning the weight to each of the first and second edges.

[0018] According to one embodiment, the step of generating the driving scenario involves determining at least one situation node from among the situation nodes connected to the central node and the first edge, based on the weight assigned to the first edge, and generating the driving scenario using the at least one situation node.

[0019] Furthermore, methods for generating driving scenarios according to other embodiments of the present disclosure may further include the step of generating video corresponding to the driving scenario using a video generation model.

[0020] A driving scenario generation device according to another embodiment of the present disclosure includes a data analysis unit that uses artificial intelligence to extract situations in text form from collected data, a data management unit that configures each of the situations extracted in text form as a node, assigns weights to the edges between the nodes and generates a situation graph, and a scenario generation unit that generates a driving scenario using the situation graph.

[0021] According to one embodiment, the data analysis unit can use generative artificial intelligence to extract the situation in text form from the collected data related to the accident situation.

[0022] According to one embodiment, the data analysis unit can extract the situation from the collected data in phrases or sentences.

[0023] According to one embodiment, the data management unit constructs the situation extracted in the text form with situation nodes, constructs information regarding each of the collected data with accident nodes, connects edges between the situation nodes and the accident nodes, and then assigns weights to the edges to generate the situation graph.

[0024] According to one embodiment, the data management unit connects a first edge between a situation node related to the accident node, connects a second edge between mutually exclusive situation nodes among the situation nodes, and then assigns the weight to each of the first edge and the second edge to generate the situation graph.

[0025] According to one embodiment, the first edge can be assigned different weights according to the degree of relationship between the accident node and the situation node, and the second edge can be assigned different weights according to the degree of exclusivity between the situation nodes.

[0026] According to one embodiment, the scenario generation unit determines at least one or more situation nodes among the situation nodes connected to the accident node by the first edge based on the weight given to the first edge, and uses the at least one or more situation nodes to generate the driving scenario.

[0027] Furthermore, the driving scenario generation device according to another embodiment of the present disclosure can further include a video generation unit that generates a video corresponding to the driving scenario using a video generation model.

[0028] The features briefly summarized above regarding the present disclosure are exemplary aspects of the detailed description of the present disclosure to be described later, and do not limit the scope of the present disclosure.

Advantages of the Invention

[0029] According to the present disclosure, based on generative artificial intelligence, a driving scenario can be generated, and a video corresponding to the generated driving scenario can be automatically generated.

[0030] According to the present disclosure, by generating a huge number of driving scenarios that may cause accidents for devices such as vehicles and robots performing autonomous driving, the learning and performance verification reliability of the autonomous driving algorithm can be improved.

[0031] According to the present disclosure, by automating the generation of driving scenarios based on generative artificial intelligence, the costs and time invested in the generation and verification of driving scenarios can be reduced.

[0032] According to the present disclosure, it is possible to construct a driving scenario based on generative artificial intelligence and reduce the input of labor costs, and it is possible to perform centralized verification for autonomous driving vulnerable cases, so the performance verification of autonomous driving products can be made robust.

[0033] The effects obtained in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those with ordinary knowledge in the technical field to which the present disclosure belongs from the following description.

Brief Description of the Drawings

[0034] [Figure 1] It is a diagram showing an operation flowchart for a method of generating a driving scenario according to an embodiment of the present disclosure. [Figure 2] It is a diagram showing an example diagram for explaining the data collection process. [Figure 3] It is a diagram showing an example diagram for explaining the process of extracting a situation from the collected data. [Figure 4] It is a diagram showing an example diagram for explaining the process of generating a situation graph. [Figure 5]This diagram illustrates an example of the process of generating a driving scenario using a situation graph. [Figure 6] This figure shows the configuration of a driving scenario generation device according to another embodiment of the present disclosure. [Figure 7] This figure shows a block diagram of a computing system for performing a method for generating a driving scenario according to one embodiment of the present disclosure. [Modes for carrying out the invention]

[0035] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings, so that they can be easily implemented by a person with ordinary skill in the art to which the present disclosure pertains. However, the present disclosure can be implemented in a variety of different forms and is not limited to the embodiments described herein.

[0036] In describing embodiments of this disclosure, if it is determined that a specific description of a known configuration or function may obscure the gist of this disclosure, such detailed description will be omitted. Furthermore, parts of the drawings that are not relevant to the description of this disclosure will be omitted, and similar parts will be denoted by similar reference numerals.

[0037] In this disclosure, when one component is described as “connected,” “joined,” or “linked” to another component, this can include not only direct connections but also indirect connections where other components exist in between. Furthermore, when one component is described as “containing” or “having” another component, this means, unless otherwise stated to the contrary, that it may contain other components rather than excluding them.

[0038] In this disclosure, terms such as "first," "second," etc., are used solely for the purpose of distinguishing one component from another, and do not limit the order or importance of the components unless specifically mentioned. Therefore, within the scope of this disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment.

[0039] In this disclosure, components that are distinguished from each other are used to clearly describe their respective characteristics and do not necessarily imply that the components are separate. That is, multiple components may be integrated and constitute a single hardware or software unit, or a single component may be distributed and constitute multiple hardware or software units. Accordingly, such integrated or distributed embodiments are also included in the scope of this disclosure, without needing to be specifically mentioned.

[0040] In this disclosure, the components described in various embodiments are not necessarily essential components, and some may be optional components. Therefore, embodiments consisting of a subset of the components described in one embodiment are also included in the scope of this disclosure. Furthermore, embodiments that include other components in addition to the components described in various embodiments are also included in the scope of this disclosure.

[0041] In this disclosure, the terms used herein to describe positional relationships, such as top, bottom, left, and right, are provided for explanatory purposes only. If the drawings shown herein are viewed in reverse, the positional relationships described in the specification may be interpreted in the opposite way.

[0042] In this disclosure, each phrase such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include any one of the items listed together with the phrase, or any possible combination thereof.

[0043] Autonomous driving is currently in a transitional period from Level 2 to Level 3, which is considered the stage of full-fledged autonomous driving, and there is a demand for more accurate and reliable performance quality. Manufacturers are building more robust scenarios and verifying performance. However, driving conditions and circumstances do not match due to differences in weather, roads, regions, and countries, and driving laws and driver tendencies also differ from country to country, making it difficult to guarantee performance using the same scenario.

[0044] While scenarios are created and evaluated for rare or difficult corner / edge cases that occur during driving, the scope of these scenarios is limited in order to confirm performance at Level 3 or higher. This is cited as one of the reasons why Level 3 autonomous driving has not yet appeared on the market.

[0045] Accidents occur due to unforeseen variables and environmental factors, making it difficult for humans to consider all possible scenarios with their limited imagination and thinking. Even if autonomous driving achieves Level 5 functionality, humans are far from being able to establish evaluation methods that can verify and guarantee this.

[0046] The embodiments of this disclosure aim to improve the learning and performance verification reliability of autonomous driving algorithms by automatically generating a vast number of driving scenarios for verifying autonomous driving performance in devices such as vehicles and robots that perform autonomous driving based on generative artificial intelligence.

[0047] Figure 1 is a flowchart showing the operation of a method for generating a driving scenario according to one embodiment of the present disclosure. Here, Figure 1 may be a flowchart of the operation of a method performed by an apparatus, server, or system for generating a driving scenario.

[0048] Referring to Figure 1, the method for generating a driving scenario according to one embodiment of the present disclosure uses artificial intelligence, for example, generative artificial intelligence, to extract the situation from collected data in text form (S110).

[0049] The data collected using the method disclosed herein may be data collected in real time from a data collection device, or data collected in advance. The data collection method will be explained in detail in Figure 2.

[0050] In describing the technology of this disclosure, we will assume that the collected data is accident-related accident data. Needless to say, the collected data applicable to the technology of this disclosure is not limited to or restricted to accident data, and may include any data for generating scenarios related to the operation of equipment such as vehicles and robots.

[0051] Before describing the technology of this disclosure, the terms used in this disclosure are defined as follows:

[0052] An accident refers to an event in which loss of life or property has occurred or is highly likely to occur, a situation is a set of individual elements that may contribute to an accident, and can include things like "fog," "pedestrians crossing the street against a traffic light," or "a vehicle cutting in," and a scenario can refer to a real-world event that is created by a combination of various situations.

[0053] Depending on the embodiment, step S110 can analyze accident data using generative artificial intelligence that takes pre-collected or stored accident data as input and extract it in text form that describes the situation. Here, step S110 can extract the situation not as words, but as phrases or sentences, and by extracting as phrases or sentences, the ambiguity of the results processed / analyzed by generative artificial intelligence can be minimized.

[0054] Depending on the embodiment, step S110 can extract several situations that may contribute to an accident from accident data using a language model or the like.

[0055] Depending on the embodiment, step S110 can use an image-to-text model or the like to extract several situations that may have contributed to the accident from a schematic diagram of the accident situation, a photograph of the accident scene, etc.

[0056] Depending on the embodiment, the situations extracted in step S110 can be categorized according to the situation, such as weather, vehicle type, road type, etc., and similar situations can be managed together and mutually exclusive situations can be identified by embedding / clustering the situations within a single category using a language model. For example, the method of this disclosure can manage "fog appeared" and "fog is rolling in" together, and "clear weather" can be identified as a situation mutually exclusive with fog.

[0057] In step S110, when the situation is extracted from the accident data, each of the extracted situations is configured in the first node (hereinafter referred to as the "situation node"), and each piece of information from the accident data is configured in the second node (hereinafter referred to as the "accident node") (S120).

[0058] Depending on the embodiment, step S120 can consist of detailed situation nodes that classify the situation nodes by category. For example, step S120 can consist of detailed situation nodes such as road situation nodes, driving situation nodes, vehicle situation nodes, and weather situation nodes. In other words, step S120 can consist of each situation and each accident in its respective node.

[0059] Once the situation nodes and accident nodes are configured in step S120, weights are assigned to the edges between the situation nodes and accident nodes to generate a situation graph (S130).

[0060] Depending on the embodiment, step S130 may involve connecting a first edge, such as a relationship edge, between situation nodes that have a relationship with an accident node, and connecting a second edge, such as an exclusivity edge, between mutually exclusive situation nodes among the situation nodes, and then assigning weights to the first and second edges respectively to generate a situation graph.

[0061] Here, the first edge may be given different weights depending on the degree of relationship between the accident node and the situation node, and the second edge may be given different weights depending on the degree of exclusivity between the situation nodes.

[0062] Depending on the embodiment, step S130 may assign different weights to the first edge in relation to the accident node, taking into consideration the order in which the edges are highly related to the accident and had a significant impact on the accident. The values ​​of the weights assigned to the first edge may be determined by the business or individual providing the technology of this disclosure, and the highest and lowest weights assigned to the first edge may be predetermined.

[0063] Depending on the embodiment, step S130 may assign a weight to a second edge connected between mutually exclusive situation nodes that is lower than the weight assigned to the first edge, but depending on the situation, the second edge may be assigned a different weight, or it may be assigned a weight of a predetermined small value. The second edge may also be assigned a weight based on the degree of mutual exclusivity between situation nodes, if such a degree can be calculated, and it goes without saying that the higher the degree of mutual exclusivity, the higher the weight that can be assigned to the second edge, or the lower the weight that can be assigned. Here, the method for calculating the degree of mutual exclusivity between situation nodes and the weight assigned to the second edge can be determined by the operator or individual providing the technology of this disclosure.

[0064] Furthermore, the situation graph structure for representing the relationship between accident nodes and situation nodes is modifiable. The modified situation graph structure may include the first edges described above connected between situation nodes according to the degree of relationship, and a second edge connected between accident nodes and situation nodes. In other words, the situation graph generated by the method of this disclosure can have various structures depending on the method.

[0065] Once a situation graph is generated in step S130, a driving scenario is generated using the situation graph, and images or videos corresponding to the generated driving scenario are generated (S140, S150).

[0066] In one embodiment, step S140 can determine at least one situation node based on the weight assigned to the first edge among the situation nodes connected to the accident node by the first edge, and generate a driving scenario using at least one situation node.

[0067] Depending on the embodiment, step S140 can generate high-quality or highly likely driving scenarios by using graph algorithms such as GNN and PageRank to identify situation nodes with a high accident contribution based on weights assigned to the first edge, and by ensuring that these are included in a predetermined proportion or higher.

[0068] In other words, if step S140 selects only situation nodes unrelated to accidents, or selects situation nodes that are mutually exclusive or cannot occur simultaneously to generate a driving scenario, only unrealistic or low-quality driving scenarios may be generated. Therefore, by selecting highly related situation nodes based on the weights assigned to the first edge and generating the driving scenario, a high-quality driving scenario can be generated.

[0069] Depending on the embodiment, step S150 can generate an image or video of the driving scenario using a generative artificial intelligence model such as a text-to-image or text-to-video model.

[0070] The method of this disclosure will be explained in detail below with reference to Figures 2 to 5.

[0071] Figure 2 is an illustrative diagram illustrating the data collection process, specifically illustrating the process of collecting driving data using a data acquisition means, such as a data logging device installed in a vehicle.

[0072] As illustrated in Figure 2, a vehicle equipped with a data logging device can acquire driving data, which may include data collected by cameras, radar, lidar, and ultrasonic sensors used for autonomous driving, as well as data from the vehicle's controller and actuators.

[0073] Here, driving data can be acquired using a sliding window method. Specifically, data is collected every millisecond, and as time passes, previous data is deleted while retaining data for a predetermined time prior to the current time, for example, one hour's worth of data. In other words, as time passes, data is acquired by sliding a window of one hour in size. Here, the data collection time unit may be 1 ms, and the collection time size may be one hour, but such data collection time unit and collection time size can be varied depending on the situation.

[0074] If an abnormal driving event 210 occurs during the process of acquiring driving data from the vehicle, or if the user explicitly turns the data storage function "ON", the sliding window function will be turned off until the abnormal driving event 210 ends or the storage function is turned "OFF", and the data from that point onward will not be deleted. In other words, all data from the hour before the abnormal event occurred up to the end of the event will be acquired.

[0075] Here, abnormal driving events can include all dangerous events that could lead to an accident, such as lane departure, skidding, collision, driver inattention (e.g., distracted driving), drowsiness, and other drivers' driving negligence.

[0076] Driving data acquired from each vehicle is transferred to a pre-configured server, for example, a server that performs the method of this disclosure. The server can collect the driving data acquired from all vehicles, and based on the driving data thus collected, a driving scenario can be generated.

[0077] The data collected using the method of this disclosure is not limited to or restricted to data obtained from autonomous vehicles, and it goes without saying that various highly reliable data collected and stored in advance may also be used.

[0078] Because accident situations often present challenges not only for humans but also for autonomous driving systems, utilizing highly reliable accident data allows for the construction of high-quality driving scenarios that are highly probable, rare, and challenging.

[0079] Accident analysis organizations in various countries, such as GIDAS (German In-Depth Accident Study), conduct in-depth investigations and analyses of traffic accidents when they occur, creating databases of accident diagrams, accident scene photographs, and other relevant data. Therefore, this disclosure method may utilize such pre-collected and stored, highly reliable accident data.

[0080] Figure 3 is an illustrative diagram illustrating the process of extracting information from collected data.

[0081] As illustrated in Figure 3, the method according to the embodiment of this disclosure can use generative artificial intelligence to extract situations from collected data 310 and categorize the extracted situations according to weather conditions, vehicle conditions, road conditions, etc.

[0082] Furthermore, the categorized weather conditions, vehicle conditions, and road conditions 320 can be managed together by embedding / clustering them within a single category using a language model, thereby grouping similar situations together.

[0083] For example, as illustrated in Figure 3, "cloudy weather," "overcast weather," and "mostly cloudy weather" can be managed as similar situations under a single weather condition. Although not illustrated, "sunny weather" is mutually exclusive with "cloudy weather" and can therefore be managed as a separate weather condition.

[0084] Thus, the method according to the embodiments of this disclosure allows for the management of extracted similar situations as a single cluster, and enables the generation of a situation graph using each of these situations.

[0085] Figure 4 is an example diagram illustrating the process of generating a situation graph.

[0086] As illustrated in Figure 4, the situation graph can consist of a situation node 410, an accident node 420, a first edge which is a relationship edge 430, and a second edge which is an exclusivity edge 440.

[0087] In Figure 4, the situation node 410 includes road condition node, driving condition node, vehicle condition node, and weather condition node. The road condition node includes two conditions: "It is a two-lane rural road" and "It is an eight-lane main road." The driving condition node includes two conditions: "The vehicle is speeding" and "The vehicle was driving slowly." The vehicle condition node includes two conditions: "It is a hatchback" and "It is a motorcycle." The weather condition node includes three conditions: "It is foggy," "The weather is sunny," and "It is raining." The situation node 410 that constitutes the situation graph is not limited to or restricted to the conditions illustrated in Figure 4. Needless to say, for the sake of explanation, the conditions have been described restrictively, and the explanation will use the conditions illustrated in Figure 4.

[0088] Accident node 420 contains three accident data: Accident 1, Accident 2, and Accident 3. Relationship edge 430 is an edge that connects accident node 420 and situation node 410, and can be connected if they are related to the accident in question. Exclusion edge 440 is an edge that connects situation nodes, and can be connected if the two situations are mutually exclusive.

[0089] In Figure 4, accident node 1 is connected by relational edges to the situation nodes "It is foggy", "It is a hatchback", "It is a two-lane road in the countryside", and "The driver is speeding". Accident node 2 is connected by relational edges to the situation nodes "It is a motorcycle", "The weather is sunny", "The driver was going slowly", and "It is an eight-lane highway". Accident node 3 is connected by relational edges to the situation nodes "The driver is speeding", "It is an eight-lane highway", and "It is raining". Here, relational edges 430 can be assigned different weights depending on the degree of relationship between the accident node and the situation node.

[0090] The exclusivity edge 440 connects mutually exclusive situation nodes such as "It is foggy" and "The weather is sunny," "The weather is sunny" and "It is raining," "It is a hatchback" and "It is a motorcycle," "It is a two-lane rural road" and "It is an eight-lane highway," and "It is speeding" and "It was driving slowly." Here, different weights can be assigned depending on the degree of mutual exclusivity between the exclusivity edge 440 and the situation nodes. Here, the fact that "It is a hatchback" and "It is a motorcycle," which are included in the vehicle situation node, are connected by the exclusivity edge 440, is because the subject performing the driving scenario cannot be both a hatchback and a motorcycle at the same time. In other words, since the subject included in the vehicle situation node is the subject performing the driving scenario, it can be connected to other subjects by exclusivity edges.

[0091] Figure 5 is an illustrative diagram illustrating the process of generating a driving scenario using a situation graph, and shows how to generate a driving scenario using the situation graph in Figure 4.

[0092] As illustrated in Figure 5, the method according to the embodiment of the present disclosure can generate driving scenarios using situation nodes connected by relationship edges to accident nodes in a situation graph. Here, the method according to the embodiment of the present disclosure can generate at least one driving scenario using weights assigned to relationship edges in the situation graph.

[0093] For example, the method according to the embodiment of the present disclosure can select a situation node 510 "It is foggy" connected to accident node 1 by a relationship edge, a situation node 520 "It is a motorcycle" connected to accident node 2 by a relationship edge, a situation node 530 "It is a speeding situation" connected to accident node 1 and accident node 3 by relationship edges, and a situation node 540 "It is an 8-lane avenue" connected to accident node 2 and accident node 3 by relationship edges, and generate a driving scenario 550 "A motorcycle speeds on an 8-lane avenue with fog" using the selected situation nodes 510, 520, 530, and 540.

[0094] Needless to say, in the situation graph, other situation nodes connected by relational edges may be selected to generate other driving scenarios. For example, the method according to the embodiment of this disclosure can select the situation node "It is a two-lane rural road" connected to the accident 1 node by a relational edge, the situation node "It is a motorcycle" connected to the accident 2 node by a relational edge, the situation node "It is a speeding situation" connected to the accident 1 node and the accident 3 node by a relational edge, and the situation node "It is raining" connected to the accident 3 node by a relational edge, and use the selected situation nodes to generate a driving scenario "A motorcycle speeds on a two-lane rural road in the rain."

[0095] In addition, the method according to the embodiments of this disclosure can generate a driving scenario using at least two accident nodes and situation nodes connected by relationship edges.

[0096] The driving scenarios generated through the process described above can be used to generate images or videos corresponding to the driving scenarios using various image generation modules, such as image generation models and video generation models. It goes without saying that the generated images or videos will always include the parts corresponding to the driving scenario, and may also include various other situations and surrounding objects. These generated images or videos can be used in the learning and verification process of autonomous driving. By using images or videos corresponding to highly reliable driving scenarios in the learning and verification process, the learning and performance verification reliability of the autonomous driving algorithm can be improved.

[0097] Thus, the method according to the embodiments of this disclosure can generate driving scenarios based on generative artificial intelligence and automatically generate video corresponding to the generated driving scenarios.

[0098] Furthermore, the method according to the embodiments of this disclosure can improve the learning and performance verification reliability of autonomous driving algorithms by generating a vast number of driving scenarios in which accidents may occur for devices such as autonomous vehicles and robots.

[0099] Furthermore, the method according to the embodiments of this disclosure can reduce the cost and time required for generating and verifying driving scenarios by automating the generation of driving scenarios based on generative artificial intelligence.

[0100] Furthermore, the method according to the embodiments of this disclosure enables the construction of driving scenarios based on generative artificial intelligence, thereby reducing labor costs, and allows for focused verification of vulnerable cases in autonomous driving, thus making the performance verification of autonomous driving products more robust.

[0101] Furthermore, the methods of this disclosure are not limited to or restricted to collecting accident data and generating accident-related driving scenarios and video, but can be applied to various fields for generating scenarios, and the data collected may differ depending on the field to which they are applied. In addition, the methods of this disclosure can be applied to all types of devices or systems that can be learned and validated using scenarios, and the fields to which they are applied, devices or systems, etc., can be determined by the business or individual providing the technology of this disclosure.

[0102] Furthermore, the method disclosed herein is not necessarily limited to generating a situation graph before generating a driving scenario; it may also involve connecting edges between nodes, assigning weights to them, and then using the weights assigned to the edges to generate a driving scenario.

[0103] Figure 6 is a diagram showing the configuration of a driving scenario generation device according to another embodiment of the present disclosure, and is a diagram showing the configuration of a driving scenario generation device that implements the methods of Figures 1 to 5. For example, the driving scenario generation device may be a server.

[0104] Referring to Figure 6, a driving scenario generation device 600 according to another embodiment of the present disclosure includes a data acquisition unit 610, a data analysis unit 620, a data management unit 630, a scenario generation unit 640, a video generation unit 650, and a storage unit 660.

[0105] The storage unit 660 is a means for storing all data related to the technology of this disclosure, and stores collected data, generative artificial intelligence models, situation nodes, driving scenario generation algorithms, generated driving scenarios and video, etc.

[0106] The data acquisition unit 610 is a means for collecting data to generate a driving scenario, and receives data acquired by various data acquisition devices, such as a vehicle, and collects data to generate a driving scenario.

[0107] Here, the data acquisition unit 610 can also acquire highly reliable data that has been collected and stored in advance.

[0108] The data analysis unit 620 uses artificial intelligence, such as generative artificial intelligence, to extract the situation from the collected data in text form.

[0109] Depending on the embodiment, the data analysis unit 620 can analyze accident data using generative artificial intelligence that takes pre-collected or stored accident data as input, and extract it in text form that describes the situation. Here, the data analysis unit 620 can minimize ambiguity in the results processed / analyzed by generative artificial intelligence by extracting the situation as phrases or sentences rather than as individual words.

[0110] Depending on the embodiment, the data analysis unit 620 can use a language model or the like to extract several situations from accident data that may contribute to the accident.

[0111] Depending on the embodiment, the data analysis unit 620 can use an image-text conversion model or the like to extract several situations that may contribute to the accident from a schematic diagram of the accident situation, a photograph of the accident scene, etc.

[0112] In this embodiment, the situations extracted by the data analysis unit 620 can be categorized according to the situation, such as weather, vehicle type, road type, etc. Within a single category, similar situations can be managed together by embedding / clustering the situations using a language model, and mutually exclusive situations can be identified.

[0113] The data management unit 630 constructs each of the extracted situations in text format as a node and generates a situation graph by assigning weights to the edges between the nodes.

[0114] In this embodiment, the data management unit 630 can generate a situation graph by configuring each extracted situation as a situation node, configuring each piece of accident data information as an accident node, and then assigning weights to the edges between the situation nodes and the accident nodes.

[0115] Depending on the embodiment, the data management unit 630 can be composed of detailed status nodes that are classified by category.

[0116] In this embodiment, the data management unit 630 can generate a situation graph by connecting relationship edges between situation nodes that have a relationship with an accident node, connecting exclusivity edges between mutually exclusive situation nodes, and then assigning weights to the relationship edges and exclusivity edges, respectively.

[0117] Here, relational edges may be given different weights depending on the degree of relationship between the accident node and the situation node, and exclusivity edges may be given different weights depending on the degree of exclusivity between the situation nodes.

[0118] Depending on the embodiment, the data management unit 630 may assign different weights to relationship edges in relation to the accident node, taking into consideration the order in which they are highly related to the accident and had a significant impact on the accident, or it may predetermine the highest and lowest weights to be assigned to the relationship edges.

[0119] Depending on the embodiment, the data management unit 630 may assign a weight lower to the exclusivity edge connected between mutually exclusive status nodes than to the relationship edge, or, depending on the situation, different weights may be assigned to the exclusivity edge, or a similar weight of a preset small value may be assigned. If the degree of mutual exclusivity between status nodes can be calculated, the exclusivity edge may be assigned a weight based on that degree of mutual exclusivity, and it goes without saying that the higher the degree of mutual exclusivity, the higher the weight that can be assigned to the exclusivity edge, or the lower the weight it may be assigned.

[0120] The scenario generation unit 640 generates a driving scenario using a situation graph.

[0121] In this embodiment, the scenario generation unit 640 can determine at least one situation node from among the situation nodes connected to the accident node by a relationship edge, based on the weights assigned to the relationship edge, and generate a driving scenario using at least one situation node.

[0122] Depending on the embodiment, the scenario generation unit 640 can generate high-quality or highly likely driving scenarios by using graph algorithms such as GNN and PageRank to identify situation nodes with a high accident contribution based on weights assigned to relational edges, and by ensuring that these nodes are included in a predetermined ratio or higher.

[0123] In other words, the scenario generation unit 640 can generate high-quality driving scenarios by selecting situation nodes with high relationships based on the weights assigned to the relationship edges.

[0124] The video generation unit 650 generates video, such as an image or video, corresponding to the generated driving scenario.

[0125] In one embodiment, the video generation unit 650 can generate a driving scenario as an image or video using a generative artificial intelligence model such as a text-to-image or text-to-video model.

[0126] Even if such description is omitted in the apparatus of other embodiments of the present disclosure, the apparatus of other embodiments of the present disclosure may include all of the things described in the manner of Figures 1 to 5, which will be obvious to those skilled in the art.

[0127] Figure 7 shows a block diagram of a computing system for performing a method for generating a driving scenario according to one embodiment of the present disclosure.

[0128] Referring to Figure 7, the method for generating a driving scenario according to one embodiment of the present disclosure described above can be implemented via a computing system. The computing system 1000 may include at least one processor 1100, memory 1300, user interface input device 1400, user interface output device 1500, storage 1600, and network interface 1700, all connected via a system bus 1200.

[0129] The processor 1100 may be a semiconductor device that processes instruction words stored in a central processing unit (CPU) or memory 1300 and / or storage 1600. The memory 1300 and storage 1600 may include various volatile or non-volatile storage media. For example, the memory 1300 may include ROM (Read Only Memory) 1310 and RAM (Random Access Memory) 1320.

[0130] Accordingly, steps of the methods or algorithms described in relation to the embodiments disclosed herein can be directly implemented by hardware, software modules, or a combination of both, executed by the processor 1100. The software modules may reside in a storage medium (i.e., memory 1300 and / or storage 1600) such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, or CD-ROM. An exemplary storage medium is coupled to the processor 1100, which can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor 1100. The processor 1100 and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a user terminal. Alternatively, the processor 1100 and the storage medium may reside as separate components in a user terminal.

[0131] The above description is merely illustrative of the technical concept of this disclosure, and any person with ordinary skill in the art to which this disclosure pertains can make various modifications and variations without departing from the essential characteristics of this disclosure. Accordingly, the embodiments disclosed in this disclosure are for illustrative purposes only, and not to limit the technical concept of this disclosure, and such embodiments do not limit the scope of the technical concept of this disclosure. The scope of protection of this disclosure should be interpreted in accordance with the following claims, and all technical concepts within an equivalent scope should be construed as being included in the scope of rights of this disclosure.

Claims

1. The steps involve using artificial intelligence to extract the situation from the collected data in text format, The steps include: constructing each of the situations extracted in the aforementioned text format as a node, assigning weights to the edges between the nodes, and generating a situation graph; A method for generating a driving scenario, comprising the step of generating a driving scenario using the aforementioned situation graph.

2. The extraction step described above is: A method for generating a driving scenario according to claim 1, wherein generative artificial intelligence is used to extract the situation in text form from the collected data related to the accident situation.

3. The extraction step described above is: A method for generating a driving scenario according to claim 2, comprising extracting the situation from the collected data as a phrase or sentence.

4. The step of generating the aforementioned situation graph is: A method for generating a driving scenario according to claim 2, comprising: configuring the situation extracted in the text format as a situation node; configuring information relating to each of the collected data as an accident node; connecting edges between the situation nodes and the accident nodes; then assigning weights to the edges; and finally generating the situation graph.

5. The step of generating the aforementioned situation graph is: A method for generating a driving scenario according to claim 4, comprising connecting a first edge between situation nodes that have a relationship with the accident node, connecting a second edge between mutually exclusive situation nodes among the situation nodes, and then assigning the weights to the first edge and the second edge respectively to generate the situation graph.

6. The first edge is, Different weights are assigned depending on the degree of the relationship between the accident node and the situation node. The aforementioned second edge is A method for generating a driving scenario according to claim 5, wherein different weights are assigned depending on the degree of exclusivity between the situation nodes.

7. The step of generating the aforementioned driving scenario is: A method for generating a driving scenario according to claim 5, comprising determining at least one situation node from among the situation nodes connected to the accident node and the first edge, based on the weight assigned to the first edge, and generating the driving scenario using the at least one situation node.

8. The method for generating a driving scenario according to claim 1, further comprising the step of generating video corresponding to the driving scenario using a video generation model.

9. The steps involve using artificial intelligence to extract the situation from the collected data in text format, The steps include: configuring each of the situations extracted in the aforementioned text format as a situation node, and configuring information about each of the collected data as a central node; The steps include connecting an edge between the situation node and the center node and assigning a weight to the edge, A method for generating a driving scenario, comprising the step of generating a driving scenario based on the weights assigned to the aforementioned edges.

10. The step of assigning weights is, A method for generating a driving scenario according to claim 9, comprising connecting a first edge between situation nodes that have a relationship with the central node, connecting a second edge between mutually exclusive situation nodes among the situation nodes, and then assigning the weights to the first edge and the second edge, respectively.

11. The step of generating the aforementioned driving scenario is: A method for generating a driving scenario according to claim 10, comprising determining at least one situation node from among the situation nodes connected to the central node and the first edge, based on the weight assigned to the first edge, and generating the driving scenario using the at least one situation node.

12. A method for generating a driving scenario according to claim 9, further comprising the step of generating video corresponding to the driving scenario using a video generation model.

13. A data analysis unit that uses artificial intelligence to extract the situation from collected data in text format, A data management unit generates a situation graph by configuring each of the situations extracted in the aforementioned text format as a node, assigning weights to the edges between the nodes, and A driving scenario generation device, including a scenario generation unit that generates a driving scenario using the aforementioned situation graph.

14. The aforementioned data analysis unit, A device for generating driving scenarios according to claim 13, which uses generative artificial intelligence to extract the situation in text form from the collected data related to the accident situation.

15. The aforementioned data analysis unit, A device for generating driving scenarios according to claim 14, which extracts the situation from the collected data in the form of a phrase or sentence.

16. The aforementioned data management unit, A device for generating a driving scenario according to claim 14, comprising: configuring the situation extracted in the text format as a situation node; configuring information relating to each of the collected data as an accident node; connecting edges between the situation nodes and the accident nodes; then assigning weights to the edges; and finally generating the situation graph.

17. The aforementioned data management unit, A device for generating a driving scenario according to claim 16, comprising: connecting a first edge between situation nodes that have a relationship with the accident node; connecting a second edge between mutually exclusive situation nodes among the situation nodes; and then assigning the weights to the first edge and the second edge respectively to generate the situation graph.

18. The first edge is, Different weights are assigned depending on the degree of the relationship between the accident node and the situation node. The aforementioned second edge is A device for generating driving scenarios according to claim 17, wherein different weights are assigned depending on the degree of exclusivity between situation nodes.

19. The scenario generation unit, A driving scenario generation device according to claim 17, which determines at least one situation node from among the situation nodes connected to the accident node and the first edge, based on the weight assigned to the first edge, and generates the driving scenario using the at least one situation node.

20. The driving scenario generation apparatus according to claim 13, further comprising a video generation unit that generates video corresponding to the driving scenario using a video generation model.

Images