Apparatus and method for generating traffic flow training data

Abstract

The present invention relates to an apparatus for generating traffic flow training data, comprising: a scenario setting module for setting multiple scenarios that can occur due to unexpected situations at an intersection; a real-environment traffic flow data generation module for generating real-environment traffic flow data including dynamic information about a vehicle for each scenario; a virtual-environment traffic flow data generation module for generating virtual-environment traffic flow data for each scenario; a virtual-environment traffic flow data augmentation module which trains a virtual-environment traffic flow data augmentation model by using the real-environment traffic flow data and the virtual-environment traffic flow data, and which augments the virtual-environment traffic flow data by using other virtual-environment traffic flow data; and a processor, which reviews the validity of the augmented virtual-environment traffic flow data so as to determine final usability of same, thereby constructing a training data set for each artificial intelligence model by using the augmented virtual-environment traffic flow data for each scenario, which has been determined to be used.

Description

Device and method for generating traffic flow learning data

[0001] The present invention relates to a device and method for generating traffic flow learning data.

[0002] Recently, in order to support autonomous vehicles to drive more safely to their destinations, the need for deep learning is increasing to automatically determine whether a lane change is necessary by considering the shape of the road ahead, the relationship between roads, the number of lanes, and traffic conditions or traffic flow (i.e., traffic flow), and to effectively determine the timing of lane changes when lane changes are necessary. In addition, there is a growing demand for abundant traffic flow learning data to increase the efficiency of such deep learning learning.

[0003] Furthermore, the need for deep learning is increasing to enable rapid and safe responses to minimize traffic congestion and accidents when autonomous and non-autonomous vehicles encounter complex road environments without prior information. Furthermore, there is a growing demand for abundant traffic flow learning data to enhance the efficiency of such deep learning.

[0004] In addition, there is a growing need for deep learning training of artificial intelligence models for predicting paths and making control decisions for non-autonomous vehicles in response to traffic conditions to support autonomous vehicles in complex road environments where autonomous and non-autonomous vehicles coexist. Furthermore, a high-quality, rich dataset (i.e., traffic flow learning data) is needed to enhance the learning efficiency of these artificial intelligence models.

[0005] For example, supporting autonomous vehicles requires road infrastructure information (e.g., route prediction information for non-autonomous vehicles). For example, autonomous vehicles can share their planned routes (e.g., through vehicle-to-vehicle (V2V) communication), allowing them to understand and assess each other's behavioral plans in advance and utilize this information for route planning. However, for conventional vehicles (i.e., non-autonomous vehicles), future route plans are unknown, requiring abundant training data for route prediction.

[0006] In addition, although it is possible to build a rich learning data set for predicting the path of general vehicles (i.e., non-autonomous vehicles) under general road conditions (i.e., road conditions of normal driving without accidents), it is difficult to secure learning data (i.e., learning data that satisfies the quantity and quality) for predicting the path of general vehicles (i.e., non-autonomous vehicles) under unexpected situations such as vehicle-to-vehicle accidents / objects falling on the road / illegally parking, so there is a need for abundant learning data for predicting the path of general vehicles under unexpected situations.

[0007] The background technology of the present invention is disclosed in Republic of Korea Patent No. 10-2341475 (December 16, 2021).

[0008] The present invention was created to solve the above problems, and the purpose of the present invention is to provide a traffic flow learning data generation device and method that automatically determines whether a lane change is necessary by considering the shape of the road ahead, the connection relationship between roads, the number of lanes, and the traffic situation or traffic flow (i.e., traffic flow), and if a lane change is necessary, can effectively determine the timing of the lane change, and can automatically generate abundant traffic flow learning data necessary for deep learning to predict the path of a non-autonomous vehicle in various traffic situations including unexpected situations.

[0009] According to one aspect of the present invention, a traffic flow learning data generation device comprises: a scenario setting module that sets a plurality of scenarios that may occur as an unexpected situation at an intersection; a real-environment traffic flow data generation module that generates real-environment traffic flow data including dynamic information of a vehicle for each scenario; a virtual-environment traffic flow data generation module that generates virtual-environment traffic flow data for each scenario; a virtual-environment traffic flow data multiplication module that learns a virtual-environment traffic flow data multiplication model using the real-environment traffic flow data and the virtual-environment traffic flow data, and multiplies the virtual-environment traffic flow data using other virtual-environment traffic flow data; and a processor that examines the effectiveness of the multiplied virtual-environment traffic flow data to determine whether to use it in the end, and configures a learning data set for each artificial intelligence model using the multiplied virtual-environment traffic flow data for each scenario for which use has been determined.

[0010] In the present invention, the plurality of scenarios are characterized by including scenarios for (a) an emergency situation in the direction of an intersection straight exit lane, (b) an emergency situation occurring in an unprotected right turn lane, (c) an emergency situation occurring within an intersection, and (d) an emergency situation occurring in an intersection left turn exit lane.

[0011] In the present invention, the real environment traffic flow data generation module is characterized by acquiring real environment traffic flow data on a real road, or generating real environment traffic flow data through an experiment after configuring a simulated environment similar to a real road.

[0012] In the present invention, data on dynamic information of the real-world traffic flow data is characterized in that it includes longitudinal position (x) of the vehicle, lateral position (y) of the vehicle, heading angle (yaw) of the vehicle, velocity, and acceleration.

[0013] In the present invention, the virtual environment traffic flow data generation module is characterized in that it is configured with an environment identical to the environment that constitutes the real environment traffic flow data.

[0014] In the present invention, the virtual environment traffic flow data multiplication module is characterized in that it uses an artificial intelligence learning model to which a specialized technique for data multiplication is applied, uses other virtual environment traffic flow data as input data, and outputs a real road data set acquired in a real environment as a result value.

[0015] In the present invention, the processor is characterized in that it uses a method of examining the effectiveness by utilizing the augmented virtual environment traffic flow data and the acquired real environment traffic flow data, and utilizing the error between the real environment traffic flow data and the augmented virtual environment traffic flow data.

[0016] In the present invention, the processor determines whether virtual environment traffic flow data is generated or not through the derived error value, and if it is determined that it cannot be generated, it is reused as an input value of a virtual environment traffic flow data multiplication module to obtain a new result value, and a process of reviewing the effectiveness of the new result value is performed again.

[0017] According to another aspect of the present invention, a method for generating traffic flow learning data includes: a step in which a processor sets a plurality of scenarios that may occur as unexpected situations at an intersection through a scenario setting module; a step in which the processor generates real-world traffic flow data including dynamic information of vehicles for each scenario through a real-world traffic flow data generation module; a step in which the processor generates virtual environment traffic flow data for each scenario through a virtual environment traffic flow data generation module; a step in which the processor learns a virtual environment traffic flow data multiplication model using the real-world traffic flow data and the virtual environment traffic flow data through a virtual environment traffic flow data multiplication module, and multiplies the virtual environment traffic flow data using other virtual environment traffic flow data; and a step in which the processor examines the effectiveness of the multiplied virtual environment traffic flow data to determine whether to use it and configures a learning data set for each artificial intelligence model using the multiplied virtual environment traffic flow data for each scenario for which use has been determined.

[0018] In the present invention, the plurality of scenarios are characterized by including scenarios for (a) an emergency situation in the direction of an intersection straight exit lane, (b) an emergency situation occurring in an unprotected right turn lane, (c) an emergency situation occurring within an intersection, and (d) an emergency situation occurring in an intersection left turn exit lane.

[0019] In the present invention, the real environment traffic flow data generation module is characterized by acquiring real environment traffic flow data on a real road, or generating real environment traffic flow data through an experiment after configuring a simulated environment similar to a real road.

[0020] In the present invention, data on dynamic information of the real-world traffic flow data is characterized in that it includes longitudinal position (x) of the vehicle, lateral position (y) of the vehicle, heading angle (yaw) of the vehicle, velocity, and acceleration.

[0021] In the present invention, the virtual environment traffic flow data generation module is characterized in that it is configured with an environment identical to the environment that constitutes the real environment traffic flow data.

[0022] In the present invention, the virtual environment traffic flow data multiplication module is characterized in that it uses an artificial intelligence learning model to which a specialized technique for data multiplication is applied, uses other virtual environment traffic flow data as input data, and outputs a real road data set acquired in a real environment as a result value.

[0023] In the present invention, the processor is characterized in that it uses a method of examining the effectiveness by utilizing the augmented virtual environment traffic flow data and the acquired real environment traffic flow data, and utilizing the error between the real environment traffic flow data and the augmented virtual environment traffic flow data.

[0024] In the present invention, the processor determines whether virtual environment traffic flow data is generated or not through the derived error value, and if it is determined that it cannot be generated, it is reused as an input value of a virtual environment traffic flow data multiplication module to obtain a new result value, and a process of reviewing the effectiveness of the new result value is performed again.

[0025] According to one aspect of the present invention, the present invention automatically determines whether a lane change is necessary by considering the shape of the road ahead, the connection relationship between roads, the number of lanes, and the traffic situation or traffic flow (i.e., traffic flow), and if a lane change is necessary, the timing of the lane change can be effectively determined, and abundant traffic flow learning data required for deep learning to predict the path of a non-autonomous vehicle in various traffic situations including unexpected situations can be automatically generated.

[0026] Figure 1 is an exemplary diagram showing a schematic configuration of a traffic flow learning data generation device according to one embodiment of the present invention.

[0027] Figure 2 is a flowchart illustrating a method for generating traffic flow learning data according to one embodiment of the present invention.

[0028] Figures 3 (a) to (d) are examples shown to explain a plurality of representative scenarios for an emergency situation in Figure 1.

[0029] Hereinafter, an embodiment of the present invention will be described with reference to the attached drawings.

[0030] In this process, the thickness of lines and the sizes of components depicted in the drawings may be exaggerated for clarity and convenience. Furthermore, the terms described below are defined based on their functions within the present invention and may vary depending on the intent or custom of the user or operator. Therefore, the definitions of these terms should be based on the overall content of this specification.

[0031] Figure 1 is an exemplary diagram schematically illustrating the configuration of a traffic flow learning data generation device according to one embodiment of the present invention. Figures 3 (a) to (d) are exemplary diagrams illustrating multiple representative scenarios for unexpected situations in Figure 1.

[0032] Referring to FIG. 1, a traffic flow learning data generation device according to the present embodiment includes a scenario setting module (110), a real environment traffic flow data generation module (120), a virtual environment traffic flow data generation module (130), a virtual environment traffic flow data multiplication module (140), and a processor (150).

[0033] The scenario setting module (110) sets multiple scenarios (e.g., four representative scenarios) that may occur in an emergency situation (emergency area) within an intersection.

[0034] The scenario setting module (110) can receive multiple scenarios from the user through an HMI (Human Machine Interface).

[0035] At this time, for data acquisition, a traffic environment (or traffic flow) is set, including traffic regulation speed, movement route information of each vehicle, and accident information.

[0036] However, information such as emergency situations (accident information, etc.) is very limited in obtaining from real-world roads.

[0037] Therefore, the scenario must be dependent on the traffic flow (or traffic environment), and must have generality in setting the scope for data acquisition, and the goal of the scenario must be to set up multiple representative scenarios for traffic flow that cover various situations (the combination of scenarios for each representative situation covers almost all situations that can occur on the road).

[0038] For example, in order to construct a traffic flow dataset for unexpected situations that are difficult to acquire in a real environment, four representative scenarios including general situations for unexpected situations (unexpected areas) that may occur at intersections can be constructed as shown below (see (a) to (d) of Figure 3).

[0039] (a) Emergency situation in the direction of the straight-line exit at the intersection

[0040] (b) An emergency situation occurs in an unprotected right turn lane.

[0041] (c) Situation in which an emergency occurs within an intersection

[0042] (d) An emergency situation occurs when making a left turn at an intersection.

[0043] The real-world traffic flow data generation module (120) generates real-world traffic flow data for each scenario.

[0044] The real-world traffic flow data generation module (120) acquires real-world traffic flow data on dynamic information of vehicles by utilizing infrastructure sensors such as RSU (Road Side Unit), acquires real-world traffic flow data on a real road, or acquires (generates) real-world traffic flow data through experiments after configuring a simulated environment similar to a real road.

[0045] Data on dynamic information of real-world traffic flow data is structured as follows (x: longitudinal position of vehicle, y: lateral position of vehicle, yaw: heading angle of vehicle, velocity: speed, acceleration: acceleration, etc.).

[0046] The virtual environment traffic flow data generation module (130) generates virtual environment traffic flow data for each scenario.

[0047] The virtual environment traffic flow data generation module (130) generates virtual environment traffic flow data for each configured scenario.

[0048] The virtual environment traffic flow data generation module (130) configures an environment identical to the environment that configures the real environment traffic flow data, thereby minimizing the problem of similarity (feasibility) with the real environment traffic flow data (i.e., ensuring that the similarity is as high as possible).

[0049] At this time, the configuration of the emergency situation (emergency area) (e.g., car accidents, falling objects, illegal parking, etc.) may differ from the actual environment. However, when interpreted from the perspective of an impassable situation (area), the configuration can be identical. Furthermore, by diversifying vehicle parameters, it is possible to generate diverse virtual data similar to actual vehicle data.

[0050] Data generated through the virtual environment traffic flow data generation module (130) may include the following essential information (x: longitudinal position of the vehicle, y: lateral position of the vehicle, yaw: heading angle of the vehicle, velocity: speed, acceleration: acceleration, etc.).

[0051] At this time, the virtual environment traffic flow data generation module (130) may use one of the disclosed simulation devices (or apps).

[0052] The virtual environment traffic flow data multiplication module (140) learns a virtual environment traffic flow data multiplication model using real environment traffic flow data and virtual environment traffic flow data, and multiplies the virtual environment traffic flow data using other virtual environment traffic flow data.

[0053] Virtual environment traffic flow data is implemented using a specific mathematical model, so it accumulates information on limited patterns and methods. Therefore, by linking real-world traffic flow data with a proliferation model, it is possible to build a substantial and diverse dataset.

[0054] For example, a virtual environment traffic flow data augmentation module (140) can utilize an artificial intelligence learning model (e.g., LSTM, GAN, etc.) that applies a specialized technique for data augmentation, use virtual environment traffic flow data as input data, and output a real road data set (or data corresponding to this data set) acquired in a real environment as a result value. Through this, data augmentation is possible by utilizing virtual environment traffic flow data to acquire values ​​corresponding to real environment traffic flow data.

[0055] The processor (150) examines the effectiveness of the proliferated virtual environment traffic flow data and determines whether to use it for the final purpose.

[0056] For example, the processor (150) examines the effectiveness by utilizing the proliferated virtual environment traffic flow data and the acquired real environment traffic flow data.

[0057] As a method for review at this time, a method that utilizes the error between real environment traffic flow data and augmented virtual environment traffic flow data (e.g., RMSE and MAE) can be used.

[0058] For reference,

[0059]

[0060] , where y i : Real-world data, y' i: Augmented data, n: Total number of data used when a vehicle draws a trajectory. Also

[0061]

[0062] , where y': augmented data, y: real environment data, and n: the total number of data used when the vehicle draws a trajectory.

[0063] The processor (150) determines whether virtual environment traffic flow data is generated (i.e., whether it can be used) through the derived error value, and if it is determined that it cannot be generated (i.e., cannot be used), it is reused as an input value of the virtual environment traffic flow data multiplication module to obtain a new result value, and the process of reviewing the effectiveness of the new result value can be performed again.

[0064] The processor (150) examines the effectiveness of the proliferated virtual environment traffic flow data to determine whether to use it, and uses the proliferated virtual environment traffic flow data for each scenario for which use has been determined to form a learning data set for each artificial intelligence model.

[0065] Meanwhile, in the above-described embodiment, each component (110 to 150) is described separately, but depending on the embodiment, it may be implemented by integrating it by the processor (150).

[0066] Figure 2 is a flowchart illustrating a method for generating traffic flow learning data according to one embodiment of the present invention.

[0067] Referring to FIG. 2, the processor (150) selects multiple scenarios (e.g., four representative scenarios) that may occur in an emergency situation (emergency area) within an intersection through the scenario setting module (110) (S101).

[0068] The processor (150) generates real-world traffic flow data for each scenario through the real-world traffic flow data generation module (120) (S102).

[0069] The processor (150) generates virtual environment traffic flow data for each scenario through a virtual environment traffic flow data generation module (130) (or simulation device) (S103).

[0070] The processor (150) learns a designated virtual environment traffic flow data multiplication model using real environment traffic flow data and virtual environment traffic flow data through the virtual environment traffic flow data multiplication module (140) (S104).

[0071] The processor (150) generates other virtual environment traffic data using the virtual environment traffic data generation module (130) (or simulation device) and multiplies the virtual environment traffic data through the virtual environment traffic data multiplication model of the virtual environment traffic data multiplication module (140) (S105).

[0072] The processor (150) examines the effectiveness of the proliferated virtual environment traffic flow data and determines whether to use it for the final purpose (S106).

[0073] At this time, although not shown in the drawing, the processor (150) determines whether virtual environment traffic flow data is generated (i.e., whether it can be used) through the derived error value, and if it is determined that it cannot be generated (i.e., cannot be used), it is reused as an input value of the virtual environment traffic flow data multiplication module (140) to obtain a new result value, and the process of reviewing the effectiveness of the new result value can be performed again.

[0074] And the processor (150) uses the virtual environment traffic flow data proliferated for each scenario for which use has been decided to form a learning data set for each artificial intelligence model (S107).

[0075] As described above, the present invention automatically determines whether a lane change is necessary by considering the shape of the road ahead, the connection relationship between roads, the number of lanes, and the traffic situation or traffic flow (i.e., traffic flow), and if a lane change is necessary, it enables the effective determination of the timing of the lane change, and automatically generates abundant traffic flow learning data necessary for deep learning to predict the path of a non-autonomous vehicle in various traffic situations, including unexpected situations. In other words, since the learning effect is higher with more learning data, abundant traffic flow learning data means as much traffic flow learning data as possible.

[0076] Although the present invention has been described with reference to the embodiments shown in the drawings, these are merely exemplary, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible from the drawings. Accordingly, the technical protection scope of the present invention should be defined by the following claims. In addition, the implementations described in this specification may be implemented as, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., a device or a program). The device may be implemented by suitable hardware, software, firmware, etc. The method may be implemented in a device such as a processor, which generally refers to a processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device such as a computer, a cell phone, a personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end-users.

Claims

1. A scenario setting module that sets multiple scenarios that may occur due to unexpected situations at an intersection; A real-world traffic flow data generation module that generates real-world traffic flow data containing vehicle dynamic information for each scenario; A virtual environment traffic flow data generation module that generates virtual environment traffic flow data for each scenario; A virtual environment traffic flow data multiplication module that learns a virtual environment traffic flow data multiplication model using real environment traffic flow data and virtual environment traffic flow data, and multiplies the virtual environment traffic flow data using other virtual environment traffic flow data; and A traffic flow learning data generation device characterized by including a processor that examines the effectiveness of the proliferated virtual environment traffic flow data to determine whether to use it for the final purpose, and configures a learning data set for each artificial intelligence model using the proliferated virtual environment traffic flow data for each scenario for which use has been determined.

2. In paragraph 1, The above multiple scenarios are, A traffic flow learning data generation device characterized by including scenarios for (a) occurrence of an emergency situation in a straight-line exit lane of an intersection, (b) occurrence of an emergency situation in an unprotected right turn lane, (c) occurrence of an emergency situation within an intersection, and (d) occurrence of an emergency situation in a left turn exit lane of an intersection.

3. In paragraph 1, The above real-world traffic flow data generation module is: A traffic flow learning data generation device characterized by acquiring real-environment traffic flow data on a real road, or generating real-environment traffic flow data through experiments after configuring a simulated environment similar to a real road.

4. In paragraph 1, Data on the dynamic information of the above real-world traffic flow data is A traffic flow learning data generation device characterized by including a longitudinal position (x) of a vehicle, a lateral position (y) of the vehicle, a heading angle (yaw) of the vehicle, a velocity, and an acceleration of the vehicle.

5. In paragraph 1, The above virtual environment traffic flow data generation module is: A traffic flow learning data generation device characterized by being configured with an environment identical to the environment that constitutes real-world traffic flow data.

6. In paragraph 1, The above virtual environment traffic flow data amplification module is: A traffic flow learning data generation device characterized by utilizing an artificial intelligence learning model to which a specialized technique in data augmentation is applied, using traffic flow data from another virtual environment as input data, and outputting a real road data set acquired in a real environment as a result value.

7. In paragraph 1, The above processor, A traffic flow learning data generation device characterized by using a method of examining the effectiveness by utilizing proliferated virtual environment traffic flow data and acquired real environment traffic flow data, and utilizing the error between the real environment traffic flow data and the proliferated virtual environment traffic flow data.

8. In paragraph 7, The above processor, A traffic flow learning data generation device characterized in that it determines whether or not virtual environment traffic flow data is generated through the derived error value, and if it is determined that generation is not possible, it is reused as an input value of a virtual environment traffic flow data multiplication module to obtain a new result value, and a process of reviewing the effectiveness of the new result value is performed again.

9. A step in which the processor sets multiple scenarios that may occur as emergency situations within an intersection through a scenario setting module; A step in which the above processor generates real-world traffic data including dynamic information of vehicles for each scenario through a real-world traffic data generation module; A step in which the above processor generates virtual environment traffic data for each scenario through a virtual environment traffic data generation module; The above processor learns a virtual environment traffic data multiplication model using real environment traffic data and virtual environment traffic data through a virtual environment traffic data multiplication module, and multiplies the virtual environment traffic data using other virtual environment traffic data; and A method for generating traffic flow learning data, characterized in that it includes a step of examining the effectiveness of the virtual environment traffic flow data multiplied by the processor to determine whether to use it for the final purpose, and using the multiplied virtual environment traffic flow data for each scenario for which use has been determined, to form a learning data set for each artificial intelligence model.

10. In paragraph 9, The above multiple scenarios are, A method for generating traffic flow learning data, characterized by including scenarios for (a) occurrence of an emergency situation in a straight-line exit lane of an intersection, (b) occurrence of an emergency situation in an unprotected right turn lane, (c) occurrence of an emergency situation within an intersection, and (d) occurrence of an emergency situation in a left turn exit lane of an intersection.

11. In paragraph 9, The above real-world traffic flow data generation module is: A method for generating traffic flow learning data, characterized by acquiring real-environment traffic flow data on a real road or generating real-environment traffic flow data through experiments after configuring a simulated environment similar to a real road.

12. In paragraph 9, Data on the dynamic information of the above real-world traffic flow data is A method for generating traffic flow learning data, characterized in that it includes longitudinal position (x) of a vehicle, lateral position (y) of a vehicle, heading angle (yaw) of the vehicle, velocity, and acceleration.

13. In paragraph 9, The above virtual environment traffic flow data generation module is: A method for generating traffic flow learning data, characterized in that the data is configured in an environment identical to the environment that constitutes real-world traffic flow data.

14. In paragraph 9, The above virtual environment traffic flow data amplification module is: A method for generating traffic flow learning data, characterized by using an artificial intelligence learning model with a specialized technique applied in data augmentation, using traffic flow data from another virtual environment as input data, and outputting a real road data set acquired in a real environment as a result value.

15. In paragraph 9, The above processor, A method for generating traffic flow learning data characterized by using a method for examining the effectiveness of the data by utilizing the propagated virtual environment traffic flow data and the acquired real environment traffic flow data, and utilizing the error between the real environment traffic flow data and the propagated virtual environment traffic flow data.

16. In paragraph 15, The above processor, A method for generating traffic flow learning data, characterized in that the generation of virtual environment traffic flow data is determined through the derived error value, and if generation is determined not to be possible, the data is reused as an input value of a virtual environment traffic flow data multiplication module to obtain a new result value, and a process for reviewing the effectiveness of the new result value is performed again.

Images