Reinforcement learning-based autonomous driving system capable of responding to dynamic environmental changes and operating method thereof

Abstract

According to an embodiment of the present application, provided is an operating method of a reinforcement learning-based autonomous driving system capable of responding to dynamic environmental changes. The method may comprise: a step of acquiring point cloud data corresponding to driving environment information via at least one sensor unit; a step in which a map generation module generates a three-dimensional point cloud map on the basis of the point cloud data; a step in which a feature extraction module including a neural network extracts driving environment features from the three-dimensional point cloud map; a step in which a reinforcement learning module determines a driving operation of a moving object on the basis of the driving environment features; and a step in which the reinforcement learning module evaluates whether the driving operation is successful according to the state of the moving object after performing the driving operation, and gives a reward.

Description

Reinforcement learning-based autonomous driving system capable of responding to dynamic environmental changes and its operation method

[0001] The present application relates to an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes and an operating method thereof.

[0002] Autonomous driving refers to a technology that enables a vehicle to perceive its surroundings, analyze driving conditions in real time, and safely navigate to its destination without driver intervention. Recently, autonomous driving technology has been utilized not only in automobiles but also in small mobility devices such as smart delivery systems and security robots, leading to a surge in demand for autonomous driving technology across mobility devices, robots, and vehicles.

[0003] Existing methods for implementing autonomous driving mainly involve using devices such as LiDAR or cameras to create a map in advance and then estimating the location on the map using sensor data measured during driving, or using a precise GPS device to receive satellite signals and estimate the absolute location.

[0004] Meanwhile, in the autonomous driving field, artificial intelligence technologies such as deep learning are actively being utilized to further improve driving performance. Deep learning, a type of machine learning, primarily utilizes supervised learning, which trains models using labeled data (i.e., data with correct answers). However, in situations such as road conditions where dynamic environmental changes frequently occur, assuming that learning through supervised learning classification using existing deep learning methods is performed, a significant limitation arises: the need to predict and determine the correct action for every situation requires a massive number of examples.

[0005] Therefore, in response to a dynamically changing environment where all cases cannot be specified through learning, a technology is needed that can learn by identifying a specific path through recognition, judgment, and movement control algorithms from information acquired through sensors such as image sensors.

[0006] The purpose of this application is to provide an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes and an operating method thereof.

[0007] According to an embodiment of the present application, an operating method of an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes is provided. The method may include: acquiring point cloud data corresponding to driving environment information through at least one sensor unit; generating a three-dimensional point cloud map based on the point cloud data by a map generation module; extracting driving environment features from the three-dimensional point cloud map by a feature extraction module including a neural network; determining a driving operation of a mobile device based on the driving environment features by a reinforcement learning module; and evaluating, by the reinforcement learning module, whether the driving operation is successful and providing a reward based on a state of the mobile device after performing the driving operation.

[0008] Additionally, the driving environment information may include information on at least one of a road, a person, another vehicle, a building, and an obstacle on which the vehicle is driving.

[0009] In addition, the feature extraction module extracts a global feature from the 3D point cloud map using max pooling, and in the step of determining the driving motion of the moving object, the driving motion can be determined based on the global feature.

[0010] In addition, the reinforcement learning module may include an action module that determines a driving operation of the mobile device based on the driving environment characteristics, and evaluates whether the driving operation is successful and determines a reward value or a loss value based on the state of the mobile device according to the performance of the driving operation; and a learning module that performs reinforcement learning for autonomous driving of the mobile device based on at least one of the reward value or loss value received from the action module, the driving environment characteristics, the driving operation, and information on the state of the mobile device after the driving operation.

[0011] In addition, the driving record module may further include a step of storing driving information including the compensation value or the loss value, the driving environment information or the driving environment characteristics before performing the driving action, and information about the state of the moving object after performing the driving action.

[0012] In addition, the state of the mobile body includes information on whether a collision has occurred after performing the driving action, and in the step of evaluating whether the driving action is successful and providing a reward, the reinforcement learning module can provide a reward value if a collision does not occur to the mobile body, and can provide a loss value if a collision occurs to the mobile body.

[0013] A computer program is provided according to an embodiment of the present application. The program can be stored on a recording medium to execute a method according to an embodiment of the present application.

[0014] According to embodiments of the present application, precise control of a mobile device is possible by implementing reinforcement learning to identify and learn a specific path and optimal driving motion in response to a road situation in which the driving environment changes dynamically and in which all cases cannot be specified through learning.

[0015] According to embodiments of the present application, memory can be saved by implementing a method of storing only extracted feature points without directly storing a 3D map using point cloud data.

[0016] The effects that can be obtained from the embodiments of the present application are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by a person having ordinary skill in the technical field to which the present application belongs from the description below.

[0017] To facilitate a more thorough understanding of the drawings cited in this application, a brief description of each drawing is provided.

[0018] Figure 1 is a diagram for explaining reinforcement learning according to an embodiment of the present application.

[0019] FIG. 2 is a block diagram illustrating the configuration of an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes according to an embodiment of the present application.

[0020] FIG. 3 is a functional block diagram for explaining the operation of an autonomous driving system according to an embodiment of the present application.

[0021] FIG. 4 is a flowchart of an operation method of an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes according to an embodiment of the present application.

[0022] FIG. 5 is a flowchart of an operation method of an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes according to an embodiment of the present application.

[0023] FIG. 6 is a drawing for explaining point cloud data including driving environment information according to an embodiment of the present application.

[0024] The technical concept of this application is susceptible to various modifications and embodiments. Specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the technical concept of this application to specific embodiments, and it should be understood that all modifications, equivalents, and alternatives fall within the scope of the technical concept of this application.

[0025] In explaining the technical idea of ​​this application, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of this application, the detailed description is omitted.

[0026] The terminology used in this specification is for the purpose of describing embodiments and is not intended to limit or restrict the present application. Singular expressions include plural expressions unless the context clearly dictates otherwise. Furthermore, numbers used in this specification (e.g., "first," "second," etc.) are merely identifiers used to distinguish one component from another.

[0027] When a part in this specification is said to be connected to another part, this includes not only direct connections but also indirect connections with other components intervening. Furthermore, when a part is said to include a component, this does not exclude other components, but rather implies the inclusion of additional components, unless otherwise specifically stated.

[0028] Furthermore, the term "or" in this application is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X utilizes A or B" is intended to mean either of the natural inclusive permutations. That is, if X utilizes A; X utilizes B; or X utilizes both A and B, "X utilizes A or B" can apply to any of the above cases. Furthermore, the term "and / or" as used herein should be understood to refer to and encompass all possible combinations of one or more of the associated configurations listed.

[0029] In addition, terms such as “~part”, “~device”, “~sub-subject”, and “~module” described in the present application mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software, such as a processor, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processor unit (APU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

[0030] It should be noted that the distinction between components in this application is merely a distinction based on the primary function of each component. In other words, two or more components described below may be combined into a single component, or a single component may be further subdivided into two or more components with more detailed functions. Furthermore, each component described below may, in addition to its own primary function, additionally perform some or all of the functions of other components. It should also be noted that some of the primary functions of each component may be exclusively performed by other components.

[0031]

[0032] The method according to the embodiment of the present application may be performed on a personal computer, workstation, server computer device, etc. having computing capabilities, or on a separate device for the same.

[0033] Additionally, the method may be performed on one or more computing devices. For example, at least one or more steps of the method according to an embodiment of the present application may be performed on a client device, and other steps may be performed on a server device. In such a case, the client device and the server device may be connected via a network to transmit and receive computational results. Alternatively, the method may be performed using distributed computing technology.

[0034]

[0035] In this specification, the term "artificial intelligence learning model" may be used interchangeably with terms such as artificial intelligence model, computational model, and machine learning model. The artificial intelligence learning model may be trained using various algorithms, such as, for example, decision trees, random forests, Gaussian naive Bayes, k-nearest neighbors, ada boost, support vector machines, voting, bagging, neural networks, and deep learning. However, the present invention is not limited thereto.

[0036] An AI learning model can be trained using at least one of the following methods: supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Training an AI learning model can be the process of applying knowledge to the model to enable it to perform a specific action.

[0037] When algorithms such as neural networks and deep learning are applied to an AI learning model, the AI ​​learning model can be referred to as a network function. The term "network function" can be used interchangeably with "neural network." A neural network can be composed of a set of interconnected computational units, generally referred to as nodes. These nodes can also be referred to as neurons. A neural network consists of at least one node, and the nodes can be interconnected by one or more links.

[0038] The neural network may include a deep neural network (DNN). A deep neural network may include, but is not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), an autoencoder, a restricted Boltzmann machine (RBM), a deep belief network (DBN), a Q network, a U network, a Siamese network, and a generative adversarial network (GAN).

[0039]

[0040] Hereinafter, embodiments of the present application will be described in detail one by one.

[0041]

[0042] Figure 1 is a diagram for explaining reinforcement learning according to an embodiment of the present application.

[0043] As illustrated in Figure 1, reinforcement learning is a learning method for selecting an optimal action (A) to maximize a reward (R) in a given state (S). More specifically, reinforcement learning is a learning method implemented so that a policy (policy) performs an action (A) based on a state (S) and obtains a reward (R) from the environment.

[0044] In this case, the entity that becomes the subject of learning is called an agent, and the agent is implemented to establish a policy for selecting actions that maximize the expected reward (R) from the environment through learning. In other words, the agent learns a decision-making strategy (i.e., policy) that maximizes reward, and its ultimate goal is to maximize the expected value of the sum of rewards (R). In other words, the agent is trained to select the optimal action (A) that maximizes the reward (R).

[0045] Meanwhile, an episode in reinforcement learning represents a series of states (S), actions (A), and rewards (R) that an agent experiences from its initial state to its final state. In reinforcement learning, at the end of an episode, information about the actions (A) taken in the previous state (S) is recorded. This recorded information can be considered a type of experience. This recorded information is used to make decisions in the next episode. At the end of an episode, the process of recording and updating the information gained during that episode is repeated.

[0046] In this application, an agent may be a vehicle driving on a road, and the vehicle may include not only a vehicle but also small mobility devices, robots, and the like. Furthermore, the state (S) may refer to the state of the external driving environment in which the vehicle is driving. The vehicle may acquire information about the state of the external driving environment, determine and perform actions—i.e., driving actions—that maximize reward (R), and each driving episode may be recorded and utilized for decision-making in other driving situations.

[0047]

[0048] FIG. 2 is a block diagram illustrating the configuration of an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes according to an embodiment of the present application.

[0049] Referring to FIG. 2, the autonomous driving system (200) may include a communication unit (210), an input unit (120), a memory (230), and a processor (240). In an embodiment, each component of the system (200) may be mounted on a moving object such as a vehicle.

[0050] The communication unit (210) can receive or transmit data from inside or outside. The communication unit (210) can include a wired or wireless communication unit. When the communication unit (210) includes a wired communication unit, the communication unit (210) can include one or more components that enable communication through a Local Area Network (LAN), a Wide Area Network (WAN), a Value Added Network (VAN), a mobile radio communication network, a satellite communication network, and a combination thereof. In addition, when the communication unit (210) includes a wireless communication unit, the communication unit (210) can wirelessly transmit and receive data or signals using cellular communication, wireless LAN (e.g., Wi-Fi), etc. In an embodiment, the communication unit (210) can transmit and receive data or signals with an external device or an external server under the control of the processor (240).

[0051] The input / output unit (220) may include an input unit for receiving input from a user and an output unit for displaying information such as the result of performing a task (i.e., a driving action or episode) or the status of the system (200). That is, the input / output unit (220) is configured to receive various user commands and data through external manipulation, and to output the results of processing the input commands and data. For example, the input / output unit (220) may be connected to various interfaces for input, such as a keypad or a mouse, to receive user commands, and for this purpose, may include an interface, such as a Thunderbolt port as well as a USB port. In addition, the input / output unit may include or be combined with various input devices and / or output devices, such as a touch screen or buttons, to receive user commands or data, or output the results of processing commands and data.

[0052] In an embodiment, the input / output unit (220) may include at least one sensor unit for acquiring external driving environment information. For example, the sensor unit may be configured to include at least one sensor, such as a LiDAR sensor, a camera, an infrared sensor, and / or an ultrasonic sensor.

[0053] The memory (230) can store programs and / or program commands for the operation of the processor (240), and can temporarily or permanently store input / output data. The memory (230) can include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, magnetic memory, magnetic disk, and optical disk.

[0054] In addition, the memory (230) can store various network functions, artificial intelligence learning models and / or algorithms, and can store various data, programs (one or more instructions), applications, software, commands, codes, etc. for driving and controlling the autonomous driving system (200).

[0055] In an embodiment, the memory (230) may store data and programs that enable the processor (240) described below to perform reinforcement learning operations for autonomous driving according to the process presented below. In addition, in an embodiment, the memory (230) may be configured with a database for storing driving information.

[0056] The processor (240) can control the overall or at least part of the operation of the autonomous driving system (200). The processor (240) can execute one or more programs or software stored in the memory (230). The processor (240) can mean a dedicated processor (240) on which the methods according to the embodiments of the present application are performed, such as a central processing unit (CPU), a graphics processing unit (GPU), or a field programmable gate array (FPGA).

[0057] In an embodiment, the processor (240) may control an operation to obtain point cloud data corresponding to driving environment information through at least one sensor unit, a map generation module to generate a 3D point cloud map based on the point cloud data, a feature extraction module including a neural network to extract driving environment features from the 3D point cloud map, a reinforcement learning module to determine a driving operation of a mobile device based on the driving environment features, and to evaluate whether the driving operation is successful and provide a reward based on the state of the mobile device after performing the driving operation.

[0058] Here, the driving environment information may include information on at least one of the road on which the vehicle is driving, people, other vehicles, buildings, and obstacles.

[0059] In an embodiment, the feature extraction module extracts global features from a 3D point cloud map using max pooling, and the reinforcement learning module can determine the driving motion of the mobile object based on the global features.

[0060] In an embodiment, the reinforcement learning module may include an action module that determines a driving operation of a vehicle based on driving environment characteristics, and evaluates whether the driving operation is successful and determines a reward value or a loss value based on changes in the state of the vehicle or driving environment information according to the performance of the driving operation; and a learning module that performs reinforcement learning for autonomous driving of the vehicle based on at least one of the reward value or loss value, the driving environment characteristics, the driving operation, and information about the state of the vehicle after the driving operation received from the action module.

[0061] In an embodiment, the processor (240) may control the driving record module to store driving information including a compensation result (compensation value or loss value), driving environment information or driving environment characteristics before performing a driving action, and information about the state of the moving object after performing the driving action.

[0062] In an embodiment, the state of the mobile body includes information on whether a collision occurs after performing a driving action, and the processor (240) can control the reinforcement learning module to assign a reward value if a collision does not occur to the mobile body, and to assign a loss value if a collision occurs to the mobile body.

[0063] The configuration of the autonomous driving system (200) illustrated in FIG. 2 is exemplary, and various configurations may be applied according to the embodiment of the present application.

[0064]

[0065] FIG. 3 is a functional block diagram for explaining the operation of an autonomous driving system according to an embodiment of the present application.

[0066] The configuration illustrated in FIG. 3 may be implemented through at least one piece of hardware constituting the autonomous driving system (200), or by a processor (240) executing at least one program and / or algorithm stored in a memory (230).

[0067] Referring to FIG. 3, the operation of an autonomous driving system (200) based on reinforcement learning capable of responding to dynamic environmental changes can be performed through an information collection unit (310), a reinforcement learning module (320), and a driving recording module (330).

[0068] The information collection unit (310) can collect external driving environment characteristics while the vehicle is driving. The driving environment characteristics may be characteristics corresponding to information on at least one of the road, people, other vehicles, buildings, and obstacles while the vehicle is driving.

[0069] The information collection unit (310) may include a sensor unit (311), a map generation module (312), and a feature extraction module (312).

[0070] The sensor unit (311) can acquire point cloud data corresponding to driving environment information. The sensor unit (311) can be mounted on at least one area of ​​a moving object, and can generate coordinates of 3D points through the direction of light irradiated and reflected toward an object and the return time (time of flight), and can collect these coordinates to generate point cloud data corresponding to driving environment information. For example, the sensor unit (311) can be a ToF (Time of Flight) type sensor, and more specifically, can be a LiDAR (Light Detection and Ranging) scanner. However, this is merely exemplary, and various sensors capable of acquiring point clouds can be applied.

[0071] The map generation module (312) can generate a 3D point cloud map based on point cloud data. Since the raw acquired 3D point cloud data may have noise or uneven sampling issues, the map generation module (312) can generate a 3D point cloud map by removing noise or outliers or reducing the amount of data through downsampling.

[0072] The feature extraction module (312) can extract driving environment features from a 3D point cloud map. The feature extraction module (312) can be configured as a neural network including a multi-perceptron (Shared-MLP, Multi-Layer Perceptron) and a max pooling layer, and the feature extraction module (312) can be implemented to learn point-by-point features through the multi-perceptron and extract global features for the driving environment through the max pooling layer.

[0073] The reinforcement learning module (320) can determine the driving behavior of a vehicle based on driving environment characteristics, evaluate success or failure based on the state of the vehicle after performing the driving behavior, and then provide a corresponding reward. Furthermore, the reinforcement learning module (320) can perform reinforcement learning on a network (or neural network) for autonomous driving based on the reward results.

[0074] The reinforcement learning module (320) may include an action module (321) and a learning module (322).

[0075] The action module (321) can determine the driving behavior of a vehicle based on driving environment characteristics. For example, the action module (321) can output a Q value, which is a combination of multiple possible states and behaviors (i.e., driving behaviors), based on driving environment characteristics, and determine the driving behavior based on the Q value with the highest expected reward. In addition, the action module (321) can evaluate the success or failure of a driving behavior and provide a reward based on the state of the vehicle after performing the driving behavior. For example, after performing a driving behavior, if no impact occurs to the vehicle or the impact is below a certain level, the action module (321) can evaluate the driving behavior determined in response to the corresponding driving environment characteristics as successful and provide a positive (+) reward value. Conversely, if an impact exceeding a certain level occurs to the vehicle, the action module (321) can evaluate the driving behavior performed in the corresponding driving environment characteristics as failed and provide a negative (-) loss value.

[0076] The learning module (322) can perform reinforcement learning for autonomous driving of a mobile device based on a reward result. That is, it can perform learning on at least one network constituting the action module (321) and / or the learning module (322). Specifically, the action module (321) transmits information on a reward result (reward value or loss value), driving environment characteristics and driving actions performed in response thereto, and / or the state of the mobile device after performing the driving actions to the learning module (322), and the learning module (322) can perform reinforcement learning on the network based on this.

[0077] The driving record module (330) can create a driving record database by storing driving information including compensation results, driving environment information or driving environment characteristics before performing a driving action, and information on the state of a moving object after performing a driving action.

[0078] For example, the driving record module (330) can store driving information (i.e., driving experience) in the form of a tuple of (driving environment characteristics, driving motion, reward or loss value, and whether a collision occurred). The driving information stored by the driving record module (330) can be used in the reinforcement learning module (320).

[0079] In an embodiment, the reinforcement learning module (320) and the driving record module (330) may be configured as a DQN (Deep Q-Network). DQN is a type of deep reinforcement learning, which is a model that extends the reinforcement learning algorithm called Q-learning using a deep learning model, mainly an artificial neural network.

[0080] A DQN can be composed of, for example, a Q-network, an Experience Replay memory, and a target network. The Q-network uses an artificial neural network to receive a state as input and output a Q-value for each possible action. The Experience Replay memory stores the experiences the agent collects while interacting with the environment. Randomly collected experiences can be sampled from this memory to train the Q-network. The target network calculates a target Q-value to increase the stability of DQN learning. The target network is updated with the weights of the Q-network at regular intervals.

[0081] In an embodiment, the Q-network and the target network may constitute at least a portion of the action module (321) and the learning module (322). Additionally, the experience replay memory may constitute the driving record module (330).

[0082] However, this is an example and is not limited thereto, and depending on the embodiment, various reinforcement learning algorithms such as A3C (Asynchronous Advantage Actor-Critic) may be applied.

[0083] The functional configuration illustrated in FIG. 3 is exemplary, and various configurations may be applied according to embodiments of the present application.

[0084]

[0085] FIG. 4 is a flowchart of an operation method of an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes according to an embodiment of the present application.

[0086] At step S410, point cloud data corresponding to driving environment information can be acquired through at least one sensor unit (311).

[0087] Here, the driving environment information is information about the external environment in which the mobile vehicle is driving, and may include information about at least one of the road, people, other mobile vehicles, buildings, and obstacles in which the mobile vehicle is driving.

[0088] The sensor unit (311) can be mounted on at least one area of ​​the mobile body, and can generate coordinates of three-dimensional points through the direction of light irradiated and reflected toward the object and the return time (time of flight), and can collect these to generate point cloud data corresponding to driving environment information. In an embodiment, the sensor unit (311) can be a ToF (Time of Flight) type sensor, and more specifically, can be a LiDAR (Light Detection and Ranging) scanner. However, this is exemplary, and various sensors capable of acquiring a point cloud can be applied.

[0089] Referring to Figure 6, point cloud data can express information such as roads, people (or pedestrians), vehicles, and buildings through point clouds.

[0090] At step S420, the map generation module (312) can generate a 3D point cloud map based on point cloud data.

[0091] In an embodiment, the map generation module (312) can generate a 3D point cloud map by removing noise from the raw point cloud data. That is, the raw acquired 3D point cloud data may have noise or uneven sampling issues. For example, there may be noise such as outliers or missing data. Here, an outlier refers to a value that is incorrectly evaluated and is observed to deviate from the overall pattern of the sample. The map generation module (312) can detect such outliers using a visualization tool (box plot, histogram, scatter plot, etc.) and remove them through transformation or binning. In addition, since the 3D point cloud data acquired through the sensor unit (311) typically generates hundreds of thousands of points, the map generation module (312) can reduce the number of data by downsampling when the number of points increases rapidly.

[0092] At step S430, the feature extraction module (312) can extract driving environment features from a 3D point cloud map.

[0093] In an embodiment, the feature extraction module (312) may be configured as a neural network including a multi-layer perceptron (Shared-MLP, Multi-Layer Perceptron) and a max pooling layer. For example, the feature extraction module (312) may be implemented to learn point-by-point features through a multi-perceptron and extract global features through a max pooling layer.

[0094] The driving environment features extracted from the 3D point cloud map by the feature extraction module (312) may include features related to road conditions, surrounding vehicles or pedestrians, buildings, etc.

[0095] At step S440, the reinforcement learning module (320) can determine the driving motion of the mobile object based on the driving environment characteristics.

[0096] In an embodiment, the reinforcement learning module (320) may be configured as a Deep Q-Network (DQN). At this time, the action module (321) may output a Q value, which is a combination of multiple possible states and actions (i.e., driving actions), based on driving environment characteristics, and determine a driving action based on the Q value with the largest expected reward.

[0097] For example, if a driving environment feature corresponding to an obstacle such as a rock is input to the action module (321), the driving action can be determined to step on the obstacle rather than avoid it. Accordingly, the moving object can perform a driving action of stepping on the rock and passing by.

[0098] At step S450, depending on the state of the moving object after performing the driving action, the reinforcement learning module (320) can evaluate whether the driving action was successful and provide a reward.

[0099] In an embodiment, the state of the vehicle after the driving action may include whether or not an impact has occurred to the vehicle. Accordingly, the action module (321) may evaluate the success of the driving action based on whether a certain level of impact has occurred to the vehicle due to the driving action, and provide a corresponding reward. That is, if no impact has occurred or the impact is below a certain level, the action module (321) may evaluate the driving action determined in response to the corresponding driving environment feature as successful and provide a positive (+) reward value. Conversely, if a certain level of impact has occurred to the vehicle, the action module (321) may evaluate the driving action performed in the corresponding driving environment feature as unsuccessful and provide a negative (-) loss value.

[0100] For example, as described above, if the driving action is determined by stepping on a stone as an obstacle and passing by, and the stone is large and a collision of a certain level or greater occurs with the moving object, the action module (321) can assign a loss value of -1, and if no collision occurs, a reward value of +1 can be assigned.

[0101] At step S460, the reinforcement learning module (320) can perform reinforcement learning for autonomous driving of a mobile device based on the reward result.

[0102] Specifically, the action module (321) transmits information about a reward result (reward value or loss value), driving environment characteristics and driving actions performed in response thereto, and / or the state of the moving object after performing the driving actions to the learning module (322), and the learning module (322) can perform reinforcement learning on the network based on this.

[0103] For example, when a collision occurs with a moving object, a loss value (-1) can be transmitted to the learning module (322) to learn that the encountered rock is large and thus a collision occurred, and when a collision does not occur even after stepping on a rock, a reward value (+1) can be transmitted to the learning module (322) to learn that the encountered rock is very small and thus it is acceptable to pass over such small rocks in the future.

[0104] The method (400) illustrated in FIG. 4 is exemplary, and various configurations may be applied according to embodiments of the present application.

[0105]

[0106] FIG. 5 is a flowchart of an operation method of an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes according to an embodiment of the present application.

[0107] The operating method (500) may further include step S510 in addition to the operating method (4000) described above with reference to FIG. 4.

[0108] In step S510, the driving record module (330) can create a driving record database by storing driving information including compensation results, driving environment information or driving environment characteristics before performing a driving action, and information on the state of the moving object after performing a driving action.

[0109] For example, the driving record module (330) can store driving information (i.e., driving experience) in the form of a tuple of (driving environment characteristics, driving behavior, reward or loss value, and whether a collision occurred). The driving information stored in the driving record module (330) can be used for learning or re-learning of the reinforcement learning module (320).

[0110] When a driving motion of stepping on a rock as exemplified in Fig. 4 is performed, (the size of the rock, stepping on it, whether or not there was a collision, compensation value or loss value) can be stored by the driving record module (330).

[0111] The method (500) illustrated in FIG. 5 is exemplary, and various configurations may be applied according to embodiments of the present application.

[0112]

[0113] The method according to the embodiment of the present application may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present application or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0114] Additionally, the methods according to the disclosed embodiments may be provided as a computer program product. The computer program product may be traded as a product between a seller and a buyer.

[0115] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) distributed electronically by an electronic device manufacturer or through an electronic marketplace (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server, an electronic marketplace server, or a relay server that temporarily stores the software program.

[0116] In a system comprising a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, if a third device (e.g., a smartphone) exists that communicates with the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include a software program itself that is transmitted from the server to the client device or the third device, or from the third device to the client device.

[0117] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0118] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) may execute a computer program product stored on the server, thereby controlling a client device in communication with the server to perform a method according to the disclosed embodiments.

[0119]

[0120] Although the embodiments have been described in detail above, the scope of the present application is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present application defined in the following claims also fall within the scope of the present application.

Claims

1. An operation method of an autonomous driving system based on reinforcement learning capable of responding to dynamic environmental changes, A step of acquiring point cloud data corresponding to driving environment information through at least one sensor unit; A step in which a map generation module generates a 3D point cloud map based on the point cloud data; A step of extracting driving environment features from the 3D point cloud map by a feature extraction module including a neural network; A step in which a reinforcement learning module determines the driving behavior of a mobile device based on the driving environment characteristics; and A method comprising a step of the reinforcement learning module evaluating whether the driving action is successful and providing a reward based on the state of the moving object after the driving action is performed.

2. In paragraph 1, A method wherein the driving environment information includes information on at least one of a road, a person, another vehicle, a building, and an obstacle on which the vehicle is driving.

3. In paragraph 1, The above feature extraction module extracts global features from the 3D point cloud map using max pooling, A method for determining the driving motion of the above-mentioned mobile body, wherein the driving motion is determined based on the global features.

4. In paragraph 1, The above reinforcement learning module, An action module that determines the driving motion of the mobile body based on the driving environment characteristics, and evaluates whether the driving motion is successful and determines a reward value or a loss value based on the state of the mobile body according to the performance of the driving motion by the reinforcement learning module; and A method comprising a learning module that performs reinforcement learning for autonomous driving of the mobile device based on at least one of a reward value or loss value received from the action module, characteristics of the driving environment, the driving action, and information about the state of the mobile device after the driving action.

5. In paragraph 4, A method further comprising a step of storing driving information including the compensation value or the loss value, the driving environment information or the driving environment characteristics before performing the driving action, and information on the state of the moving object after performing the driving action.

6. In paragraph 1, The state of the above moving object includes information on whether a collision occurred after performing the driving action, In the step of evaluating the success or failure of the above driving action and providing a reward, The above reinforcement learning module is a method for assigning a reward value when a collision does not occur with the moving object, and assigning a loss value when a collision occurs with the moving object.

7. A computer program stored in a recording medium for executing a method according to any one of paragraphs 1 to 6.

Images