Road extraction apparatus

Abstract

To provide a road extraction apparatus and method capable of extracting a road from a satellite image using a deep learning model previously learned through multi-task learning.SOLUTION: The road extraction apparatus includes a data collection module for collecting road images related to a road and a deep learning module for executing a task to extract the road from the road images using deep learning technology. The deep learning module can include a first deep learning unit for performing segmentation on the road image and a second deep learning unit for performing object detection on the road image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a device and method for road extraction.

[0002] Specifically, the present invention relates to a road extraction device and method capable of extracting roads from satellite images using a deep learning model pre-trained through multi-task learning. [Background technology]

[0003] The material described in this section is merely intended to provide background information for the present embodiment and may not constitute prior art.

[0004] Currently, hundreds of artificial satellites around the world are performing their specific tasks in the sky above the earth for the purposes of meteorology, communications, broadcasting, agriculture, space development, military, etc. The satellite images taken by these artificial satellites are used in a variety of ways as public data in various fields.

[0005] Because such satellite images are taken at very high altitudes, there has been a recent surge in interest in technologies for processing, analyzing, and / or storing these satellite images using deep learning techniques.

[0006] In particular, these satellite images are often used to generate or update maps, and there has been a substantial need for a technique that attempts to accurately extract road configurations from satellite images in order to generate or update the maps. Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present invention is to provide a road extraction device and method capable of more accurate road extraction from satellite images by using a deep learning model that has been trained in advance through multi-task learning.

[0008] Specifically, the object of the present invention is to provide a road extraction device and method capable of performing different tasks by arranging multiple decoders of an autoencoder model that performs segmentation, some of which perform segmentation and other of which perform object detection.

[0009] Another object of the present invention is to provide a road extraction device and method capable of more accurate road extraction by additionally executing tasks related to a texture indicating the texture of a road, a road edge indicating an edge of a road, a road direction indicating an angle of a road, a not-road mark (e.g., a building, a plant, a water body), a road mark (e.g., a vehicle), etc., in addition to tasks related to a road surface, a road center line, etc.

[0010] The object of the present invention is not limited to the object mentioned above, and other objects and advantages of the present invention not mentioned will be understood by the following description and will become more clearly understood by the embodiments of the present invention. Also, it will be easily understood that the object and advantages of the present invention can be realized by the means and combinations thereof as set forth in the claims. [Means for solving the problem]

[0011] A road extraction device according to some embodiments of the present invention includes a data collection module that collects road images related to roads and a deep learning module that performs a task for extracting the roads from the road images using deep learning technology, and the deep learning module may include a first deep learning unit that performs segmentation on the road images and a second deep learning unit that performs object detection on the road images.

[0012] The road image may also include a satellite image relating to the road.

[0013] Additionally, the deep learning module can learn via multi task learning.

[0014] The deep learning module may also use an autoencoder to perform the task.

[0015] In addition, the first deep learning unit and the second deep learning unit may include at least one encoding unit that encodes the road image, which is input data, and converts it into a latent representation, and at least one decoding unit that converts the latent representation to generate output data.

[0016] Also, at least one of the first deep learning unit and the second deep learning unit may include a plurality of decoding units.

[0017] Also, each of the plurality of decoding units may be pre-trained to perform a different type of task.

[0018] In addition, the 1-1 decoding unit included in the first deep learning unit can perform tasks related to at least one of a road surface, a road center line, a texture indicating the texture of the road, an edge indicating the edge of the road, and a road direction indicating the angle of the road, the 1-2 decoding unit included in the first deep learning unit can perform tasks related to a not-road mark including at least one of a building, a plant, and a body of water, and the second decoding unit included in the second deep learning unit can perform tasks related to a road mark including a vehicle.

[0019] The present invention may further include a post-processing module that combines and processes the tasks performed by the first deep learning unit and the second deep learning unit to generate final data.

[0020] The post-processing module may also convert unconnected regions related to the roads, if any, into connected regions. Effect of the Invention

[0021] The road extraction apparatus and method according to some embodiments of the present invention have a novel effect of enabling more accurate road extraction from satellite images by using a deep learning model that is pre-trained through multi-task learning.

[0022] Specifically, in some embodiments of the road extraction apparatus and method of the present invention, some of the autoencoders perform segmentation and others perform object detection, but by arranging multiple decoders of the autoencoding model that perform segmentation, various types of tasks can be performed.

[0023] In addition, the road extraction device and method according to some embodiments of the present invention have the significant effect of enabling more accurate road extraction by additionally performing tasks related to the texture indicating the texture of the road, the road edge indicating the edge of the road, the road angle indicating the direction of the road, etc., in addition to tasks related to the road surface, road center line, etc.

[0024] Furthermore, the road extraction device and method according to some embodiments of the present invention have a new effect of further improving the accuracy of road extraction by additionally executing tasks related to non-road marks (e.g., buildings, plants, water bodies), road marks (e.g., cars), etc.

[0025] In addition to the above, specific effects of the present invention will be described together with the following detailed description of the implementation of the invention. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 illustrates a road extraction system according to some embodiments of the present invention. [Diagram 2] FIG. 2 is a block diagram of a road extraction device according to some embodiments of the present invention. [Figure 3a] FIG. 3a is a diagram for explaining the neural network structure of a deep learning module according to some embodiments of the present invention. [Figure 3b] FIG. 3b is a diagram illustrating an autoencoder according to some embodiments of the present invention. [Figure 4a] 1 is a diagram illustrating the structures of a first deep learning unit and a second deep learning unit according to some embodiments of the present invention. [Figure 4b]1 is a diagram illustrating the structures of a first deep learning unit and a second deep learning unit according to some embodiments of the present invention. [Figure 4c] 1 is a diagram illustrating the structures of a first deep learning unit and a second deep learning unit according to some embodiments of the present invention. [Figure 4d] 1 is a diagram illustrating the structures of a first deep learning unit and a second deep learning unit according to some embodiments of the present invention. [Diagram 5] FIG. 5 is a conceptual diagram illustrating the operation of a deep learning module according to some embodiments of the present invention. [Figure 6] FIG. 6 is a conceptual diagram illustrating the operation of a post-processing module according to some embodiments of the present invention. [Figure 7] FIG. 7 is a flow chart of a road extraction method according to some embodiments of the present invention. [Figure 8] FIG. 8 is a diagram illustrating the hardware implementation of a road extraction device for performing a road extraction method according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The terms or words used in this specification and claims should not be interpreted as being limited to their general or dictionary meanings. They should be interpreted as meanings and concepts consistent with the technical idea of ​​the present invention, based on the principle that the inventor can define the concept of the term or word to best describe his / her invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are merely one embodiment in which the present invention is realized, and do not represent the entire technical idea of ​​the present invention, so it should be understood that there may be various equivalents, modifications, and applicable examples that can replace them at the time of this application.

[0028] Terms such as first, second, A, B, etc., used in this specification and claims can be used to describe various components, but the components should not be limited by the terms. The terms are used only to distinguish one component from another. For example, a first component can be named as a second component, and similarly, the second component can be named as the first component, without departing from the scope of the invention. The term "and / or" includes any combination of multiple related listed items or multiple related listed items.

[0029] The terms used in the present specification and claims are merely used to describe specific embodiments and are not intended to limit the present invention. The singular expressions include the plural expressions unless otherwise clearly indicated in the context. In this application, the terms "comprise" or "have" and the like should be understood not to preclude the presence or additional possibility of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification.

[0030] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0031] Terms as defined in commonly used dictionaries should be interpreted to have a meaning consistent with the contextual meaning of the relevant art, and are not to be interpreted as idealized or overly formal, unless expressly defined in this application.

[0032] Furthermore, the configurations, processes, steps, or methods included in the respective embodiments of the present invention may be shared to the extent that there is no technical contradiction between them.

[0033] In the following, road extraction apparatus and methods according to some embodiments of the present invention will be described with reference to FIGS.

[0034] FIG. 1 illustrates a road extraction system according to some embodiments of the present invention.

[0035] Referring to FIG. 1, a road extraction system (1) according to some embodiments of the present invention may include an external database (100), a road extraction device (200), and a communication network (300).

[0036] The external database (100) may be a database that stores, manages, and / or transmits the raw data that is the subject of road extraction.

[0037] In some examples, the external database 100 may be in the form of various types of electronic devices, such as a computer, a laptop, a mobile device, a wearable device, a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, and a redundant array of inexpensive disks, or redundant array of independent disks (RAID) system, but the embodiments of the present invention are not limited thereto.

[0038] The external database (100) can transmit raw data for the road extraction device (200) to perform road extraction to the road extraction device (200). In other words, the road extraction device (200) can receive raw data related to road extraction from the external database (100).

[0039] The original data may refer to data that is a target of road extraction. For example, the original data may include a video, a video frame, an image, a panorama, etc., including a road configuration. In other words, the original data may include video data, video frame data, image data, panorama data, etc., including a road configuration. However, the embodiment of the present invention is not limited thereto.

[0040] For convenience of explanation, the following description will be given assuming that the original data is a road image.

[0041] In one example, the road image may include a satellite image including road morphology. The satellite image may refer to an image taken from a satellite located at a certain height in the sky. In this case, the external database (100) may be a database that stores, preserves, analyzes, and / or manages the satellite image, which is public data.

[0042] The road extraction device (200) is capable of extracting roads from road images, in other words, the road extraction device (200) is capable of performing road extraction from road images received from the external database (100).

[0043] In this case, the road extraction device (200) may be in the form of a workstation, a data center, an internet data center (IDC), a DAS (direct attached storage) system, a SAN (storage area network) system, a NAS (network attached storage) system, a RAID (redundant array of inexpensive disks, or redundant array of independent disks) system, etc., but the embodiment of the present invention is not limited thereto.

[0044] In some examples, the road extraction unit (200) may perform a task to extract roads from a road image using deep learning techniques.

[0045] In this case, the task may refer to a series of operations performed by the deep learning module. In other words, the task may refer to a series of processes or operations in which the deep learning module inputs input values ​​and outputs output values ​​to derive a predetermined result.

[0046] For example, the road extraction device 200 can perform tasks related to computer vision on the road image, or in other words, the road extraction device 200 can perform tasks related to image analysis on the road image.

[0047] For example, the road extraction device 200 can perform segmentation and object detection related to the shape and / or characteristics of the road on the road image. Here, segmentation refers to the task of classifying which class each pixel in the image belongs to, and object detection refers to the task of identifying the position and type of a specific object in the image and representing it with a bounding box or the like.

[0048] At this time, the deep learning module used by the road extraction device (200) of the present invention may be trained in advance through multi-task learning. In other words, the deep learning module included in the road extraction device (200) of the present invention may be trained so that one deep learning model processes various tasks. At this time, the deep learning model used by the deep learning module of the present invention may include an autoencoder, but the embodiment of the present invention is not limited thereto.

[0049] For example, a first deep learning unit included in the road extraction device 200 of the present invention may perform segmentation based on an autoencoder, and a second deep learning unit included in the road extraction device 200 of the present invention may perform object detection based on the autoencoder. At least one of the first deep learning unit and the second deep learning unit may include a plurality of encoding units and / or a plurality of decoding units, and each decoding unit may be pre-trained to perform a different task.

[0050] As a specific example, the 1-1 decoding unit included in the first deep learning unit may be pre-trained to perform segmentation on a road surface, a road center line, a texture indicating the texture of a road, an edge indicating an edge of a road, and a direction indicating an angle of a road. Also, the 1-2 decoding unit included in the first deep learning unit may be pre-trained to perform segmentation on a not-road mark including a building, a plant, and a water body. Also, the second decoding unit included in the second deep learning unit may be pre-trained to perform object detection on a road mark including a vehicle.

[0051] In this case, a non-road mark generally means an object that is not included in a road, and conversely, a road mark generally means an object that is included in a road, and the types of non-road marks and road marks can be defined in advance by an administrator of the road extraction device (200) of the present invention.

[0052] In other words, the road extraction device (200) of the present invention goes further than general road extraction methods by not determining that an area in which a non-road mark (e.g., a building) is detected is a road, and by determining that an area in which a road mark (e.g., a vehicle) is detected is a road, thereby achieving the new effect of being able to perform more accurate road extraction using non-road marks and road marks.

[0053] The detailed operation process of the road extraction device (200) will be described later.

[0054] Meanwhile, the communication network (300) serves to connect the external database (100) and the road extraction device (200). That is, the communication network (300) refers to a communication network that provides a connection path so that the road extraction device (200) can transmit and receive data from the external database (100). The communication network (300) can include, for example, wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), and ISDNs (Integrated Service Digital Networks), and wireless networks such as wireless LANs, CDMA, Bluetooth (registered trademark), and satellite communications, but the scope of the present invention is not limited thereto.

[0055] The operation of the road extraction device (200) will now be described in more detail with reference to FIG.

[0056] FIG. 2 is a block diagram of a road extraction device according to some embodiments of the present invention.

[0057] Referring to Figures 1 and 2, a road extraction device (200) receives a road image (RI), and then performs road extraction from the received road image (RI) to generate final data (FD).

[0058] Specifically, the road extraction device 200 may include a data collection module 210, a deep learning module 220, and a post-processing module 230. The deep learning module 220 may include a first deep learning unit 221 and a second deep learning unit 222.

[0059] The data collection module (210) is capable of receiving the road image (RI), in other words, the data collection module (210) is capable of receiving the road image (RI) from the external database (100).

[0060] In this case, the road image (RI) may include a satellite image including the shape of the road. The satellite image may mean an image taken from an artificial satellite located at a certain height in the sky. That is, the road extraction device (200) may receive the road image (RI) including the satellite image from the external database (100), which is a database that stores, preserves, analyzes, and / or manages satellite images that are public data.

[0061] The data collection module 210 can transmit the received road image (RI) to other components in the road extraction device 200. For example, the data collection module 210 can transmit the road image (RI) to the deep learning module 220, but the present invention is not limited thereto.

[0062] The data collection module (210) can use various communication modules to facilitate data exchange between the external database (100) and the road extraction device (200) via a communication network (300 in FIG. 1).

[0063] The deep learning module 220 can execute a task for extracting roads from the road image RI. In other words, the deep learning module 220 can execute a task related to road extraction on the road image RI to generate a task result (hereinafter referred to as "TR").

[0064] In this case, the deep learning module (220) may include a first deep learning unit (221) and a second deep learning unit (222), whereby the first deep learning unit (221) may generate a first task execution result (TR1), and the second deep learning unit (222) may generate a second task execution result (TR2).

[0065] The task performed by the deep learning module 220 may refer to a series of operations performed by the deep learning module 220. In other words, the task may refer to a series of processes or operations in which the deep learning module 220 inputs input values ​​and outputs output values ​​to derive a predetermined result.

[0066] For example, the deep learning module 220 may perform tasks related to computer vision on road images. In other words, the deep learning module 220 may perform tasks related to image analysis on road images. For example, the deep learning module 220 may perform segmentation and object detection related to road characteristics on road images. Here, segmentation refers to a task of classifying which class each pixel in an image belongs to, and object detection refers to a task of identifying the position and type of a specific object in an image and representing it with a bounding box, etc.

[0067] At this time, the deep learning module (220) can learn based on machine learning.

[0068] More specifically, deep learning technology, which is a type of machine learning, is a technology that learns at a deep level in multiple steps based on data. In other words, deep learning refers to a collection of machine learning algorithms that extract key data from multiple data by increasing the level.

[0069] In some examples, the deep learning module (220) may utilize various well-known deep learning structures, such as a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), a graph neural network (GNN), a generative adversarial network (GAN), a transformer, an autoencoder, or the like.

[0070] Specifically, CNN (Convolutional Neural Network) is a model that replicates the functions of the human brain, based on the assumption that when humans recognize an object, they extract basic features of the object, then perform complex calculations in the brain, and recognize the object based on the results. CNN can include, but is not limited to, well-known structures such as LeNet, AlexNet, VGGNet, GoogleNet, and ResNet.

[0071] Recurrent Neural Networks (RNNs) are often used in natural language processing and other fields. They are an effective structure for processing time-series data that changes over time, and can form an artificial neural network structure by stacking layers at each moment.

[0072] DBN (Deep Belief Network) is a deep learning structure that consists of stacking multiple layers of RBM (Restricted Boltzmann Machine), a deep learning technology. When a certain number of layers are reached through repeated RBM (Restricted Boltzmann Machine) learning, a DBN (Deep Belief Network) with the corresponding number of layers can be constructed.

[0073] GNN (Graphic Neural Network, hereinafter referred to as "GNN") refers to an artificial neural network structure implemented in a manner that derives similarities and feature points between modeling data using modeling data modeled based on data mapped between specific parameters.

[0074] GAN (Generative Adversarial Network, hereinafter referred to as "GAN") refers to an artificial neural network structure that creates new data in a form similar to input data using a generative neural network and a discriminative neural network. GAN may include well-known DCGAN (Deep Convolutional GAN), CGAN (Conditional GAN), WGAN (Wasserstein GAN), StyleGAN (Style-Based GAN), CycleGAN, etc., but the embodiment of the present invention is not limited thereto.

[0075] Transformer is an attention-driven, encoder-decoder-structured artificial neural network that can grasp the overall meaning between input and output sequences. Transformer uses the attention mechanism to ensure that all elements of the input sequence affect the output sequence, allowing both the encoder and decoder to consider the entire sequence. Transformer can use natural language, time series data, as well as image patches as input.

[0076] An autoencoder is a deep learning structure that extracts and reconstructs the features of data. Typically, an autoencoder includes an encoder that compresses input values ​​and a decoder that restores the compressed data. The encoder converts the input values ​​into a low-dimensional latent representation, and the decoder restores the latent representation to the same dimensions as the input values. In this case, the encoder and decoder can each be configured as a multi-layer perceptron (MLP). When training an autoencoder, input data is input and weights and biases are learned in a direction that minimizes the difference between the output value and the input value. The trained autoencoder can successfully extract the features of the input data and restore noisy input data. Autoencoders are mainly used in fields such as data compression, dimensionality reduction, noise removal, and data generation, and can also be used in fields such as image recognition, natural language processing, and voice recognition.

[0077] Meanwhile, the artificial neural network learning of the deep learning module (220) can be performed by adjusting the weights of the connection lines between the nodes (and adjusting the bias values, if necessary) so that a desired output is obtained for a given input. Furthermore, the artificial neural network can continuously update the weight values ​​through learning. Also, methods such as back propagation can be used to train the artificial neural network.

[0078] In this case, the machine learning method of the artificial neural network may be unsupervised learning, semi-supervised learning, supervised learning, etc. Furthermore, the deep learning module (220) may be controlled to automatically update the artificial neural network structure for outputting analysis data after learning according to settings.

[0079] At this time, the deep learning module (220) according to some embodiments of the present invention may be pre-trained through multi-task learning. In other words, the deep learning module (220) may be trained so that one deep learning model processes various tasks. At this time, the deep learning model used by the deep learning module of the present invention may include the above-mentioned autoencoder, but the embodiments of the present invention are not limited thereto.

[0080] For example, the first deep learning unit 221 may perform segmentation based on an autoencoder, and the second deep learning unit 222 may perform object detection based on the autoencoder. At least one of the first deep learning unit and the second deep learning unit may include a plurality of encoding units and / or a plurality of decoding units, and each decoding unit may be pre-trained to perform a different task.

[0081] As a specific example, the 1-1 decoding unit included in the first deep learning unit (221) may be pre-trained to perform segmentation on a road surface, a road center line, a texture indicating the texture of a road, an edge indicating the edge of a road, and a direction indicating the angle of a road. In addition, the 1-2 decoding unit included in the first deep learning unit (221) may be pre-trained to perform segmentation on a not-road mark including a building, a plant, and a water body. In addition, the second decoding unit included in the second deep learning unit (222) may be pre-trained to perform object detection on a road mark including a vehicle.

[0082] In this case, a non-road mark generally means an object that is not included in a road, and conversely, a road mark generally means an object that is included in a road, and the types of non-road marks and road marks can be defined in advance by an administrator of the road extraction device (200) of the present invention.

[0083] In other words, the deep learning module (220) of the present invention goes further than general road extraction methods by not determining that an area where a non-road mark (e.g., a building) is detected is a road, and by determining that an area where a road mark (e.g., a vehicle) is detected is a road, thereby achieving the new effect of being able to perform more accurate road extraction using non-road marks and road marks.

[0084] The neural network structure of the deep learning module (220) according to some embodiments of the present invention will now be described in further detail with reference to FIG. 3a.

[0085] FIG. 3a is a diagram for explaining the neural network structure of a deep learning module according to some embodiments of the present invention.

[0086] Referring to FIG. 3a, a deep learning module (220) according to some embodiments of the present invention may include an input layer (input) with road images as input nodes, an output layer (Output) with task execution results as output nodes, and M hidden layers arranged between the input layer and the output layer.

[0087] Here, weights can be set for the edges connecting the nodes of each layer. These weights or the presence or absence of edges can be added, deleted, or updated during the learning process. Thus, the weights of the nodes and edges placed between the k input nodes and the i output nodes can be updated through the learning process.

[0088] Before the deep learning module (220) performs learning, all nodes and edges can be set to initial values. However, when information is input in a cumulative manner, the weights of the nodes and edges are changed, and during this process, matching may be performed between the parameters (road images) input to the learning factors and the values ​​(task execution results) assigned to the output nodes.

[0089] Additionally, when using a cloud server, the deep learning module 220 can receive and process a large number of parameters, and thus the deep learning module 220 can perform learning based on a large amount of data.

[0090] Weights of nodes and edges between input nodes and output nodes constituting the deep learning module 220 can be updated through the learning process of the deep learning module 220. In addition, it goes without saying that parameters input or output by the deep learning module 220 can be expanded to various data other than road images and task execution results.

[0091] The autoencoder used in the present invention will now be described in more detail with reference to FIG. 3b.

[0092] FIG. 3b is a diagram illustrating an autoencoder according to some embodiments of the present invention.

[0093] Referring to FIG. 3b, an autoencoder (hereinafter referred to as "AE") includes an encoder neural network (hereinafter referred to as encoder unit (EN)) and a decoder neural network (hereinafter referred to as decoder unit (DN)), and may include a middle layer (ML) disposed between the encoder unit (EN) and the decoder unit (DN).

[0094] The autoencoder (AE) is a kind of deep neural network model that compresses data (i.e., input data) input through the encoder unit (EN) to reduce the data size, and then converts the reduced data using the decoder unit (DN) to the same size as the input data in the encoder unit (EN) and outputs it, thereby making the output data of the autoencoder (AE) the same as the input data.

[0095] An autoencoder (AE) learns the features of input data in an unsupervised manner. To this end, the autoencoder (AE) converts the input data through the encoder (EN) into low-dimensional data (latent representation) that effectively expresses the features, and the converted data can then be restored to the original data through the decoder (DN).

[0096] An autoencoder (AE) can learn patterns inherent in the original data with the goal of minimizing the reconstruction error, which corresponds to the difference between the original data (X1, X2, X3, X4; i.e., input data (road image)) and the restored data (X1', X2', X3', X4'; i.e., output data (task execution result)).

[0097] Hereinafter, the structure of the deep learning module (220) according to some embodiments of the present invention will be described with reference to Figures 4a and 4b. That is, the configurations of the first deep learning unit (221) and the second deep learning unit (222) implemented through the above-mentioned autoencoder will be described with reference to Figures 4a and 4b.

[0098] 4a to 4d are diagrams for explaining structures of a first deep learning unit and a second deep learning unit according to some embodiments of the present invention.

[0099] Referring to Figures 4a to 4d, as described above, the deep learning module (220) according to some embodiments of the present invention may be pre-trained based on multi-task learning, and in order to implement this, the deep learning module (220) may be designed separately into a first deep learning unit (221) and a second deep learning unit (222), and in this case, at least one of the first deep learning unit (221) and the second deep learning unit (222) may include a plurality of encoder units and / or a plurality of decoder units.

[0100] Specifically, FIG. 4a illustrates a structure in which the deep learning module (220) is designed to be separated into a first deep learning unit (221) and a second deep learning unit (222), in which the first deep learning unit (221) and the second deep learning unit (222) each include an encoding unit (221a, 222a) and a decoding unit (221b, 222b), and the first decoding unit (221b) is designed to be separated into a 1-1 decoding unit (221b_1) and a 1-2 decoding unit (221b_2), etc.

[0101] FIG. 4b illustrates a structure in which the deep learning module (220) is designed to be separated into a first deep learning unit (221) and a second deep learning unit (222), in which the first deep learning unit (221) and the second deep learning unit (222) each include an encoding unit (221a, 222a) and a decoding unit (221b, 222b), and the second decoding unit (222b) is designed to be separated into a 2-1 decoding unit (222b_1) and a 2-2 decoding unit (222b_2), etc.

[0102] FIG. 4c illustrates a structure in which the deep learning module (220) is designed to be separated into a first deep learning unit (221) and a second deep learning unit (222), in which the first deep learning unit (221) and the second deep learning unit (222) each include an encoding unit (221a, 222a) and a decoding unit (221b, 222b), in which the first decoding unit (221b) is designed to be separated into a 1-1 decoding unit (221b_1) and a 1-2 decoding unit (221b_2), etc., and the second decoding unit (222b) is designed to be separated into a 2-1 decoding unit (222b_1) and a 2-2 decoding unit (222b_2), etc.

[0103] FIG. 4d illustrates a structure in which the deep learning module (220) is designed to be separated into a first deep learning unit (221) and a second deep learning unit (222), and in this case, the first deep learning unit (221) and the second deep learning unit (222) each include an encoding unit (221a, 222a) and a decoding unit (221b, 222b), and the first encoding unit (221a) is designed to be separated into a 1-1 encoding unit (221a_1) and a 1-2 encoding unit (221a_2), etc., and the second decoding unit (222b) is designed to be separated into a 2-1 decoding unit (222b_1) and a 2-2 decoding unit (222b_2), etc.

[0104] In this case, the multiple encoding units and / or multiple decoding units can be pre-trained to perform different tasks, and the deep learning module (220) of the present invention having undergone multi-task learning through this configuration can process various tasks through one deep learning model (autoencoder).

[0105] Hereinafter, a specific example of a multi-task learning method of the deep learning module (220) of the present invention will be described with reference to Fig. 5. Hereinafter, for convenience of explanation, the structure of the deep learning module (220) of the present invention for implementing multi-task learning will be described assuming the structure shown in Fig. 4a.

[0106] 5 is a conceptual diagram for explaining the operation of a deep learning module according to some embodiments of the present invention. Specifically, FIG 5 illustrates an operation method when the structure of the deep learning module (220) of the present invention is the structure illustrated in FIG 4a.

[0107] Referring to FIG. 5, a deep learning module (220) according to some embodiments of the present invention may include a first deep learning unit (221) and a second deep learning unit (222), and in this case, the first deep learning unit (221) and the second deep learning unit (222) may each include an encoding unit (221a, 222a) and a decoding unit (221b, 222b).

[0108] The first deep learning unit (221) may perform segmentation on the road image (RI) through the first encoding unit (221a) and the first decoding unit (221b), and the second deep learning unit (222) may perform object detection on the road image (RI) through the second encoding unit (222a) and the second decoding unit (222b). Segmentation may refer to a task of classifying which class each pixel in the road image (RI) belongs to, and object detection may refer to a task of identifying the position and type of a specific object in the road image (RI) and expressing it as a bounding box (BB), etc.

[0109] In some examples, the first encoding unit 221a included in the first deep learning unit 221 may be pre-trained in a manner for performing segmentation, and the second encoding unit 222a included in the second deep learning unit 222 may be pre-trained in a manner for performing object detection. In this case, the first encoding unit 221a and the second encoding unit 222a may have different encoder structures to perform their respective tasks. For example, the first encoding unit 221a and the second encoding unit 222a may be pre-trained to have different neural network depths, presence or absence of downsampling, types of layers included, presence or absence of skip connections, etc.

[0110] In order to implement the above-mentioned multi-task learning, the first decoding unit (221b) of the present invention may include a plurality of decoding units (221b_1, 221b_2, etc.) In other words, the first decoding unit (221b) of the present invention may include a 1-1 decoding unit (221b_1), a 1-2 decoding unit (221b_2), etc.

[0111] At this time, each decoding unit (221b_1, 221b_2, etc.) included in the first decoding unit (221b) may be trained in advance to perform different tasks. In other words, the first deep learning unit (221) is trained to perform segmentation, and at this time, each decoding unit (221b_1, 221b_2, etc.) included in the first decoding unit (221b) may be trained in advance to perform different segmentation tasks among segmentation.

[0112] As a specific example, the 1-1 decoding unit (221b_1) included in the first deep learning unit (221) may be pre-trained to perform segmentation regarding a road surface, a road center line, a texture indicating the texture of the road, an edge indicating the edge of the road, and a road direction indicating the angle of the road.

[0113] The task execution result (TR1_1) of the 1-1 decoding unit (221b_1) shown in Figure 5 illustrates a case where the 1-1 decoding unit (221b_1) performs segmentation of the road surface on the latent representation after the 1st encoding unit (221a) converts the road image (RI) into a latent representation.

[0114] In addition, the first-second decoding unit (221b_2) included in the first deep learning unit (221) can be pre-trained to perform segmentation on non-road marks including buildings, plants, water bodies, etc. In this case, the non-road marks generally refer to objects that are not included in roads, and the types of non-road marks can be pre-defined by the administrator of the road extraction device (200) of the present invention.

[0115] The task execution result (TR1_2) of the 1-2 decoding unit (221b_2) shown in Figure 5 illustrates a case where the 1-2 decoding unit (221b_2) performs segmentation of a building, which is an example of a non-road mark, on the latent representation after the 1-2 encoding unit (221a) converts the road image (RI) into a latent representation.

[0116] As described above, in order to perform different segmentation tasks, the 1-1 decoding unit 221b_1 and the 1-2 decoding unit 221b_2 of the present invention may have different decoder structures. For example, the 1-1 decoding unit 221b_1 and the 1-2 decoding unit 221b_2 may differ in the decoder neural network architecture, the type of loss function, the presence and type of normalization function, the type of activation function, etc. In this case, the neural network structure may include the number of output layers, the type of layers included (e.g., U-Net, FCL (Fully Connected Layer)), etc.

[0117] Meanwhile, the second decoding unit included in the second deep learning unit 222 can be pre-trained to perform object detection related to road marks. In this case, a road mark generally means an object included in a road, and the type of road mark can be pre-defined by an administrator of the road extraction device 200 of the present invention.

[0118] The task execution result (TR2) of the second decoding unit (222b) shown in Fig. 5 illustrates a case where the second encoding unit (222a) converts the road image (RI) into a latent representation, and then the second decoding unit (222b) performs object detection on the latent representation for a vehicle (car), which is an example of a road mark. As a result of the object detection, the task execution result (TR2) is illustrated in Fig. 5 to include a bounding box (BB).

[0119] The deep learning module (220) of the present invention, in which such multi-task learning is performed, can process various tasks through one deep learning model (autoencoder).

[0120] In summary, the deep learning module (220) of the present invention detects the road surface, the road center line, the texture indicating the road texture, the road edge indicating the road edge, and the road angle indicating the road angle, and then determines that an area where a non-road mark (e.g., a building) is detected is not a road, and determines that an area where a road mark (e.g., a vehicle) is detected is a road. As a result, the deep learning module (220) of the present invention has a new effect of going beyond the general road extraction method and performing more accurate road extraction using non-road marks and road marks.

[0121] 1 and 2, the deep learning module (220) can transmit the task execution results (TR1, TR2) to the post-processing module (230). In other words, the first deep learning unit (221) can transmit the first task execution result (TR1) to the post-processing module (230), and the second deep learning unit (222) can transmit the second task execution result (TR2) to the post-processing module (230).

[0122] The post-processing module (230) can generate final data (FD) based on the task execution result of the deep learning module (220). In other words, the post-processing module (230) can generate final data (FD) based on the first task execution result (TR1) and the second task execution result (TR2).

[0123] The operation of the post-processing module (230) according to some embodiments of the invention will now be described in further detail with reference to FIG.

[0124] FIG. 6 is a conceptual diagram illustrating the operation of a post-processing module according to some embodiments of the present invention.

[0125] 2 and 6, the post-processing module 230 may merge and process the first task execution result TR1 and the second task execution result TR2 to generate final data FD. As described above, in order to implement multi-task learning, the first decoding unit 221b in FIG. 5 may be implemented with a plurality of decoding units 221b_1 and 221b_2 in FIG. 5, so that the first task execution result TR1 may include the 1-1 task execution result TR1_1 and the 1-2 task execution result TR1_2.

[0126] In some examples, the post-processing module 230 may process the first task execution result TR1 and the second task execution result TR2 after merging them to generate the final data FD. For example, the post-processing module 230 may include a refinement network, and the final data FD may be generated through the refinement network.

[0127] In this case, if there is an unconnected area related to the road, the post-processing module 230 can convert the unconnected area into a connected area. In other words, if there is an unconnected area in the merged result of the first task execution result TR1 and the second task execution result TR2, the post-processing module 230 can convert the unconnected area into a connected area.

[0128] In one example, the post-processing module 230 may perform common noise reduction techniques, smoothing, binarization, morphological operations, etc. to convert non-connected regions into connected regions, although the present invention is not limited thereto.

[0129] Fig. 7 is a flow chart of a road extraction method according to some embodiments of the present invention. Each step (S100 to S300) in Fig. 7 can be performed by the road extraction device (200 in Figs. 1 and 2). Below, we will briefly explain it without overlapping contents.

[0130] 1, 2 and 7, first, a road image relating to a road can be collected (S100).

[0131] In some examples, the data collection module (210) can receive the road image (RI), in other words, the data collection module (210) can receive the road image (RI) from the external database (100).

[0132] In this case, the road image (RI) may include a satellite image including the shape of the road. The satellite image may mean an image taken from an artificial satellite located at a certain height in the sky. That is, the road extraction device (200) may receive the road image (RI) including the satellite image from the external database (100), which is a database that stores, preserves, analyzes, and / or manages satellite images that are public data.

[0133] Next, a task can be performed to extract the road from the road image using deep learning techniques (S200).

[0134] The task performed by the deep learning module (220) may refer to a series of operations performed by the deep learning module (220). In other words, the task may refer to a series of processes or operations in which the deep learning module (220) inputs input values ​​and outputs output values ​​to derive a predetermined result. For example, the deep learning module (220) may perform a task related to computer vision for a road image. In other words, the deep learning module (220) may perform a task related to image analysis for a road image. For example, the deep learning module (220) may perform segmentation and object detection related to road characteristics for a road image. Here, segmentation refers to a task of classifying which class each pixel in an image belongs to, and object detection refers to a task of identifying the position and type of a specific object in an image and expressing it as a bounding box, etc. At this time, the deep learning module (220) may include a first deep learning unit (221) and a second deep learning unit (222), whereby the first deep learning unit (221) may generate a first task execution result (TR1), and the second deep learning unit (222) may generate a second task execution result (TR2).

[0135] In some examples, the deep learning module (220) according to some embodiments of the present invention may be pre-trained through multi-task learning. In other words, the deep learning module (220) may be trained to have one deep learning model to handle various tasks. In this case, the deep learning model used by the deep learning module of the present invention may include the above-mentioned autoencoder, but the embodiments of the present invention are not limited thereto.

[0136] For example, the first deep learning unit 221 may perform segmentation based on an autoencoder, and the second deep learning unit 222 may perform object detection based on the autoencoder. At least one of the first deep learning unit and the second deep learning unit may include a plurality of encoding units and / or a plurality of decoding units, and each decoding unit may be pre-trained to perform a different task.

[0137] As a specific example, the 1-1 decoding unit included in the first deep learning unit (221) may be pre-trained to perform segmentation on a road surface, a road center line, a texture indicating the texture of a road, an edge indicating the edge of a road, and a direction indicating the angle of a road. In addition, the 1-2 decoding unit included in the first deep learning unit (221) may be pre-trained to perform segmentation on a not-road mark including a building, a plant, and a water body. In addition, the second decoding unit included in the second deep learning unit (222) may be pre-trained to perform object detection on a road mark including a vehicle.

[0138] In this case, a non-road mark generally means an object that is not included in a road, and conversely, a road mark generally means an object that is included in a road, and the types of non-road marks and road marks can be defined in advance by an administrator of the road extraction device (200) of the present invention.

[0139] In other words, the deep learning module (220) of the present invention goes further than general road extraction methods by not determining that an area where a non-road mark (e.g., a building) is detected is a road, and by determining that an area where a road mark (e.g., a vehicle) is detected is a road, thereby achieving the new effect of being able to perform more accurate road extraction using non-road marks and road marks.

[0140] Each executed task can then be post-processed and output (S300).

[0141] In some examples, the post-processing module (230) can generate final data (FD) based on the task execution result of the deep learning module (220). In other words, the post-processing module (230) can generate final data (FD) based on the first task execution result (TR1) and the second task execution result (TR2).

[0142] For example, the post-processing module 230 may merge the first task execution result TR1 and the second task execution result TR2 and process them to generate final data FD. For example, the post-processing module 230 may include a refinement network and may generate final data FD through the refinement network.

[0143] In this case, if there is an unconnected area related to the road, the post-processing module 230 can convert the unconnected area into a connected area. In other words, if there is an unconnected area in the merged result of the first task execution result TR1 and the second task execution result TR2, the post-processing module 230 can convert the unconnected area into a connected area.

[0144] FIG. 8 is a diagram illustrating a hardware implementation of a road extraction device for performing a road extraction method according to some embodiments of the present invention.

[0145] Referring to Fig. 8, a road extraction device (200) for performing a road extraction method according to some embodiments of the present invention may be embodied as an electronic device (1000). The electronic device (1000) may include a controller (1010), an input / output device (1020), a memory device (1030), an interface (1040), and a bus (1050). The controller (1010), the input / output device (1020), the memory device (1030), and / or the interface (1040) may be connected to each other via a bus (1050). In this case, the bus (1050) corresponds to a path through which data moves.

[0146] Specifically, the controller (1010) may include at least one of a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphic processing unit (GPU), a microprocessor, digital signal processing, a microcontroller, an application processor (AP), and logic elements capable of performing similar functions.

[0147] The input / output device (1020) may include at least one of a keypad, a keyboard, a touch screen, and a display device.

[0148] The memory device (1030) may store data and / or programs, etc.

[0149] The interface (1040) may perform a function of transmitting data to a communication network or receiving data from a communication network. The interface (1040) may be in the form of a wired or wireless connection. For example, the interface (1040) may include an antenna or a wired or wireless transceiver. Although not shown, the memory device (1030) is an operating memory for improving the operation of the controller (1010) and may further include a high-speed dynamic random access memory (DRAM) and / or static random access memory (SRAM). The memory device (1030) may store a program or application therein.

[0150] The road extraction device 200 and the external database 100 according to the embodiment of the present invention may each be a system formed by connecting a plurality of electronic devices 1000 to each other via a network. In this case, each module or a combination of modules may be embodied as the electronic device 1000. However, the present embodiment is not limited thereto.

[0151] Additionally, the road extraction device (200) may be embodied in at least one of a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, a redundant array of inexpensive disks (RAID), or redundant array of independent disks (RAID) system, and an electronic document management (EDMS) system, but the present embodiment is not limited thereto.

[0152] The road extraction device (200) can also transmit data to the external database (100) via a network. The network can include a network using wired Internet technology, wireless Internet technology, and short-range communication technology. The wired Internet technology can include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

[0153] The wireless Internet technology may include at least one of, for example, Wireless LAN (WLAN), Digital Living Network Alliance (DMNA), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed ​​Downlink Packet Access (HSDPA), High Speed ​​Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS), and 5G New Radio (NR) technologies. However, the present embodiment is not limited thereto.

[0154] The short-range communication technology may include at least one of, for example, Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee (registered trademark), Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G New Radio (NR). However, the present embodiment is not limited thereto.

[0155] The road extraction device (200) communicating through the network may comply with technical standards and standard communication methods for mobile communication. For example, the standard communication methods may include at least one of GSM (registered trademark), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (registered trademark), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTEA (Long Term Evolution-Advanced), and 5G NR (New Radio). However, the present embodiment is not limited thereto.

[0156] The above description is merely an illustrative example of the technical idea of ​​the present embodiment, and various modifications and variations may be made by a person having ordinary knowledge in the technical field to which the present embodiment belongs, without departing from the essential characteristics of the present embodiment. Therefore, the present embodiment is intended to explain, not to limit, the technical idea of ​​the present embodiment, and the scope of the technical idea of ​​the present embodiment is not limited by such an embodiment. The scope of protection of the present embodiment should be interpreted according to the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

Claims

1. a data collection module for collecting road images relating to a road; and A deep learning module performs a task for extracting the road from the road image using deep learning techniques, The deep learning module includes: A first deep learning unit that performs segmentation on the road image; A second deep learning unit that performs object detection on the road image; Road extractor.

2. The road image is Includes satellite imagery relating to said road The road extraction device according to claim 1 .

3. The deep learning module includes: Learning through multi-task learning The road extraction device according to claim 1 .

4. The deep learning module includes: Performing the task using an autoencoder The road extraction device according to claim 3.

5. The first deep learning unit and the second deep learning unit, At least one encoding unit for encoding the road image as input data and converting it into a latent representation; at least one decoding unit that transforms the latent representation to generate output data; The road extraction device according to claim 4.

6. At least one of the first deep learning unit and the second deep learning unit, Includes multiple decoding units The road extraction device according to claim 5.

7. Each of the plurality of decoding units comprises: They are pre-trained to perform different types of tasks The road extraction device according to claim 6.

8. The first-1 decoding unit included in the first deep learning unit includes: Executing a task related to at least one of a road surface, a road center line, a texture indicating the texture of the road, a road edge indicating an edge of the road, and a road direction indicating an angle of the road; The first-second decoding unit included in the first deep learning unit includes: performing a task on a not-road mark including at least one of a building, a plant, and a body of water; The second decoding unit included in the second deep learning unit includes: Perform tasks related to road marks including vehicles The road extraction device according to claim 7.

9. The deep learning unit further includes a post-processing module that processes the tasks performed by the first deep learning unit and the second deep learning unit to generate final data. The road extraction device according to claim 1 .

10. The post-processing module includes: If there is an unconnected area related to the road, convert the unconnected area into a connected area. The road extraction device according to claim 9.

Images