Road extraction device and method
Through the deep learning model and autoencoder of multi-task learning, combined with segmentation and object detection tasks, the accuracy of road extraction in satellite images is solved, and the accurate identification of road shape, texture, edge, direction and non-road markings is achieved. Improved accuracy of map generation and update.
Patent Information
- Application Number
- CN202311807996.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-09-08
- Filing Date
- 2023-12-26
- Publication Date
- 2024-11-08
AI Technical Summary
In existing satellite images, road extraction technology is difficult to accurately extract the shape, texture, edges, directions and distinguish road from non-road markers, resulting in difficulties in generating and updating maps.
Using a deep learning model of multi-task learning, combined with the autoencoder, segmentation and object detection tasks, different tasks are performed through multiple decoding departments, including road shape, texture, edge, direction, identification of buildings, plants, waters and vehicles. To extract the road accurately.
It realizes more accurate extraction of roads in satellite images, can distinguish roads from non-road markings, and improves the accuracy of map generation and updates.
Smart Images

Figure CN118918488A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a road extraction device and method.
[0002] Specifically, the present invention relates to a road extraction device and method that can extract roads from satellite images using a deep learning model that is pre-learned through multi-task learning. Background Art
[0003] The contents described herein are only used to provide background technology for this embodiment and cannot constitute prior art.
[0004] Currently, there are hundreds of artificial satellites around the world flying above the Earth, performing fixed missions for purposes such as meteorology, communications, broadcasting, agriculture, space development, and military. Satellite images taken by these artificial satellites can be used as public data in a wide range of fields.
[0005] The above-mentioned satellite images are images taken at very high altitudes, and therefore, technologies for processing, analyzing and / or storing the above-mentioned satellite images using deep learning technologies are currently attracting much attention.
[0006] In particular, the satellite images are often used to generate or update maps. In this case, a technology for accurately extracting road shapes from satellite images is highly required in order to generate or update corresponding maps. Summary of the invention
[0007] An object of the present invention is to provide a road extraction device and method that can further accurately extract roads from satellite images using a deep learning model that has been pre-learned through multi-task learning.
[0008] Specifically, the object of the present invention is to provide a road extraction device and method, that is, to enable a part of the autoencoder to perform segmentation and the other part to perform object detection, thereby performing different tasks by configuring the decoders of multiple autoencoder models for performing segmentation.
[0009] Furthermore, another object of the present invention is to provide a road extraction device and method, that is, in addition to tasks related to road shape (road surface), road center line (center line), etc., the road can be further accurately extracted by additionally performing tasks related to texture (texture) representing road texture, road boundary (edge) representing road edge, road direction (direction) representing road angle, non-road marks (not-road marks, such as buildings, plants, water), road marks (road marks, such as vehicles (car)), etc.
[0010] The purpose of the present invention is not limited to the purpose mentioned above. Other purposes and advantages not mentioned in the present invention can not only be understood based on the following description, but also can be further clearly understood through the embodiments of the present invention. Moreover, it is obvious that the purpose and advantages of the present invention can be achieved by the devices and combinations thereof represented by the scope of protection of the invention.
[0011] The road extraction device of multiple embodiments of the present invention includes: a data collection module for collecting road images related to the road; and a deep learning module for performing a task for extracting the road from the road image using deep learning technology. The deep learning module may include: a first deep learning unit for performing segmentation using the road image as an object; and a second deep learning unit for performing object detection using the road image as an object.
[0012] Furthermore, the road image may include a satellite image related to the road.
[0013] Furthermore, the above-mentioned deep learning module can be learned through multi-task learning.
[0014] Furthermore, the deep learning module may utilize an autoencoder to perform the above tasks.
[0015] Furthermore, the first deep learning unit and the second deep learning unit may include: at least one encoding unit that encodes the road image as input data and converts it into a latent representation; and at least one decoding unit that generates output data by converting the latent representation.
[0016] Furthermore, at least one of the first deep learning unit and the second deep learning unit may include a plurality of decoding units.
[0017] Furthermore, the plurality of decoding units may perform pre-learning respectively to execute different types of tasks.
[0018] Furthermore, the 1-1 decoding unit included in the above-mentioned first deep learning unit performs tasks related to at least one of a road shape (road surface), a road center line (center line), a texture (texture) representing the texture of the above-mentioned road, a road boundary (edge) representing the edge of the above-mentioned road, and a road direction (direction) representing the angle of the above-mentioned road; the 1-2 decoding unit included in the above-mentioned first deep learning unit performs tasks related to non-road marks (not-road marks) including at least one of a building (building), a plant (plant) and a water area (water); and the second decoding unit included in the above-mentioned second deep learning unit can perform tasks related to road marks (road marks) including vehicles (car).
[0019] Furthermore, the road extraction device of various embodiments of the present invention may further include a post-processing module for merging the tasks respectively performed by the first deep learning unit and the second deep learning unit and processing them to generate final data.
[0020] Furthermore, when there is an unconnected area associated with the road, the post-processing module may convert the corresponding unconnected area into a connected area.
[0021] The road extraction device and method according to the various embodiments of the present invention have the following effect, that is, a deep learning model pre-learned by multi-task learning can be used to further accurately extract roads from satellite images.
[0022] Specifically, the road extraction device and method of multiple embodiments of the present invention enable a part of the autoencoder to perform segmentation while the other part performs object detection. Thus, a variety of different tasks can be performed by configuring the decoders of multiple autoencoder models for performing segmentation.
[0023] Furthermore, in addition to tasks related to road surface, road center line, etc., the road extraction device and method of multiple embodiments of the present invention can further accurately extract the road by additionally performing tasks related to texture representing the texture of the road, road boundary representing the edge of the road, and road direction representing the angle of the road.
[0024] Moreover, the road extraction device and method of multiple embodiments of the present invention can further accurately extract roads by additionally performing tasks related to non-road marks (e.g., buildings, plants, water), road marks (e.g., cars), etc.
[0025] In addition to the above contents, the specific effects of the present invention will also be described in the following description of the specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 Schematic diagram of a road extraction system according to multiple embodiments of the present invention.
[0027] Figure 2 FIG. 4 is a block diagram of a road extraction device according to multiple embodiments of the present invention.
[0028] Figure 3a Schematic diagram of the structure of a neural network of a deep learning module for illustrating multiple embodiments of the present invention.
[0029] Figure 3b Schematic diagram of an autoencoder for illustrating multiple embodiments of the present invention.
[0030] Figures 4a to 4d Schematic diagram of the structure of a first deep learning unit and a second deep learning unit for illustrating multiple embodiments of the present invention.
[0031] Figure 5 A conceptual diagram for illustrating the related work of the deep learning module of multiple embodiments of the present invention.
[0032] Figure 6 The diagram is a conceptual diagram for explaining the related operations of the post-processing modules according to various embodiments of the present invention.
[0033] Figure 7 Flowcharts of road extraction methods according to multiple embodiments of the present invention.
[0034] Figure 8 The diagram is a schematic diagram for explaining that a road extraction device according to multiple embodiments of the present invention is implemented by hardware in order to execute a road extraction method. DETAILED DESCRIPTION
[0035] In this specification and the scope of protection of the invention, the terms or words used should not be interpreted as ordinary meanings or dictionary meanings. According to the principle that the inventor can define the concept of terms or words in order to explain his own invention in the best way, they should be interpreted as meanings and concepts that conform to the technical ideas of the present invention. In addition, the embodiments recorded in this specification and the structures shown in the accompanying drawings are only one embodiment for implementing the present invention. It should be understood that since they cannot represent the entire technical ideas of the present invention, at the time of applying for the present invention, there may be a variety of equivalent technical solutions and examples of deformable applications that can replace them.
[0036] In this specification and the scope of protection claimed in the invention, the terms "first", "second", "A", "B", etc. used may be used to describe a variety of structural elements, but the above structural elements are not limited to the above terms. The above terms are only used to distinguish one structural element from other structural elements. For example, without departing from the scope of protection claimed in the present invention, the first structural element may be named as the second structural element, and similarly, the second structural element may be named as the first structural element. The term "and / or" includes a combination of multiple related recorded items or an item among multiple related recorded items.
[0037] In this specification and the scope of protection of the invention, the terms used are only used to illustrate specific embodiments and do not limit the present invention. Unless otherwise clearly defined in the context, the expression in the singular includes the expression in the plural. In this specification, the terms such as "including" or "having" should not be understood as excluding in advance the existence or additional possibility of other features, numbers, steps, operations, structural elements, components or their combinations recorded in the specification.
[0038] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meanings as commonly understood by one of ordinary skill in the art to which the present invention belongs.
[0039] Terms defined in commonly used dictionaries should be interpreted as meaning the same as the meaning they have in the context of the relevant technology, and should not be interpreted as idealized or overly formal meanings unless clearly defined in this specification.
[0040] Furthermore, the various structures, processes, procedures or methods included in the various embodiments of the present invention may be shared within the scope of non-interference with each other at the technical level.
[0041] Below, refer to Figures 1 to 8 The road extraction device and method according to multiple embodiments of the present invention are described.
[0042] Figure 1 Schematic diagram of a road extraction system according to multiple embodiments of the present invention.
[0043] Reference Figure 1 , a road extraction system 1 according to various embodiments of the present invention may include an external database 100 , a road extraction device 200 , and a communication network 300 .
[0044] The external database 100 may be a database for storing, managing, and / or transmitting raw data as a road extraction object.
[0045] As multiple examples, the external database 100 can be various electronic devices such as computers, laptops, mobile devices, wearable devices, workstations (workstations), data centers, Internet data centers (IDC), direct attached storage (DAS) systems, storage area network (SAN) systems, network attached storage (NAS) systems, disk arrays (RAID) systems, etc. However, the embodiments of the present invention are not limited to these.
[0046] The external database 100 may transmit raw data used by the road extraction device 200 to perform road extraction to the road extraction device 200. In other words, the road extraction device 200 may receive raw data related to road extraction from the external database 100.
[0047] The original data refers to the data used as the road extraction object. For example, the original data refers to an image, an image frame, an image, a panorama, etc. including the road shape. In other words, the original data refers to image data, image frame data, image data, panorama data, etc. including the road shape. However, the embodiment of the present invention is not limited to this.
[0048] In the following, for the sake of convenience, the original data is assumed to be a road image.
[0049] As an example, the road image may be a satellite image including the road shape. A satellite image refers to an image taken by an artificial satellite from a specified altitude. In this case, the external database 100 may be used as a database for saving, storing, analyzing and / or managing satellite images as public data.
[0050] The road extraction device 200 may extract a road from a road image. In other words, the road extraction device 200 may perform road extraction on a road image received from the external database 100.
[0051] In this case, the road extraction device 200 can be a workstation, a data center, an Internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, a disk array (RAID) system, etc. However, the embodiments of the present invention are not limited to this.
[0052] As multiple examples, the road extraction device 200 may utilize deep learning technology to perform a task for extracting a road from a road image.
[0053] In this case, a task refers to a series of work performed by a deep learning module. In other words, a task refers to a series of processes or work in which a deep learning module receives input values and outputs output values in order to derive a specified result.
[0054] For example, the road extraction device 200 may use the road image as an object to perform tasks related to computer vision. In other words, the road extraction device 200 may perform tasks related to image analysis on the road image.
[0055] For example, the road extraction device 200 can use the road image as an object to perform segmentation and object detection related to the road shape and / or characteristics. Segmentation refers to the task of classifying each pixel in the image in order to determine the category, and object detection refers to the task of using a bounding box or the like to identify the position and type of a specific object in the image.
[0056] In this case, the deep learning module used by the road extraction device 200 of the present invention can be pre-learned by multi-task learning. In other words, the deep learning module included in the road extraction device 200 of the present invention can learn so that one deep learning module processes multiple tasks. In this case, the deep learning model used by the deep learning module of the present invention may include an auto encoder, but the embodiment of the present invention is not limited thereto.
[0057] For example, the first deep learning unit included in the road extraction device 200 of the present invention may perform segmentation based on an autoencoder, and the second deep learning unit included in the road extraction device 200 of the present invention may perform target detection based on an autoencoder. In this case, at least one of the first deep learning unit and the second deep learning unit may include multiple encoding units and / or multiple decoding units, and in this case, each decoding unit may be pre-learned to perform different tasks.
[0058] As a specific example, the 1-1 decoding unit included in the first deep learning unit can perform pre-learning to perform segmentation related to road shape (road surface), road center line (center line), texture (texture) representing road texture, road boundary (edge) representing road edge, and road direction (direction) representing road angle, etc. In addition, the 1-2 decoding unit included in the first deep learning unit can perform pre-learning to perform segmentation related to non-road marks (not-road marks) including buildings (buildings), plants (plants), and water areas (water). In addition, the second decoding unit included in the second deep learning unit can perform pre-learning to perform target detection related to road marks (road marks) including vehicles (cars).
[0059] In this case, non-road signs refer to objects that are not included in the road, whereas road signs refer to objects that are usually included in the road. The types of non-road signs and road signs can be preset by the administrator of the road extraction device 200 of the present invention.
[0060] That is, the road extraction device 200 of the present invention does not judge an area where non-road signs (e.g., buildings) are not detected as a road, but may judge an area where road signs (e.g., vehicles) are detected as a road. Therefore, compared to ordinary road extraction methods, it has a new effect of being able to further accurately extract roads by utilizing non-road signs and road signs.
[0061] The following describes the working process of the road extraction device 200 in detail.
[0062] On the other hand, the communication network 300 performs the function of connecting the external database 100 with the road extraction device 200. That is, the communication network 300 refers to a communication network for providing an access path so that the road extraction device 200 can send and receive data from the external database 100. For example, the communication network 300 can be a wired network such as a local area network (LANs), a wide area network (WANs), a metropolitan area network (MANs), an integrated service digital network (ISDNs), or a wireless network such as a wireless local area network, code division multiple access, Bluetooth, or satellite communication, but the scope of the present invention is not limited thereto.
[0063] Below, refer to Figure 2 The operation of the road extraction device 200 will be described in further detail.
[0064] Figure 2 FIG. 4 is a block diagram of a road extraction device according to multiple embodiments of the present invention.
[0065] Reference Figure 1 and Figure 2 After the road extraction device 200 receives a road image (Road Image; hereinafter referred to as "RI"), it can perform road extraction on the received road image RI to generate final data (Final Data; hereinafter referred to as "FD").
[0066] Specifically, the road extraction device 200 may include a data collection module 210, a deep learning module 220, and a post-processing module 230. The deep learning module 220 may include a first deep learning unit 221 and a second deep learning unit 222.
[0067] The data collection module 210 may receive the road image RI. In other words, the data collection module 210 may receive the road image RI from the external database 100.
[0068] In this case, the road image RI may be a satellite image including the road shape. A satellite image refers to an image taken by an artificial satellite from a predetermined height. That is, as the external database 100 is used as a database for saving, storing, analyzing and / or managing satellite images as public data, the road extraction device 200 may receive the road image RI including the satellite image from the external database 100.
[0069] The data collection module 210 may transmit the received road image RI to other structural elements in the road extraction device 200. For example, the data collection module 210 may transmit the road image RI to the deep learning module 220, but the present invention is not limited thereto.
[0070] The data collection module 210 can utilize a variety of communication modules and can communicate through a communication network ( Figure 1 300) performs data exchange between the external database 100 and the road extraction device 200.
[0071] The deep learning module 220 may execute a task for extracting a road from the road image RI. In other words, the deep learning module 220 may execute a task related to road extraction on the road image RI to generate a task execution result (TaskResult; hereinafter referred to as "TR").
[0072] In this case, the deep learning module 220 may include a first deep learning unit 221 and a second deep learning unit 222 , whereby the first deep learning unit 221 may generate a first task execution result TR1 , and the second deep learning unit 222 may generate a second task execution result TR2 .
[0073] The task performed by the deep learning module 220 refers to a series of operations performed by the deep learning module 220. In other words, the task refers to a series of processes or operations in which the deep learning module 220 receives input values and outputs output values in order to derive a prescribed result.
[0074] For example, the deep learning module 220 may use the road image as an object to perform tasks related to computer vision. In other words, the deep learning module 220 may perform tasks related to image analysis on the road image. For example, the deep learning module 220 may use the road image as an object to perform segmentation and object detection related to road characteristics. In this case, segmentation refers to the task of classifying each pixel in the image in order to determine the category, and object detection refers to the task of using a bounding box or the like to identify the position and type of a specific object in the image.
[0075] In this case, the deep learning module 220 may perform learning based on machine learning.
[0076] In more detail, deep learning, a type of machine learning, refers to a technique for in-depth learning based on data in stages. That is, deep learning refers to a collection of machine learning algorithms that extract core data from multiple data as the stages gradually increase.
[0077] As multiple examples, the deep learning module 220 can utilize a variety of well-known deep learning structures. For example, the deep learning module 220 can utilize structures such as convolutional neural network (CNN), recurrent neural network (RNN), deep belief network (DBN), graph neural network (GNN), generative adversarial network (GAN), transformer network, autoencoder, etc.
[0078] Specifically, a convolutional neural network (CNN) refers to a model that simulates the human brain function based on the assumption that the basic features of an object are extracted by a person in the process of identifying the object, and then the object is identified based on the complex calculations in the brain. Convolutional neural networks may include well-known structures such as LeNet neural network, AlexNet neural network, VGGNet neural network, GoogleNet neural network, ResNet neural network, etc., but are not limited thereto.
[0079] Recurrent Neural Network (RNN) is commonly used in natural language processing, etc. As a structure that effectively processes time-series data that changes over time, it can stack layers at each moment to form an artificial neural network structure.
[0080] A Deep Belief Network (DBN) is a deep learning structure that is composed of multiple layers of a Restricted Boltzmann Machine (RBM), a deep learning technology. If a Restricted Boltzmann Machine (RBM) is repeatedly learned to form a specified number of layers, a Deep Belief Network (DBN) with a corresponding number of layers can be formed.
[0081] Graph Neural Network (GNN) refers to an artificial neural network structure that is implemented by deriving similarities and feature points between modeling data based on modeling data mapped between specific parameters.
[0082] Generative Adversarial Network (GAN, Generative Adversarial Network; adversarial generative neural network; hereinafter referred to as GAN) refers to an artificial neural network structure that uses a generative neural network and a discriminative neural network to generate new data with a similar form to the input data. GAN may include the well-known deep convolutional generative adversarial network (DCGAN, Deep ConvolutionalGAN), conditional generative adversarial network (CGAN, Conditional GAN), distance generative adversarial network (WGAN, WassersteinGAN), style generative adversarial network (StyleGAN, Style-Based GAN), cycle generative adversarial network (CycleGAN), etc., but the embodiments of the present invention are not limited to this.
[0083] Transformer network is an artificial neural network with an encoder-decoder structure that applies language sense, and can grasp the overall meaning between the input sequence and the output sequence. Transformer network can use the attention mechanism to make all elements of the input sequence affect the output sequence, so that both the encoder and the decoder consider all the overall sequences. Transformer network can not only use natural language and time series data as input, but also can repair images and use them as input.
[0084] As a deep learning structure, the autoencoder performs the function of extracting and reconstructing data features. Typically, the autoencoder includes: an encoder for compressing input values; and a decoder for recovering the compressed data. The encoder converts the input value into a low-dimensional latent representation, and the decoder recovers the latent representation to the same dimension as the input value. In this case, the encoder and the decoder are respectively composed of multi-layer perceptrons (MLPs). When learning the autoencoder, the input data is input and the weighted values and biases are learned in the direction of minimizing the difference between the output value and the input value. The autoencoder learned in this way can effectively extract the features of the input data and recover the input data with noise. Autoencoders are mainly widely used in the fields of data compression, dimensionality reduction, noise removal, data generation, etc., and can also be applied to the fields of image recognition, natural language processing, and sound recognition.
[0085] On the other hand, the artificial neural network learning of the deep learning module 220 can output the desired output to the given input by adjusting the weights of the connection lines between the nodes (and the bias value if necessary). In addition, the artificial neural network can continuously update the weight value through learning. In addition, the learning of the artificial neural network can use methods such as back propagation.
[0086] In this case, unsupervised learning, semi-supervised learning, supervised learning, etc. can be used as the machine learning method of the artificial neural network. In addition, the deep learning module 220 can automatically update the artificial neural network structure for outputting analysis data after learning according to the setting.
[0087] In this case, the deep learning module 220 of multiple embodiments of the present invention can perform pre-learning through multitask learning. In other words, the deep learning module 220 can learn to process multiple tasks through one deep learning model. In this case, the deep learning model used by the deep learning module of the present invention may include the above-mentioned auto encoder, but the embodiments of the present invention are not limited thereto.
[0088] For example, the first deep learning unit 221 performs segmentation based on the autoencoder, and the second deep learning unit 222 may perform object detection based on the autoencoder. In this case, at least one of the first deep learning unit and the second deep learning unit may include multiple encoding units and / or multiple decoding units, and each decoding unit may perform pre-learning to perform different tasks.
[0089] As a specific example, the 1-1 decoding unit included in the first deep learning unit 221 can perform pre-learning to perform segmentation related to the road shape (road surface), the road center line (center line), the texture (texture) representing the road texture, the road boundary (edge) representing the road edge, and the road direction (direction) representing the road angle. In addition, the 1-2 decoding unit included in the first deep learning unit 221 can perform segmentation related to non-road marks (not-road marks) including buildings (buildings), plants (plants), and water areas (water). In addition, the second decoding unit included in the second deep learning unit 222 can perform pre-learning to perform target detection related to road marks (roadmarks) including vehicles (cars).
[0090] In this case, non-road markings refer to objects that are not usually included in the road, and conversely, road markings refer to objects that are usually included in the road. The types of non-road markings and road markings can be preset by the administrator of the road extraction device 200 of the present invention.
[0091] That is, the deep learning module 220 of the present invention judges the area where road markings (for example, vehicles) are detected as roads, but does not judge the area where non-road markings (for example, buildings) are detected as roads. Therefore, compared with the ordinary road extraction method, it has a new effect of being able to use non-road markings and road markings to further accurately extract roads.
[0092] Below, refer to Figure 3a The neural network structure of the deep learning module 220 according to various embodiments of the present invention is further described in detail.
[0093] Figure 3a A diagram for illustrating the structure of a neural network of a deep learning module according to multiple embodiments of the present invention.
[0094] Reference Figure 3a The deep learning module 220 of multiple embodiments of the present invention may include: an input layer (input), which uses the road image as an input node; an output layer (output), which uses the task execution result as an output node; and M hidden layers, which are configured between the input layer and the output layer.
[0095] Among them, weight values can be set at the edges for connecting nodes of each layer. The presence or absence of such weight values or edges can be added, deleted, or updated during the learning process. Therefore, the weight values of nodes and edges configured between k input nodes and i output nodes can be updated through the learning process.
[0096] Before the deep learning module 220 performs learning, all nodes and edges may be set to initial values. However, in the case of accumulating input information, as the weighted values of nodes and edges are changed, a match can be achieved between the parameters (road image) input as learning factors in this process and the values (task execution results) assigned to the output nodes.
[0097] Additionally, in the case of using a cloud server, the deep learning module 220 can receive and process a large number of parameters. Therefore, the deep learning module 220 can perform learning based on huge data.
[0098] The weighted values of the nodes and edges between the input nodes and output nodes constituting the deep learning module 220 may be updated through the learning process of the deep learning module 220. In addition, in addition to the road image and the task execution result, the parameters received or output by the deep learning module 220 may be further extended to a variety of data.
[0099] Below, refer to Figure 3b The autoencoder used in the present invention is described in further detail.
[0100] Figure 3b A diagram of an autoencoder for illustrating multiple embodiments of the present invention.
[0101] Reference Figure 3b The autoencoder (Autoencoder; hereinafter referred to as "AE") includes an encoder neural network (Encoder Network; hereinafter referred to as the encoding unit EN) and a decoding neural network (Decoder Network; hereinafter referred to as the decoding unit DN), and may include an intermediate layer (ML, Middle Layer) configured between the encoding unit EN and the decoding unit DN.
[0102] As a deep neural network model, the autoencoder AE makes the output data of the autoencoder AE the same as the input data. After compressing the data received by the encoding unit EN (i.e., the input data) to reduce the data, the decoding unit DN can be used to convert the reduced data into the same size as the input data of the encoding unit EN and output it.
[0103] The autoencoder AE learns the features of the input data in an unsupervised manner. To this end, the autoencoder AE can convert the data received through the encoding unit EN into low-dimensional data (latent representation) that effectively represents the corresponding features, after which the converted data can be restored to the source data again through the decoding unit DN.
[0104] The autoencoder AE is able to learn the intrinsic patterns of the source data by aiming to minimize the reconstruction error corresponding to the difference between the source data (X1, X2, X3, X4; i.e., the input data (road image)) and the restored data (X1', X2', X3', X4'; i.e., the output data (task execution result)).
[0105] Below, refer to Figure 4a and Figure 4b The structure of the deep learning module 220 of multiple embodiments of the present invention is described. Figure 4a and Figure 4b The structures of the first deep learning unit 221 and the second deep learning unit 222 implemented by the above-mentioned autoencoder will be described.
[0106] Figures 4a to 4d The diagrams are used to illustrate the structures of the first deep learning unit and the second deep learning unit according to multiple embodiments of the present invention.
[0107] Reference Figures 4a to 4d As described above, the deep learning module 220 of multiple embodiments of the present invention can perform pre-learning based on multi-task learning. To this end, the deep learning module 220 can be designed to be divided into a first deep learning unit 221 and a second deep learning unit 222. In this case, at least one of the first deep learning unit 221 and the second deep learning unit 222 may include multiple encoding units and / or multiple decoding units.
[0108] As a specific example, Figure 4a As shown, the deep learning module 220 is designed to be divided into a first deep learning unit 221 and a second deep learning unit 222. In this case, the first deep learning unit 221 and the second deep learning unit 222 respectively include encoding units 221a, 222a and decoding units 221b, 222b, and the first decoding unit 221b is divided into a 1-1 decoding unit 221b_1 and a 1-2 decoding unit 221b_2, etc.
[0109] exist Figure 4b In the embodiment, the deep learning module 220 is designed to be divided into a first deep learning unit 221 and a second deep learning unit 222. In this case, the first deep learning unit 221 and the second deep learning unit 222 respectively include encoding units 221a, 222a and decoding units 221b, 222b, and the second decoding unit 222b is divided into a 2-1 decoding unit 222b_1 and a 2-2 decoding unit 222b_2, etc.
[0110] like Figure 4c In the embodiment, the deep learning module 220 is designed to be divided into a first deep learning unit 221 and a second deep learning unit 222. In this case, the first deep learning unit 221 and the second deep learning unit 222 respectively include encoding units 221a, 222a and decoding units 221b, 222b. The first decoding unit 221b is divided into a 1st-1 decoding unit 221b_1 and a 1st-2 decoding unit 221b_2, etc., and the second decoding unit 222b is divided into a 2nd-1 decoding unit 222b_1 and a 2nd-2 decoding unit 222b_2, etc.
[0111] exist Figure 4d In the embodiment, the deep learning module 220 is divided into a first deep learning unit 221 and a second deep learning unit 222. In this case, the first deep learning unit 221 and the second deep learning unit 222 respectively include encoding units 221a, 222a and decoding units 221b, 222b. The first encoding unit 221a is divided into a 1-1 encoding unit 221a_1 and a 1-2 encoding unit 221a_2, etc., and the second decoding unit 222b is divided into a 2-1 decoding unit 222b_1 and a 2-2 decoding unit 222b_2, etc.
[0112] In this case, multiple encoding units and / or multiple decoding units can be pre-learned to perform different tasks, and the deep learning module 220 of the present invention that performs multi-task learning based on the above structure can process multiple tasks through one deep learning model (autoencoder).
[0113] Below, refer to Figure 5 A specific example of the multi-task learning method of the deep learning module 220 of the present invention is described. In this case, for the sake of convenience, the structure of the deep learning module 220 of the present invention for implementing multi-task learning is assumed to be Figure 4a The structure shown is described.
[0114] Figure 5 1 is a conceptual diagram for illustrating the operation of the deep learning module in various embodiments of the present invention. Specifically, Figure 5 The deep learning module 220 of the present invention is shown in Figure 4a The structure shown works in this case.
[0115] Reference Figure 5 The deep learning module 220 of multiple embodiments of the present invention may include a first deep learning unit 221 and a second deep learning unit 222. In this case, the first deep learning unit 221 and the second deep learning unit 222 may include encoding units 221a, 222a and decoding units 221b, 222b, respectively.
[0116] The first deep learning unit 221 can perform segmentation on the road image RI through the first encoding unit 221a and the first decoding unit 221b, and the second deep learning unit 222 can perform object detection on the road image RI through the second encoding unit 222a and the second decoding unit 222b. Segmentation refers to the task of classifying each pixel in an image in order to determine the category, and object detection refers to the task of identifying the position and type of a specific object in the road image RI using a bounding box BB or the like.
[0117] As multiple examples, the first encoding unit 221a included in the first deep learning unit 221 can be pre-learned in a manner for performing segmentation, and the second encoding unit 222a included in the second deep learning unit 222 can be pre-learned in a manner for performing target detection. In this case, the first encoding unit 221a and the second encoding unit 222a can have different encoding structures in order to perform tasks respectively. For example, the first encoding unit 221a and the second encoding unit 222a can be pre-learned so that the neural network depth, the presence or absence of low sampling, the type of layer included, the presence or absence of skip connection, etc. become different.
[0118] In order to realize the above multi-task learning, the first decoding unit 221b of the present invention may include a plurality of decoding units 221b_1, 221b_2, etc. In other words, the first decoding unit 221b of the present invention may include a 1-1 decoding unit 221b_1, a 1-2 decoding unit 221b_2, etc.
[0119] In this case, the various decoding units 221b_1, 221b_2, etc. included in the first decoding unit 221b can be pre-learned to perform different tasks. In other words, the first deep learning unit 221 learns to perform segmentation. At this time, the various decoding units 221b_1, 221b_2, etc. included in the first decoding unit 221b can be pre-learned so that they perform different segmentation tasks during the segmentation process.
[0120] As a specific example, the 1-1 decoding unit 221b_1 included in the first deep learning unit 221 can perform pre-learning to perform segmentation related to road shape (road surface), road center line (center line), texture (texture) representing road texture, road boundary (edge) representing road edge, and road direction (direction) representing road angle.
[0121] exist Figure 5 In the example, the task execution result TR1_1 of the 1-1 decoding unit 221b_1 is as follows: after the first encoding unit 221a converts the road image RI into a latent representation, the 1-1 decoding unit 221b_1 performs segmentation related to the road shape (road surface) on the corresponding latent representation.
[0122] Furthermore, the first 1-2 decoding unit 221b_2 included in the first deep learning unit 221 may perform pre-learning to perform segmentation related to non-road marks including buildings, plants, and water areas, etc. In this case, the non-road mark refers to an object that is not usually included in the road, and the type of the non-road mark may be preset by the manager of the road extraction device 200 of the present invention.
[0123] exist Figure 5 In the example, the task execution result TR1_2 of the 1-2 decoding unit 221b_2 is as follows: after the first encoding unit 221a converts the road image RI into a latent representation, the 1-2 decoding unit 221b_2 performs segmentation related to a building as an example of a non-road sign on the corresponding latent representation.
[0124] As described above, in order to perform different segmentation tasks, the 1-1 decoding unit 221b_1 and the 1-2 decoding unit 221b_2 of the present invention may have different decoding structures. For example, the decoding neural network structure (layer, architecture), the type of loss function (Loss Function), the presence and type of normalization function (Normalization Function), the type of activation function (Activation Function), etc. of the 1-1 decoding unit 221_1 and the 1-2 decoding unit 221b_2 may be different from each other. In this case, the neural network structure may include the number of output layers, the types of layers included (for example, U-Net, fully connected layer (FCL)), etc.
[0125] On the other hand, the second decoding unit included in the second deep learning unit 222 can perform pre-learning to perform target detection related to road marks. In this case, road marks refer to objects that are usually included on roads, and the types of road marks can be preset by the manager of the road extraction device 200 of the present invention.
[0126] exist Figure 5 In the example, the task execution result TR2 of the second decoding unit 222b is as follows: after the second encoding unit 222a converts the road image RI into a latent representation, the second decoding unit 222b performs object detection related to a vehicle (car) as an example of a road sign on the corresponding latent representation. As a result of the object detection, Figure 5 The task execution result TR2 shown includes a bounding box BB.
[0127] The deep learning module 220 of the present invention that performs the above-mentioned multi-task learning can process multiple tasks through one deep learning model (autoencoder).
[0128] In summary, after detecting the road shape (road surface), the road center line (center line), the texture (texture) representing the road texture, the road boundary (edge) representing the road edge, and the road direction (direction) representing the road angle, etc., the deep learning module 220 of the present invention does not determine the area where non-road signs (e.g., buildings) are detected as roads, and can determine the area where road signs (e.g., vehicles) are detected as roads. Therefore, compared with the ordinary road extraction method, the deep learning module 220 of the present invention has a new effect of being able to use non-road signs and road signs to further accurately extract roads.
[0129] Refer again Figure 1 and Figure 2, the deep learning module 220 may transmit the task execution results TR1 and TR2 to the post-processing module 230. In other words, the first deep learning unit 221 may transmit the first task execution result TR1 to the post-processing module 230, and the second deep learning unit 222 may transmit the second task execution result TR2 to the post-processing module 230.
[0130] The post-processing module 230 may generate the final data FD based on the task execution result of the deep learning module 220. In other words, the post-processing module 230 may generate the final data FD based on the first task execution result TR1 and the second task execution result TR2.
[0131] Below, refer to Figure 6 The operation of the post-processing module 230 according to various embodiments of the present invention will be described in further detail.
[0132] Figure 6 The diagram is a conceptual diagram for explaining the related operations of the post-processing modules according to various embodiments of the present invention.
[0133] Reference Figure 2 and Figure 6 After merging the first task execution result TR1 and the second task execution result TR2, the post-processing module 230 can process and generate the final data FD. In this case, in order to implement the above multi-task learning, the first decoding unit ( Figure 5 221b) can be composed of multiple decoding units ( Figure 5 221b_1, 221b_2), thereby, the first task execution result TR1 may include the 1-1 task execution result TR1_1 and the 1-2 task execution result TR1_2.
[0134] As multiple examples, after merging the first task execution result TR1 and the second task execution result TR2, the post-processing module 230 may process and generate the final data FD. For example, the post-processing module 230 may include a refinement network, and the final data FD may be generated by the refinement network.
[0135] In this case, if there is an unconnected area related to the road, the post-processing module 230 can convert the corresponding unconnected area into a connected area. In other words, if there is an unconnected area in the merged result of the first task execution result TR1 and the second task execution result TR2, the post-processing module 230 can convert the corresponding unconnected area into a connected area.
[0136] As an example, the post-processing module 230 may convert the unconnected region into a connected region by performing common denoising techniques, smoothing, binarization, morphological operations, etc. However, the present invention is not limited thereto.
[0137] Figure 7 Flowcharts of road extraction methods according to multiple embodiments of the present invention. Figure 7 Each step (step S100 to step S300) of the road extraction device ( Figure 1 and Figure 2 200) is executed. Hereinafter, repeated contents will be omitted and briefly described.
[0138] Reference Figure 1 , Figure 2 and Figure 7 First, a road image related to a road may be collected (step S100 ).
[0139] As multiple examples, the data collection module 210 may receive the road image RI. In other words, the data collection module 210 may receive the road image RI from the external database 100.
[0140] In this case, the road image RI may be a satellite image including the road shape. A satellite image refers to an image taken by an artificial satellite located at a specified altitude. That is, as the external database 100 is used as a database for saving, storing, analyzing and / or managing satellite images as public data, the road extraction device 200 may receive the road image RI including the satellite image from the external database 100.
[0141] Next, a task for extracting the road from the road image may be performed using deep learning technology (step S200 ).
[0142] The task performed by the deep learning module 220 refers to a series of work performed by the deep learning module 220. In other words, the task refers to a series of processes or work in which the deep learning module 220 receives input values and outputs output values in order to derive a specified result. For example, the deep learning module 220 can perform tasks related to computer vision using a road image as an object. In other words, the deep learning module 220 can perform tasks related to image analysis on a road image. For example, the deep learning module 220 can perform segmentation and object detection related to road characteristics using a road image as an object. In this case, segmentation refers to a task of classifying each pixel in an image in order to determine a category, and object detection refers to a task of using a bounding box or the like to identify the position and type of a specific object in an image. At this time, the deep learning module 220 may include a first deep learning unit 221 and a second deep learning unit 222, whereby the first deep learning unit 221 may generate a first task execution result TR1, and the second deep learning unit 222 may generate a second task execution result TR2.
[0143] As multiple examples, the deep learning module 220 of multiple embodiments of the present invention can perform pre-learning through multitask learning. In other words, the deep learning module 220 can learn to process multiple tasks through one deep model. In this case, the deep learning model used by the deep learning module of the present invention may include the above-mentioned auto encoder, but the embodiments of the present invention are not limited thereto.
[0144] For example, the first deep learning unit 221 performs segmentation based on the autoencoder, and the second deep learning unit 222 may perform object detection based on the autoencoder. In this case, at least one of the first deep learning unit and the second deep learning unit may include multiple encoding units and / or multiple decoding units, and each decoding unit may perform pre-learning to perform different tasks.
[0145] As a specific example, the 1-1 decoding unit included in the first deep learning unit 221 can perform pre-learning to perform segmentation related to road shape (road surface), road center line (center line), texture (texture) representing road texture, road boundary (edge) representing road edge, and road direction (direction) representing road angle, etc. In addition, the 1-2 decoding unit included in the first deep learning unit 221 can perform pre-learning to perform segmentation related to non-road marks (not-road marks) including buildings (buildings), plants (plants), and water areas (water). In addition, the second decoding unit included in the second deep learning unit 222 can perform pre-learning to perform target detection related to road marks (road marks) including vehicles (cars).
[0146] In this case, non-road markings refer to objects that are not usually included in the road, and conversely, road markings refer to objects that are usually included in the road. The types of non-road markings and road markings can be preset by the administrator of the road extraction device 200 of the present invention.
[0147] That is, the deep learning module 220 of the present invention judges the area where road markings (for example, vehicles) are detected as roads, but does not judge the area where non-road markings (for example, buildings) are detected as roads. Therefore, compared with the ordinary road extraction method, it has a new effect of being able to use non-road markings and road markings to further accurately extract roads.
[0148] Subsequently, each executed task may be post-processed and output (step S300 ).
[0149] As multiple examples, the post-processing module 230 may generate the final data FD based on the task execution results of the deep learning module 220. In other words, the post-processing module 230 may generate the final data FD based on the first task execution result TR1 and the second task execution result TR2.
[0150] For example, after combining the first task execution result TR1 and the second task execution result TR2, the post-processing module 230 may process and generate the final data FD. For example, the post-processing module 230 may include a refinement network, and the final data FD may be generated by the refinement network.
[0151] In this case, if there is an unconnected area related to the road, the post-processing module 230 can convert the corresponding unconnected area into a connected area. In other words, if there is an unconnected area in the merged result of the first task execution result TR1 and the second task execution result TR2, the post-processing module 230 can convert the corresponding unconnected area into a connected area.
[0152] Figure 8 The diagram is a schematic diagram for explaining that a road extraction device according to multiple embodiments of the present invention is implemented by hardware in order to execute a road extraction method.
[0153] Reference Figure 8 In order to perform the road extraction method of the various embodiments of the present invention, the road extraction device 200 may be implemented by an electronic device 1000. The electronic device 1000 may include a controller 1010, an input / output device (I / O) 1020, a memory device 1030, an interface 1040, and a bus 1050. The controller 1010, the input / output device 1020, the memory device 1030, and / or the interface 1040 may be connected via a bus 1050. In this case, the bus 1050 corresponds to a data movement path.
[0154] Specifically, the controller 1010 may include a central processing unit (CPU), a microprocessor (MPU), a graphics processing unit (GPU), a microprocessor, a digital signal processor, a microcontroller, an application processor (AP), and at least one of logic elements capable of performing similar functions.
[0155] The input-output device 1020 may include at least one of a keypad, a keyboard, a touch device, and a display device.
[0156] The storage device 1030 may store data and / or programs, etc.
[0157] The interface 1040 can perform the function of transmitting data through a communication network or receiving data from a communication network. The interface 1040 can be in a wired or wireless form. For example, the interface 1040 can include an antenna or a wired or wireless transceiver. Although not shown, the storage device 1030 is used as a dynamic memory to improve the operation of the controller 1010, and can also include a high-speed dynamic random access memory and / or a static random access memory. The storage device 1030 can store programs or applications internally.
[0158] The road extraction device 200 and the external database 100 of the embodiment of the present invention may be systems constructed by connecting multiple electronic devices 1000 via a network. In this case, each module or a combination of modules may be implemented by the electronic device 1000. However, the embodiment of the present invention is not limited thereto.
[0159] Additionally, the road extraction device 200 can be implemented by at least one of a workstation, a data center, an Internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, a disk array (RAID, redundant array of inexpensive disks, or redundant array of independent disk) system and an electronic document management (EDMS, Electronic Document Management) system, but the embodiments of the present invention are not limited to this.
[0160] Furthermore, the road extraction device 200 can transmit data to the external database 100 through a network. The network may include wireless Internet technology, a network implemented by wireless Internet technology and short-range communication technology. For example, the wired Internet technology may include at least one of a local area network (LAN) and a wide area network (WAN).
[0161] For example, wireless Internet technology may include wireless local area network (WLAN, Wireless LAN), Digital Living Network Alliance (DMNA, Digital Living Network Alliance), wireless broadband (Wibro, Wireless Broadband), World Interoperability for Microwave Access (Wimax, World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA, High Speed Downlink Packet Access), High Speed Uplink Packet Access (HSUPA, High Speed Uplink Packet Access), IEEE 802.16, Long Term Evolution (LTE, Long Term Evolution), Long Term Evolution-Advanced (LTE-A, Long Term Evolution-Advanced), Wireless Mobile Broadband Service (WMBS, Wireless Mobile Broadband Service) and 5G New Wireless Standard (5G NR (New Radio)) technology at least one. However, the embodiments of the present invention are not limited to this.
[0162] For example, the short-range communication technology may include at least one of Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication (NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio). However, the embodiments of the present invention are not limited thereto.
[0163] The road extraction device 200 communicating through the network may comply with technical standards and standard communication methods for mobile communication. For example, the standard communication method may include at least one of digital mobile communication standard (GSM, Global System for Mobilecommunication), code division multiple access (CDMA, Code Division Multi Access), code division multiple access 2000 (CDMA2000, Code Division Multi Access 2000), evolution data optimization (EV-DO, Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), wideband code division multiple access (WCDMA, WidebandCDMA), high speed downlink packet access (HSDPA, High Speed Downlink Packet Access), high speed uplink packet access (HSUPA, High Speed Uplink Packet Access), long term evolution technology (LTE, Long Term Evolution), advanced long term evolution technology (LTE-A, Long Term Evolution-Advanced) and 5G new wireless standard (5G NR (New Radio)). However, the embodiment of the present invention is not limited to this.
[0164] The above description is only an illustrative description of the technical ideas of the embodiments of the present invention, and ordinary technicians in the technical field to which the present invention belongs can make various modifications and deformations without departing from the essential features of the present invention. Therefore, the embodiments of the present invention are only used for illustration and are not used to limit the technical ideas of the present invention, and the scope of the technical ideas of the present invention is not limited to the above embodiments. The protection scope of the present invention should be interpreted based on the scope of protection claimed by the invention, and all technical ideas within the equivalent scope thereof belong to the scope of protection claimed by the present invention.
Claims
1. A road extraction device, characterized in that: include: A data collection module, for collecting road images related to the road; as well as a deep learning module, which uses deep learning technology to perform a task for extracting the road from the road image, The deep learning module includes: A first deep learning unit performs segmentation on the road image; as well as The second deep learning unit performs object detection on the road image.
2. The road extraction device according to claim 1, characterized in that: The road image includes a satellite image associated with the road.
3. The road extraction device according to claim 1, characterized in that: The deep learning module learns through multi-task learning.
4. The road extraction device according to claim 3, characterized in that: The deep learning module utilizes an autoencoder to perform the task.
5. The road extraction device according to claim 4, characterized in that: The first deep learning unit and the second deep learning unit include: at least one encoding section that encodes the road image as input data and converts it into a latent representation; and At least one decoding unit generates output data by transforming the latent representation.
6. The road extraction device according to claim 5, characterized in that: At least one of the first deep learning unit and the second deep learning unit includes a plurality of decoding units.
7. The road extraction device according to claim 6, characterized in that: The plurality of decoding parts are respectively pre-learned to perform different types of tasks.
8. The road extraction device according to claim 7, characterized in that: The 1-1 decoding unit included in the first deep learning unit performs a task related to at least one of a road shape, a road centerline, a texture representing a texture of the road, a road boundary representing an edge of the road, and a road direction representing an angle of the road, The first-second decoding unit included in the first deep learning unit performs a task related to a non-road sign including at least one of a building, a plant, and a water area, The second decoding section included in the second deep learning section performs a task related to a road sign including a vehicle.
9. The road extraction device according to claim 1, characterized in that: It also includes a post-processing module that combines the tasks respectively performed by the first deep learning unit and the second deep learning unit and processes them to generate final data.
10. The road extraction device according to claim 9, characterized in that: When there is an unconnected area associated with the road, the post-processing module converts the corresponding unconnected area into a connected area.
Citation Information
Patent Citations
Method for extracting road various information of multi-level knowledge driven panchromatic remote sensing image
CN101364259A
Road segmentation method based on supervised learning deep autoencoder
CN108537132A
Method for extracting lane lines based on monitoring video detection traffic flow
CN109871752A
Method for lane and roadway detection
CN111937004A
Road structure prediction and target detection method based on multi-task neural network
CN114048536A
