Method for training a deep learning model for extracting road features based on multitasks and computing device using the same

Abstract

To provide a method for training a deep learning model for extracting road features based on multi-tasks.SOLUTION: A method comprises: setting an initial value for learning; determining the number of learning steps required to complete one epoch for each of learning tasks; determining the total number of steps included in the entire epoch by referring to the number of learning steps; comparing the total sum of the current numbers of steps of the learning tasks with the total number of steps, if the total sum of the current numbers of steps of the learning tasks is less than the total number of steps, randomly selecting a specific learning task for learning out of the learning tasks by referring to each of initial selection probability values for each of the learning tasks; and calculating losses for the specific task by referring to a prediction result output in a specific learning task layer and Ground Truth corresponding to the specific learning task.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a method for training a deep learning model for extracting road features based on multitasking, and a computing device using the same. [Background technology]

[0002] When an autonomous vehicle is traveling, the technology to estimate its current position is very important.

[0003] However, existing autonomous vehicle position estimation technologies simply use lane detection information to match HD maps, and simply apply a lane detection model based on the edge of image processing. As a result, lane estimation performance is limited by noise such as changes in image quality, shadows, and occlusion or damage to lanes. Furthermore, since only lanes are considered when matching HD maps with lane detection information, performance degradation due to incorrect lane estimation is very serious.

[0004] Therefore, the inventors of the present invention propose a road feature estimation model for performing HD map matching in locations where there are no lanes or the lanes are occluded on the road. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to solve the above-mentioned problems.

[0006] Another object of the present invention is to perform HD map matching in locations where there are no lanes on the road or where the lanes are blocked.

[0007] It is yet another object of the present invention to apply a multi-task model structure to minimize overlapping operations for a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures associated with the road, and a third learning task for semantic segmentation of the lanes of the road and the structures.

[0008] It is yet another object of the present invention to propose a multi-task learning method based on heterogeneous data in order to minimize the cost of constructing the learning data.

[0009] A further object of the present invention is to propose a road feature estimation model that is robust against image noise and can simplify post-processing. [Means for solving the problem]

[0010] According to one embodiment of the present invention, in a method for training a deep learning model for extracting road features based on multiple tasks, (a) a computing device sets a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures associated with the road, and a third learning task for semantic segmentation of the lanes on the road and the structures, sets initial selection probability values ​​for each of the first learning task, the second learning task, and the third learning task, and sets first, second, and third numbers of learning steps required to complete one epoch for each of the first learning task, the second learning task, and the third learning task, and sets the first number of learning steps, the third number of learning steps, etc. (b) when it is determined that the total learning progress rate is equal to or less than a first threshold progress rate or a first threshold number of steps, the computing device randomly selects a specific learning task to be learned from the first learning task, the second learning task, or the third learning task by referring to the initial selection probability values ​​for each of the first learning task, the third learning task, and calculates a loss for the specific learning task; and (c) the computing device updates model parameters related to the specific learning task by referring to the loss value for the specific learning task.

[0011] In one example, when it is determined in step (b) that the total learning progress rate exceeds a first threshold progress rate or exceeds a first threshold number of steps, the computing device calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific learning current performance to the specific learning best performance of the specific learning task, calculates a specific learning weight corresponding to the specific learning task by referring to the specific performance achievement rate, calculates a specific learning selection probability adjustment value by referring to the specific learning weight and the number of steps of the specific learning task, and determines the probability that the specific learning task will be selected from the first learning task to the third learning task as the specific learning selection probability adjustment value. In one example, in step (b), the specific learning weight is represented by a value obtained by subtracting the ratio of the specific learning weight to the sum of the first performance achievement rate to the third performance achievement rate for each of the first learning task to the third learning task from 1, and the specific learning selection probability adjustment value is represented by a value obtained by multiplying the specific learning weight by the ratio of the specific number of learning steps for the specific learning task to the sum of all of the first number of learning steps to the third number of learning steps.

[0012] In one example, (d) when the model parameters are updated, the computing device evaluates the specific current learning performance of the specific learning task based on the specific current step of the specific learning task, calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific current learning performance to the specific best learning performance of the specific learning task, calculates a total learning progress rate by referring to the first learning performance achievement rate to the third learning performance achievement rate for each of the first learning task to the third learning task (any one of the first learning performance achievement rate to the third learning performance achievement rate is the specific performance achievement rate), and, if the sum of the first current step number to the third current step number when 1 is added to the specific current step of the specific learning task is less than the total step number, supports the computing device to repeat step (b) and subsequent steps by adding 1 to the step number of the specific current step.

[0013] In one example, the deep learning model includes, as shared layers, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and a decoder layer to which encoder output data output from the encoder layer is input; and, as individual layers, a first learning task layer including a first task header corresponding to the first learning task to which decoder output data output from the decoder layer is input, and a first output function for applying an output function operation to the first task header output data output from the first task header; a second learning task layer including a second task header corresponding to the second learning task to which decoder output data output from the decoder layer is input, and a second output function for applying an output function operation to the second task header output data output from the second task header; and a third learning task layer including a third task header corresponding to the third learning task to which decoder output data output from the decoder layer is input, and a specific softmax function for applying a classification operation to the third task header output data output from the third task header.

[0014] In one example, in step (c), when it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or a second threshold number of steps, the computing device trains the shared layer and the individual layers included in the deep learning model, and when it is determined that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, the computing device freezes the shared layer and stops learning, and updates specific model parameters of a specific individual layer among the individual layers that is related to the specific learning task.

[0015] In one example, the deep learning model includes, as a shared layer, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and, as individual layers, a first learning task layer corresponding to the first learning task and including a first decoder to which encoder output data output from the encoder layer is input and a first output function for applying an output function operation to the first decoder output data output from the first decoder, a second learning task layer corresponding to the second learning task and including a second decoder to which the encoder output data output from the encoder layer is input and a second output function for applying an output function operation to the second decoder output data output from the second decoder, and a third learning task layer corresponding to the third learning task and including a third decoder to which the encoder output data output from the encoder layer is input and a specific softmax function for applying a classification operation to the third decoder output data output from the third decoder.

[0016] In one example, in step (c), when it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or a second threshold number of steps, the computing device trains the shared layer and the individual layers included in the deep learning model, and when it is determined that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, the computing device freezes the shared layer and stops learning, and updates specific model parameters of a specific individual layer among the individual layers that is related to the specific learning task.

[0017] In one example, the method further includes: (e) when lane candidate group data corresponding to the first learning task output from the deep learning model is input, the computing device labels the lane candidate group data for each channel, selects the largest specific labeled blob (bLob), removes the remaining labeled blobs, calculates the coordinates of the blob center points for the specific labeled blob, and then applies a moving average filter to the coordinates of the blob center points to determine predicted information about the lane; and (f) the computing device performs matching by referencing the predicted information about the lane and an HD map, thereby estimating the position and attitude of the autonomous vehicle.

[0018] In one example, after step (e), when the lane-related prediction information and the semantic segmentation information are obtained, the computing device further includes steps of: classifying a specific overlapping portion between the lane-related prediction information and the semantic segmentation information as a left boundary line and a right boundary line of a specific lane; determining the coordinates of the center point of overlap between the left boundary line and the right boundary line as a center point of an exposed lane; restoring an area for an unexposed lane by referring to the left boundary line and the right boundary line of the exposed lane; and determining lane-related adjustment information; and in step (f), the computing device performs matching by referring to the lane-related adjustment information and the HD map, thereby estimating the position and attitude of the autonomous vehicle.

[0019] In one example, in step (a), the computing device further sets a fourth learning task for simultaneous learning of the first learning task through the third learning task, further sets a selection probability initial value for the fourth learning task, further sets a fourth number of learning steps required to complete one epoch for the fourth learning task, and determines the total number of steps included in the entire epoch by referring to the first number of learning steps through the fourth number of learning steps; and in step (b), when it is determined that the total learning progress rate is less than or equal to the first threshold progress rate or less than or equal to the first threshold number of steps, the computing device randomly selects the specific learning task to be learned from the first learning task through the fourth learning task by referring to the selection probability initial value for each of the first learning task through the fourth learning task, and calculates the loss for the specific learning task.

[0020] In one example, (g) when the model parameters are updated, the computing device evaluates the specific current learning performance of the specific learning task based on the specific current step of the specific learning task, calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific current learning performance to the specific best learning performance of the specific learning task, calculates a total learning progress rate by referring to the first learning performance achievement rate to the fourth learning performance achievement rate (any one of the first learning performance achievement rate to the fourth learning performance achievement rate is the specific performance achievement rate), and if the sum of the first current step number to the third current step number when 1 is added to the specific current step of the specific learning task is less than the total step number, supports re-performing step (b) and subsequent steps by adding 1 to the step number of the specific current step.

[0021] According to another embodiment of the present invention, a computing device for training a deep learning model for extracting road features based on multitasking includes at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor (I) sets a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures associated with the road, and a third learning task for semantic segmentation of the lanes on the road and the structures, sets initial selection probability values ​​for each of the first learning task, the second learning task, and the third learning task; sets first, the third, and fourth numbers of learning steps required to complete one epoch for each of the first, the second, and third learning tasks; a process of: (II) determining the total number of steps included in all epochs by referring to the number of learning steps in the first epoch; (II) when it is determined that the total learning progress rate is equal to or less than a first threshold progress rate or a first threshold number of steps, randomly selecting a specific learning task to be learned from the first learning task, the second learning task, or the third learning task by referring to the initial selection probability values ​​for each of the first learning task, the third learning task, or the third learning task, and calculating a loss for the specific learning task; and (III) updating a model parameter related to the specific learning task by referring to the loss value of the specific learning task.

[0022] In one example, when the processor determines in process (II) that the total learning progress rate exceeds a first threshold progress rate or exceeds a first threshold number of steps, it can calculate a specific performance achievement rate for the specific learning task by referring to the ratio of the specific learning current performance to the specific learning best performance of the specific learning task, calculate a specific learning weight corresponding to the specific learning task by referring to the specific performance achievement rate, calculate a specific learning selection probability adjustment value by referring to the specific learning weight and the number of steps of the specific learning task, and determine the probability that the specific learning task will be selected from the first learning task to the third learning task as the specific learning selection probability adjustment value.

[0023] In one example, in the (II) process, the specific learning weight can be represented by a value obtained by subtracting the ratio of the specific learning weight to the sum of the first performance achievement rate to the third performance achievement rate for each of the first learning task to the third learning task from 1, and the specific learning selection probability adjustment value can be represented by a value obtained by multiplying the specific learning weight by the ratio of the specific number of learning steps for the specific learning task to the sum of all of the first number of learning steps to the third number of learning steps.

[0024] In one example, the processor can further perform the following process: (IV) when the model parameters are updated, evaluate the specific current learning performance of the specific learning task based on the specific current step of the specific learning task, calculate a specific performance achievement rate for the specific learning task by referring to the ratio of the specific current learning performance to the specific best learning performance of the specific learning task, calculate a total learning progress rate by referring to the first learning performance achievement rate to the third learning performance achievement rate for each of the first learning task to the third learning task (any one of the first learning performance achievement rate to the third learning performance achievement rate is the specific performance achievement rate), and if the sum of the first current step number to the third current step number when 1 is added to the specific current step of the specific learning task is less than the total step number, support the process from (II) onwards to be performed again with 1 added to the step number of the specific current step.

[0025] In one example, the deep learning model may include, as shared layers, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and a decoder layer to which encoder output data output from the encoder layer is input; and, as individual layers, a first learning task layer including a first task header corresponding to the first learning task to which decoder output data output from the decoder layer is input, and a first output function for applying an output function operation to the first task header output data output from the first task header; a second learning task layer including a second task header corresponding to the second learning task to which decoder output data output from the decoder layer is input, and a second output function for applying an output function operation to the second task header output data output from the second task header; and a third learning task layer including a third task header corresponding to the third learning task to which decoder output data output from the decoder layer is input, and a specific softmax function for applying a classification operation to the third task header output data output from the third task header.

[0026] In one example, in the process (III), when the processor determines that the total learning progress rate is equal to or less than a second threshold progress rate or a second threshold number of steps, the processor trains the shared layer and the individual layers included in the deep learning model, and when the processor determines that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, the processor freezes the shared layer and stops learning, and updates specific model parameters of a specific individual layer among the individual layers that is related to the specific learning task.

[0027] In one example, the deep learning model may include, as a shared layer, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and as individual layers, a first learning task layer corresponding to the first learning task, the first learning task layer including a first decoder to which encoder output data output from the encoder layer is input and a first output function for applying an output function operation to the first decoder output data output from the first decoder, a second learning task layer corresponding to the second learning task, the second decoder to which encoder output data output from the encoder layer is input and a second output function for applying an output function operation to the second decoder output data output from the second decoder, and a third learning task layer corresponding to the third learning task, the third decoder to which encoder output data output from the encoder layer is input and a specific softmax function for applying a classification operation to the third decoder output data output from the third decoder.

[0028] In one example, in the process (III), when the processor determines that the total learning progress rate is equal to or less than a second threshold progress rate or a second threshold number of steps, the processor trains the shared layer and the individual layers included in the deep learning model, and when the processor determines that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, the processor freezes the shared layer and stops learning, and updates specific model parameters of a specific individual layer among the individual layers that is related to the specific learning task.

[0029] In one example, the processor can further perform the following process: (V) when lane candidate group data corresponding to the first learning task output from the deep learning model is input, labeling the lane candidate group data for each channel, selecting the largest specific labeled blob, removing the remaining labeled blobs, calculating the coordinates of the blob center points for the specific labeled blob, and then applying a moving average filter to the coordinates of the blob center points to determine predicted information about the lane; and (VI) performing matching by referencing the predicted information about the lane and an HD map, thereby estimating the position and attitude of the autonomous vehicle.

[0030] In one example, after the process (V), the processor further performs a process of (VI) when the prediction information about the lane and the information about the semantic segmentation are obtained, classifying a specific overlapping portion between the prediction information about the lane and the information about the semantic segmentation as the left and right boundary lines of a specific lane, determining the coordinates of the center point of overlap between the left and right boundary lines as the center point of an exposed lane, restoring an area for the unexposed lane by referring to the left and right boundary lines of the exposed lane, and determining adjustment information about the lane, and in the process (VI), performing matching by referring to the adjustment information about the lane and the HD map, thereby estimating the position and attitude of the autonomous vehicle.

[0031] In one example, in the process (I), the processor further sets a fourth learning task for simultaneous learning of the first learning task through the third learning task, further sets a selection probability initial value for the fourth learning task, further sets a fourth number of learning steps required to complete one epoch for the fourth learning task, and determines the total number of steps included in all epochs by referring to the first number of learning steps through the fourth number of learning steps; and in the process (II), when it is determined that the total learning progress rate is equal to or less than the first threshold progress rate or the first threshold number of steps, the processor can randomly select the specific learning task to be learned from the first learning task through the fourth learning task by referring to the selection probability initial value for each of the first learning task through the fourth learning task, and calculate the loss for the specific learning task.

[0032] In one example, the processor can further perform the following process: (VII) when the model parameters are updated, evaluate the specific current learning performance of the specific learning task based on the specific current step of the specific learning task, calculate a specific performance achievement rate for the specific learning task by referring to the ratio of the specific current learning performance to the specific best learning performance of the specific learning task, calculate a total learning progress rate by referring to the first learning performance achievement rate to the fourth learning performance achievement rate (any one of the first learning performance achievement rate to the fourth learning performance achievement rate is the specific performance achievement rate), and if the sum of the first current step number to the third current step number when 1 is added to the specific current step of the specific learning task is less than the total step number, support the re-execution of process (II) and subsequent steps by adding 1 to the step number of the specific current step. [Effects of the Invention]

[0033] The present invention has the following advantages.

[0034] The present invention has the advantage that HD map matching can be performed in a location where there are no lanes on the road or where the lanes are blocked.

[0035] The present invention has the advantage of being able to apply a multi-task model structure to minimize overlapping operations for a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures associated with the road, and a third learning task for semantic segmentation of the lanes of the road and the structures.

[0036] The present invention has an effect of realizing a multi-task learning method based on heterogeneous data for minimizing the cost of constructing learning data.

[0037] The present invention has the effect of realizing a road feature estimation model that is resistant to image noise and allows for simplified post-processing.

[0038] The following drawings attached for use in explaining embodiments of the present invention are only a part of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter referred to as "ordinary engineer") can derive other drawings based on these drawings without performing any inventive work. [Brief explanation of the drawings]

[0039] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of a computing device for training a deep learning model for extracting road features based on multitasks, according to one embodiment of the present invention. [Figure 2] FIG. 2 is a flowchart that schematically illustrates a process of training a deep learning model for multi-task based road feature extraction according to one embodiment of the present invention. [Figure 3]Figure 3 shows a configuration for estimating road characteristics based on a multi-task deep learning model, and illustrates (i) a structure capable of performing a lane estimation task (i.e., a first learning task), a road structure line estimation task (i.e., a second learning task), and a semantic segmentation task (i.e., a third learning task), and (ii) a process of matching the results of the lane estimation task with the results of the semantic segmentation task, and estimating the position and attitude of an autonomous vehicle using the adjustment information and HD map obtained by matching the results of the road structure line estimation task with the results of the semantic segmentation task. [Figure 4] FIG. 4 is a diagram showing examples of lane learning GT data, road structure learning GT data, and semantic segmentation learning GT data corresponding to image data according to an embodiment of the present invention. [Figure 5] FIG. 5 illustrates an example of a decoder-shared architecture divided into (i) a shared layer including an encoder layer and a decoder layer, and (ii) individual layers including first through third learning task layers, according to one embodiment of the present invention. [Figure 6] FIG. 6 illustrates an example of a decoder-independent architecture partitioned into (i) a shared layer including an encoder layer and (ii) separate layers including respective learning task layers each including a respective decoder, according to one embodiment of the present invention. [Figure 7] FIG. 7 is a diagram illustrating an example of a process of labeling lane-related data for each channel, removing noise, and then applying a moving average filter to obtain predicted lane-related information according to an embodiment of the present invention. [Figure 8] FIG. 8 is a diagram illustrating an example of a process for determining exposed lanes by applying prediction information related to the results of semantic segmentation, restoring areas for non-exposed lanes by referring to the exposed lanes, and determining adjustment information related to the lanes, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0040] In order to clarify the objectives, technical solutions, and advantages of the present invention, the following detailed description of the present invention refers to the accompanying drawings which show, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those of ordinary skill in the art to practice the invention.

[0041] Moreover, throughout the detailed description of the present invention and the claims, the word "comprises" and variations thereof are not intended to exclude other technical features, additives, components, or steps. Other objects, advantages, and characteristics of the present invention will become apparent to those of ordinary skill in the art, in part from the description and in part from the practice of the present invention. The following examples and figures are offered by way of illustration and are not intended to limit the present invention.

[0042] Furthermore, the present invention covers all possible combinations of the embodiments shown herein. It should be understood that the various embodiments of the present invention, although different from one another, are not necessarily mutually exclusive. For example, a specific shape, structure, and characteristic described herein may be implemented in one embodiment without departing from the spirit and scope of the present invention. It should also be understood that the location or arrangement of individual components within each disclosed embodiment may be modified without departing from the spirit and scope of the present invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention is limited only by the appended claims, along with the full scope of equivalents to which such claims, when properly interpreted, are entitled. In the drawings, like reference numerals refer to the same or similar functionality throughout the various aspects.

[0043] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention.

[0044] FIG. 1 is a diagram illustrating a schematic configuration of a computing device for training a deep learning model for extracting road features based on multitasks, according to one embodiment of the present invention.

[0045] As shown in FIG. 1 , a computing device 100 for training a deep learning model for extracting road features based on multitasking according to the present invention may include a memory 110 and a processor 120.

[0046] The memory 110 of the computing device 100 for training a deep learning model for extracting road features based on multitasking can store instructions to be executed by the processor 120, specifically, the instructions are code generated to cause the computing device 100 to function in a particular manner for providing content, and can be stored in computer-usable or computer-readable memory for computers and other programmable data processing devices. The instructions can perform processes to perform the functions described in the present specification.

[0047] The processor 120 of the computing device 100 may include hardware components such as a microprocessing unit (MPU) or central processing unit (CPU), cache memory, and data bus, etc. The computing device 100 may also include software components such as an operating system and applications that perform specific purposes.

[0048] The computing device 100 may also be linked to a database. The database may include, but is not limited to, at least one of the following types of storage media: flash memory, hard disk, micro multimedia card, card-type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The database may be installed separately from the computing device 100, or may be installed inside the computing device 100 to record transmitted and received data. Unlike the illustrated example, the database may be implemented as two or more separate databases, depending on the implementation conditions of the invention.

[0049] A method using the computing device 100 according to an embodiment of the present invention configured as above will be described below with reference to FIG.

[0050] FIG. 2 is a flowchart illustrating a process of training a deep learning model for extracting road features based on multiple tasks, according to one embodiment of the present invention.

[0051] 2, initial values ​​for learning may be set in step S211. Specifically, in step S211, computing device 100 may register a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures associated with the road, and a third learning task for semantic segmentation of the lanes on the road and the structures, set initial selection probability values ​​for each of the first to third learning tasks, and set initial values ​​of a performance achievement rate (details regarding the performance achievement rate will be described later in steps S215 and S242) and a total learning progress rate (details regarding the total learning progress rate will be described later in step S242) for each of the first to third learning tasks to 0.

[0052] Then, proceed to step S212, determine the first to third numbers of learning steps required to complete one epoch for each of the first to third learning tasks, and refer to the first to third numbers of learning steps to determine the total number of steps included in all epochs.

[0053] Then, the process proceeds to step S213, where the sum of the current step numbers for the first learning task through the third learning task is compared with the total number of steps. If the sum of the current step numbers for the first learning task through the third learning task is less than the total number of steps, the process proceeds to step S214, which will be described later. If the sum of the current step numbers for the first learning task through the third learning task is the same as the total number of steps, the process proceeds to step S250, where learning is completed.

[0054] As described in step S211, the current state is one in which only the initial values ​​are set, so the process can proceed to step S214.

[0055] If it is determined in step S214 that the total learning progress rate is equal to or less than the first threshold progress rate or equal to or less than the first threshold number of steps, the computing device 100 proceeds to step S221 and can randomly select a specific learning task to be learned from the first learning task to the third learning task by referring to the initial selection probability values ​​for each of the first learning task to the third learning task.

[0056] Here, the number of steps of the nth learning task (where n is an integer between 1 and 3) can be expressed as the number of data of the nth learning task / the batch size (batch_size) corresponding to the nth learning task, and the total number of steps = the number of epochs set by the user ×

number

[0057] Returning to step S221, after a specific learning task is randomly selected, the process proceeds to step S222, where the loss for the specific task is calculated by referring to the prediction result output in the specific learning task layer and the GT corresponding to the specific learning task, i.e., the Ground Truth.

[0058] In step S214, contrary to the above example, if the total learning progress rate exceeds the first threshold progress rate or the first threshold number of steps, the computing device 100 proceeds to step S215, calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific learning current performance (the current performance value of the specific learning task) to the specific learning best performance of the specific learning task (the best performance value of the specific learning task calculated in advance through a test on a single learning task), calculates a specific learning weight corresponding to the specific learning task by referring to the specific performance achievement rate (a value obtained by subtracting the ratio of the specific performance achievement rate to the sum of the first performance achievement rate to the third performance achievement rate corresponding to the first learning task to the third learning task, respectively) by referring to the specific performance achievement rate, calculates a specific learning selection probability adjustment value by referring to the specific learning weight and the number of steps of the specific learning task, determines the probability that the specific learning task will be selected from the first learning task to the third learning task as the specific learning selection probability adjustment value, and then proceeds to the above-mentioned steps S221 and S222.

[0059] Here, the specific performance achievement rate can be expressed as the specific learning current performance / the specific learning best performance, the specific learning weight can be expressed as 1-(the specific performance achievement rate / the first performance achievement rate+the second performance achievement rate+the third performance achievement rate), and the specific learning selection probability adjustment value can be expressed as the specific learning weight x (the number of steps in the specific learning task / the number of steps in the first learning task+the number of steps in the second learning task+the number of steps in the third learning task).

[0060] Returning to step S222, the loss for a specific learning task is calculated in step S222. To facilitate understanding of this step, the loss calculation process will be outlined with reference to FIGS. 3 and 4.

[0061] Figure 3 shows a configuration for estimating road characteristics based on a multi-task deep learning model, illustrating (i) a structure capable of simultaneously performing a lane estimation task (i.e., a first learning task), a road structure line estimation task (i.e., a second learning task), and a semantic segmentation task (i.e., a third learning task), and (ii) a process of matching the results of the lane estimation task with the results of the semantic segmentation task, and estimating the position and attitude of an autonomous vehicle using the adjustment information and HD map obtained by matching the results of the road structure line estimation task with the results of the semantic segmentation task.

[0062] Referring to FIG. 3 , image data 300 acquired through at least one camera mounted on an autonomous vehicle is input to an encoder layer 310, and then the output of the encoder layer 310 is input to a decoder layer 320, resulting in lane candidate data 321 for a first learning task for estimating lanes on a road, structure line candidate data 322 for a second learning task for estimating lines of structures related to the road, and semantic segmentation 323 for a third learning task for semantic segmentation of the lanes and structures on the road, as outputs of the decoder layer 320.

[0063] First, in the deep learning model, the encoder layer 310 has a structure that shares calculation information for all three tasks, and the decoder layer 320 can be configured so that the calculation results can be shared by the three tasks or can be calculated independently, depending on the performance of the selected basic model. This will be described later with reference to Figures 5 and 6.

[0064] The first learning task is for a model that can connect and predict lanes that are partially hidden by multiple lanes or that are interrupted between lanes. The second learning task is for a model that can connect and predict vertical structures (such as utility poles and traffic light supports) and road structures (such as curb boundaries) when they are partially hidden. The third learning task is for a model that can perform general segmentation of lanes, curbs, traffic lights, utility poles, cars, drivable roads, etc.

[0065] On the other hand, post-processing 340 and the like in FIG. 3 will be described later with reference to FIGS.

[0066] FIG. 4 is a diagram showing examples of lane learning GT data, road structure learning GT data, and semantic segmentation learning GT data corresponding to image data according to an embodiment of the present invention.

[0067] Specifically, the lane learning GT data 331 indicates correct answer values ​​for lines to display lanes, stop lines, guidelines, speed bump lines, etc., with left and right directions and corresponding indices (e.g., L3, L2, L1, R1, R2, R3, etc.) from the center of the autonomous vehicle, for image data 300 acquired through at least one camera mounted on the autonomous vehicle. The road structure learning GT data 332 indicates correct answer values ​​for lines to display road structures such as utility poles, traffic lights, traffic sign board supports, and curbs, for the image data 300. The semantic segmentation learning GT data 333 indicates correct answer values ​​that display the visible states of lanes, drivable roads, curbs, traffic lights, automobiles, pedestrians, etc. as surface information for the image data 300. The losses of such correct answers, i.e., lane learning GT data 331, road structure learning GT data 332, and semantic segmentation learning GT data 333, can be calculated by comparing them with the estimated values ​​of the decoder layer 320, i.e., lane candidate group data 321 for the first learning task, structure line candidate group data 322 for the second learning task, and semantic segmentation 323 for the third learning task.

[0068] Then, proceeding to step S223, the computing device 100 compares the total learning progress rate with a second threshold progress rate and a second threshold number of steps. If it is determined that the total learning progress rate is equal to or less than the second threshold progress rate or the second threshold number of steps, the computing device 100 proceeds to step S230 and backpropagates the loss calculated in step S222 to update model parameters associated with the specific learning task.

[0069] In step S223, if it is determined that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, contrary to the above example, the process proceeds to step S224, where the shared layer is frozen and learning is stopped, and the process proceeds to step S230, where specific model parameters of a specific individual layer related to the specific learning task among the individual layers can be updated.

[0070] Below we will consider shared and individual layers in more detail.

[0071] FIG. 5 illustrates an exemplary decoder-shared architecture divided into (i) a shared layer including an encoder layer and a decoder layer, and (ii) individual layers including first through third learning task layers, according to one embodiment of the present invention.

[0072] Specifically, the decoder-shared deep learning model includes, as shared layers, an encoder layer 310 to which image data 300 acquired through at least one camera mounted on an autonomous vehicle is input, and a decoder layer 320 to which encoder output data output from the encoder layer 310 is input, and, as individual layers, a first learning task layer 510 including a first task header 511 corresponding to the first learning task to which decoder output data output from the decoder layer is input, and a first output function 512 for applying an output function operation to the first task header output data output from the first task header 511. The decoder has a structure including a second learning task layer 520 including a second task header 521 corresponding to the second learning task to which the decoder output data output from the decoder layer 320 is input and a second output function 522 for applying an output function operation to the second task header output data output from the second task header 521, and a third learning task layer 530 including a third task header 531 corresponding to the third learning task to which the decoder output data output from the decoder layer 320 is input and a specific softmax function 532 for applying a classification operation to the third task header output data output from the third task header 531.

[0073] FIG. 6 is a diagram illustrating an example of a decoder-independent structure according to one embodiment of the present invention, including (i) an encoder layer as a shared layer and (ii) an nth learning task layer, each of which includes an nth decoder (n is an integer from 1 to 3) as a separate layer.

[0074] Specifically, the decoder-independent deep learning model includes, as a shared layer, an encoder layer 310 to which image data 300 acquired through at least one camera mounted on an autonomous vehicle is input, and, as individual layers, a first learning task layer 610 corresponding to the first learning task and including a first decoder 611 to which encoder output data output from the encoder layer is input and a first output function 612 for applying an output function operation to the first decoder output data output from the first decoder 611; The learning task layer 620 corresponds to the third learning task and includes a second decoder 621 to which the encoder output data output from the encoder layer is input and a second output function 622 for applying an output function operation to the second decoder output data output from the second decoder 621, and a third learning task layer 630 corresponding to the third learning task and includes a third decoder 631 to which the encoder output data output from the encoder layer is input and a specific softmax function 632 for applying a classification operation to the third decoder output data output from the third decoder 631.

[0075] Here, the first output function and the second output function of the decoder-independent deep learning model and the decoder-shared deep learning model, respectively, may be, but are not limited to, a sigmoid function.

[0076] Next, when the model parameters are updated in step S230, the computing device 100 proceeds to step S241, where it evaluates the specific learning current performance of the specific learning task based on the specific current step of the specific learning task, and then proceeds to step S242, where it calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific learning current performance to the specific learning best performance of the specific learning task, and calculates the first learning performance achievement rate to the third learning performance achievement rate (the first learning performance achievement rate) for each of the first learning task to the third learning task. The process then proceeds to step S243, where if the sum of the first current step count through the third current step count in the state where 1 is added to the specific current step of the specific learning task (the sum of all the step counts in all the epochs accumulated up to now) is less than the total number of steps, the process can be guided to repeat step S213 and onwards with 1 added to the number of steps of the specific current step.

[0077] Here, the total learning progress rate can be expressed as the first performance achievement rate + the second performance achievement rate + the third performance achievement rate / 3.

[0078] Meanwhile, the learning process of the deep learning model from step S211 to step S243 has been illustrated assuming that there is no integrated learning data, but if there is integrated learning data, in addition to the first to third learning tasks, a fourth learning task that simultaneously learns the first to third learning tasks can be set, and a process similar to steps S211 to S243 can be performed. In other words, the process proceeds through the same process except that three learning tasks are changed to four learning tasks, so a description thereof will be omitted.

[0079] Next, the post-processing process will be described with reference to FIGS.

[0080] To explain the post-processing 340 and subsequent steps in Figure 3, when the lane candidate group data 321, structure line candidate group data 322, and semantic segmentation 323 are obtained from the deep learning model, line components are extracted from each of the lane candidate group data 321 and structure line candidate group data 322, and these are fused with the results of the semantic segmentation 323. Adjustment information 341, which is the result of post-processing to obtain line information and / or surface information for each of the lane candidate group data 321 and structure line candidate group data 322, can be matched with an HD map 350 to estimate the position and attitude of the autonomous vehicle.

[0081] 7 and 8, a process of matching the result of the lane estimation task with the result of the semantic segmentation task to obtain adjustment information 341 and performing post-processing using the adjustment information 341 and the HD map 350 will be described as an example. The process of matching the result of the road feature line estimation task with the result of the semantic segmentation task and performing post-processing is similar, so a description thereof will be omitted.

[0082] First, FIG. 7 is a diagram illustrating an example of a process of labeling lane-related data for each channel, removing noise, and then applying a moving average filter to obtain predicted lane-related information according to an embodiment of the present invention.

[0083] Specifically, when lane candidate group data 321 is input from the deep learning model, the computing device 100 performs labeling 710, 720 on the lane candidate group data 321 for each channel, selects and leaves the largest specific labeled blob 710, and removes the remaining labeled blobs 720. Using the y-axis on the screen of the lane candidate group data 321 as a reference, the computing device 100 obtains the minimum position 711 and the maximum position 713 on the x-axis corresponding to each of the y-axis values ​​of the specific labeled blob 710. Then, the computing device 100 extracts coordinates 712 of multiple blob center points as the median values ​​of the minimum position 711 and the maximum position 713 on the x-axis. A moving average filter is applied to the coordinates 712 of the blob center points to determine a point determined as predicted information regarding the lane.

[0084] Here, to explain the process of applying the moving average filter in more detail, four center points 712a, 712b, 712c, and 712d are selected from the plurality of blob center points 712, and one of the four center points 712a through 712d, located midway between the two end points 712a and 712d, can be set as point 714 determined as predicted information about the lane. Similarly to the above, the moving average filter can be applied multiple times in pixel units of the four points to set multiple points 714 determined as predicted information about the lane. The multiple points 714 determined as predicted information about the lane in this way can be sampled at a user-specified number of points, for example, eight points, and finally determined as predicted information about the lane 715.

[0085] In the process of explaining the moving average filter in Figure 7, four center points 712a to 712d are selected from the multiple blob center points 712, and the point located at the center of the two center points 712a and 712d located at both ends of the four center points 712a to 712d is set as point 714 determined as predicted information regarding the lane, and predicted information 715 regarding the lane is extracted. However, the method of applying the moving average filter is not necessarily limited to selecting four points, and the user can proceed by setting a different number of selected blob center points 712.

[0086] Similarly, in FIG. 7, eight points 714 determined as predicted information regarding the lane are sampled and determined as predicted information regarding the lane 715, but this is not limited to this, and M points 714 determined as predicted information regarding the lane can also be sampled and determined as predicted information regarding the lane 715.

[0087] In the content of FIG. 7, the process of extracting point components of a lane has been specifically illustrated and described. However, the process of extracting point components is not limited to lanes. It goes without saying that point information of line estimation data of road-related structures such as curbs and utility poles as well as stop lines and speed bumps included in lane estimation data such as lanes can be similarly extracted.

[0088] Next, a process of fusing the lane prediction information and the semantic segmentation will be described with reference to FIG.

[0089] FIG. 8 is a diagram illustrating an example of a process of determining exposed lanes by applying predicted information about lanes (e.g., the results obtained in FIG. 7) to the results of semantic segmentation, restoring areas for non-exposed lanes by referring to the exposed lanes, and determining adjustment information about lanes, according to one embodiment of the present invention.

[0090] When the semantic segmentation result 323a for a specific lane is obtained from the lane prediction information 715 and the semantic segmentation result 323 of Figure 3, the computing device 100 distinguishes a specific overlapping portion between the lane prediction information 715 and the semantic segmentation result 323a for the specific lane as the left boundary line 811 and the right boundary line 813 of the specific lane, determines the coordinates of the center point of overlap between the left boundary line 811 and the right boundary line 813 as the exposed lane center point 812, and uses the lane center point 812 to restore an unexposed lane (an area that exists in the lane prediction information 715 but is not visible in the semantic segmentation result 323a for the specific lane is considered to be an unexposed lane area, which is an area where there is no lane or where the lane is hidden by a structure or the like).

[0091] For example, the previously predicted center point 812 of the past exposed lane can be applied as the current lane center point 812 to restore the area for the unexposed lane and calculate adjustment information 341 for the lane, and the user can decide whether to use the information for the unexposed lane.

[0092] Next, once the lane adjustment information 341 is calculated, the computing device 100 can reference and match the lane adjustment information 341 and the HD map 350, thereby estimating the position and attitude of the autonomous vehicle 360.

[0093] The above-described embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer components and stored on a computer-readable storage medium. The computer-readable storage medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions stored on the computer-readable storage medium may be specially designed and constructed for the present invention, or may be well-known and available to those skilled in the computer software arts. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processes according to the present invention, or vice versa.

[0094] Although the present invention has been described above using specific details such as specific components and limited examples and drawings, these are merely provided to facilitate a more general understanding of the present invention, and the present invention is not limited to the above examples. Those skilled in the art will be able to make various modifications and variations from such descriptions.

[0095] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and all modifications equivalent to or equivalent to the scope of the claims, as well as the scope of the claims described below, are considered to fall within the scope of the spirit of the present invention.

Claims

1. A method for training a deep learning model for extracting road features based on multitasks, (a) a computing device sets a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures associated with the road, and a third learning task for semantic segmentation of the lanes on the road and the structures, sets initial selection probability values ​​for each of the first learning task, the second learning task, and the third learning task, sets first, the third, and the fourth learning step numbers required to complete one epoch, and determines the total number of steps included in all epochs by referring to the first, the second, and the third learning step numbers; (b) when it is determined that the total learning progress rate is equal to or less than a first threshold progress rate or equal to or less than a first threshold number of steps, the computing device selects a specific learning task to be learned from the first learning task to the third learning task by referring to the initial selection probability values ​​for each of the first learning task to the third learning task, and calculates a loss for the specific learning task; and (c) updating model parameters associated with the specific learning task by the computing device with reference to the loss value of the specific learning task; A method comprising:

2. In the step (b), 2. The method of claim 1, wherein, when it is determined that the total learning progress rate exceeds a first threshold progress rate or exceeds a first threshold number of steps, the computing device calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific learning current performance to the specific learning best performance of the specific learning task, calculates a specific learning weight corresponding to the specific learning task by referring to the specific performance achievement rate, calculates a specific learning selection probability adjustment value by referring to the specific learning weight and the number of steps of the specific learning task, and determines the probability that the specific learning task will be selected from the first learning task to the third learning task as the specific learning selection probability adjustment value.

3. In the step (b), 3. The method of claim 2, wherein the specific learning weight is represented by a value obtained by subtracting the ratio of the specific learning weight to the sum of the first performance achievement rate through the third performance achievement rate for the first learning task through the third learning task, respectively, from 1, and the specific learning selection probability adjustment value is represented by a value obtained by multiplying the specific learning weight by the ratio of the specific number of learning steps for the specific learning task to the sum of all of the first number of learning steps through the third number of learning steps.

4. (d) when the model parameters are updated, the computing device evaluates the specific current learning performance of the specific learning task based on the specific current step of the specific learning task, calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific current learning performance to the specific best learning performance of the specific learning task, calculates a total learning progress rate by referring to the first learning performance achievement rate to the third learning performance achievement rate for each of the first learning task to the third learning task (one of the first learning performance achievement rate to the third learning performance achievement rate is the specific performance achievement rate), and, if the sum of the first current step number to the third current step number in a state where 1 is added to the specific current step of the specific learning task is less than the total step number, supports the computing device to repeat step (b) and subsequent steps by adding 1 to the step number of the specific current step; 10. The method of claim 1, further comprising:

5. 2. The method of claim 1, wherein the deep learning model includes, as shared layers, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input and a decoder layer to which encoder output data output from the encoder layer is input; and, as individual layers, a first learning task layer including a first task header corresponding to the first learning task to which decoder output data output from the decoder layer is input and a first output function for applying an output function operation to the first task header output data output from the first task header; a second learning task layer including a second task header corresponding to the second learning task to which decoder output data output from the decoder layer is input and a second output function for applying an output function operation to the second task header output data output from the second task header; and a third learning task layer including a third task header corresponding to the third learning task to which decoder output data output from the decoder layer is input and a specific softmax function for applying a classification operation to the third task header output data output from the third task header.

6. In the step (c), 6. The method of claim 5, wherein when it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or a second threshold number of steps, the computing device trains the shared layer and the individual layers included in the deep learning model, and when it is determined that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, the computing device freezes the shared layer and stops learning, and updates specific model parameters of specific individual layers among the individual layers that are related to the specific learning task.

7. 2. The method of claim 1, wherein the deep learning model includes, as a shared layer, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and as individual layers, a first learning task layer corresponding to the first learning task and including a first decoder to which encoder output data output from the encoder layer is input and a first output function for applying an output function operation to the first decoder output data output from the first decoder, a second learning task layer corresponding to the second learning task and including a second decoder to which the encoder output data output from the encoder layer is input and a second output function for applying an output function operation to the second decoder output data output from the second decoder, and a third learning task layer corresponding to the third learning task and including a third decoder to which the encoder output data output from the encoder layer is input and a specific softmax function for applying a classification operation to the third decoder output data output from the third decoder.

8. In the step (c), 8. The method of claim 7, wherein when it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or a second threshold number of steps, the computing device trains the shared layer and the individual layers included in the deep learning model, and when it is determined that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, the computing device freezes the shared layer and stops learning, and updates specific model parameters of specific individual layers related to the specific learning task among the individual layers.

9. (e) when lane candidate group data corresponding to the first learning task output from the deep learning model is input, the computing device labels the lane candidate group data for each channel, selects the largest specific labeled blob (bLob), removes the remaining labeled blobs, obtains coordinates of blob center points for the specific labeled blob, and then applies a moving average filter to the coordinates of the blob center points to determine prediction information about the lane; and (f) the computing device references and matches the predicted lane information with an HD map, thereby estimating the position and attitude of the autonomous vehicle; The method of claim 1 further comprising:

10. After the step (e), When the prediction information about the lane and the information about the semantic segmentation are obtained, the computing device further includes the steps of: classifying a specific overlapping portion between the prediction information about the lane and the information about the semantic segmentation as a left boundary line and a right boundary line of the specific lane; determining the coordinates of the center point of overlap between the left boundary line and the right boundary line as a center point of an exposed lane; restoring an area for an unexposed lane by referring to the left boundary line and the right boundary line of the exposed lane; and determining adjustment information about the lane; 10. The method of claim 9, wherein in step (f), the computing device references and matches the lane adjustment information and the HD map, thereby estimating the position and attitude of the autonomous vehicle.

11. In the step (a), the computing device further sets a fourth learning task for simultaneous learning of the first learning task through the third learning task, further sets a selection probability initial value for the fourth learning task, further sets a fourth number of learning steps necessary to complete the one epoch for the fourth learning task, and determines the total number of steps included in all the epochs by referring to the first number of learning steps through the fourth number of learning steps; In the step (b), 2. The method of claim 1, wherein when it is determined that the total learning progress rate is equal to or less than the first threshold progress rate or equal to or less than the first threshold number of steps, the computing device selects the specific learning task to be learned from the first learning task through the fourth learning task by referring to the initial selection probability values ​​for each of the first learning task through the fourth learning task, and calculates a loss for the specific learning task.

12. (g) when the model parameters are updated, the computing device evaluates the specific current learning performance of the specific learning task based on the specific current step of the specific learning task, calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific current learning performance to the specific best learning performance of the specific learning task, calculates a total learning progress rate by referring to the first learning performance achievement rate to the fourth learning performance achievement rate (any one of the first learning performance achievement rate to the fourth learning performance achievement rate is the specific performance achievement rate), and if the sum of the first current step number to the third current step number in a state where 1 is added to the specific current step of the specific learning task is less than the total step number, supports the computing device to repeat step (b) and subsequent steps by adding 1 to the step number of the specific current step; 12. The method of claim 11 further comprising:

13. A computing device for training a deep learning model for extracting road features based on multiple tasks, comprising: at least one memory for storing instructions; and at least one processor configured to execute said instructions; The processor (I) sets a first learning task for estimating lanes on a road, a second learning task for estimating lines of structures associated with the road, and a third learning task for semantic segmentation of the lanes of the road and the structures, sets initial selection probability values ​​for each of the first learning task through the third learning task, sets first through third numbers of learning steps required to complete one epoch for each of the first learning task through the third learning task, and calculates a total number of learning steps by referring to the first through third numbers of learning steps. A computing device that performs (i) a process of determining the total number of steps included in an epoch; (ii) a process of selecting a specific learning task to be learned from the first learning task through the third learning task by referring to the initial selection probability values ​​for each of the first learning task through the third learning task when it is determined that the total learning progress rate is equal to or less than a first threshold progress rate or equal to or less than a first threshold number of steps; and (iii) a process of updating model parameters related to the specific learning task by referring to the loss value of the specific learning task.

14. The processor: In the process (II), 14. The computing device of claim 13, wherein, when it is determined that the total learning progress rate exceeds a first threshold progress rate or exceeds a first threshold number of steps, the computing device calculates a specific performance achievement rate for the specific learning task by referring to the ratio of the specific learning current performance to the specific learning best performance of the specific learning task, calculates a specific learning weight corresponding to the specific learning task by referring to the specific performance achievement rate, calculates a specific learning selection probability adjustment value by referring to the specific learning weight and the number of steps of the specific learning task, and determines the probability that the specific learning task will be selected from the first learning task to the third learning task as the specific learning selection probability adjustment value.

15. In the process (II), 15. The computing device of claim 14, wherein the specific learning weight is represented by a value obtained by subtracting the ratio of the specific learning weight to the sum of the first performance achievement rate through the third performance achievement rate for the first learning task through the third learning task, respectively, from 1, and the specific learning selection probability adjustment value is represented by a value obtained by multiplying the specific learning weight by the ratio of the specific number of learning steps for the specific learning task to the sum of all of the first number of learning steps through the third number of learning steps.

16. The processor:

14. The computing device of claim 13, further comprising: (IV) a process for, when the model parameters are updated, evaluating a specific current learning performance of the specific learning task based on a specific current step of the specific learning task; calculating a specific performance achievement rate for the specific learning task by referring to a ratio of the specific current learning performance to a specific best learning performance of the specific learning task; calculating a total learning progress rate by referring to first to third learning performance achievement rates for each of the first to third learning tasks (one of the first to third learning performance achievement rates is the specific performance achievement rate); and, if the sum of the first to third current step numbers, obtained by adding 1 to the specific current step of the specific learning task, is less than the total number of steps, supporting the re-execution of process (II) and subsequent steps with 1 added to the number of steps of the specific current step.

17. 14. The computing device of claim 13, wherein the deep learning model includes, as shared layers, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input and a decoder layer to which encoder output data output from the encoder layer is input, and as individual layers, a first learning task layer including a first task header corresponding to the first learning task to which decoder output data output from the decoder layer is input and a first output function for applying an output function operation to the first task header output data output from the first task header, a second learning task layer including a second task header corresponding to the second learning task to which decoder output data output from the decoder layer is input and a second output function for applying an output function operation to the second task header output data output from the second task header, and a third learning task layer including a third task header corresponding to the third learning task to which decoder output data output from the decoder layer is input and a specific softmax function for applying a classification operation to the third task header output data output from the third task header.

18. The processor: In the process (III), 18. The computing device of claim 17, wherein when it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or a second threshold number of steps, the shared layer and the individual layers included in the deep learning model are trained, and when it is determined that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, the shared layer is frozen and training is stopped, and specific model parameters of specific individual layers among the individual layers related to the specific learning task are updated.

19. 14. The computing device of claim 13, wherein the deep learning model includes, as a shared layer, an encoder layer to which image data acquired through at least one camera mounted on an autonomous vehicle is input, and as individual layers, a first learning task layer corresponding to the first learning task and including a first decoder to which encoder output data output from the encoder layer is input and a first output function for applying an output function operation to the first decoder output data output from the first decoder, a second learning task layer corresponding to the second learning task and including a second decoder to which the encoder output data output from the encoder layer is input and a second output function for applying an output function operation to the second decoder output data output from the second decoder, and a third learning task layer corresponding to the third learning task and including a third decoder to which the encoder output data output from the encoder layer is input and a specific softmax function for applying a classification operation to the third decoder output data output from the third decoder.

20. The processor: In the process (III), 20. The computing device of claim 19, wherein when it is determined that the total learning progress rate is equal to or less than a second threshold progress rate or a second threshold number of steps, the shared layer and the individual layers included in the deep learning model are trained, and when it is determined that the total learning progress rate exceeds the second threshold progress rate or the second threshold number of steps, the shared layer is frozen and training is stopped, and specific model parameters of specific individual layers among the individual layers related to the specific learning task are updated.

21. The processor:

14. The computing device of claim 13, further performing: (V) a process of, when lane candidate group data corresponding to the first learning task output from the deep learning model, labeling the lane candidate group data for each channel, selecting the largest specific labeled blob, removing the remaining labeled blobs, calculating coordinates of blob center points for the specific labeled blob, and then applying a moving average filter to the coordinates of the blob center points to determine predicted information about the lane; and (VI) a process of referencing the predicted information about the lane and an HD map to perform matching, thereby estimating the position and orientation of an autonomous vehicle.

22. The processor: After the process (V), (VI) when the prediction information on the lane and the information on the semantic segmentation are obtained, further performing a process of distinguishing a specific overlapping portion between the prediction information on the lane and the information on the semantic segmentation as a left boundary line and a right boundary line of the specific lane, determining the coordinate of the center point of overlap between the left boundary line and the right boundary line as a center point of an exposed lane, restoring an area for the non-exposed lane by referring to the left boundary line and the right boundary line of the exposed lane, and determining adjustment information for the lane; 22. The computing device of claim 21, wherein in the process (VI), the lane adjustment information and the HD map are referenced and matched to thereby estimate the position and attitude of the autonomous vehicle.

23. the processor: In the process (I), further setting a fourth learning task for simultaneous learning of the first learning task through the third learning task, further setting a selection probability initial value for the fourth learning task, further setting a fourth number of learning steps necessary to complete one epoch for the fourth learning task, and determining the total number of steps included in all of the epochs by referring to the first number of learning steps through the fourth number of learning steps; 14. The computing device of claim 13, wherein, in process (II), when it is determined that the total learning progress rate is equal to or less than the first threshold progress rate or equal to or less than the first threshold number of steps, the processor refers to the initial selection probability values ​​for each of the first learning task through the fourth learning task, selects the specific learning task to be learned from the first learning task through the fourth learning task, and calculates a loss for the specific learning task.

24. The processor:

24. The computing device of claim 23, further comprising: (VII) a process for, when the model parameters are updated, evaluating a specific current learning performance of the specific learning task based on a specific current step of the specific learning task; calculating a specific performance achievement rate for the specific learning task by referring to a ratio of the specific current learning performance to a specific best learning performance of the specific learning task; calculating a total learning progress rate by referring to first to fourth learning performance achievement rates (any one of the first to fourth learning performance achievement rates is the specific performance achievement rate); and, if the sum of first to third current step numbers in a state where 1 is added to the specific current step of the specific learning task is less than the total number of steps, supporting the re-execution of process (II) and subsequent steps with 1 added to the number of steps of the specific current step.

Images